Is finetuning from a pretrained model always better than training from scratch? At the worst case scenario, we could treat the pretrained weights as a random initialization, same as what we would do for training from scratch, right? If that is the case, then wouldn't it be better to always start with a pretrained model, as the lower layers of weights has already probably learnt general patterns of images that are transferable across all data sets?

My worry is what if the dataset I want to use for finetuning is highly specialized, highly unnatural and very different from the dataset the pretrained model is trained on. Would this still mean finetuning from a pretrained model wouldn't be the best idea? (e.g. training on X-Ray images instead of natural images for a cifar-1000 pretrained model.)

One limitation of using pretrained model is that you are forced to use the same architecture and weights. There could many scenarios where the pretrained architecture is limiting. If you train from scratch, you can define a custom architecture for the specific problem.


Can Single Node Hadoop Cluster be installed on a system with 1gb RAM I am trying to learn hadoop, would like to know if for basic single node installation 1gb RAM system would be enough or we need more RAM. It would be helpful if someone can share what other minimum system requirements which I can setup single node setup.
I trying to check on Apache Hadoop site but there is no specific mention of minimum system requirements for installation.
Thanks

Based on this serverfault answer and this recommendation page it seems you need more RAM. The only way to know for sure is to just try.


Combine 100 2D arrays into one 3D array I have 100 2D lists a0 to a99, it is easy to convert them into arrays (I verified already):

print(np.array(a0).shape)   # (2,150)
print(np.array(a1).shape)   # (5,150)
print(np.array(a2).shape)   # (10, 150)
# the sizes of a0 to a99 vary from (1,150) to (10, 150) only

I want to combine these 100 3D arrays into ONE 3D array, for example combined_array:

print(combined_array.shape)  # (100,10,150)
print(combined_array[0].shape) # (2,150)
print(conbined_array[1].shape) # (5,150)
print(combined_array[2].shape) # (10,150)

I use Python 3.7.

Your arrays have different shapes on the 0 axis, so you cannot use numpy.stack directly.

You can either use padding or put all arrays in a list. Using padding:

import numpy as np

a0 = np.empty((2,150))
a1 = np.empty((5,150))
a2 = np.empty((10,150))

max_shape = [0,0]
for a in [a0, a1, a2]:
    if max_shape[0] < a.shape[0]:
        max_shape[0] = a.shape[0]
    if max_shape[1] < a.shape[1]:
        max_shape[1] = a.shape[1]
arrays = []
for a in [a0, a1, a2]:
    arrays.append(np.pad(a, pad_width=((0, max_shape[0] - a.shape[0]),
                                       (0, max_shape[1] - a.shape[1])),
                         mode='constant'))
stacked_array = np.stack(arrays)


Which ML method would be best for deriving a rough formula for prediction based on existing data? Which ML method would you say is the easiest to derive a mathematical formula from based on already existing data of predictor stats and outcome?

I have this data:

Opponent 1:

Strength: x
Battle Score: y

I also have a model that I put against the opponent:

Opponent 2:

Strength: z
Battle Score: k

Finally, all outcomes of fights are written into a database (which currently has around 2800 outcomes) and look something like this model:

Fight:

Strength: x - z
Battle Score: y - k
Outcome: win/lose

I would want to get proper weights for Strength and Battle Score, so I can derive a simple formula from it and thus somewhat predict whether the next fight will be won or lost.

If you want "the easiest" for "a simple formula", then for sure it will be a linear regression on the battle score, or a logistic regression on "win/lose". That way you'll have the coefficients of the model, and they will be interpretable (which you won't get from a neural network with hundreds of parameters).


How to Save Model that has a TensorFlow Probability Regularizer? Consider the following minimal VAE:
import tensorflow as tf
import tensorflow_probability as tfp

tfk = tf.keras
tfkl = tf.keras.layers
tfpl = tfp.layers
tfd = tfp.distributions

#Fake dataset
cim = np.random.randint(2, size=(10,10))

#Parameters

vector_size = 10
input_shape = (vector_size,)
encoded_size = 3
latent_dim = 5

#Model

prior = tfd.Independent(tfd.Normal(loc=tf.zeros(encoded_size), scale=1), reinterpreted_batch_ndims=1)
vae = tfk.Sequential([
    #Encoder
    tfkl.InputLayer(input_shape=input_shape),
    tfkl.Dense(
        tfpl.MultivariateNormalTriL.params_size(encoded_size),
        activation=None,
        use_bias = False
    ),
    tfpl.MultivariateNormalTriL(
        encoded_size,
        activity_regularizer=tfpl.KLDivergenceRegularizer(prior)
    ),
    #Decoder
    tfkl.Dense(
        units = vector_size,
        activation = tf.nn.leaky_relu,
        use_bias = False
    )
])

vae.compile(optimizer=tf.optimizers.Adam(learning_rate=1e-3), loss='mse')
history = vae.fit(cim, cim, epochs = 1)

tf.saved_model.save(vae, './VAE')

The final line throws an error: AttributeError: 'Tensor' object has no attribute 'log_prob'
Tracing the issue, it seems that the activity regularizer tfpl.KLDivergenceRegularizer(prior) is not being serialized corrected. Any ideas for how to save this Model with the regularizer intact?

One solution I found was saving the model in HD5 format. That seems to bypass the serialization of the regularizer:
vae.save("./VAE.h5")
You can also try saving the weights only: vae.save_weights("./VAE-weights")


Doubt regarding the number of weights in 2 layer neural network Considering a hypothetical scenario , where we have 10 input layers, and 5 output layers.

How many weights are there in the neural network?

If this is implemented in pytorch, the answer will be 50. But shouldn't it be 15 since 10 weights from input layer, and 5 weights from output layer? Please resolve my doubt.

Please forgive my english, not a native speaker.

All units in one layer gets connected to all units in the next layer. To accomplish this all units in the first layer will need a weight per unit in the last layer. Therefore you get the number of weight be multiplication and not addition like you assumed.

Let's for example add one more layer with 3 units in the end and maybe it will be clearer. We have layers with units [10, 5, 3]. Then we will have $W_1: 10\times5=50$ weights between the first two layers and $W_2: 5\times3=15$ between the last two layers and together they will be $W_1 + W_2 = W <=> 50 + 15 = 65$.


Training with many CPU cores doesn't improve performance I ran my job on a computing cluster: first with 1 node / 4 cores, then with 2 nodes / 32 cores. But the training time is pretty much exactly the same for both of them! 67 seconds per step.

I am trying to fine-tune GPT-2 for my text dataset (chat logs).

What can I do to get a performance increase corresponding to the increase in processing power with CPUs?

Have you done any analysis as to what steps in your procedure consume most time? For example, it may be that GPU operations are responsible for most of the time. Then, of course, adding CPU cores will not give any benefit. And have you looked at CPU statistics, i.e. is the full capacity used?


Convolutional Neural Network: learning capacity and image coverage I was looking through a CNN tutorial and towards the end they refer to learning capacity and image coverage during network learning diagnostics

What do those 2 terms mean in the context of a convolutional neural network?

You can look at nolearn/lasagne/util.py to see how learning capacity and image coverage are computed for each layer:

real_filters = get_real_filter(layers, img_size)
receptive_fields = get_receptive_field(layers, img_size)
capacity = 100. * real_filters / receptive_fields
capacity[np.logical_not(np.isfinite(capacity))] = 1
img_coverage = 100. * receptive_fields / img_size

Xudong Cao. "A practical theory for designing very deep convolutional neural networks". 2015. https://www.kaggle.com/c/datasciencebowl/forums/t/13166/happy-lantern-festival-report-and-code explains how to compute the capacity of a layer:

  To quantitatively measure the learning capacity of a convolutional
  layer we define the c-value of a convolutional layer as follows.

  c-value = Real Filter Size / Receptive Field Size

  where the real filter size of a k-by-k convolutional layer is k if
  there is no down sampling, it doubles after each down sampling i.e. 2k
  after one down sampling and 4k after two down sampling etc. The
  receptive field size is defined as the maximum size of a neuron can
  see on the raw image. It grows proportionally as the convolutional
  neural network goes deep. Figure 3 shows how the receptive fields
  grows in an exemplar convolutional neural network.

This is why the coverage increases as you go deeper in the network, e.g.:

# Neural Network with 122154 learnable parameters

## Layer information

name        size        total    cap.Y    cap.X    cov.Y    cov.X
----------  --------  -------  -------  -------  -------  -------
input0      1x28x28       784   100.00   100.00   100.00   100.00
conv2d1     32x26x26    21632   100.00   100.00    10.71    10.71
maxpool2d2  32x13x13     5408   100.00   100.00    10.71    10.71
conv2d3     64x11x11     7744    85.71    85.71    25.00    25.00
conv2d4     64x9x9       5184    54.55    54.55    39.29    39.29
maxpool2d5  64x4x4       1024    54.55    54.55    39.29    39.29
conv2d6     96x2x2        384    63.16    63.16    67.86    67.86
maxpool2d7  96x1x1         96    63.16    63.16    67.86    67.86
dense8      64             64   100.00   100.00   100.00   100.00
dropout9    64             64   100.00   100.00   100.00   100.00
dense10     64             64   100.00   100.00   100.00   100.00
dense11     10             10   100.00   100.00   100.00   100.00

Explanation
    X, Y:    image dimensions
    cap.:    learning capacity
    cov.:    coverage of image
    magenta: capacity too low (<1/6)
    cyan:    image coverage too high (>100%)
    red:     capacity too low and coverage too high


Train a final model with the full data I have trained a few NLP models, measured their performances and now I want to create a final model for production trained with all the data I have available.
I'm working in text classification and I'm using transformers with the library PyTorch Lightning. The issue here is that I can't find an example of how to turn off the validation step (with pytorch lightning) and after several google searches I was not able to find discussions about that.
So, now I'm having second thoughts and questioning if I'm thinking correctly. Is it common practice to train the production model with the full dataset without the validation? Or should I just use the best model I have already trained? If so, why?

The last step for production models is typically to train with the entire set (train + validation) after tuning the hyperparameters using the validation set(s). The difference is typically not too drastic as the validation set should only be a fraction of the dataset but more data is always helpful especially for DL-based models.
I'm not familiar with the library you are using but perhaps you could just set the validation data to 0 and ignore the evaluation it prints.


In Deep Learning, how many kinds of Attention exist? And what is the history of Attention models? How many definitions of attention are commonly employed for Deep Learning tasks?

That's what I've encountered up to now:

Self-attention
Bahdanau
Luong
Multi-Head (used in Transformers)

Could you provide formal explanations of each of these (and others in case the list is incomplete) and tips on when to prefer one to the other?

And what is the history of Attention models? How did they develop through time, and how did they improve previous formulations?

As far as I know, one of the first "successful" use of attention came in 2014 by Badhanau, Cho, and Bengio with their paper "Neural Machine Translation by Jointly Learning to Align and Translate"

Since then, attention has been refined and improved in multiple papers. Attention is All You Need and the ever-growing use of transformers has made it even more popular.

However, I think it's crucial to understand that even though it has shown incredible value, it is ultimately just a mechanism that weights inputs in an attempt to pool them together in a smarter way than max or averaging pooling.


How is it possible to process an image with a few neurons? An 1024*1024 pixel image has around one million pixels. If I would like to connect each pixel to an R,G,B input neuron, then more than 3 million neurons are needed.

It would be really hard, to train a neural network, which has millions of inputs. How is it possible, to reduce the number of neurons?

There are several ways to make this big number trainable:

Use CNNs
Auto-Encoders (see Reducing the Dimensionality of Data with Neural Networks)

Dimensionality reduction of the input

Scale the image down
PCA / LDA

Troll-Answer

If you really meant "only a few neurons" then you might want to have a look at Spiking neural networks. Those are incredibly computationally intensive, need a lot of hand-crafting and still get worse performance than normal neural networks for most tasks ... but you only need very little of them.


Is there a theorem of prominent Russian mathematicians that played an important role in the development of machine learning? I once attended a seminar in which a statement answering the question asked above was answered in the affirmative.
I do not recall further specifics, however.

A search on Google Scholar yielded
NeuralComputation 1991 article
That is

Kolmogorov's Theorem Is Relevant
Abstract We show that Kolmogorov's theorem on representations of
continuous functions of n-variables by sums and superpositions of
continuous functions of one variable is relevant in the context of
neural networks. We give a version of this theorem with all of the
one-variable functions approximated arbitrarily well by linear
combinations of compositions of affine functions with some given
sigmoidal function. We derive an upper estimate of the number of
hidden units."

This theorem of Kolmogorov is surely the one referred to in the seminar I attended. But I somewhat erred I first thought in using the plural of mathematician.
But then I see the terminology
"Kolmogorov–Arnold representation theorem (or superposition theorem)" is also employed.
Wikipedia gives the following references:

Andrey Kolmogorov, "On the representation of continuous functions of
several variables by superpositions of continuous functions of a
smaller number of variables", Proceedings of the USSR Academy of
Sciences, 108 (1956), pp. 179–182; English translation: Amer. Math.
Soc. Transl., 17 (1961), pp. 369–373.
Vladimir Arnold, "On functions
of three variables", Proceedings of the USSR Academy of Sciences, 114
(1957), pp. 679–681; English translation: Amer. Math. Soc. Transl., 28
(1963), pp. 51–54.


When using Data augmentation is it ok to validate only with the original images? I'm working on a multi-classification deep learning algorithm and I was getting big over-fitting:

My model is supposed to classify sunglasses on 17 different brands, but I only had around 400 images from each brand so I created a folder with data augmented x3 times, generating images with these parameters:

datagen = ImageDataGenerator(
    rotation_range=30,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    fill_mode='nearest')

After doing so i got these results:

I don't know if it's correct to do the validation only using the original images or if I have to use also the augmented images for the validation, also is strange for me to get higher accuracy on the validation than the training.

You should validate only on the original images. The augmentation is there so that it can help your model generalize better, but to evaluate your model you need actual images, not transformed ones.

To do this in keras you need to define two instances of the ImageDataGenerator, one for training and one for validating. To train the model you need to set both generators to the fit_generator function.

train_gen = ImageDataGenerator(aug_params).flow_from_directory(train_dir)
valid_gen = ImageDataGenerator().flow_from_directory(valid_dir)

model.fit_generator(train_gen, validation_data=valid_gen)

It is possible to achieve a higher validation accuracy than a train accuracy if you heavily augment the training data.


What is the best way to pick the optimized configuration from this dataset? I have about 8000 configurations in an excel sheet. each configuration has four scores as seen in the image below. I would like to choose the best solution that has the highest lighting level score, lowest energy consumption score, the highest visual comfort score, the highest thermal comfort score, and the lowest CO2 level score. I'm new to data science but familiar with Python. I'm hoping that someone who is kind enough to explain a possible solution to rank these configurations from best to worst. Also, I'm interested to know the mathematical algorithm behind of the possible ranking system.

I have not executed the below code but I would do something like this,
df.loc[(df['Lighting Levela (Max)'] == max(df['Lighting Levela (Max)']) &
       (df['Energy Consumption (Min)'] == min(df['Energy Consumption (Max)']) &
       (df['Visual Comfort (Max)'] == max(df['Visual Comfort (Max)']) &
       (df['Thermal Comfort (Max)'] == max(df['Thermal Comfort (Max)']) &
       (df['CO2 levels (Min)'] == min(df['CO2 levels (Max)']), :)


What is the difference between 'if the data is of good quality' and 'if the data is tidy'? I'm doing Data Analyst nanodegree from Udacity. I'm confused between the difference even after going through the lecture a few times.

Let me try to explain by intuitively. First let me take the easy one.

Data being tidy

As per definition Tidy means Arranged in Order, Neat, Uncluttered.
All of these explain the physical aspects of the data representation. For example, data arranged in proper columns, with good headings, with relevance etc. You can think of this being syntactic in nature

Good Quality Data

This refers to actual data elements in the file. Like not having too many blank (missing) values. Every column having proper data types. Data without special (unreadable) characters. Properly encoded. Not having too much unstructured data. You can think this as the semantics of the data.

Hope this helps


The robustness of the Frobenius and L2,1 norms to the outlier I have a question about the properties of the Frobenius and L$_{2,1}$ norms. Why is the L$_{2,1}$ norm more robust to the outlier than the Frobenius norm?
PS: For a matrix $A\in\mathbb{R}^{n\times d}$, it can be easily seen that
$$
\text{Frobenius norm:}\qquad\Vert A\Vert_F=
\left(\sum_{i=1}^{n}\sum_{j=1}^{d}\vert a_{i,j}\vert^2 \right)^{\frac{1}{2}}=\sum_{i=1}^{n}\Vert A(i,:)\Vert_2^2,
$$
and
$$
L_{2,1}\,\,norm: \qquad\Vert A\Vert_{2,1}=\sum_{i=1}^{n}\left(\sum_{j=1}^{d}\vert a_{i,j}\vert^2 \right)^{\frac{1}{2}}=\sum_{i=1}^{n}\Vert A(i,:)\Vert_2,$$
where $A(i,:)$ is the $i$-th row of $A$.
I would be very grateful if some could answer my question.

I have only a couple of hints:

Frobenius norm, by definition, takes equal account of all data in matrix (all rows and columns).
Whereas $L_{2,1}$ norm is Frobenius norm but per row instead, so outliers in other rows do not affect (equally) norm of current row.


Market-basket: calculating support/confidence/lift/rules How can I calculate support/confidence/lift on a dataset in order to find frequent itemsets and determine association rules, in python? What would be the most effective method for predicting and offering recommendations on a test set of incomplete "shopping carts"? I am limited to the Anaconda distribution so I cant use packages such as orange3, etc.

I've done this (using anaconda) with the following libraries.

from mlxtend.frequent_patterns import apriori
from mlxtend.frequent_patterns import association_rules

Have a look here.


How to sum all price values having in count the quantity of each one Using pandas, I would like to sum all of the price to get the count value for each one. The Dataframe looks like this:

   price  quantity
0     25         2
1     20         3
2     20        -1

My code:

import pandas as pd

df = pd.DataFrame({'price': [25, 20, 20], 'quantity': [2, 3, -1]})
print(df)

So, I would expect the output to be something like $(25*2)+(20*3)+(20*-1)$:

>>> 90

You could always call on the pandas DataFrame's columns and work with that.

values = df['price'] * df['quantity']
sum(values)

if you want more information, I recommend https://stackoverflow.com/questions/14059094/i-want-to-multiply-two-columns-in-a-pandas-dataframe-and-add-the-result-into-a-n


Q learning for blackjack, reward function? I am currently learning reinforcement learning and am have built a blackjack game.

There is an obvious reward at the end of the game (payout), however some actions do not directly lead to rewards (hitting on a count of 5), which should be encouraged, even if the end result is negative (loosing the hand).

My question is what should the reward be for those actions ?

I could hard code a positive reward (fraction of the reward for winning the hand) for hits which do not lead to busting, but it feels like I am not approaching the problem correctly.

Also, when I assign a reward for a win (after the hand is over), I update the q-value corresponding to the last action/state pair, which seems suboptimal, as this action may not have directly lead to the win.

Another option I thought is to assign the same end reward to all of the action/state pairs in the sequence, however, some actions (like hitting on count <10) should be encouraged even if it leads to a lost hand.

Note: My end goal is to use deep-RL with an LSTM, but I am starting with q-learning.

It depends on the goal of the project. Projects can be a spectrum from "pure" to "applied":

A completely pure project only give minimal information. In the case of the Blackjack reward signal, that would just provide the win, loss, or tie information to the agent at the end of each hand.

A more applied project could give more information to the agent before, during, or after each hand.

An applied project will most likely start training better, learn faster, and end training higher. However, an applied project is less likely to provide evidence about the specific capacities of the system.

For both pure and applied projects, the amount and type of reward signal is a hyperparameter that is best tuned through empirical experimentation.


selecting variable randomly at each node in a tree in Random Forest In Random Forest method, for each tree we randomly select a set of variables (features) of fixed size. But once this set is frozen for that particular tree, does the tree behave like a regular decision tree algorithm?

I am assuming that random forest is nothing but generating a bunch of classical 'decision trees' and taking their votes towards the final classification. But in many places whatever description I have read seems to suggest that; for a given decision tree within the forest even at each node we randomly select variables. Is that the case?

Does it mean that at each node in the tree, we randomly select m variables from the variable set which is fixed for that tree? Or from the global variable set of the training dataset? And then from the selected set of variables we select 1 variable heuristically (e.g. whichever variable maximises information gain) -- is that a correct statement?

To train a tree in a random forest, at each split m variables are randomly selected from the global variable set, and the variable that maximizes information gain is chosen to be split on. Source: https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf (the original random forest paper).


Difference between OrdinalEncoder and LabelEncoder I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing:

In the Documentation it is given about sklearn.preprocessing.OrdinalEncoder() whereas in the book it was given about sklearn.preprocessing.LabelEncoder(), when I checked their functionality it looked same to me. Can Someone please tell me the difference between the two please?

Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while LabelEncoder is for converting target variable.

That's why OrdinalEncoder can fit data that has the shape of (n_samples, n_features) while LabelEncoder can only fit data that has the shape of (n_samples,) (though in the past one used LabelEncoder within the loop to handle what has been becoming the job of OrdinalEncoder now)


What are stovepipes? I am reading the book The Data Warehouse Lifecycle Toolkit by Ralph Kimball. I come across the term Stovepipes fairly often. After doing some research I read that Stovepipes are when you don't have conformed dimensions to link data marts. This short description was all I could find every other resource I looked at simply said that Stovepipe models are bad.

What is is meant by "conformed dimensions"? What exactly are these Stovepipes and can somebody provide me with an example?

It seems "stovepiping" is a term that legendary author on the subject of data warehousing Ralph Kimball adopted, and it looks that it actually originates from intelligence community. In its original (intelligence community) meaning, it is mostly related to the usage of raw data, instead of processed data.

From Wikipedia:

  Stovepiping (also stove piping) is a metaphorical term which recalls a stovepipe's function as an isolated vertical conduit, and has been used, in the context of intelligence, to describe several ways in which raw intelligence information may be presented without proper context. It is a system created to solve a specific problem. The lack of context may be due to the specialized nature, or security requirements, of a particular intelligence collection technology. It also has limited focus and data within is not easily shared.

From RationalWiki:

  Stovepiping is a term originating in the intelligence community to describe a process by which raw data is funneled directly to high-ranking officials or the media.

Also, there is an interesting article in PhraseFinder.

Even better explanation from Relational Solutions Blog:

  Most data marts are designed as one off reporting solutions, designed to solve an immediate problem that the business users need to solve. When designed as a stand-alone, they are often referred to as “stove pipes” or “silos” of information.

  Analysts new to the space think these are new terms, but Data warehousing consultants have used these terms since the 90’s. They’re used to describe stand-alone reporting solutions. Typically these stand-alone solutions are developed by individual teams or departments.

  These groups develop “silo’s” or “stove pipe” reporting databases to achieve a specific goal that they were unable to get financial approval for. If they have a need for something that you can't get approval for, you resort to building something on your own. It happens in every company and every department.


How to treat input that inherently has a tree structure? When having a single vector you use an MLP neural network
When having a 2D structure you use a CNN neural network
When having a sequence you use a RNN neural network

Now you have preprocessed an instance and the result is a tree structure.
Let's say for simplicity that the tree structure is always the same tree; only the node values differ among instances.

What kind of neural network architecture would be required to consume the information of a tree structure but also leveraging the connections between the tree nodes?

As @Emre mentioned, RNN is a good option. It's worth noting that if the number of possible nodes in each tree is the same or at least has the same upper bound, you could use literally any architecture you want and just pass in the adjacency matrix. Alternatively, you could build an intermediate model to convert your graph into a graph embedding and then once again, you can do basically whatever you want with that.

A pretty big piece of potentially important information here is what you are trying to accomplish, which could have significant consequences for how you want to represent your inputs.


Where is the output in the LSTM? I'm trying to understand where the output of the LSTM is. Please refer to the following picture:

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

It seems that at each tilmestep, we output h_t and C_t which correspond to the hidden state and the memory cell.

Now suppose I'm trying to model the stock price movements which is binary [0,1], 0 for down, 1 for up which is my y_i.

I feed in x_t which is a feature vector at each tilmestep and I expect to get a 1 dimensional output y_t after the last tilmestep.

Are h_t what I'm looking for? This would imply that h_t matches the output dimension, but for some reason I though it is independent of the output dimension.

Usually you have to add a Dense layer after the LSTM unit. That will try to understand how to use the output of LSTM.

For example in Keras:

model = Sequential()
model.add(LSTM(4, input_shape=(1, 5)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam')


How would you encode missing pixels in image data? I am working through an example on the MNIST dataset, and was just curious, if your image input data were missing some pixels, how would you encode it.  Since the values are always positive, and normalized between 0 and 1, would it make sense just to encode it as -1 or something.
Thanks

You'd substitute a statistical aggregator, such as the median or mean, for the missing data. Calculating the aggregator for a neighborhood region would "smooth out" the missing pixels.
You wouldn't set it to -1 because lost pixels are effectively noise, which you want to suppress, not exaggerate.


Clustering for multiple variable There are total 50 students(john, Roy..) and used some action to do a job. My dataSet something like this

    John    Roy    Micheal   Ron  ....... Smith
     A       B        B       A            C
     A       A        C       B            B
     C       A        A       B            B
     .       .        .       .            .
     .       .        .       .            .
     .       .        .       .            .
     F       G        E       A            G

Here A,B,C...G are strings. Final data is like this...

         A   B   C   D   E   F   G
 John    3  34  23  34   4   3   1
 Roy     5  23  12  3    5   39  46
 ...................................

This means John used "A" 3 times whereas Roy used "A" 5 times.
So based on their action I want to cluster them (i.e. the student used the same number of activities will be in the same group).

Which clustering method can be used and how ?

K-means

Your data has $7$ dimensions so k-means is worth to try. See the PCA of your data and check if any cluster is visible there as K-means will have a tough time if clusters are not Gaussian. the setup is:

Each person is a point in $7D$ space (a $50\times7$ matrix)
Apply PCA and inspect it.
If different clusters visible then you will have a result

Fuzzy C-Means

I would suggest soft clustering algorithms. Soft clustering means that output is not binary (each sample belong only to one cluster and does not belong to others) but it assigns a membership score for belongness of each sample to each cluster. It minimizes the same objective function as K-means but with a weight which is calculated in each iteration and can be found here. Libraries usually have this algorithm under the name FCM.

K-means Through GMM

Another soft version of K-means is called Gaussian Mixture Models in which you try to estimate Gaussian kernels whose superposition describes the data (as you see again the Gaussian distribution of data is crucial here). The setup is:

Choose a $k$ as the initial number of clusters and produce $k$ arbitrary Gaussian kernels (i.e. with arbitrary $\mu$ and $\sigma$)
Use Expectation Maximization algorithm for updating the new clustering structure at each step.

Spectral Clustering

Define a Similarity Matrix from the data by any means. For example calculate the distances between points in $7D$ space and reverse that. Or apply a RBF kernel for determining the similarity between them. Then calculate the second eigenvalue-eigenvector pair according to sorted eigenvalues. Use K-means to cluster those elements of eigenvectors.

If you have more detailed questions just drop a comment here. Good Luck!


Are there trained neural networks, that can distinguish a book's author point from what he stands against? Let's take some specific book to narrow this example. Atlas shrugged by Ayn Rand. When i'll be saying something like "Ayn Rand's ideas", i'll mean only those, which are clearly stated in the book.

Humans who read this book, can clearly distinguish sentences, that describe Ayn Rand's point of view from sentences, that oppose it.

Are there trained neural networks, that could distinguish sentences with Ayn Rand's ideas vs sentences with opposing ideas?

If not, are there any reasons, why it's impossible to train such network, if each sentence in such book would be marked as pro-, contra- or neutral to Ayn Rand's ideas? Successfully trained network should be able to distinguish author's point of view in books, where humans can clearly do it.

Absolutely. You've described topic modeling and sentiment analysis. You might want to train separate models for each part of the problem and stack them, but this is a totally solvable problem.


Deciding tools to use for ML I want to setup a decent framework for building and deploying ML code on our servers. The server side code is all java. I have inherited a system that uses weka as it is based on java and makes life easy for the server side folks. But  data scientists who come in, use R/python to do their modelling and then write some glue code to make the models run in java environment on weka. This library is getting outdated(works alright though), but mainly writing this glue code is getting cumbersome/repetitive and can be done away with. I'm looking for an optimized pipeline that helps me get models deployed quickly.

How is this done in the industry? What does your pipeline look like?

I would like to be able to quickly model using one language that data science team uses(I'm tending to python) and be able to deploy/call these models for prediction on the server side without having to redo a bunch of stuff to get it working there.

Well, I think your question does not have a "correct" answer since you are looking for a piece of advice on ML model deployment.

As you mentioned, you have a complete back-end in java, and you should not drop it.
Instead, (1) you can take advantage of a micro-services oriented approach. With micro-services, you can communicate different pieces of code using HTTP. You may have some of your back-end controllers in java, and some kind of ML-managers to wrap prediction routines with web routes.

I think it's a good idea for you to learn python, and also python web frameworks (flask is a very simple one, you can google "ML deployment flask" or something like that, and there is plenty of tutorials). Also, (2) you can convince your data scientists to write their micro-services for you because it is simple to do. Then, you can get their code and adapt to better look like a web-service.

Another way to take advantage of your java back-end is to (3) try to translate the python/R ML code to Spark MLlib, which has many powerful methods for you to write your java code. However, you will eventually have to code some operations that your data scientists made in python/R, which is not available in MLlib.

It depends, but I hope that it helps!


Why is my Count (distinct) not working the way I would expect it to be? PowerBI My total is set as Count (distinct). However, the total itself doesn't make any sense, as you can see in my attachment. Any ideas what I'm doing wrong? By not making any sense, I mean that the numbers above are not being added.

Any suggestions would be greatly appreciated.

Count distinct is a measure (aggregate function). When the measure is used in the total row, it is not summing the distinct count values about but is calculating the function in the context of all the table data. Without seeing the data I would make the assumption that there are 85 distinct values between those 4 months. March, for example, happens to contain 83 of 85 of those values.

If you want a sum of the values above, you can use SUMX.

Example:

Count of ProjectName =
SUMX (
    VALUES ( 'put table name here'[MonthNameShort] ),
    CALCULATE ( DISTINCTCOUNT ( 'put table name here'[ProjectName] ) )
)

Comparison in Power BI:


What are the best practices for data formatting? I used to do some analysis on Excel but my company want to use Python to increase the analysis.
I am novice on python and overwhelmed by all of this knowledge I have to learn :'(

So for my first project, I have imported my csv file and I had this error message "C:\Users\xxx\AppData\Local\Continuum\anaconda3\lib\site-packages\IPython\core\interactiveshell.py:2785: DtypeWarning: Columns (8,9,31,32,35,36,37,38,39,40) have mixed types. Specify dtype option on import or set low_memory=False.interactivity=interactivity, compiler=compiler, result=result) "

Sadly I have no clue about all of this so I looked on the web and guess what?it's still not clear for me. So my questions are

What should I do to avoid this error message ?
What are the best practices/ formatting when you work with dataset ?

Thanks

This warning is trying to let you know that some of the columns of your DataFrame failed to resolve to a single data type (at least from pandas' perspective, it didn't know what to interpret the columns as!). Since it's a warning, pandas is happy to give you the DataFrame after finishing the read_csv but it's not advisable to proceed without addressing your mixed type columns in some fashion.

1) You can avoid this warning by explicitly identifying the dtype. The parameter in read_csv is conveniently called dtype. The usage specified by the doc is either single data type (e.g. dtype = str) or a dictionary of columns and data types (e.g. dtype = {0:str, 1:int}). Some options for the data type are:

the pandas specific data types (object)
the built in Python types (str, int, float)
or numpy data types (numpy.int32, numpy.float64)

The nice thing about specifying per column data types is that Python will throw a hard error if something breaks that type convention. It's a great, immediate check that your data is in a format you expect and it's always nice for Python to break loudly. However, it's possible you want to just read all your data in and then fix it in the DataFrame. You generally can get away with specifying dtype=str and pandas will (usually) perform the read_csv without complaint.

2) Since this question is unfortunately best answered as "it depends," I would advise looking into data cleaning methodologies (since that's what this problem largely is). Some specific (but generic) advice is: think hard about how you want to deal with your missing/malformed data, be explicit about the transformations you'll apply to the data to handle those cases, and keep track of the fact you've made those changes.

I hope this was helpful!


Do I need to square a column if I want a neural network to try using that? In the classical linear regression implementation, if I suspect the square of the values of the column is correlated to the target, then I actually need to create a new column with the squares for the algorithm to make use of that.
Is this also necessary when using neural networks? I know it's a broad question - are there cases where this is necessary and cases where it isn't?

You don’t necessarily need to, according to the universal function approximation theorem.

It is easier for a neural network to learn an identity function than some other function, so if one of the inputs definitely needs to be squared your network will learn faster if you pass the input already squared
If your network is sufficiently large it should work out that squaring that input is helpful and approximate the squaring function as part of the overall learning process


Validation loss is not decreasing I am trying to train a LSTM model. Is this model suffering from overfitting?

Here is train and validation loss graph:

The model is overfitting right from epoch 10, the validation loss is increasing while the training loss is decreasing.
Dealing with such a Model:

Data Preprocessing: Standardizing and Normalizing the data.
Model compelxity: Check if the model is too complex. Add dropout, reduce number of layers or number of neurons in each layer.
Learning Rate and Decay Rate: Reduce the learning rate, a good starting value is usually between 0.0005 to 0.001. Also consider a decay rate of 1e-6.

There are many other options as well to reduce overfitting, assuming you are using Keras, visit this link.


How is y=mx+b different from hθ(x)=θ0+θ1x? I could not quite comprehend the hypothesis represented by  hθ(x)=θ0+θ1x
To find out good values for the parameters θ0 and θ1 we want to minimize the difference between the calculated result and the actual result of our test data. So we subtract hθ(x(i))−y(i) for all i from 1 to m. Hence we calculate the sum over this difference and then calculate the average by multiplying the sum by 1/2m.  This would result in:

1/2m∑mi=1(hθ(x(i))−y(i))2

So, I googled further and landed on a youtube video that talked about
y=mx+b, where m is the slope and b is y intercept. This is called a Linear Model.

Also, in the linear model, the following formula is used to determine m and b.

Now my questions are:

1) Is the hypotheses and the linear model the same?
2) Is there a cost function for the linear model?
3) Why would anyone want to guess, choose and arrive at θ0 and θ1 when there is a straight forward formula (linear model)?

1) Your hypothese $hθ(x)$ is clearly a linear model with $b \leftrightarrow θ_0$ and $m \leftrightarrow θ_1$ as you expected (don't be too hesitant and your colleges could have been able to confirm this).

2) Lets call $hθ(x_i)$ the prediction $p_i$. A cost function $C$ can be any function that when you try to minimize it holds a good solution.
One example would be the mean squared error (mse):  $C(p_i, y_i)=1/2 \cdot (p_i - y_i)^2$. I'm quite sure the mse is used to optimize the linear model, however other cost functions are possible.

3) I suppose this is an assignment? You clearly gave a very good suggestion to approximate your data. I suppose the goal of your task was to come to the solution of the linear model yourself. Try to understand how $b$ and $m$ is estimated in the linear model.

If you are in school, it is very useful to understand how people before you came to certain formula. It seems like you are reinventing the wheel, but the insights you gather with this can only help your education :).

Good luck.


Choosing an algorithm with normalized data(Classification) I am having a dataset which all the features are from 0 to 1(real numbers) and the output is 0 or 1(integer numbers). Example:

var1   var2   var3  output
0.01    0.1    0.7       1
0.01    0.1    0.7       1
 0.1    0.2    0.3       0
 0.2    0.4    0.4       0
 0.4    0.1    0.9       1

Which classification algorithm is recommended when the variables are 'normalized' from 0 to 1? Does SVM or logistic regression 'react' good in these types of data?

I have noticed that most of the people that are using SVM,NN or logistic regression when making feature scaling they use stardadization( (value-mean)/std ) ). Is there a reason not to rescale the values between from 0 to 1?

Re-scaling or any other form of standardization/normalization is very useful when dealing with models that are trained with gradient descent: (SVM,NN, LogReg).

This question explains the effect of normalization on the gradient rather well: https://stats.stackexchange.com/questions/111467/is-it-necessary-to-scale-the-target-value-in-addition-to-scaling-features-for-re

Decision trees for example are invariant to linear transformations, therefore feature scaling, in theory, does not effect the model in any way.

In the case of your set-up, you have a binary classification problem.

I would recommend trying out first a linear classifier on your data. Logistic regressions is a good first choice. If you decide that LogReg is not working as well as you would like it to then you can move on to more complicated models. I would recommend using decision trees coupled with a gradient boosting model.

In problems that are similar to what you described, specifically when there is no inherent structure in the data (as there is in image classification, speech recognition, etc.), gradient boosting models tend to outperform neural networks.

Hope this helps.


pandas count values for last 7 days from each date There are two pd.DataFrame. First is like this:
print df1

        id        date    month  is_buy
     0  17  2015-01-16  2015-01       1
     1  17  2015-01-26  2015-01       1
     2  17  2015-01-27  2015-01       1
     3  17  2015-02-11  2015-02       1
     4  17  2015-03-14  2015-03       1
     5  18  2015-01-28  2015-01       1
     6  18  2015-02-12  2015-02       1
     7  18  2015-02-25  2015-02       1
     8  18  2015-03-04  2015-03       1

In second data frame there are some aggregated data by month from the first one:
df2 = df1[df1['is_buy'] == 1].groupby(['id', 'month']).agg({'is_buy': np.sum})

print df2

        id    month       buys
     0  17  2015-01          3
     1  17  2015-02          1
     2  17  2015-03          1
     3  18  2015-01          1
     4  18  2015-02          2
     5  18  2015-03          1

I'm trying to get new df2 column named 'last_week_buys' with aggregated buys by last 7 days from first day of each df1['month']. In other words, I want to get this:
        id    month       buys    last_week_buys
     0  17  2015-01          3               NaN
     1  17  2015-02          1                 2
     2  17  2015-03          1                 0
     3  18  2015-01          1               NaN
     4  18  2015-02          2                 1
     5  18  2015-03          1                 1

Are there any ideas to get this column?

The main obstacle is figuring out whether a date is within the last 7 days of the month. I'd recommend something hacky like the following:

from datetime import datetime, date, timedelta
def last7(datestr):
    orig = datetime.strptime(datestr,'%Y-%m-%d')
    plus7 = orig+timedelta(7)
    return plus7.month != orig.month

Once you have that, it's relatively simple to adapt your previous code:

df3 = df1[df1['is_buy'] == 1 && last7(df1['date'])].groupby(['id', 'month']).agg({'is_buy': np.sum})

Now we just join together df2 and df3 and we're done.


Sci-kit learn function to select threshold for higher recall than precision When we care more that there should be no false negatives, as far as possible… ie. higher recall (video is suitable for kid or not), we should use (receiver operating characteristic) ROC (area under the curve) AUC and try to maximize it.

Scikit-Learn provides a function to compute this directly:

from sklearn.metrics import roc_auc_score
roc_auc_score(y_train_5, y_scores)

Similarly, when we care more about the false positives than the false negatives, example shop lifting case then what should we do?

Should we just try to maximize the recall ignoring the precision or is there a better metric?

I had read somewhere "Use the precision vs recall curve and get that max" but not very sure what that means... Can you please explain the same?

If there a direct function for this metric in sci-kit learn, like above for the first case, please do let me know.

As far as a fuction in scikit to implement a certain threshold for a higher recall, I don't think there is one.

But, depending on what model you're using, you can vary the threshold for probability outputs of the model to obtain a higher Recall.

For example, say you are using a Random Forest or Logistic Regression classifier model, then you can use the predict_proba function on your data set; this will return probabilities for each sample belonging to a particular class. If you then create a new array from the output of predict_proba using your threshold, then the model will have a Recall with your desired accuracy when it classifies new unseen data.

e.g pseudo code for binary classification

clf = sklearn.ensemble.RandomForestClassifier()
model = fit(X,y) # fit model to training datset
probs = model.predict_proba(X_new) # prediction on a new dataset X_new

threshold = 0.7 # threshold we set where the probability prediction must be above this to be classified as a '1'
classes = probs[:,1] # say it is the class in the second column you care about predictint
classes[classes>=threshold] = 1
classes[classes<threshold] = 0


Is this the correct way to apply a recommender system based on KNN and cosine similarity to predict continuous values? My data is:
userID, gameID, rating (1.0 through 10.0)
First, I normalize the values the ratings of each row.
I use cosine similarity to create a similarity matrix where each cell represents similarity between a pair of userIDs (value 0.0 through 1.0).
For each unrated gameID, I find the 10 most similar users that have rated the gameID. The predicted rating is equal to the sum of each neighbors rating times similarity, then divided by 10 (number of neighbors).

This seems fine for finding the best predictions, just take the top N rating values, but for actually predicting the values it doesn't perform so well. Intuitively, the average of a group of similar userID's ratings would be an accurate prediction. When each rating is made smaller by multiplying by a value that is always less than 1.0, the predicted value is consistently smaller than expected.

This seemed to follow formulas that I found, but I feel like I missed something. I am using Python, Pandas, and Numpy.

Edit
I am seeing that this final value is sometimes referred to as a "weighted predicted rating". I am wondering how to "de-weight".

The predicted rating is equal to the sum of each neighbors rating times similarity, then divided by 10 (number of neighbors).

You want to obtain a weighted-average of ratings, where the weight is the similarity score. Instead of dividing by 10 above, you should divide by the sum of similarities, to get a correct normalized score. Dividing by 10 is the likely reason for the predictions to be smaller than expected.

The formula you should use is:

  Final score = Sum_over_neighbours(Rating * Similarity) / Sum_over_neighbours(Similarity)


ARMA/ARIMA on energy forecasts timeseries: strange prediction I'm trying to use ARMA/ARIMA with the statsmodel Python package, in order to predict the gas consumption. I tried with a dataset of this format:

Using only the gas column.

from pandas.tseries.offsets import *

arma_mod20 = sm.tsa.ARMA(januaryFeb[['gas [m3]']], (5,3)).fit()
predict_sunspots = arma_mod20.predict('2012-01-13', '2012-01-14', dynamic=True)
ax = januaryFeb.ix['2012-01-13 00:00:00':'2012-01-15 22:00:00']['gas [m3]'].plot(figsize=(12,8))
ax = predict_sunspots.plot(ax=ax, style='r--', label='Dynamic Prediction');
ax.legend();

Why is the prediction so bad?

I'm not an expert on time series, but I have a general advice: may I suggest you to try other packages (and various parameters) to see, if there are any differences in results.

Also, unless you have to use Python, I'd recommend to take a look at the R's extensive ecosystem for time series analysis: see http://www.statmethods.net/advstats/timeseries.html and http://cran.r-project.org/web/views/TimeSeries.html.

In particular, you may want to check the standard stats package (including functions arima() and arima0), as well as some other packages: FitARMA (http://cran.r-project.org/web/packages/FitARMA), forecast (http://cran.r-project.org/web/packages/forecast) and education-focused fArma (cran.r-project.org/web/packages/fArma), to mention just a few. I hope this is helpful.


Different t-test results from R and LibreCalc while I was trying to apply one-tailed t-test to two vector of data on R, I just wanted to apply same thing on librecalc also and then I noticed that p-values are different. Here is a summary what I have done so far:

in R I applied: t.test(data[,1],data[,2],paired = TRUE,alternative = 'greater') and then result is:

Paired t-test

data:  ttest[, 2] and ttest[, 3]
t = -134.33, df = 49, p-value = 1
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
-8.318141       Inf
sample estimates:
mean of the differences
            -8.2156

It returns 1 as p-value. However, same thing on Librecalc, it gives following:

Paired t-test
Alpha   0,05
Hypothesized Mean Difference    0
    Variable 1  Variable 2
Mean    33,003568   41,219168
Variance    0,155220041404082   0,038756230791837
Observations    50  50
Pearson Correlation 0,04472285907661
Observed Mean Difference    -8,2156
Variance of the Differences 0,187038752653061
df  49
t Stat  -134,325507363656
P (T<=t) one-tail   7,16710643183049E-65 //(Here is the p-value)
t Critical one-tail 1,67655089261685
P (T<=t) two-tail   1,4334212863661E-64
t Critical two-tail 2,00957523712924

Except the p-values, all the other results are same as you see. Can someone tell me what is going on here, or what should I do for a one-tailed t-test to compare two algorithms. (I have 50 different runs' results of both algorithm). Here is the data:

31,3888 41,028
33,3064 41,2048
32,6824 41,5216
33,4552 40,9848
32,6832 41,3152
33,2704 41,212
32,8344 41,2104
32,6376 41,2904
33,3552 41,22
32,6    41,2152
32,6752 41,4784
32,9464 41,1352
33,2064 41,4216
33,1752 41,0568
33,0952 41,2808
32,5272 41,7328
33,1624 40,8696
33,0192 41,0544
33,1744 41,1048
33,4464 41,0488
33,7448 41,3832
33,0568 41,4656
32,836  41,1192
33,1    41,0832
33,2144 41,4944
33,0544 41,2736
33,3232 41,4232
32,8616 41,2304
33,228  41,2184
33,2472 41,02
32,8952 40,9808
33,5056 41,3808
33,5248 41,348
33,4416 41,1072
33,0416 41,016
33,0504 41,168
32,4064 41,184
32,6256 41,572
32,488  40,7848
33,4304 41,2536
33,2128 41,2248
32,9576 41,5392
33,0136 41,4608
32,9296 41,0872
32,632  41,0128
33,2616 40,968
32,7664 41,0192
32,8952 41,3344
33,3512 41,2984
32,4408 41,1216

If alternative = 'greater' means that alternative hypothesis is "x has greater mean than y". In your code:

t.test(data[,1],data[,2],paired = TRUE,alternative = 'greater')

it translates to data[,1] has grater mean that data[,2]. Which is obviously not true. Just change the order of inputs and you'll get a correct p-value

t.test(data[,2],data[,1],paired = TRUE,alternative = 'greater')

I suppose that in LibreCalc arguments are flipped w.r.t. R


Suggestions for binary classification algorithm I am new to machine learning, so was hoping you might have some insights one what classification algorithm might be best suited for my data.

I have a data set of hundreds of predictions from each of ~100 forecasters. Each prediction is binary, and the outcome variable is binary.

So basically let's say I have data on 200 events. For each event, I have 1 prediction from each of 100 forecasters. Each of the 100 forecasters predicts each of the 200 events.

I am hoping for an output of some formula, for which I input the predictions of the forecasters for that particular outcome, and it renders an output for the probability of that outcome.

My main interest is in predictive accuracy.

Do you have any suggestions for what algorithms might suit this problem?

I would say support vector machine, support vector classifier. This will allow you to have as many predictors as you want. Also, you can improve your accuracy by tuning your model parameters.

Here is another nice link for support vector classification

Since you did not mention any language, I would use python as there are so much online support and you resources available online.

I would also check xgboost classifier.

Let us know how did it go.


In general, when does TF-IDF reduce accuracy? I'm training a corpus consisting of 200000 reviews into positive and negative reviews using a Naive Bayes model, and I noticed that performing TF-IDF actually reduced the accuracy (while testing on test set of 50000 reviews) by about 2%. So I was wondering if TF-IDF has any underlying assumptions on the data or model that it works with, i.e. any cases where accuracy is reduced by the use of it?

The IDF part of TF-IDF gives less weight to a word if it occurs in a large fraction of the documents in your corpus.  However, this doesn't necessarily mean that the word is unimportant for distinguishing your two classes.  A word which is common in your corpus, but which also occurs substantially more often in one class than the other, could very well be quite valuable in distinguishing the classes.  This can especially be true if your set is not balanced between the two classes.

For example:
Suppose that 95% of negative reviews contained the term "boring", while only 5% of the positive reviews contained "boring".  If this occurred in a balanced set of negative and positive reviews, that means that half the total reviews would contain "boring", and this might be one of the terms whose weight is more strongly suppressed by IDF, despite the fact that it's obviously a valuable term for distinguishing the classes.  If instead 90% of the reviews are negative (with "boring" again occurring in 95% of negative reviews and 5% of positive ones), then a full 86% of your documents contained the term, and its weight might be highly suppressed by IDF despite its obvious importance.

In essence, the problem is that TF-IDF is agnostic to the class labels in your training set.  One alternative to TF-IDF for text classification that doesn't have this flaw is Bi-Normal Separation (BNS), see for example this paper:

Forman, G.  BNS Feature Scaling: An Improved Representation over TF-IDF for SVM Text Classification.  Hewlett-Packard Labs Tech Report HPL-2007-32R1, 2007.(link to PDF)


Split data into linear regression I am looking for a way that could help me create more precise models.

Let's say these are real estate prices for different areas. Only in the data I do not have a clear division into these areas I suppose this is the relationship.

At present I have one model (red line) and I would like to have n models, eg 3. Additional two green lines. And use them for points that are closer to this line.

How convenient to go about it?

What measure should be used to divide this data and apply linear regression so that the variance is as low as possible?

May I have some inspiration :)?

You can check the histogram of y/x. Using only values of x far from zero would be necessary, in your case its not a problem because prices are far from zero.

If in your case the three different distributions have intercept near 0, you may have useful results. You can also try to check the histogram of (y-intercept)/x.

You should find different modes in the distribution of this quotient. You could then divide the dataset according to different values of this quotient.


Is there any library available for balancing imbalanced text dataset? I have a text dataset similar to newsgroup dataset, the problem with the dataset is that it is highly imbalanced. So is there any readily built library that will do upsampling or downsampling with a function call?

from imblearn.over_sampling import ADASYN, SMOTE, RandomOverSampler
from imblearn.under_sampling import NearMiss, RandomUnderSampler

ros = RandomOverSampler(random_state=777)
X_ROS, y_ROS = ros.fit_sample(testing_tfidf, testing_target)

smt = SMOTE(random_state=777, k_neighbors=1)
rus = RandomUnderSampler(random_state=777)

Good article for reference


Concept of Cross-Entropy Can anyone say the cross entropy Conceptually in machine learning for selecting best features, in simple way?
I confused about that.

You do not really understand something unless you can explain it to
your grandmother.
Albert Einstein

Entropy can be calculated using formula:-

$Entropy = -p\log_2 p − q\log_2 q$

Here $p$ and $q$ are probability of success and failure respectively in that node.
Entropy is also used with categorical target variable. It chooses the split which has
lowest entropy compared to parent node and other splits. The lesser the entropy,
the better it is.

Steps to calculate entropy for a split:

Calculate entropy of parent node
Calculate entropy of each individual node of split and calculate weighted
average of all sub-nodes available in split.


Decision Tree: Efficient splitting of nodes, minimize number of gini evaluations I have a dataset specific problem where i need to use a splitting function other than gini_index. This requires me to re-write a decision tree from scratch. I have a working model, but itis highly inefficient.

To make a split i currently iterate though each feature and then through each unique datapoint in that dataset for each node (total of nodes x features x unique levels gini evaluations). Cause of this my DT on a 300k X 145 dataset has been running for 2 days.

How can I cut down on the number of splitting evaluations, or speed up the program. I read Fisher Yates algorithm in Sklean's code, but I don't understand the logic. Any help would be appreciated.

In general, to reduce the amount of time needed to run your dataset through the See4.5 (C4.5) algorithm, you'll want to reduce the number of nodes in the tree needing to be processed.

This can be done utilizing pruning, optimal operator selection, and incorporating a heuristic into your decision tree search.

Alpha-beta pruning, bidirectional search, and the Minmax algorithm for operator selection are a good choice when it comes to decision-tree time reduction.

I'm not going to write an entire book here, however look into artificial intelligence and see what they've accomplished so far. Alot of it needs tweaking because they keep switching (but that's another story), however, if you come across any books saying bidirectional search is in any way non-optimal, just ignore that because it's inherent that researchers can't code that well.

A good implementation of the Gini algorithm in practical use is available through Ross Quinlan's website. If you look in to and understand the C5.0 source code, you should be on decision tree research level as there isn't a clear explanation available online as far as I can tell detailing the new algorithm's additions.


What regression model can handle tiny amounts of data? I'm trying to use machine learning to predict properties of a material during a crash test, but each data point requires physically crashing an expensive toy car, so I can only gather a few hundred data points. The end goal is to use the trained model to figure out how to optimize material strength given the parameters I can tweak in the material, which I'm using as inputs.

What type of model would be able to:

Handle very few data points (minimizing number of crashes necessary)
Accurately learn a likely very complex relationship between the inputs (design choices) and the output (performance in crash)
Be able to then be explored to figure out, in as few crashes as possible, which changes to the inputs should be made to maximize the output

The description of your problem is vague. In case you want to do causal analysis, you need to consider the bias-variance-tradeoff. For a causal analysis you would stick to low bias (e.g. using a "best linear unbiased estimator" like ordinary least square regression), while most ML models, such as neural nets or boosting, go for low variance.

Still not clear to me: how does your data look like. If you have more features/variables ($X$) than observations ($i$) your problem is high dimensional. For a high dimensional problem, use the Lasso. If $i$>$X$ and you have say at least 40+ degrees of freedom, you could simply apply OLS regression. If you (can) use OLS and you care for significance and confidence bands, use robust standard errors (hc2, hc3).

As mentioned above, you will likely have high variance using OLS or Lasso. What you can do to work on that in a causal model, is to make linear transformations of $x_i$, e.g. add polynomials, or you can add interactions of two $x$, e.g. $x_3=x_1 x_2$.

Finally, you could use generalized additive models (e.g. with regression splines) to model "complexity" if it originates from non-linearity in $X$. However, in this case you go in the direction of decreasing variance (at the cost of probably introducing unwanted bias).

I guess the nature of your problem is of "high dimension". This is a little special (and I'm no expert here). Have a look at "Elements of Statistical Learning", Chapter 18. The book gives a really good overview of possible options to deal with this problem.


Choosing regularization method in neural networks When training neural networks, there are at least 4 ways to regularize the network:

 L1 Regularization
 L2 Regularization
 Dropout
 Batch Normalization

plus of course other things like weight sharing and reducing the number of connections, which might not be regularization in the strictest sense.

But how would one choose which of those regularization methods to use? Is there a more principled way than "just try everything and see what works"?

There are not any strong, well-documented principles to help you decide between types of regularisation in neural networks. You can even combine regularisation techniques, you don't have to choose just one.

A workable approach can be based on experience, and following literature and other people's results to see what gave good results in different problem domains. Bearing this in mind, dropout has proved very successful for a broad range of problems, and you can probably consider it a good first choice almost regardless of what you are attempting.

Also sometimes just picking a option you are familiar with can help - working with techniques you understand and have experience with may get you better results than trying a whole grab bag of different options where you are not sure what order of magnitude to try for a parameter. A key issue is that the techniques can interplay with other network parameters - for instance, you may want to increase size of layers with dropout depending on the dropout percentage.

Finally, it may not matter hugely which regularisation techniques you are using, just that you understand your problem and model well enough to spot when it is overfitting and could do with more regularisation. Or vice-versa, spot when it is underfitting and that you should scale back the regularisation.


Clustering a very large number of very small clusters with most data unrelated I'm trying to detect duplicates in a data set of about 34k distinct items. When I say "duplicate," I don't mean identical items, just very similar. I have an algorithm that will Cartesian join the items and return a sparse matrix of similar items with a similarity from 0.0 to 1.0. This matrix has about 5k non-zero entries in it, so it's very sparse.

I already know what the similar items are by pairs. What I need is a good way to cluster them together. I expect there to be many clusters with only 2 items in them, and a couple thousand with more than 2 items, with the vast majority of items being unclustered.

Is there a clustering algorithm that fits this scenario well? I've tried several, but they either give poor results, or are designed with few large clusters in mind instead of many small clusters, and so tend to crash.

Is clustering the wrong approach?

I'm using Spark, but the sparse matrix is small enough that I don't mind exporting to something else if necessary.

At this data size, you can still use hierarchical clustering.

You can stop the clustering early when the similarity is too low.

But all of these approaches are pretty inefficient. The proper approach is to not use clustering at all. Instead, use similarity search or, e.g., minhash.


Filter dimensionality in ConvNNs This is a major confusion for me - I've always thought filters (those small sliding windows size 3x3 or 5x5) are strictly 2D, this among other is specified in Caffe API:

  kernel_size / kernel_h / kernel_w. The filter dimensions, given by
  kernel_size for square filters or kernel_h and kernel_w for
  rectangular filters

Now, I've never been 100% clear on what these convolutions mean (taken from Matt Zeiler's article (Visualizing and Understanding Convolutional Networks) and how they were obtained:

After searching it turned out that filters can be 3D?? How does it work then? I found a bit of information on CS 231n course, but it's not explained in detail anywhere.

The diagram below Example 2 in this CS231 lecture is a good visualization of how filters work: http://cs231n.github.io/convolutional-networks/. Since each layer of a ConvNet has a depth corresponding to the number of filters (in an input RGB image, for example, the depth of the input layer would be 3), each filter of size $k\times k$ applied to a layer of delth $d$ slides over a volume of $k\times k\times d$ (note that the filter stays at a constant depth).


Interpretation of coefficient on interactive effects term in regression analysis I have two continuous variables: $X1$ and $X2$, both have a positive correlation on the dependent variable $y$ (continuous). I found that the interaction term $(X1*X2)$ is statistically significant for $p\lt(0.00)$ when added to the model. The final model is the following:
$$y=aX_1+bX_2-c(X_1*X_2)$$
Where $a$, $b$, and $c$ are the regression coefficients. However, I am wondering why the regression coefficient for $c$ of the interaction term is negative? What does it mean?
If I only add the interaction term $(X1*X2)$ to the model why is the coefficient positive when if I add it with $(X1)$ and $(X2)$ the regression coefficient becomes negative? It doesn't make sense!

If the following statement is true, Then it means y should increase as either X1 or X2  increases.

I have 2 continuous variables; X1 and X2, both have a positive
correlation on the dependent variable y (continuous)

This is your final model:

$y=aX_1+bX_2-c(X_1*X_2)$

but I wonder why the regression coefficient c of the interaction term
is negative?  what does it mean?

you have a negative sign before the c coefficient which means it tries to decrease y as X1,X2 increases but the data necessitates that it should increase. Hence c is negative.

why if I add only the interaction term (X1*X2) to the model, the
coefficient is positive while if I add it with (X1) and (X2) the
regression coefficient becomes negative? it doesn't make sense!

By adding just X1*X2 to your does it look like the equation below?

$y=c(X_1*X_2)$

Then it makes sense only if c is positive if not then Y will decrease with increase in X1, X2. It makes sense.


What does the limit of xgboost max_depth=1 represent? In my mind, this means that each tree just takes one feature, and produces a step function based upon it.

In the limit of n_estimators being very large and max_depth=1, does xgboost become a linear model?

On my dataset, a gridsearch found max_depth to be 1, so I'm wondering if I should be building a linear model instead.

A decision tree model is a non-linear mapping from x to y where XGBoost (or LightGBM) is a level-wise decision-tree ensembling algorithm, so your model will still be nonlinear with max_depth = 1. But max_depth = 1 will most probably block your algorithm from your model getting complex enough to capture complex patterns from the data, since you permit only for one split of the tree. Max_depth is used for preventing overfitting, to avoid tree growing very deep.

I suggest you looking at XGBoost documentation, since tree parameters have a strong correlation with each others upon hyperparameter tuning in the way: https://xgboost.readthedocs.io/en/latest/parameter.html


Just a suggestion:

Let the length of the most transaction over all ID be $n$.

Prepare a list of size $n$, $A$, initialize all entries to $0$.

Now go through each ID, suppose a particular ID has $m$ transaction.  We add the value of the $i$-th transaction to the $i$-th entry of $A$.

The list $A$ would have your desired information. The $i$-entry stores the sum of values of the $i$-th day.

Alternatively, you can also introduce a new feature that encode which transaction  is it for each ID and then use groupby to sum it up again.


Can Reinforcement Learning work for Dutch auctions? I had the following idea of Reinforcement Learning:

There is the RL agent and the environment. The agent can execute actions, sends them to the environment and receives an observation and a reward (a float).
Accounting problem: The reward might be delayed.
Learning algorithms: Q-learning is one example. It basically tries to learn the mapping between State x Action to reward and works like this

Can Reinforcement Learning work for a setting similar to Dutch auctions?

The agent can make 3 actions: Increase, keep or lower the price
The agents goal is to sell the item for the highest price. So the agent is the auctioneer.
The auction item does not change. The bidders are anonymous and thus don't give a hint about their behavior.
This means the observations are the same, only the time to the end gets lower - independently of the action of the agent.

Whether or not a Dutch Auction can be framed as a reinforcement learning problem depends on whether there is meaningful state that can be changed by the auctioneer's actions.

If the bidders are behaving in a simplified rational manner* - e.g. they are working to a simple mechanic of buying the item when it reaches a price they have decided to pay - then the behaviour of the auctioneer will not make much difference, it is very formulaic as it is already part of a price optimising strategy. There is no meaningful state change between actions - the bidder valuations and attention stay fixed no matter what the auctioneer does. In an extreme, automated case, the optimal action would be to set a very high initial price and then reduce it in many small increments so as to trigger the highest purchase possible from bidders. This is not done in real auctions because it is a waste of everyone's time (and it would be weird, socially). However, it shows you a problem with the framing of the problem. In a basic rational setting, with nothing to gain or lose apart from money and value of the item, there is little room for altering strategy.

The problem becomes far more interesting when bidders are not purely rational optimisers themselves (or have some other unknown values related to the auction such as personal time they are willing to invest in the process, or rivalry with other bidders), and when some information is available about their past behaviour, or at least in general about human behaviour. In that case the problem can modelled as a Partially Observable Markov Decision Process (POMDP), and you can concern yourself with rewards that might vary due to your actions - e.g. risk of making bidders skittish with "odd" behaviour as the auctioneer. There are limits to this due to the anonymous bidders. If you kept bidder identity between items (if a single auction consisted if multiple item sales with the same bidders) then you might gain some information on each sale in order to optimise better your actions within that auction.

Another possible way to have stateful behaviour is if item ownership has value, such that market forces of supply/demand come into effect (once a bidder has one item maybe they value the next one less, or maybe more if multiple items have higher value to the bidder when kept together). Again this is only going to be relevant to the auctioneer if bidder identities remain constant for a multi-item auction, and even then only makes a difference if there is non-rational pricing behaviour in the bidders.

The main problem you will have with training RL auctioneer to work with non-rational bidders, is modelling those bidders. You could maybe bootstrap with data from real auctions with human auctioneers - this can still be done in a Q learning framework. Q-Learning can learn about optimal policy from observing non-optimal behaviour within limits, optimal actions still need to be observed, but don't have to all be in the same episodes. However, in general to improve from that model and apply reinforcement learning, you have to you put it in control. When it explored unusual actions in order to learn, that could make it a liability if real money was at stake - to defend against this you may need to keep action choice limited.

* Staying with "purely rational" approaches, you could get more sophisticated. You could maybe model the bidders as trying to predict the auctioneer and other bidder's next action, and extract the best expected value, in which case you have framed the problem more like adversarial game theory. This could make the auctioneer's actions have more impact. The most interesting part of this is occurring in the bidder though, not the auctioneer, as the bidder would have to incorporate a meaningful value to owning the item and be smart enough to second-guess other bidder's valuations.

The auctioneer's goal would then be to find the best price that the highest bidder was willing to pay, and behave so that the highest bidder predicted that a rival would bid on the next lower increment. This would be very similar to the non-rational approach in practice, but more tractable to do in simulation - you would need a good simulated model of the bidders before you could train an auctioneer, and the bidders would probably need to be RL or similar too, with both auctioneer and bidder models evolving together.

Intuitively, and assuming the game theory model was stable here, I would expect the auctioneer, with no other information to go on than the number of bidders, would have an optimal strategy where selling price decreased in increments inversely sized to some fractional power of the number of bidders, as more rivals means that bidders will predict rivals willing to pay values closer to their own evaluation. A partially stochastic policy might also be favourable to the auctioneer, as being unpredictable means that bidders cannot rely on guessing the next price reduction accurately (although they might counter that by choosing whether to bid stochastically).


Geometric Interpretation of Whether SVMs are performing well or not I came across this research paper which contained this figure

which talks about the center of mass (presumably, of the training dataset's datapoints?) and represents the solution of an SVM as polygon (or is it a point?).
I'm having trouble understanding this figure, and since it seems to provide a geometric interpretation of when an SVM performs poorly or not, I'm interested in it. Any thoughts?

A most interesting paper! Following Opper and Haussler$\dagger$, the authors define a version space; the set of unit vectors which separate the training samples:

$\mathcal V \equiv \{ \mathtt w | y_i f(\mathtt x_i) >0 , i= 1, \ldots, n, \| \mathtt w \|_2 = 1 \}$

Remember that we are dealing with a classification problem where $y_i \in \{-1, +1\}$, so we want our classifier $f(x_i)$ to have the same sign as $y_i$. It might help to recall that $\mathtt w = \sum_i \alpha_i \phi(\mathtt x_i)$ and $f(\mathtt x) = \left< \mathtt w, \phi(\mathtt x) \right> = \sum_i \alpha_i k(\mathtt x_i, \mathtt x)$

Normally $y_i \left( \left< \mathtt w , \phi(\mathtt x_i) \right> + b \right) \ge 1$. What they've done is to set $b=0$ and eliminate the margin (RHS). The length constraint is to ensure uniqueness.

  The version space is illustrated as a region on the sphere as shown in
  Figs. 5 and 6. If the version space is shaped as in Fig. 5, the SVM
  solution is near to the optimal point. However, if it has an elongated
  shape as in Fig. 6, the SVM solution is far from the optimal one.

The reason there is an inscribed sphere is that:

  the SVM solution coincides with the Tchebycheff-center of the version
  space, which is the center of the largest sphere contained
  in $\mathcal V$. However, the theoretical optimal point in version
  space yielding a Bayes-optimal decision boundary is the
  Bayes point, which is known to be closely approximated by
  the center of mass.

In other words, they are saying that the SVM classifier is not always Bayes optimal. Please see the references for proof. The hypersphere comes about when you consider the margin $\gamma \equiv \min y_i f(\mathtt x_i)$. The support vectors are then the boundaries of the version space which are tangent to the hypersphere.

A cited paper* follows this idea to develop an algorithm to actually estimate the center of mass "by averaging over the trajectory of a billiard ball bouncing in version space." If you're interested in diagnosing SVM failures, perhaps it would help to read that paper too, since it claims to offer a better algorithm instead.

$\dagger$ M. Opper and D. Haussler, “Generalization performance of Bayes optimal
classification algorithm for learning a perceptron,” Phys. Rev. Lett.,
vol. 66, p. 2677, 1991.

* T. Graepel, R. Herbrich, and C. Campbell, “Bayes point machines: Estimating
the bayes point in kernel space,” in Proc. IJCAI Workshop Support Vector Machines, 1999, pp. 23–27


3-way factorization machines Factorization machines are a great way to model interactions between different features, in a way that is more efficient than full-blown quadratic interactions. There exist efficient implementations like libFM and Vowpal Wabbit's --lrq option.

My question is about higher order interactions: the FM paper mentions that it is "easy" to extend it to higher orders (Tensor FM?), see. Eq. (5) in the paper. Are there any good implementations of 3-way factorization machines?

[moved from cross-validated]

I have recently started polylearn, a scikit-learn-contrib package for efficient implementation of factorization machines and polynomial networks, in Python with Cython. Polylearn currently coordinate descent solvers for 2nd and 3rd order factorization machines, and for arbitrary order polynomial networks, using the approach described in the papers cited on the polylearn website.

I'm currently working on an implementation of arbitrary higher order FMs using SG and CD style algorithms, using the approach from this recent paper, I should have it ready within a week.

Polylearn has not had a release yet, so I would love to hear any feedback on the API and functionality, in order to round things off before the 0.1 release!


Xgboost predict probabilities When using the python / sklearn API of xgboost are the probabilities obtained via the predict_proba method "real probabilities" or do I have to use logit:rawand manually calculate the sigmoid function?

I wanted to experiment with different cutoff points. Currently using binary:lgisticvia the sklearn:XGBClassifier the probabilities returned from the prob_a method rather resemble 2 classes and not a continuous function where changing the cut-off point impacts the final scoring.

Is this the right way to obtain probabilities for experimenting with the cutoff value?

Curious Georg if you ran across this article in your pursuit of trying to generate probabilities. It is worth noting that binary:logistic and multi:softprob return predicted probability of each data point belonging to each class.

You can look here to see how the following code is used:


How are precision and recall better metrics than accuracy for classification in my example? I'm trying to understand precision and recall with an intuitive example, but my calculation doesn't seem right.
For example, there are 8 red balls and 2 blue ones. I'm stupid and just predict all of them are red. The accuracy would be 0.8 and looks good. But it doesn't reflect that it's just a lucky, stupid guess, and I missclassify all of the blue balls. If I'm not mistaken, we have 8 true positives (TP = 8), 2 false positives (FP = 2), and neither true nor false negatives (TN = 0, FN = 0) in this case. Then isn't precision = TP/(TP+FP) = 0.8 and recall = TP/(TP+FN) = 1? Don't they still look good?

Your calculation is correct, but you forgot to ask yourself one question: why should we consider "red" as the positive class? Precision and recall can be calculated for every class (i.e. considering the current class as positive), as opposed to accuracy.
So if we take "blue" as positive we get:

precision = NaN (because there's 0 predicted positive)
recall = 0 / 2 = 0

Usually after that one would calculate micro- and/or macro-precision/recall:

macro-precision = NaN because the precision is NaN for blue
micro-precision = totalTP / totalTP+totalFP = 8/10
macro-recall = ( 1 + 0 ) / 2 = 0.5
micro-recall = totalTP / totalTP+totalFN = 8/10

By definition, micro-performance favours the majority class so it's still high. However macro-performance would clearly show the problem that accuracy cannot show.


What is the best Big-Data framework for stream processing? I found that Apache-Storm, Apache-Spark, Apache-Flink and TIBCO StreamBase are some powerful frameworks for stream processing. but I don't know which one of them has the best performance and capabilities.

I know Apache-Spark and Apache-Flink are two general purpose frameworks that support stream processing. but Apache-Storm and TIBCO StreamBase are built for stream processing specially. Is there any considerable advantage between these frameworks?

Thanks

You might also want to take a look at Apache Flink. It is a new contender to Apache Spark as it can do both, realtime and batch analysis. It also claims to be faster than Apache Spark, but I think mostly in depends on your usecase.

Good thing is: Apache Flink project has some Storm compatibility layer in its development version. In this way you can take your existing Storm project and simply run it inside Flink. For me it was at least 2 times faster than Storm in a very simple topology. So give it a try! :)


Math PhD (Nonlinear Programming) switching to Data Science? I am a math Ph.D. student who is interested in going to the industry as a Data Scientist after graduation. I will briefly give some background on my education before posing my question, so that it is better understood:

Maths Coursework:

This has been mostly in pure maths: topology, functional analysis, etc, but also include more applied ones(on which I have specialized for the dissertation): convex optimization, nonlinear programming, numerical analysis, linear programming, multiobjective optimization. In addition, I have right now 0 knowledge of inferential stat, but I am confident in probability theory.

Programming:

I just took a year-long course in the Bachelor, but it was mostly Mathematica and some Java, of which I remember nothing honestly. In this course, the content did not include anything of data structures or design and analysis of algorithms, nor databases management systems. I also learned Matlab on my own for implementing algorithms in the bachelor thesis.

The above background was during the Bachelor and Master's program. Now, during the Ph.D. program, I discovered that Machine Learning is the perfect mix (for me) between Nonlinear Optimization, Programming and applications in the real world, i.e, it is both theoretically interesting and application-oriented. This is the reason why I became so excited to go to industry. Hence, I started learning things on my own(in my little free time) during the last 3 years.

Short summary of things learned:

Python: I am comfortable implementing optimization algorithms, work with jupyter notebooks and the numpy library (in fact, I had to do this for the dissertation), and doing basic data manipulations and cleaning tasks in pandas. This I learned online, in a platform called dataquest (https://app.dataquest.io). However, I don't think I have enough knowledge to pass an interview in data structures and algorithms(see above).
Machine Learning: I took a master level course in the topic at the uni (since I am in Germany, we don't have courses in the Ph.D., so this was all in my personal time), which I really enjoyed. Topics included: k-NN, PCA, SVM, NN, etc.
Taking a course in Databases this semester, which focuses on SQL.
Taking Deep Learning specialization on Coursera this semester.

Finally, I want to say that I feel totally capable of learning the topics. In fact, with time I intend to take more graduate-level courses available online(for example, Stanford CS231N, CS234, etc) because, in my opinion, online courses may not be rigorous enough. Hopefully, after the defense, I will be able to focus full time on this.

Hence the questions:

Can I still get hired at this point (I mean, after finishing this semester with the knowledge described above)? I honestly think I am not ready, but I feel confident that I  can get decent in a year.
Am I being too naive in thinking a company would give me a chance?
What should I do to become more hirable in any case?

Data science jobs cover a wide range of different activities so any answer is likely to be subjective. I'm in academia so my knowledge of the job market is limited, but from what I can see:

The current context is very favorable to data scientists looking for a job, so anybody with some basic knowledge of ML has a chance. You're already above this level so no worries on this front imho. The trend is reasonably likely to continue like this, but nobody knows the future.
Ideally to increase your chances you should be able to demonstrate that you have some hands-on experience: for instance github projects, participation to a ML competition, this kind of thing.
Don't neglect your maths PhD: you seem to have a lot of free time to learn data science, good for you... But make sure the PhD is your top priority, it's going to be a great asset on your resume even if it's not related to data science.


Robustness of hyperparameter tuning I use a Bayesian hyperparameter (HP) optimization approach (BOHB) to tune a deep learning model. However, the resulting model is not robust when repeatedly applied to the same data. I know, I could use a seed to fix the parameter initialization, but I wonder if there are HP optimization approaches that already account for robustness.
To illustrate the problem, let's consider a one-layer neural network with only one HP: the hidden size (h). The model performs well with a small h. With a larger h, the results start to fluctuate more, maybe due to a more complex loss landscape; the random initialization of the parameters can lead to a good performance, or to a very bad performance if the optimizer gets stuck in a local minimum (which happens more often due to the complex loss landscape). The loss vs h plot could look something like this:

I would prefer the 'robust solution', while the 'best solution' is selected by the HP optimizer algorithm. Are there HP optimization algorithms that account for the robustness? Or how would you deal with this problem?

One option is not to measure the performance of the hyperparameters on the loss function of the training data but measure performance of the hyperparameters on the elevation metric on the validation data. The end goal of the most machine learning systems is the ability to predict on unseen data. Focusing on "best solution" as measured by loss function on training will lead to overfitting / non-robust solutions.


Train neural network for regression with negative samples I have training samples which have have vector $\vec x$ as input and a vector $\vec y$ as output - both vectors have real (float) numbers $\in \mathbb R$ as entries. I want to train a neural network such that if I put an $\vec x$ I get the right $\vec y$ out. Obviously I am having an input layer of as many neurons as $\vec x$ is large in dimension and output neurons as large as $\vec y$ is in dimension.

In my training set I have a lot of sample where I have a given input and and given output and I know that the output is not what I want (a wrong sample, or a negative sample if you will). Of course I could just not use those samples for training, but then I would through away some perfectly good information.

I know there are papers out there using negative samples for classification (categorical output), but I do not know of an approach for regression (continuous output).

How do I use my negative samples?

An option might be to tinker with the cost function: Usually one uses the difference between the target output ($out_{t}$) and the actual output ($out_{a}$) as a part as the cost function. Let's say

$$
J = \sum_{i=1}^N (out_{t,i} - out_{a,i})^2
$$

is the cost function, where N is the number of samples. Now we can adjust this to the cost function

$$
J = \sum_{i=1}^N type_i \cdot (out_{t,i} - out_{a,i})^2
$$

where $type_i$ is 1 for normal samples and -1 for "negative samples". Alternative one might also use

$$
J = \left( \sum_{i=1}^M (out_{t,i} - out_{a,i})^2 \right) + \left( \sum_{i=M+1}^N \frac1{(out_{t,i} - out_{a,i})^2} \right)
$$

as the cost function where the samples up to $M$ are the normal ones and from $M+1$ to $N$ are the negative samples.


How to determine individual contribution when data is aggregated by team I want to know the individual contribution of each salesperson in terms of actual sales where the dataset I have aggregates sales data by sales "team".

The dataset looks as follows:

Month | # of Sales | SalesPerson1 | SalesPerson2 | SalesPerson3
1 | 25 | Fred | Susan | Tom
1 | 12 | John | Wayne | Connor
2 | 16 | Fred | John | Wayne
2 | 11 | Susan | Tom | Winona
3 | 34 | Connor | Susan | Tom

Is there a way to predict the individual contribution of any given salesperson based on the data above where the membership of any given sales team can change from one month to the next?

I want to be able to say that for any given sales person, they close x # of sales per month.

Thanks!

Two alternatives

TrueSkill:

You might want to try a variant of TrueSkill. As Wikipedia says:

  Factor graphs are used to "pack up" each team into $(\mu ,\sigma )$
  pairs on which the update formulas are run; the skill updates are then
  correctly distributed to each player.

Here's the TrueSkill paper with more details. From the abstract:

  The new system tracks the uncertainty about player skills, explicitly
  models draws, can deal with any number of competing entities and can
  infer individual skills from team results. Inference is performed by
  approximate message passing on a factor graph representation of the
  model.

Simple Regression:

You can try to set this up as a multivariate regression problem. Inputs are tuples representing who was involved in the sale, outputs are # of sales. So something like this, assuming tuples are (Fred, Susan, Tom, John, Wayne, Conner, Winona):

(1, 1, 1, 0, 0, 0, 0), 25
(0, 0, 0, 1, 1, 1, 0), 12
(1, 0, 0, 1, 1, 0, 0), 16
etc...

Now each person's coefficient should be proportional to their contribution, assuming no interaction effects.


How to fix these vanishing gradients? I am trying to train a deep network for twitter sentiment classification. It consists of an embedding layer (word2vec), an RNN (GRU) layer, followed by 2 conv layers, followed by 2 dense layers. Using ReLU for all activation functions.

I have just started using tensorboard & noticed that I seemingly have extremely small gradients in my convolutional layer weights (see figure)

I believe I have vanishing gradients since the distribution of CNN filter weights does not seem to change & the gradients are extremely small relative to the weights (see figure). [NOTE: the figure shows layer 1, but layers 2 looked very similar]

My questions are:

1) Am I interpreting the plots correctly that I do indeed have vanishing gradients & thus my convolutional layers arent learning? Does this mean they are currently essentially worthless?

2) What can I do to remedy this situation?

Thanks!

UPDATE 3/13/18

Few comments:

1) I have tried the network w/ just 1 layer and no layers (RNN-->FC), and having 2 kayers does empirically improve performance.

2) I have tried Xavier initialization and it doesnt do much (the previous default initialization mean value of .1 was very close to the Xander value)

3) By quick math, the gradients seem to change on the order of 1e-5, while the weights themselves are on the order of 1e-1. Thus at every iteration, the weights change 1e-5/1e-1*100% = ~.01%. Is this to be expected? What is the threshold for how much the weights change until we consider them to have converged / consider the changes to be useless in the sense that they dont change outcome?

In order to fix the problem of vanishing gradients, you can use Xavier Initilization. Also, the implementation of Xavier Initialization in tensorflow can be done by following this thread.


What statistical method can be applied in my case? I want to study the impact of 26 parameters on one variable, and therefore determine the 3 or 4 which have to most influence on it. For that, I have constructed a 10 x 26 matrix: 26 parameters with 10 observations, and a vector of 10 arrays containing the variable I want to study with 10 observations. From this, how can I determine which are the 3 or 4 most influential variables on the parameters. I was considering using PCA, but I'm not sure that's the right approach. Can anyone give me some direction here?

There are a number of methods for this. Here's a list:

You can build a Regression model and observe the p-values of the coefficients of each variable.
Pearson Correlation
Spearman Correlation
Kendall Correlation
Mutual Information
RReliefF algorithm
Decision trees
Principal Component Analysis (which you have tried)

etc.

You can search for other methods with these keywords:

feature selection, variable importance, variable ranking, parameter selection

They are all almost the same, just different terminology in different fields.

However, the issue that you MUST pay attention here is "10" observations are almost nothing. It will be extremely difficult to trust any outcome, regardless of the method you use.