Why Sckit's fit_transform causes a huge drop in accuracy and all other evaluation metrics? Trying to use sc.fit_transform(X), I get a huge drop in accuracy on the same model. Without scaling the values of my dataset, I get accuracy values of 80 - 82%. When I try to scale them, using sc.fit_transform(X), I get accuracy values of 70 - 74%.

What could be the reasons for this huge drop in accuracy?

EDIT:

Here is the code I am using:

# read the dataset file
basic_df = pd.read_csv('posts.csv', sep=';', encoding = 'ISO-8859-1', parse_dates=[2], dayfirst=True)

# One-Hot-Encoding for categorical (strings) features
basic_df = pd.get_dummies(basic_df, columns=['industry', 'weekday', 'category_name', 'page_name', 'type'])

# bring the label column to the end
cols = list(basic_df.columns.values) # Make a list of all of the columns in the df
cols.pop(cols.index('successful')) # Remove target column from list
basic_df = basic_df[cols+['successful']] # Add it at the end of dataframe

dataset = basic_df.values

# separate the data from the labels
X = dataset[:,0:45].astype(float)
Y = dataset[:,45]

#standardizing the input feature
X = sc.fit_transform(X)

# evaluate model with standardized dataset
#estimator = KerasClassifier(build_fn=create_baseline, epochs=5, batch_size=5, verbose=0)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=seed)
#estimator.fit(X_train, Y_train)
#predictions = estimator.predict(X_test)
#list(predictions)

# build the model
model = Sequential()
model.add(Dense(100, input_dim=45, kernel_initializer='normal', activation='relu'))
model.add(Dense(50, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit the model
history = model.fit(X_train, Y_train, validation_split=0.3, epochs=500, batch_size=10)

There is a part of code commented, as I tried to use the KerasClassifier in the beginning. But both methods end up with much less accuracy (as stated above), when I use fit_transform(X). Without using fit_transform(X) I get an accuracy of 80 - 82%. Without 70 - 74%. How come? Am I doing something wrong? Does scaling the input data not always lead to better (or almost same accuracy results at least) AND primarily faster fitting? Why is this big drop in accuracy when using it?

PS: 'sc' is StandardScaler() --> sc = StandardScaler()

Here is the dataframe used (in 2 photos, because it too wide to make a screenshot in just one photo) with column 'successful' as label-column:

Without seeing the actual data its kind hard to say.
I do have one speculation, when using the scaler on all data(before the train/test split) you create a data leakage.
Meaning, some of the data the model fitting should not see(the test set) is effectively included in the train set(because the scaler seen it and used if for setting the scales).
This data leakage can cause overfitting and thus lower accuracy score.
Trying doing the scale part on each train/test split(use fit_transform on the train and only transform on the test).
This is a better practice for model research and obviously closer to how the trained models will perform live(on not seen yet data).


Why does ada (adaboost) in R return different training error graphs and variable importance plots when running the same function multiple times? Question says most of it. I created a matrix of descriptors, set the vectors of responses, and input a set number of iterations. Each time I run the function with the same exact inputs, I get the same confusion table, but I get a different training error graph and different variable importance plots.

Boosting, together with bagging, falls into the realm of so cold ensemble models: you randomly draw a sample from the data, fit a model, adjust your predictions, sample once again. Unless your samples are fixed, every time you run the algo you'll get slightly different results.


when to use certain metrics for splitting decision trees? So I just very recently learned about decision trees, and the different metrics for determining the best split when training the tree. I cannot seem to be able to find anything on which metric to use for certain situations, or which one is better to use for certain kinds of data?

The ones I'm comparing are:

Chi-Square
Gini Impurity
Gini Index (Gini Coefficient)
Variance
Information Gain
Information Gain Ratio

When should I use which ones, or is there an insignificant difference between all?

This is not really a question about decision trees, but about properties to difference metrics. I would be surprised if you find significant differences in the final performance of your decision tree, for different metrics, but if you do find big differences, please comment!

So I don't think it matters very much what you choose here, but for example, the chi-squared numerical value has a specific meaning that is only valid if the data is normally distributed. If it is not, then the chi-squared can be artificially big, or small. But the best split will probably be the one with the lowest chi-squared, so that's why I don't think there is a big impact, even if your data is not normally distributed.

About the Gini coefficient: A given split results in two samples that you're comparing with the truth. The Gini coefficient has a downward bias for small samples, so if one of your samples after splitting is small, then your metric will be biased. That's not great, so probably, don't use the Gini coefficient in a decision tree.


What do I initialise each model in cross validation with in a multi-layer Perceptron? So, as far as my understanding goes, cross-validation is used to determine the best model.

I understand that once we determine the best model, we then train it on the entire dataset. I'm supposed to be using cross-validation for the multi-layer perceptron that can classify the MNIST dataset. I don't seem to get how cross-validation fits in training the model.

Let's say I'm using 5-fold cross-validation, which means I will have to make 5 different models but, how will the training of these individual model proceed? In particular, I have the following questions:

Will the training of these individual model be as usual(backward propagation)?
What do I initialise each model with? (Random Weights?)
After completing the cross-validation, I have the best model(say B) with me now, what does it mean to train this model on the entire dataset?
(Does it mean, I initialise the weights of the new model being trained on the whole dataset, with those of B).

I think you are trying to do cross validation with hyperparameter tuning, so here is how it's done with k-fold CV (from this answer):

You can split your data into 2 datasets: training and test. k-folds cross validation takes a model (and specified hyperparameters) and partitions the training dataset into k equally sized subsets. Then, it does the following k times:

Trains the model on k-1 of the subsets
Evaluates the model accuracy on the subset that wasn’t trained on.

It then reports the average error. To do hyperparameter tuning, do the steps above using every time a different hyperparameter combinations. Then, choose the set of parameters for which k-folds reports the lowest error. However, be careful to not excessively minimize the k-folds error, since it will often lead to overfitting.

Ultimately, we want a measure of how well our final model will generalize. This is why we created the test set at the beginning—evaluating the model’s accuracy on this set is a useful estimation of its success.

So, k-fold doesn't mean k different models, but k folds of the dataset!

To reply to your questions:

  Will the training of these individual model be as usual(backward propagation)?

Yes each training is as usual, just changes the training set and the hyperparameters.

  What do I initialise each model with? (Random Weights?)

Yes, (still) as usual in neural networks. You are not re-using old weights.

  After completing the cross-validation, I have the best model(say B) with me now, what does it mean to train this model on the entire dataset? (Does it mean, I initialise the weights of the new model being trained on the whole dataset, with those of B).

Well, you are mistakenly exchanging again weights and hyperparameters, but, if you have a very big dataset and cross validating on the entire dataset takes too long, you can:

take a portion of your dataset (maybe 10%), let's call it A
Use A to find the best hyperparameters using k-fold CV as I described before
Now you can use the entire dataset for training (except a test set) the model using those best hyperparameters. With the hope that's the real best model.


Is it possible to make a label automatically in supervised learning(Machine Learning)? My background knowledge: Basically, supervised learning is based on labeled data. Using the labeled data, the machine can study and determine results for unlabeled data. To do that, for example, if we handle picture issue, manpower is essentially needed to cut raw photo, label on the photos, and scan on the server for fundamental labeled data.

I know it sounds weird, but i'm just curious if there are any algorithms/system to make a label automatically for supervised learning.

The problem you ask is probably Semi-supervised learning.

In a Semi-supervised learning problem, you have a small amount of labeled data with a large amount of unlabeled data. The goal of this problem is to build a model that has higher accuracy than using labeled data alone.

On the other hand, if you have no labeled data at the beginning, then you might have to use some automatic system (e.x: sensors) to determine the label of your data. In this paper about robotic grasping, Sergey Levine automatically detects an attempt of grasp as success or failure based on the openness of the grippers (aka fingers) of the robotic arm and the difference of images before and after the attempt. He called this type of problem Self-supervised learning.

Hope this help.


Tools to extract information I have set of technical sentences extracted from few research papers.

e.g., Via analyzing the peer feedback content, it was found that the feedback provided by the mixed mode group was of better quality than that provided by the "peer comments" group; that is, the former provided more detailed feedback to individuals than the latter.

I want to extract the entities in the sentences and the relationships that take part with the identified. (e.g., SOV relationships)

What are the tools that I can use to efficiently extract them?

Open Calais is a free-to-use tool for entity recognition and relationship mapping. It's from Thompson-Reuters so may not be wholly suitable for technical language but worth a try. Has python bindings


Coursera's course "How to Win a Data Science Competition" The Coursera Course "How to Win a Data Science Competition".

Has onyone seen it? How is the quality of it? Would you recommend it?

It's a technically rigorous course.

Recommend? It depends on what your goal is.

Definitely, if your goal is to compete on Kaggle.
Not really, if you want to improve practical ML.

Remember, Kaggle is kind of Formula 1. The fastest car wins. But in the real world, you don't always need a Formula 1 car. Sometimes you need an SUV, sometimes you need a van, sometimes a Tesla ....


Back propagation and Structure of a Neural Network in scikit-neuralnetwork I am trying to learn Neural Networks using scikit-neuralnetwork framework and I know basics about Neural Networks and now trying to implement it with scikit-learn. but I am confused on 2 points.
1- what is the structure of this NN given below? Somehow, in some examples felt to me, some people don't put input layer as a layer. Otherwise, I am thinking this as a 2 layer NN has input layer with 100 nodes and 1 node at the ouput layer.
from sknn.mlp import Classifier, Layer

nn = Classifier(
   layers=[
    Layer("Maxout", units=100, pieces=2),
    Layer("Softmax")],
learning_rate=0.001,
n_iter=25)

nn.fit(X_train, y_train)

2- Does scikit-neuralnetwork do back propagation within the code that I put above?
Thank you!

1)
From what I understand, scikit-neuralnetwork tries to automatically determine the correct input and output sizes by the X and y data you give it when  calling nn.fit. Therefore structure should be:

Input layer with shape determined by X_train
Dense layer with 100 units and maxout activation with 2 linear pieces
Softmax classification layer with as many units as needed for y_train

Seems to use input shape from data here:
https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/backend/lasagne/mlp.py#L183

And output shape from data here:
https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/mlp.py#L62

However, note that maxout seems no longer supported: https://github.com/aigamedev/scikit-neuralnetwork/issues/142

2) Yes it uses backpropagation by calling appropriate lasagne/theano functions to create/compile the backpropagation training function:
https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/backend/lasagne/mlp.py#L50-L103

(Actual training seems to happen here: https://github.com/aigamedev/scikit-neuralnetwork/blob/b7fd0c089bd7c721c4d9cf9ca71eed74c6bafc5e/sknn/backend/lasagne/mlp.py#L316-L335)


Tuning SVM C parameter I would like to ask for help regarding my model.

I have a dataset of preprocessed images and I performed a binary classification with SVM on Python.
I tuned the value of the c parameter from 0.001 to 1 with 10-fold CV and I obtained the following results.

My questions are:

Does this mean that the optimal value of C is 0.001?
The rest means that there is overfitting?

The first value is not 0.000 but 0.001

Thank you.

Essentially, you only want to look at your test accuracy. The training accuracy is only relevant in comparison with the test accuracy. In your situation, it seems that your test accuracy is roughly the same across the board beside a slight peak around 0.620. So I would pick this as the best value for C.

My answer is related to the one I gave on this post, here is a summary:

  Overfitting is not when your train accuracy is really high (or even 100%). It is when your train accuracy increases and your test accuracy decreases
  is low.
  it is not abnormal that your train accuracy is higher than your test accuracy. After all, your model has an advantage with the train set
  since it's been given the correct answer already.


How to choose negative examples for recommendation system? I am building a search recommendation system for e-commerce which generates most relevant results given an input query. I have framed it as a classification problem (learning to rank) and using pointwise ranking to compute relevance scores.

Let us say for every input query Q1, we have to generate 30 most relevant results. Suppose, a user initiates a search query and chooses the 3rd result as a relevant result. My question is how should this interaction be used to generate training dataset for the model :

Q1 - Result 3 : Positive Label
Q1 - Result 1,2 : Negative Label (User obviously saw result 1,2 and didn't like it. Also mentioned as skip-highs in some papers/blogs).
Q1 - Random : Negative Label (just add some random results to add random negatives to the dataset).

What about ranks 4-30. It's possible user saw ranks 4-8, but after those ranks user didn't even see the results. Should these lower ranks be used to generate implicit negatives to feed to the model ?

Pros

Random sampling from implicit negatives in lower ranks will ensure that the model is trained on the entire feature distribution that it would have to score during predictions. If we just use the explicit positives/negatives from Step 1/2, model would have been trained only on very relevant results. Sampling from lower ranks will ensure that the model will be trained on entire spectrum of relevant results too.

Cons

We are adding noise to the training dataset, since the user didn't even interact with these results.

So, the main question is should we include the results from rank 4-30, which were probably not even seen by the user as negative examples in the training dataset or not?

Yes, we should include negative samples, more so to correct selection or sampling bias.

The user would have seen only a subset that is selected by another algorithm or rules which decided that the subset is most relevant to the user. Of that, position bias kicks in and the user may not see everything. So the choices within a query or the documents within a query are likely to be very similar to one another (perhaps small differences). This makes it diff for the algorithm to learn the parameters.

Whereas while scoring in production, almost all choices/documents have to be scored for each query. UNLESS the old model is used first to select a subset of hotels and if learning-to-rank model reranks among the subset.

In my experience adding negative samples (that we can safely tell that user would not have picked/clicked the item even if he/she saw the item) helps. But it is difficult to define such base for all use-cases.

We are still experimenting with this.


what make lightGBM run faster than XGBoost? I am curious on what differences in implementation allow speed up of lightGBM over XGBoost, some times up to magnitude of orders.

First of all both the GBM methods are great and superiority of each algorithm is dependent on the data.
Major Differences between the two is that LightGBM uses a novel technique of Gradient-based One-Side Sampling (GOSS) to filter out the data instances for finding a split value while XGBoost uses pre-sorted algorithm & Histogram-based algorithm for computing the best split.
GOSS assumes that data points with small gradients tend to be more well trained. This means that it is more efficient to concentrate on data points with larger gradients.To attenuate the problem of biased sample it also randomly samples from data with small gradients.
To read more you can refer:

Paper
LightGBM_Documentation


panda grouping by month with transpose Based on the following dataframe, I am trying to create a grouping by month, type and text,
I think I am close to what I want, however I am unable to group by month the way I want, so I have to use
the column transdate. However, when I transpose this, I lose the order

df = pd.DataFrame({'date':['6/2/2017','5/23/2017','5/20/2017','6/22/2017','4/21/2017','7/2/2017','5/23/2017','5/20/2017','8/22/2017','2/21/2017'],'rev':[100,200,300,400,500,-70,-250,-200,400,500],'text':['Car','House','Car','Truck','House','Car','House','Car','Truck','House']})
df['date'] = pd.to_datetime(df['date'])
df = df.sort_values('date')
#New Column
df['transdate'] = pd.to_datetime(df['date'])
df['transdate'] = df['transdate'].dt.strftime('%B - %Y')
#second new column
df['type'] = np.where(df['rev']>0, 'positive', 'negative')

This give me this:

Then, I create a pivot table that I am transposing

df_pivot = df.pivot_table(index='transdate',columns=['type','text'],aggfunc=sum, fill_value=0).T
df_pivot

I am wondering how I can sort the first row, starting with feb 2017, then april 2017 and so on? Or, starting the other way around, aug 2017 then july 2017 ... but keeping the order of the months?

Or, will be best to do the pivot table with index date and then, do the grouping?
If this is the case, how can I do the grouping?

# make a month column to preserve the order
df['month'] = pd.to_datetime(df['date']).dt.strftime('%m')

# create the pivot table with this numeric month column
df_pivot = df.pivot_table(index='month',columns=['type','text'],aggfunc=sum, fill_value=0).T

# create a mapping between numeric months and the English version
mapping = pd.Series(df.transdate.tolist(),index=df.month.values).drop_duplicates()

# replace the columns according to the mapping
df_pivot.columns = [i for i in map(mapping.get, df_pivot.columns)]


What is the purpose of the 'train model' step in data mining? My understanding is that training a model is something done in machine learning using training data so that the model can predict values when new data is given to it.
Data mining is the process to find patterns in the existing dataset. So what exactly is the purpose of the 'train model' step in data mining?

There is a very important difference in Machine Learning (ML) between supervised methods and unsupervised methods:

Supervised learning consists in training a model with some labelled data in order to make the final model able to predict the label on some new (unlabelled) data. This means that the task is designed by choosing exactly what what one wants to predict. For example the task of predicting the author of a text is completely different from predicting the topic of a text, even if the text might be the same.

Note: A "label" is for categorical data, and the task is called classification in this case. The same principle can be applied to numerical values, in which case this is a regression task.

Unsupervised learning consists in detecting patterns in the data with no additional information than the data itself. This means that there's no specific "label" to predict. Very often unsupervised learning is some form of clustering, i.e. grouping instances by similarity. Data mining methods usually belong to this category. In general there is no need for separate training and testing steps with unsupervised methods. However one might need to tune some parameters or sometimes to evaluate the model with some annotated data, so this would require separating training and testing again.


Physical modelling with neural networks - single output + stack ensemble vs multi-output We are trying to replace an existing physical model (8 inputs/7 outputs) with artificial neural networks. The physics behind the existing model is mainly thermodynamics of humid air for air conditioning, with some turbomachinery involved, which yields most likely complex functions between inputs and outputs.

One approach was already done:
single output neural networks (10 NN with same # hidden layers but different parameters like batch size, # epochs, optimizer, etc). Then some sort of stacking ensembled was used: every predictions were  used as new inputs for a single NN to predict a final value.

The accuracy is pretty well, however there are some test datapoints where the absolute errors are high enough to be worried about the predicted value (this could be used for example for air conditioning control strategies, so a bad prediction would result in uncontrolled system).

In order to improve accuracy some colleagues were suggesting to keep it simpler and perform just a multi-output regression with a single neural network.

From the mathematical point of view, I have following questions:

1) would a single output NNs stacking ensembled outperformed the multi-output single NN?
2) is the way of stack ensembling using NN a good approach? I saw some different techniques like arithmetic averaging the inputs

Thanks for your time!
Regards

My guess: I agree with your colleagues.  I see no reason to do anything other than a single neural network with multiple outputs.  If necessary, increase the capacity of that single neural network until you see no further improvement.

An stacking ensemble where you have a few neural nets whose inputs are fed as input into another neural network is itself equivalent to one bigger neural net.  I don't see any reason to expect the stacking approach to train more effectively or to be better for some other reason.

However, this is ultimately an empirical field.  The only way to find out what will work best is to try different approaches and see.  There's not a huge amount of theory that enables us to make predictions about what approaches will be most effective.

You might consider whether you can obtain more training data, particularly in the regimes of particular interest.  Often better training data sets offer higher gains than modifying the neural network architecture.


data size requirements for XGBoost This is a quick question. If I compare neural network and random forest, the data size requirement is huge in neural network, but a decision tree or random forest can work with less number of records too.
Does any such problem occurs with XGBoost as well? Does it also need a lot of data so that it can go in multiple iterations to reduce the error term?

No. Xgboost is more like traditional ML algorithm. It doesn't need too much data and it also perform better than almost all ML algorithm


Output vector size of a LSTM Is the size of the output vector of all machine learning algorithms the same? Can't an ML algorithm predict only one value as output?

I have trained an LSTM network with X, Y, heading, speed(from taxi sensor) against feature_x, feature_y. So given a point, I can predict the next one.

Given the LSTM is many-to-one type how does it produce two values as output(x,y)?

Could an MLP also do that?

Is the size of the output vector of all machine learning algorithms the same?

No, the output vector depends on the architecture that you define, which should itself be dependent on the data that you have.

Can't an ML algorithm predict only one value as output?

Absolutely it can. Think for instance of regression problems. I cannot think of any algorithms that don't allow you to have an arbitrary sized output vector.

Given the LSTM is many-to-one type how does it produce two values as output(x,y)?

I don't think you are using "many-to-one" correctly in this case. It seems that you have a single input which happens to be a time-series and you want to predict a vector containing two values, so it is in fact a "one-to-one" problem.

A network can produce as many outputs as you want. It all depends on the architecture.

Could an MLP also do that?

Yes, absolutely. RNNs are essentially a specific way to use MLPs to achieve a specific purpose (in this case, use the positional information). SO anything an LSTM can do could be replicated using MLPs.


Importance of variables in RandomForest in R I'm using varImpPlot() to evaluate the importance of variables from my rf model, and I can't decide wich variable I have to delete from my model. I'm not sure about the diference between "MeanDecreaseAccuracy" and "MeanDecreaseGini", I guess the last one can measure the purity of the tree nodes, but as an example of my model: I have a variable that has a MeanDecreaseGini of about 100 and at the same time a MeanDecreaseAccuracy of about zero.! Because of this I can't decide if this variable is good or bad.

If the features scale of measurement and/or number of categories vary, permutation accuracy is more reliable than the GINI index (Strob et al. 2007).

However, better yet is to use Boruta all relevant variable selection prior to running random forests to determine which variables are relevant to the target property, and which are not. This allows you to trim your dataset to only useful, or relevant variables.

Strobl, Carolin, Anne-Laure Boulesteix, Achim Zeileis, and Torsten Hothorn. “Bias in Random Forest Variable Importance Measures: Illustrations, Sources and a Solution.” BMC Bioinformatics 8 (2007): 25. doi:10.1186/1471-2105-8-25.


How to know when to stop trainning a deep network? I've been training several auto encoders containing two GRUs as encoder and decoder during last year. It occurred to me that while training there were sequences of epochs which their loss did change just a little bit(less than 0.005 or so). Every time I took this as a sign that my model is not going to be better and stopped training. Now that I think about it I wonder what if I continued to train for more epochs and loss of model started to decrease again? Is there a factor to decide how long and how many epochs are enough to train recurrent networks?

Thats the whole point behind early stopping, don’t train the network to too small training error. It is always a bit tricky with early stopping but following framework might help.

When training, also output validation error

• Every time validation error improved, store a copy of the weights

• When validation error not improved for some time, stop

• Return the copy of the weights stored

After early stopping of the first run, train a second run and reuse
validation data
How to reuse validation data

Start fresh, train with both training data and validation data up to
the previous number of epochs
Start from the weights in the first run, train with both training data
and validation data until the validation loss < the training loss at the
early stopping point


Stop CNN model at high accuracy and low loss rate? I train my CNN model with a large number of epochs, with each epoch I print the training loss and accuracy, but there is a lot of high and low in these two metrics, I want to do early stopping with for example loss at 0.2 and accuracy at %95 or more because I get this at more than one epoch, my question is:

1- is early stopping done on train set or validation set?

2- if on validation set, should I print validation loss and accuracy for each epoch with with a train set loss and accuracy?

3- can you give an idea to do early stopping with a piece of code but not in keras, because I use tensorflow not keras?

Early stopping is determined based on the validation set's results (either loss, accuracy or some other special metric).
Usually early stopping is checked every single epoch so you will need to check your validation accuracy/loss after each epoch. You don't have to print it, but if it is already calculated, there is no reason to withhold it from yourself. (you can also check validation results every x epochs if it slows your training process too much).
There are many sources that show how to use early stopping with tensorflow:

Early Stopping with TensorFlow and TFLearn

Early stopping with tf.estimator, how?

Early-stopping functionality for use with tf.estimator API.


How to handle string features in a neural network I am new to ANNs. How to handle features whicvh consist of strings? E.g. if I have group featire which are different groups conatining differnt capital Leters and numbers.

Now I have the train data set containing all such important features.

My first guess is just to replace stings with numbers. But you have different uncorrelated groups of strings like the category or the manufacturer.

Does the string add any useful data? No. You should replace categories by id's for sure, and manufacturers as well. Why would having uncorrelated groups of strings stop you from giving each an id?

So Google = 0, Apple = 1, etc. But that doesn't mean that a category can't have id 0 or 1. This is a different input, so these should start at 0 too: Books = 0, Electronics = 1. You still have to normalize these id's for your network ofcourse (value = id / maxId).

This might be useful: https://github.com/cazala/synaptic/wiki/Normalization-101


How does a convolutional ply differ from an ordinary convolutional network? I am currently working on recreating the results of this paper.
In the paper they  describe a method for using CNN for features extraction, and have a acoustic model that is Dnn-hmm and pretrained using RBM.

Section III subsection A states different ways the input data can be represented. I decided to vertically stack the spectrum plots of the static, delta and delta deltas.

So as such:

The paper then describes how the network should be.  They state that they use a convolutional network, but nothing about the structure of network?.
Further more is the network always refered to as a convolutional ply? which i am sure i see any difference in  compared to an ordinary network convolutional neural network (cnn).

The paper states this regarding the difference:

(from section III subsection B)

  A convolution ply differs from a standard, fully connected hidden
  layer in two important aspects, however. First, each convolutional
  unit receives input only from a local area of the input. This means
  that each unit represents some features of a local region of the
  input. Second, the units of the convolution ply can themselves be
  organized into a number of feature maps, where all units in the same
  feature map share the same weights but receive input from different
  locations of the lower layer

Another thing i was wondering is whether the paper actually states how many output parameter is needed to feed the dnn-hmm acoustic model. I can't seem to decode the number of filters, filters sizes.. in general details of the network?

It seems that a convolutional ply is exactly the same as an ordinary convolutional layer. From their paper, they argue that the term "CNN layer" usually refers to a convolutional layer followed by a pooling layer. In an attempt to reduce confusion, they name the convolutional part a "convolution ply" and a the pooling part a "pooling ply":

  In CNN terminology, a pair of convolution and pooling layers in Fig. 2 in succession is usually referred to as one CNN “layer.” A deep CNN thus consists of two or more of these pairs in succession. To avoid confusion, we will refer to convolution and pooling layers as convolution and pooling plies, respectively.

Ironically, this has increased the confusion, leading to this post. At the time I suppose it was not common to have several convolutional layers in a row before a pooling layer, but we see this often in modern architectures.

To answer your other question about the network structure; they state the structure of the network that they use in the Experiments section (Section V-B). To hopefully reduce confusion, I've replaced the word "ply" with "layer":

  In these experiments we used one convolution [layer], one pooling [layer] and two fully connected hidden layers on the top. The fully connected layers had 1000 units in each. The convolution and pooling parameters were: pooling size of 6, shift size of 2, filter size of 8, 150 feature maps for FWS, and 80 feature maps per frequency band for LWS.


Multi-label Text Classification I am trying to build a multi-label classifier for suggesting tags on blog posts. The textual data is full of noise. The approach I have been following until now was a BOW approach with Tf-idf weighting. However I could only get an accuracy of around 0.35 on the test set using OnevsRest approach and a SVC. I then decided to eliminate the noise and I applied TextRank on my data to obtain a more meaningful summary and then applied the same BOW approach which however resulted in further loss of accuracy. I am quite new to machine learning and data science in general. Can someone please guide me with the approach? I am looking specifically on how to create my features and what else can I try.
I have 12 labels and around 1500 blog posts. I know it isn't much data, but the data-set is still well-balanced and won't be the primary reason for such low accuracy.

What you seem to want is to identify meaningful discrepancies between your blog posts in order to identify what category they fit in.

I think it's important for you to consider the classification problem in terms of human thinking:

How would humans decide which blog gets which tags?
Can we distil this decision-making into meaningful features? In other words, does this decision-making depend on (linguistic) phenomena that can be found in the text? (it probably does)
(How) can we extract these features from the text?

Also, it might be the case that either the tag set you propose or the data set you have are too similar. This means that the tags might be too similar and that their occurrence depends on many similar feature values which makes it difficult for your training algorithm to extract discriminative features. The same thing happens when the data expresses too little variety.

On a final note: This might well be the first attempt to solving this particular problem (on this data set and this set of labels), so the number you get is not necessarily bad or sub-par, because it might simply be a difficult task.


sentence vector to sentence I have implemented an auto-encoder that takes sentence vectors as input and at decoder the last layer outputs sentence vectors. I would like to convert sentence vectors to sentences. Is there any way to convert sentence vectors to sentences?
class AutoEncoder(nn.Module):

def __init__(self, embedding_dim, hidden_dim):

    #Constructor
    super().__init__()

    self.fc=nn.Linear(embedding_dim,hidden_dim)
    self.act1=nn.ReLU(0.2)
    self.fc1=nn.Linear(hidden_dim,hidden_dim)
    self.act2=nn.ReLU(0.2)
    self.fc2=nn.Linear(hidden_dim,hidden_dim)
    self.act3=nn.ReLU(0.2)
    self.fc3=nn.Linear(hidden_dim,hidden_dim)
    self.act4=nn.ReLU(0.2)
    self.fc4=nn.Linear(hidden_dim,hidden_dim)
    self.act5=nn.ReLU(0.2)
    self.fc5=nn.Linear(hidden_dim,embedding_dim)
def forward(self, X):
    l1=self.fc(X)
    a1=self.act1(l1)
    l2=self.fc1(a1)
    a2=self.act2(l2)
    l3=self.fc2(a2)
    a3=self.act3(l3)
    l4=self.fc3(a3)
    a4=self.act4(l4)
    l5=self.fc4(a4)
    a5=self.act5(l5)
    l6=self.fc5(a5)
    return l6

Here I have considered word embeddings initially and converted them to sentence vectors by averaging. These vectors are given for auto encoder model.
Let me know whether we have any other approach where we can generate a sentence.
Any kind of reference is helpful.

You might need an encoder-decoder framework that can be implemented using two LSTM/GRU models. The Encoder takes the first phrase in, outputs the last (summary) hidden layer, it is taken by the first Decoder state, alongside BOS token, outputs the second token, uses it to predict the third, and so on until you reach the end of the (response) phrase.


Is it possible to train this image classifier? I'm writing a mobile app that will enable a user to scan a Craft Beer Label from a bottle, tap, six pack, etc.

The scan will only work for my customers who are the Brewers themselves, so I will have access to all of the artwork used for their labels.

My concern is around training the model.  As you can imagine, there will be differences in lighting conditions at bars, amoung other challenges.

As I am just getting started in Machine Learning, is this task feasible?  How difficult will it be to train the model given the various conditions found at different pubs/bars.

Thanks.

For sure, is a really feasible application! Indeed, there does exist Deep Learning models for that.

I suggest you to start by doing transfer learning on a pre-trained model like this: https://github.com/satojkovic/DeepLogo Typically you remove the last layers of the network and add layers that adapt to your data. Then, train these last new layers with your data (freezing the layers you have kept from the original model).

Another good advice is to perform data augmentantion on your dataset to include ligth, scale and rotation variants of your images.

Summary:

It's a feasible task
Check out pre-build models
Learn about transfer learning on pre-trained models
Learn about data augmentation to include robustness against different scenarios and env. conditions


Negative values in XGBoost regression I am trying to perform regression using XGBoost. My dataset has all positive values but some of the predictions are negative.
I read on this link that reducing the number of trees might help the situation.
I reduced the estimators from 700 to 570 and the number of negative predictions decreased but is there any way to remove these negative predictions? When I further tried reducing the estimators to 400, I got worse results with high rmse and more negative predictions.
I looked at the negative predictions to understand where the model went wrong(see below image):

It looks like if I take the absolute value of the negative predictions then that would be sufficient but is this the right method?
My code is:
!pip install xgboost
!pip install scikit-optimize
from xgboost import XGBRegressor

final_XGB=XGBRegressor(random_state=123,gamma= 24.47 ,learning_rate=0.1235,max_depth=10,min_child_weight=0.21509999999999999,
                      n_estimators=570,subsample=0.74,reg_lambda=0.8)

from sklearn.model_selection import cross_validate

cross_val_scores=cross_validate(final_XGB,X_train,y_train,cv=3,scoring=['neg_mean_squared_error','r2'],verbose=1,return_train_score=True,n_jobs=-1 )

cross_val_scores['test_r2'].mean() #returns 0.9595609470775659

EDIT:
A bit more about the dataset. I am trying to predict the number of people that would be present at a given location at a given time period.
In order to predict the count of people, I have taken the

hour of day,
date,
peak(is it a public holiday?),
Weekend(1/0),
Area (area of location in m2)
location name.

The location names were string values (eg: Location1, Location2, etc.) so I transformed them using JamesSteinEncoder.
There are no missing values and I have removed a few outliers from the dataset.
Regarding the relationship between "Actual" and "Error" column, I have plotted the graph below:

Snapshot of final results:

Gradient boosting machines can return values outside of the training range. Have a look at this post Can Boosted Trees predict below the minimum value of the training label?
In practice this is unlikely to happen, but it can be the case for your data.
If this is happening probably what it means is that your training data and the one you are evaluating are different.


Elements of Statistical Learning - question on p. 12 I am starting to work through Elements of Statistical Learning, and right off the bat I am coming across things that I don't understand. I would be grateful for any help from this community. Please let me know if this is not the appropriate forum to post these questions (in which case, if you're feeling extra-nice, please point me to the correct forum).

On page 12, the authors present the familiar expression for linear regression:

  Y-hat = X-transpose * β

The authors then state:

  Here we are modeling a single output, so Y-hat is a scalar; in general Y-hat
  can be a K–vector, in which case β would be a p×K matrix of
  coeﬃcients. In the (p + 1)-dimensional input–output space, (X, Y-hat)
  represents a hyperplane. If the constant is included in X, then the
  hyperplane includes the origin and is a subspace;

Questions:

Is the input-output space only p+1 dimensional assuming that Y is a scalar, rather than a K-vector? If Y is a K-vector, would the input-output space be p+K dimensional?
What does the statement that "(X, Y-hat) represents a hyperplane" mean? Assuming that X-transpose is a single column vector representing the fact that there is only one input variable, could you help me visualize what the hyperplane would look like?

Thank you in advance!

You are correct, at p+1 it is returning to talk about the K=1 case again. p inputs produce 1 output. The linear function $\beta$ you fit maps p dimensional X to a p dimensional subspace of the p+1 dimensional space that includes the 'output' and it's specifically just a hyperplane. Try plotting $z = 3x + 2y$ in a graphing calculator for a quick intuition of what single output from p=2 inputs feels like.


Continuous Bag Of Words (CBOW) network architecture? Looking into word2vec like embeddings I found this exercise on PyTorch's website which prompts the reader to implement a CBOW network in PyTorch.
My question is about the architecture to implement this CBOW network.
Here is my understanding: From a number of sources, it seems that the network should have a single hidden layer (with weights and no biases) which is connected to an activation layer (most sources say softmax). Then the network will be trained to map one-hot encoded words to likely contexts. Finally, the hidden layer's weights will be used to as the embedding matrix.
My confusion is: I see a number of solutions like this first one off google where there are multiple hidden layers. In this example, there is a embedding layer and there are two linear layers connected by a relu. Here is another that uses one linear layer.
My questions are:

What is the proper architecture to train CBOW encodings?
If this multiple hidden layer approach is correct, how do you not lose semantic information when you only use one of the layers as the encodings?
If the single hidden layer approach is correct, does anyone have examples of this being implemented using this approach in PyTorch (fine if no)?

Note: Very new to ML so feel free to correct me even on nit picky things so I can learn!

What is the proper architecture to train CBOW encodings?

The original paper by Mikolov et al uses 1 hidden layer. However, for NLP tasks (and deep learning in general), there is no "correct" number of layers to use.
For some tasks, you might find using more hidden layers to be better, for other tasks maybe one hidden layer is sufficient. The number of layers is a hyperparameter, along with other 'design choices' like the dimension of the embedding vectors, the size of the vocabulary, and many, many more.

If this multiple hidden layer approach is correct, how do you not lose semantic information when you only use one of the layers as the encodings?

I think of it this way. When training CBOW, the hidden layers learn some 'relationship function' between the input context words and the output target word. In order for the 'relationship function' to perform well, the embeddings must be arranged accordingly (in N-dimensional space), and it just happens that the optimal arrangement encodes semantic information.
So the hidden layers shouldn't really have anything to do with semantic information. However, because of the immense modeling power of hidden layers (which make up neural networks), the truth is the embeddings will inevitably lose some semantic information to the hidden layers.

If the single hidden layer approach is correct, does anyone have examples of this being implemented using this approach in PyTorch (fine if no)?

Here's literally the 1st github example off Google.


Classify documents using a set of known vocabularies I have a bunch of documents that I want to classify which ones talk about soccer (unsupervised learning, I do not want to manually label the documents).

One way I am thinking about is to go online and search for the most popular words in soccer articles to make a list of vocabularies (for example: score, shoot, World Cup, etc). Then somehow use that list of vocabularies to classify the documents (maybe if a particular contains 30% of the words in that list of vocabularies, then that document talks about soccer).

I am wondering whether it is a valid method or there are better existing methods. Really appreciate any help.

First of all you need to have available a train set, which means that you should annotate manually which document is related to soccer and which not. Then you need to process the available corpus (remove numbers, stop-words etc., stemming) and build a vocabulary. After that you should choose the appropriate feature representation. Each term is a feature and you have to decide how you are going to reprsent each feature, which means what kind of weight you will assingn. One way is the tf-idf representation. Then you will be able to train a classifier.

*The only way to avoid labeling manually the texts is to find some already labeled in the same language.


Architectures that take inputs of mixed sampling rates Let's say a model is trained on multiple datasets of 1D time series. These datasets have been gathered with different sampling rates. I plan to use a convolution neural network to process these time series for classification.
My question is, in this scenario, are there any tricks I can do to the data or the model itself to make it "aware" of the sampling frequency? I've tried naively just training them together, and I'm getting an ok result.  Are there any existing works? I tried googling but found nothing worth noting as of now.
I'm hesitant about just resampling the data is I don't want to accidentally remove/introduce important information by making the method of interpolation a hyperparameter.

With a search online, I couldn't find a satisfying answer, so I came up with a way. I tested and it gave pretty nice accuracy. This method is inspired by this paper.
Similar to the approach in the paper, I first select the number of convolution kernels I want to use, and depending on the sampling rate of a data point, I apply convolution to it using the convex hull of the set of convolution kernels.
So how exactly do I come up with the weight? There are a lot of hyperparameters, but what I did is assign a sampling rate to each kernel by dividing up the expected range of sampling frequency. I then subtract the sampling rate of the input datapoint to see which kernel is the closest to it. Then I take a softmin of the absolute difference, which makes sure the resulting weights add up to 1.
There are many things to play around here: Do we uniformly assign the rates? How many kernels to use in a set? How do you calculate the weight?...


LSTM loss function and backpropagation I'm trying to understand the connection between loss function and backpropagation.
From what I understood until now, backpropagation is used to get and update matrices and bias used in forward propagation in the LSTM algorithm to get current cell and hidden states.
And loss function takes the predicted output and real output from the training set.
But which part is the training part of the LSTM?
Are they somehow connected ?
What is the training objective for the LSTM model?

For sake of convenience, I'll go in ascending order of how the neural
network is working

The forward propagation of the LSTM cell is where the input is fed  ->  passed through the hidden states  ->  output is achieved (either at each time-step or at the end depending upon the type of problem
like classification, forecasting, etc.,)
Then after receiving the output the error between the actual value and the predicted value is calculated. This is where the loss
function comes into the picture
Classification problem - cross-entropy/log-likelihood
Regression Problem - Mean Squared Error, Mean Absolute Error functions are used.
So, after calculating the error only we backpropagate through time (BPTT) in case of RNN neural networks which updates weights of the
matrices and cell states of the LSTM cells. In practice, we refer to
these backpropagation algorithms as optimization algorithms like
Gradient Descent, etc.,
The optimization algorithms like RMSProp, Adam are much faster in practice than the standard gradient descent algorithm.

Overall, the training part of any neural network algorithm is that at
the early stages of training, it will perform very horribly and over
a period of time, with the correct loss function and backpropagation
algorithm it will try to reduce the chances of classifying error and
will converge fast depending upon the backpropagation algorithm used.


Should I use a cleaned labeled data for sentiment analysis? I wanted to work on twitter sentiment analysis.so before that I decided to collect some twitter data and label them on my own (pos,neg,neu). My doubt is that Should I clean the data before I label (i.e., removing RT,#,https,@ symbols) Or I can label them with out cleaning the Data? Does cleaning of data before labeling make any difference?

The OP asks "Does cleaning of data before labeling make any difference?" - that's an empirical question... one that should be investigated by EDA of your data.

In some cases twitter-specific conventions can be highly indicative of a specific class (e.g., http:// associated with spam/advertising tweets) or sentiment (e.g., :-) emoticon associated with positive valence). Similarly, as I discuss in this post, stop words can be great features to keep in certain types of text models. My support for the above answer is based on an (unpublished) project I did that involved about 10 people hand-scoring 50k tweets stratified across major industries (food, travel, electronics, CPG, etc.) and brands (McDonalds, Southwest Airlines, iPhone, Tide PODs).

My advice, create feature extractors for any textual feature that you believe has a theoretical or logical justification for being good indicator of text polarity. Then empirically test each feature to determine whether it significantly improves classification accuracy. Keep those that are in your model; save the other feature extractors for a rainy day.


Writing custom data analysis program I have a number of large datasets (10GBs) each with data fetched from a NoSQL database that I have remotely downloaded on my desktop. I would like to write a Python program to run some custom data analysis (plots - preferably interactive) and export custom reports in html or pdf.

I was wondering how people do the following:

1) Store the data. For the moment I have plain text files (each file has rows of a fixed number of columns - most of the data are categorical). Would it make sense to save those in some database (SQL) or hdf5? Any hints on which is preferrable?

2) Which plotting library would you propose for the graphs? I have seen about bookeh and matplotlib supports interactive widgets but I don't know what people normally use.

3) Could I export the analysis results in an IPython notebook and then in html programmatically?

1) Store the data. For the moment I have plain text files (each file
  has rows of a fixed number of columns - most of the data are
  categorical). Would it make sense to save those in some database (SQL)
  or hdf5? Any hints on which is preferable?

Yes, it would make sense to store in a local database, rather than using large csv/text files. As you say that the data is derived from a NoSQL source, I assume unstructured data. So, using a SQL/relational store is out of question. As you say you are using Python, I would suggest you use TinyDB, which is both light-weight and easy to handle.

  2) Which plotting library would you propose for the graphs? I have
  seen about bookeh and matplotlib supports interactive widgets but I
  don't know what people normally use.

Matplotlib would be good enough. Actually, this question is more opinion-based than anything else. There are a lot of visualization libraries you can use, like Bokeh, Seaborn, etc.

  3) Could I export the analysis results in an IPython notebook and then
  in html programmatically?

Yes, you can do the analytics directly in an Ipython notebook(Jupyter), which also supports Markdown and HTML cells.

In addition, you can also use widgets and interactive visualization with Jupyter Ipy notebooks and Matplotlib.  Tutorials for the same


Multiple Output Layers in Neural Networks in Deep Q Learning I am trying to train a computer to play with LEGO bricks (simulated) using DQN.
My input is an image with 4 color channels (RGB and depth) and the output of the neural network is the coordinates of the best position where the next brick should be put.

Instead of having $m_x * m_y * m_z$ (where $m_k$ is the size of positions along the k axis) different outputs, I want to have 3 output layers, each of which returns the x, y or z coordinates of the best position, so I have only $m_x + m_y + m_z$ outputs.

My idea is replacing the fully-connected output layer by three fully-connected output layers, but only have one reward value (labeled value) for three layers. Can I do this? Will the three layers returns different outputs if the reward is same?

You decide what the output layer represents, so it should be OK (and probably easier to implement) to have a single fully connected layer of size $m_x + m_y + m_z$ and to interpret the output in your own code. E.g. $n_x = N_{(x)}, n_y = N_{(m_x + y)}, n_z = N_{(m_x + m_y + z)}$, where $N_{(i)}$ is the output of the $i^{th}$ neuron. In the case of regression, this should have very similar behaviour to 3 separate fully connected layers for each dimension.

However, once you have either structure, it is not clear that you will be estimating reward directly any more. There is definitely no guarantee that the rewards will be the same between $n_x, n_y, n_z$, in fact they will not be in most cases. You will need to define a function $\hat{r}(x, y, z)$ that combined the outputs of the network and have your loss and error gradient calculated from that. The function could just be $\hat{r}(x, y, z) = n_x + n_y + n_z$ or maybe $\hat{r}(x, y, z) = n_x \times n_y \times n_z$

Clearly this approximation has limits built in - it will likely over-estimate and under-estimate rewards due to its structure. If the simplified structure matches your problem, then this will be fine - it may even speed convergence. But if not, it will limit performance.

An alternative, if you don't want to build a network that has a large number of outputs, is to have a single reward estimate as the output, and have the action choice as an input. That has the advantage of leaving the details of the approximation to the neural network. It has the disadvantage that you would need to run the network up to $m_x * m_y * m_z$ times in order to decide the policy in each state.

Because of the large number of possible actions in your case, you may want to look at a policy-based learning method, rather than DQN. There are deep NN-based versions of those algorithms. A recently-published algorithm called Asynchronous Advantage Actor-Critic (A3C) has performed well in computer gaming tasks and might be appropriate for your problem.


How exactly does a validation data-set work work in machine learning? With typical machine learning you would usually use a training data-set to create a model of some kind, and a testing data-set to then test the newly created model. For something like linear regression after the model is created with the training data you now have an equation that you would use to predict the outcome of the set of features in the testing data. You would then take the prediction that the model returned and compare that to the actual data in the testing set. How would a validation set be used here?

With nearest neighbor you would use the training data to create an n-dimensional space that has all the features of the training set. You would then use this space to classify the features in the testing data. Again you would compare these predictions to the actual value of the data. How would a validation set help here as well?

Machine learning models output some sort of function; for example, a decision tree is a series of comparisons that results in a leaf node, and that leaf node has some associated value. This decision tree predicts survival chance on the Titanic, for example:

(Image by Steven Milborrow, from Wikipedia)

This can be used exactly like the equation generated by linear regression. You feed the new data points into the model, you get predictions out, and then you compare those predictions to the actual values.

The comparison you do is applying a "cost function," and what cost function you use is determined by your application. For example, linear regression can be done with different cost functions; typically, people use "ordinary least squares" which tries to minimize mean squared error, but other possibilities exist, like minimizing the absolute value of the error. When the result is a percentage, then something like a log probability loss function is typically a better choice than mean squared difference between the actual and predicted probability.

Even k-nearest neighbors generates a function; it splits the space into regions where the k-nearest neighbors are the same set, and then has a flat value for that region. It ends up very similar to the result generated by a decision-tree based method.

There's also a terminology point here: typically, people talk about the training set and the test set, or the training set, the validation set, and the test set. The point of the validation set is different from that of the test set; the test set is used to determine out-of-sample error (that is, how much the model has overfit and what its real world error is likely to be like) whereas the validation set is used to determine what hyperparameters minimize the expected test error.

One could fit ten different models on 80% of the data, determine which one has the lowest error on another 10% of the data, and then finally estimate real-world error on the last 10% of the data.


Should i drop simple features after deriving more complex features from them? I know for a fact that complex features projects the data into higher dimensions which makes the previously non-separable data linearly separable.
But, Is this not true that these complex features will be highly correlated with the features from which they are derived.
My question is, Should i drop the simpler features from training set?

It will depend on the underlying relationship e.g.,
$Y = \alpha X_1 + \beta X_2 + \gamma X_1X_2$
In the above case, the Target is dependent on individual features as well as the Interaction features($X_1X_2$). So, you can't skip the individual.
Same will be True for polynomial too,
$Y = \alpha X + \beta X^2$
You may remove the original features for a scenario similar to this,
$Y = \alpha X_1X_2$ Or,
$Y = \alpha X^2$
Another intuition is that we never see a zero correlation to the original features nor we get a 100% correlation to the Interaction features.


ROC curve for deep learning I am working in a deep learning project for image classification. I am using Vgg16 to create the model and the dataset has 100 classes. The testing accuracy is 98.9% and loss is 0.1731

And I got the following ROC and Average Precision curves:

I do not now if the ROC and Average Precision result are good results or there is a problem in the curves because the curves are perfect. Do you know why I got ROC =1? Is there any problem in ROC and average precision?

Mathematically, it is not possible to have the area under the curve equal to 1 if you have a precision less than 1... If you have a doubt, you should check the confusion matrix and calcuate the scores yourself. You'll know directly if you did a mistake or not

A few other remarks:

Accuracy is not precision
The loss is of no help here


Neural network breaks if hidden layers are increased I am trying to make my neural network as dynamic as possible. For example, I want to be able to increase hidden_layers number. It works so far with one hidden layer, but if I increase it to two, it instantly breaks. There is a problem when the backpropagation is reaching the layer between Inputs <- 1st_hidden. Here is my code. I am trying to solve XOR problem by providing this to my NN:

inputs  = np.array([[1, 0], [0, 1], [1, 1], [0, 0]])
targets = np.array([[1], [1], [0], [0]])

Can anyone help me? I can not seem to solve it.

nn = NeuralNetwork(input_nodes = 2, hidden_layers = 1, hidden_nodes = 3, output_nodes = 1)

I initialize my neural network:

class NeuralNetwork:
def __init__(self, input_nodes, hidden_layers, hidden_nodes, output_nodes):

    self.input_nodes   = input_nodes
    self.hidden_layers = hidden_layers
    self.hidden_nodes  = hidden_nodes
    self.output_nodes  = output_nodes
    self.layers_count  = self.hidden_layers + 1

    self.weights       = []
    self.bias          = []

    min_value = -2
    max_value =  2

    # Initialize weights and bias matrices and place them in a list
    # Matrix shape within lists
    # [(2,2), (2,2), (1,2)]

    for i in range(self.layers_count):
        if i == 0:
            self.weights.append(np.random.uniform(min_value, max_value, (self.hidden_nodes, self.input_nodes)))
            self.bias.append(np.random.uniform(min_value, max_value, (self.hidden_nodes,)))
        elif i == self.hidden_layers:
            self.weights.append(np.random.uniform(min_value, max_value, (self.output_nodes, self.hidden_nodes)))
            self.bias.append(np.random.uniform(min_value, max_value, (self.output_nodes,)))
        else:
            self.weights.append(np.random.uniform(min_value, max_value, (self.hidden_nodes, self.hidden_nodes)))
            self.bias.append(np.random.uniform(min_value, max_value, (self.hidden_nodes,)))

    self.learning_rate = 0.1

def train(self, input_array, targets):

    inputs = np.transpose(input_array)

    applied_activation = nn.feedforward(inputs)

    output_errors = np.subtract(targets, applied_activation[-1])

    # Backpropogation
    for i in reversed(range(self.layers_count)):
        delta    = np.ndarray((self.input_nodes,))
        gradient = np.ndarray((self.hidden_layers,))

        if i == self.hidden_layers:
            errors = output_errors
            delta, gradient = calculate_weights_deltas(applied_activation[i], output_errors, applied_activation[i-1])
        elif i == 0:
            # IT BREAKS HERE
            errors          = calculate_errors(self.weights[i+1], output_errors)
            delta, gradient = calculate_weights_deltas(applied_activation[i], errors, inputs)
        else:
            errors          = calculate_errors(self.weights[i+1], output_errors)
            delta, gradient = calculate_weights_deltas(applied_activation[i], errors, applied_activation[i-1])

        delta, gradient = calculate_weights_deltas(applied_activation[i], errors, applied_activation[i-1])
        # Adjust weights by deltas
        self.weights[i] = np.add(self.weights[i], delta)
        # Adjust the bias by its deltas
        self.bias[i] = np.add(self.bias[i], gradient)

The error I get is the following:

            elif i == 0:
                errors          = calculate_errors(self.weights[i+1], output_errors)

            ValueError: shapes (3,3) and (1,) not aligned: 3 (dim 1) != 1 (dim 0)

I found my error. My early calculations only involved one hidden layer, however, I did not include the derivatives of previous weights when calculating the hidden_error. This means that in the scenario of a neural network of:

Input -> H1 -> H2 -> Output

to calculate the backpropagation of the last step (Input <- H1), I did not calculate derivatives of weights_hidden_output (H2 -> Output) and weights_hidden_to_hidden (H1 -> H2).


Neural Network regression negative performance I have a problem with the performance of a multi layer perceptron regressor (neural network) and I cannot figure out why.
Task: I am trying to improve a time series prediction. I have predictions of a physical parameter of the last 4 years along with the quasi true values. I train the NN with the predictions for -7 days until +1 days around the day I am interested in as features, in order to obtain a better prediction for that day.
Problem: The output of the NN is worse than the feature for the day I am interested in, both for the training and the testing data. Both in terms of RMSE and MAE. I expected the output to be at least on the same level as the feature I input to the NN.

Method: Python with sklearn. I use a grid search with cross validation to get good hyper parameter. I test for different hidden layer configurations, activation functions, learning rate and regularization penalty strengths. I split the data into 66% for training and the remaining data for testing.
I am really grateful for tips how to figure out what my problem here is.
EDIT:
I am using sklearn.neural_network.MLPRegressor which provides ‘identity’, ‘logistic’, ‘tanh’ and ‘relu’ as activation functions and I have teseted all of them in the grid search.
I did NOT scale the feature matrix because all features are in the same unit as the desired output and range from about -1 to +1.
EDIT2:
tuned_parameters = [{'hidden_layer_sizes': [int(2/3*number_features),
                                        (int(2/3*number_features), int(4/9*number_features)),
                                        (int(2/3*number_features), int(4/9*number_features), int(8/27*number_features))],
                 'alpha': 10.0 ** -np.arange(1, 4),
                 'activation': ["identity", "relu", "logistic", "tanh"],
                 'learning_rate': ['adaptive', "invscaling"],
                 'solver': ['lbfgs'],
                 'early_stopping': [True],
                 'max_iter': [600]}]

regr = GridSearchCV(MLPRegressor(), tuned_parameters, n_jobs=3, verbose=2)
regr.fit(feature_training_matrix, combined_training_target_vector)

Data:
The prediction data I use has the following structure: for every day of the last ~4 years there were predictions made for the next 90 days. I have a text file with -90d to +90d data for every day. I try to train the NN to estimate a better prediction for the next 10 days. For this I take -7 up to +1 days around the current prediction day (1-10 days after the currently used starts to predict) as features. This means that the predcition of the day I am interested in is included as a feature.
feature example: [0.16272058, 0.13296574, 0.14213905, 0.25064893, 0.23302285,
   0.21019931, 0.20733988, 0.1466959 , 0.17029025, 0.15876942]

corresponding target: 0.174652

It is quite possible that the original model does already a very good job - it is not possible to improve it.
For example the true relationship between original input variables and target might be linear, so a neural network doesn't add anything.
As a test, I would increase hidden_layer_sizes and set early_stopping=False and even just doing a grid search (without CV): RMSE Training should get better than RMSE Feature. Most likely RMSE Testing will be worse, but at least you will have evidence there are no other unexpected circumstances (eg. a bug in the code).
It does make sense to use predictions of a base model as inputs. Especially if that original base model is constrained somehow, and you expect your new model to outperform the original one because in the new model you don't have that constraint.
For example the original model might be linear (ARIMA or ARIMAX), while yours is nonlinear - a neural network.
Even if the true relationship between inputs and target is nonlinear, you don't have too much playroom here for a neural network (remember we want to outperform a base model). As I understand you have only ~ 4*365 observations. You can easily overtrain with many neurons, but with only a few neurons training can stuck in local minima.


How to improve the accuracy of a Doc2Vec model (Gensim) in case of a toy-sized data set? I'm building an NLP question-answering application using Doc2Vec technique in gensim package of Python. My training questions is very small, only 20 documents and I am getting very inaccurate and different similarities even for same document while running at multiple instances. Almost all the sources which I referred trained data set containing thousands of documents. So I infer the reason behind my model's inaccuracy is the size of my data set.

Is there any way to improve the similarity between documents, maybe by changing parameters or feature engineering? If yes, what are those parameters and by what ratio should I change them? If no, what are other ways or perhaps other neural network models to tackle the problem?

20 is a very very small number, even performing the following might not help.
However, you can try changing the parameters such as the window size, and the number of iterations. However, increasing the number of iterations a lot can cause the model to over fit.
In my case, decreasing the window size helped.


Feature and the Gaussian Distribution (classification) I have a question regarding variable following or not a random distribution.
I selected 4 features negatively correlated to the label (Fraud/No Fraud). The notebook I'm taking the inspiration from plotted the distribution of these feature regarding the label. What came out is that my feature 1 (Fraud only) is following a Normal Distribution.
Here are my questions :

Why is it important to know if my feature is following a Normal Distribution ? -> My guess : some models need it for faster convergence or better results
Is there any interest to visualize my features as Non Fraud vs Fraud and compare the distributions ?
If my features are not following a Normal Distribution but are scaled, should I still force them to a Gaussian like shape ?

Thank you very much !

It completely depends on the type of model. Some models need to represent the features with parameters: for example Naive Bayes with numerical features needs to have a way to calculate the probability based on the value, and the most common case is to assume that the features follow a normal distribution. On the other hand whether a feature is normally distributed or not doesn't matter at all for Decision Trees.
Yes, it can be very informative in order to know whether this feature is a good indicator or not: the more different the distributions, the more easily the algorithm can distinguish the classes using this feature.
No, don't change the distribution of a feature (unless you have a specific reason to do so, e.g. based on expert-knowledge for this particular data). Any way you would do that would certainly alter the overall distribution of the data and/or the way the features are related within an instance, so the model would not learn from the true distribution and therefore its predictions on real data would likely go wrong.


Siamese vs matching network for correct image category matching I have to find the closest match between my image and bunch of already collected images of different classes in the folder. Whic meta-learning approach should I select. I am thinking about the Siamese or matching network. In Siamese, I have to match my image with all existing images in the folder to find the correct match. So do you think if I can use a matching network and produce a better result? What is the parameter based on which developer decides where to use matching/prototypical network and where to use Siamese network.
Note- At the time of the test new unseen images class can be added on which model is not trained. Do you think a matching network will still work for this case?

A standard approach for this kind of problem is to take the representations of the images computed by some pre-trained neural network (e.g. imagenet). As "representation" of the image we usually take the vector after flattening the output of the last convolutional layer block.
If your images are not of general domain, but are of a very specific domain, and you have enough data, you may want to train an autoencoder on them instead of using a pre-trained model.
You now compute and store the representations of all the images you want to compare against. Once you have the representations, for each image you want to match, you simply compute the closes image in the representation space. There are libraries like Facebook's faiss that are very fast computing binary similarities over a large number of vectors.
This kind of approach is cheap in terms of compute, it is scalable and it is widely used.


What's the difference of Symbolic translation approach and statistical translation? Based on what I know, symbolic computing is based on hard rules in the code. Statistical translation is based on statistic and probabilities. If so, could you give an easy to understand example to demonstrate this difference? If it is not correct, please give a correct explanation of the differences.

By symbolic translation, I assume that you meant rule-based translation. As a simple example, consider translating the English sentence "I eat apples" to Spanish.

Rule-based translation works by first tagging the words with their part of speech ("I" as noun/subject, "eat" as verb, "apples" as noun/object). Then, create an English parse tree and transform that into a Spanish parse tree (this requires knowledge of how English phrases map to Spanish phrases). Lastly, translate each English word into its corresponding Spanish word, and use the Spanish parse tree to determine the structure of the sentence.

The simplest statistical machine translation system are n-gram models based on n-gram frequencies. For an explanation of the mathematics behind statistical translation, see this paper: http://www.mitpressjournals.org/doi/pdf/10.1162/coli.2006.32.4.527.


Enormous dataset, how to proceed? So I have started my master thesis and I have been handed a time series dataset with 2000 rows and nearly 600 columns of data. I have dealt with time series before, but nowhere similar to this level of complexity. Many courses in time series only deals with univariate time series, and now I'm supposed to work on a time series with 600 different factors with extreme loads of nan-values and present it in a simple and illustrating way. Plotting single univariate time series sort of doesn't make sense as there are still 600 other time series to consider. I was just wondering if anyone here have any tips on how to proceed? Any input would be appreciated.

Are you constrained to use only R or Python? If you have the option, try using Alteryx. It's a GUI based tool which has greatly helped me explore and analyze large volumes of data. The tool is built on R so plugging in R functions is a breeze too.


Pearson Correlation is a technique for investigating the relationship between two quantitative, continuous variables, for example, age and blood pressure. Pearson's correlation coefficient (r) is a measure of the strength of the association between the two variables, a number between -1 and 1. [reference]

This following scatter plots from SPSS tutorial explains different pearson correlations visually:

From your scatter plots, however, it looks like there is no linrar correlation between between sensor 6 vs. class label. I do not know why you expect a correlation??!! That is why using pandas.DataFrame.corr, which is by default based on Pearson correlation, should in principle should lead to a near-zero correlation. What strikes me though is that how this gives you high correlations for some sensory data namely sensor0, sensor4 and sensor8!! Maybe post part of your actual dataset rather just the statistics. I am curious why we see those correlations.

Two questions:

What are the data types of the class_label and sensory data columns? Having a wrong data types could lead to wrong correlation values in pandas. See this post
Shouldn't you be looking at the correlation between a continuous value vs. categorical varaibles? See this post as a good reference.


Reinforcement learning with sparse acting agent I'm working on a problem where the optimal policy involves the agent "doing nothing" most of the time, and "doing something" during rare critical moments. Is there any literature or best practices when it comes to solving problems like these where actions are quite rare? I tried changing the random action choice to choose inaction over action with a certain probability to bias the agent to do nothing, but that didn't really affect learning. I think part of the issue is that the agent only receives reward feedback when he acts, so this makes learning the optimal sparse action policy quite difficult. I've tried to avoid reward hacking (agent acting frequently and gathering many small rewards) by weighing the rewards in such a way that rewards given by infrequent but correctly timed action far out-weight rewards given by frequent, incorrectly timed action. But I'm not sure if that's a good solution, or even a solution at all. Currently using a dueling DQN network structure, but open to trying anything (policy gradients?). Any help would be much appreciated.

One option is to create a reward function to incentivize the behavior you want:

Small positive reward for doing nothing.
Large positive reward for doing the rare, correct action.
Negative reward for taking action when no-action is correct.
Negative reward for taking incorrect action.


Is it valid to use numpy.gradient to find slope of line as well as slope of curve at any point? what is the difference between slope of the line and slope of the curve? Is it valid to use numpy.gradient to find the slope of the line and slope of the curve  at any point?
    #slope of line at any point
    tanθ= y2-y1/x2-x1

    #slope of curve at any point
    tanθ  =dy/dx

is it valid to use numpys np.gradient() to get both slopes of curve and line ? or is it meant only to find the slope of line?
Reference slope of line vs curve

what is the difference between slope of line and slope of curve

It is really a matter of perspective. The slope of a line is the same over the entire span of that line, i.e. until the line changes direction. The slope of a curve is like the slope of millions of tiny lines all connected, so the slope is only the same value over tiny spans. So we can only talk about the slope of a curve at a give point (e.g. a given x value) and then we normally talk about the gradient of the line at that point.

Using your words, the gradient computed by numpy.gradient is the slope of a curve, using the differences of consecutive values.
However, you might like to imagine that your changes, when measured over smaller and smaller distances, become the slope (by your definitions). So when e.g. x2 - x1 almost reaches zero, you have the same meaning as $dx$.
Coarse example with 10 points
Here is a coarse example of numpy.gradient, where $dx$ has s size of 1 (equally spaced values):
In [1]: import matplotlib.pyplot as plt
In [2]: import numpy as np
In [3]: N = 10                                  # Use ten samples
In [4]: x = np.linspace(0, np.pi*2, N)          # Equally spaced x values
In [5]: y = np.sin(x)                           # Corresponding sine values
In [6]: grads = np.gradient(y)                  # compute the gradients

Plotting the values and the gradients - I shifted the gradients by 0.5 to the right, so their values line up with the middle of the segment for which they describe the slope:
In [7]: fig, ax = plt.subplots()
In [8]: ax.plot(x, y, "-b", label="values")                  # the y values
        ax.plot(x + 0.5, grads, "--r", label="gradients")    # the computed gradients
        plt.legend()
In [9]: plt.show()

Fine example with 1,000,000 points
Now we do example same as before, just use one million points: N = 1_000_000.
The blue line would look much more like a true sine wave, but the red line is now measuring the gradient with a much higher resolution than before, giving us the exact value at 1,000,000 points - for each tiny line segment of the blue line.

So the gradient values look like they are all zero! Well this is just because we made x2-x1 become almost zero (1 / 1e6), so the values of y2 - y1 also were essentially zero! We have started to approximate $\frac{dy}{dx}$.
Let change the axis scale to see that the gradients do still match the pattern we might expects, and are indeed very smooth - looking like a curve:

Much better  :)
(Notice the scale difference - 1e-6).

Here are a few explanations of what np.gradient really does.


Plotting in PySpark? I have been searching for methods to plot in PySpark.

I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are:

through method collect() which brings data into 'local' Python session and plot
through method toPandas() which converts data to 'local' Pandas Dataframe.

The problem is that these both are very time-consuming functions. Is there any method by which we can plot data residing in Spark session directly (not importing it into the local session)?

No, there is no such method, I have found out.

The reason is, plotting libraries run on a single machine and expect a rather sample dataset. Data on Spark is distributed among its clusters and hence needs to be brought to a local session first, from where it can be plotted.

That is why methods such as collect(), toPandas() are needed.


Soft SVM solving for $b$ I don't really understand how to approach this problem. I know that $w=\sum_{n=1}^{N}{a_ny_nx_n}$ and $y_n(w^T\cdot x_n+b)=1. $ So I can solve for $b$ from that equation but I can't figure out how to apply equation(1) that I'm supposed to use.

From $$y^{(n)}(w^T\cdot x^{(n)} + b)=1,$$

since $y^{(n)}$ is binary, we have $$w^T\cdot x^{(n)} + b = y^{(n)}$$

That is $$b=y^{(n)}-w^T\cdot x^{(n)}.\tag{2}$$

Now, let's examine

\begin{align}&\sum_{m \in S}  \alpha_m y^{(m)}\langle x^{(n)}, x^{(m)}\rangle\\&=\sum_{m \in S}  \alpha_m y^{(m)}\langle x^{(n)}, x^{(m)}\rangle + \sum_{m \notin S}  0\cdot y^{(m)}\langle x^{(n)}, x^{(m)}\rangle \\
&=\sum_{m \in S}  \alpha_m y^{(m)}\langle x^{(n)}, x^{(m)}\rangle + \sum_{m \notin S}  \alpha_m\cdot y^{(m)}\langle x^{(n)}, x^{(m)}\rangle\\
&=\sum_{m=1}^N \alpha_m y^{(m)}\langle x^{(n)}, x^{(m)}\rangle \\
&=\langle \sum_{m=1}^N \alpha_my^{(m)}x^{(m)}, x^{(n)}\rangle\\
&= w^Tx^{(n)} \tag{3}\end{align}

Using $(2)$ and $(3)$, you should be able to obtain the conclusion.


What are good resources to learn the basics of graph theory I just spent half a day "debugging" a script that tried to get the betweenness centrality of a weighted graph with negative weights. However the package I used for that julia's LightGraphs uses Dijkstra's shortest paths, which require non-negative weights as input. I didn't know this, and thus wasted a lot of my and other peoples time trying to find out why my script didn't work.
I the end I found it out. But I figured it would be good to learn some basic graph theory to avoid this kind of problems in the future. Can anybody recommend a good [book|course|other resource] on graph theory? I don't look for a source that rigorously proves all kind of properties, but more of an overview. If I need to know more about a specific graph/test/metric I can always look it up later.

I can think of two very good resources that may help you to acquire the foundational knowledge on graph theory and complex networks:

Network Science by Barabasi is you prefer a web tutorial: http://networksciencebook.com/
The Structure of Complex Networks by Estrada if you prefer a book: https://oxford.universitypressscholarship.com/view/10.1093/acprof:oso/9780199591756.001.0001/acprof-9780199591756


Lower loss always better for Probabilistic loss functions? I am working on an neural net int Tensorflow that predicts percentages for win, draw, loss for given data of a game. The labels I provide are always {1, 0, 0}, {0, 1, 0} or {0, 0, 1}. After some epochs my accuracy doesn't increase any further, but the loss still decreases for a many epochs (also on the validation set, though very slowly). I am using a softmax activation in the last layer and the categorical crossentropy loss function provided by Keras. I was wondering if in this case, lower loss always corresponds to better probabilities (because I obviously wouldn't want the net to output only values like 1 or 0 for probabilities), or in other words, does this net output the "true" probabilites and if so, why does it do that?

Consider loss to be something akin to implied variability of your model. When loss is extremely high, your model's output could be just about anything. As the loss lowers, your model is becoming more confident in its output and will be able to give similar output classification/regressions even if the initial weights or input data is slightly different. Lower loss in the training set is always a good thing so long as you are seeing a subsequent decrease in the loss of the validation set. As soon as that validation loss starts to stagnate though, I like to save the model and quit. Usually at this point you could get a little better loss, but it tends to just be overfitting and doesn't help any on the test set.


How to re-train a model from false positives I'm still a bit new to deep learning. What I'm still struggling, is what is the best practice in re-training a good model over time?

I've trained a deep model for my binary classification problem (fire vs non-fire) in Keras. I have 4K fire images and 8K non-fire images (they are video frames). I train with 0.2/0.8 validation/training split. Now I test it on some videos, and I found some false positives. I add those to my negative (non-fire) set, load the best previous model, and retrain for 100 epochs. Among those 100 models, I take the one with lowest val_loss value. But when I test it on the same video, while those false positives are gone, new ones are introduced! This never ends, and Idk if I'm missing something or am doing something wrong.

How should I know which of the resulting models is the best? What is the best practice in training/re-training a good model? How should I evaluate my models?

Here is my simple model architecture if it helps:

def create_model():
  model = Sequential()
  model.add(Conv2D(32, kernel_size = (3, 3), activation='relu', input_shape=(300, 300, 3)))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(128, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Conv2D(64, kernel_size=(3,3), activation='relu'))
  model.add(MaxPooling2D(pool_size=(2,2)))
  model.add(BatchNormalization())
  model.add(Dropout(0.2))
  model.add(Flatten())
  model.add(Dense(256, activation='relu'))
  model.add(Dropout(0.2))
  model.add(Dense(64, activation='relu'))
  model.add(Dense(2, activation = 'softmax'))

  return model

#....
if retrain_from_prior_model == False:
    model = create_model()
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
else:
    model = load_model("checkpoints/model.h5")

In most cases, one shouldn't retrain a trained network with only the new data. Rather, train the network from scratch with the new and old data.

Adding new data and retraining the model just on that new set of data, will probably make your model fit to only that new data, thus forgetting general features from the other data it was trained on.

Also, instead of selecting your final model based on only validation loss, you should select your model based on a validation metric. For example, in your case, it could be accuracy, precision, recall etc


May the training sets of two predictive modeling cases (with an overlap in features) be combined? Say we have dataset D1 (columns A, B, C) and D2 (columns A, B, D) with target variable E. As both datasets are rather small, their respective predictive models do not perform really well. To improve predictive performance, may/should I apply the steps below?

Remove the not-in-common columns (so column A, B and E remain in both datasets)
Split each dataset in a train and a test set (cross-validation to evaluate, so no validation set)
Train our predictive model on the combination of the training sets of D1 and D2
Assess test set performance of D1 and D2 on their respective test sets (using 1 model rather than 2)

I assume this is allowed, but am not fully sure if there are any (bad) repercussions.

Well, there is a  obvious trade-off of columns vs rows. Why don't you first check the importance of columns C and D in the individual models? The less relevant they are, the better idea it is to throw them away and comine the tables.
For example, you can try a random forest with sklearn and then check feature_importances_.


Data augmentation for multiple output heads in Keras I have a transfer learning based two output classification problem. So, accordingly, I have formatted my data to have X_train as a (number of samples, height, width, channels) numpy array, y_train1 as (number of samples,) numpy array and y_train2 as (number of samples,) numpy array.

As I am not training using directory structure, I am using ImageDataGenerator.flow(). However, I am not able to figure out how I can pass two label arrays because, it is taking the labels as (2, number of samples) when I send it as [y_train1, y_train2] list.

I am able to train the network without Keras data augmentation (for two outputs). But, I am not able to apply data augmentation.

I am trying to do the following:

datagen = ImageDataGenerator(horizontal_flip=True, vertical_flip=True, rescale=1./255, class_mode="multi-label")

model.fit(datagen.flow(X_train, [y_train1, y_train2], batch_size=batch_size), batch_size=batch_size, epochs=nb_epochs, steps_per_epoch=spe, validation_data=(X_val, [y_val1, y_val2]))

Also, ImageDataGenerator.flow does not have class_mode unlike ImageDataGenerator.flow_from_dataframe.

Any suggestions/help would be appreciated!

References:

Data Augmentation Multi Outputs (No answer. I upvoted this just now)
Get multiple output from Keras (Does not explain data augmentation)

Please refer to the source code provided at https://gist.github.com/swghosh/f728fbba5a26af93a5f58a6db979e33e which should assist you in writing custom generators (basis ImageDataGenerator) for training end to end multi-output models. In the provided example, GoogLeNet is being trained which consists of two auxiliary classifiers and thus, comprising 3 outputs in the complete model.

The output from ImageDataGenerator.flow_from_directory() have been passed into a Python function with yield statements such that the targets can be repeated thrice as per requirement.

def three_way(gen):
    for x, y in gen:
        yield x, [y, y, y]

Also, please make sure to pass a value to steps_per_epoch argument in case of model.fit() or model.fit_generator() so as to ensure that the training loop doesn't run indefinitely. (This is because the Python generator object won't be inherited from Sequence class and number of batches/steps cannot be inferred implicitly by Keras)

Also, you may consider using an image data pipeline constructed using tf.data APIs combined with Random* Layers and Preprocessing Layers which is a more modern approach for tf.keras compared to the old ImageDataGenerator. There should be a bunch of performance and usability benefits in case you transition. Let me know if you are looking for an example.

Please look at this example in case you want to implement multi-output multi-label classification model. https://www.pyimagesearch.com/2018/06/04/keras-multiple-outputs-and-multiple-losses/

I'd still advise to use tf.data.Dataset instead of raw tensors which'd be more feasible for larger datasets that do not fit into memory plus, the tf.data APIs provides for an additional bunch of flexibilities.


Why is sampling useful in machine learning? I have met that question online and I wanted to know where sampling can simulate complex processes and why?

Why is sampling useful in machine learning?

Sampling can increase the accuracy of the model
Sampling can simulate complex processes
Sampling is lower cost
Sampling can save lots of time

I can select multiple ones. As far as I know, sampling is lower cost and can save lots of time but, can it simulate complex processes?

TL:DR - If you know the posterior distribution of the complex process (i.e. the output distribution), and that distribution is one which can be modelled with reasonable accuracy, then sampling from it should reasonably represent the responses of the complex system.

An example could be a complex decision process, whereby many decisions are made consecutively, maybe with some conditional or temporal relationships along the way (basically any process that is considered to be complex). Now imagine, after all this complexity, there are e.g. two possible categorically outputs: a zero or a one. Well this something we might be able to model with a Bernoulli distribution, assuming we can estimate a reasonable parameter $p$ (specific to the Bernoulli distribution).

By sampling from the distribution, we would hope to draw samples, which are representative of the complex process. I hope this covers your main question.
I think point 4. is also clear to understand as well from my trivial example. I am unsure as to what cost refers exactly in point 3.

As for point 1., there are examples where (random) sampling, for example in a grid search of model/optimisation parameters can improve results, as it results in an improved exploration of the parameter space compared to other methods such as grid-search.


Index for efficient argmax(w.x) query ~ 20d I'm looking for a spatial index that can efficiently find the most extreme n points in a certain direction, i.e. for a given w, find x[0:n] in the dataset where x0 gives the largest value of w.x and x1 the second largest value of w.x, etc... . Is there a name for this type of query? What would be an efficient data structure to use? x might have around 20 dimensions.

Thankyou!

The query is called a 'top-k' query, and you can answer it quickly using the ranking cube approach. http://www1.se.cuhk.edu.hk/~hcheng/paper/vldb06_rankcube.pdf

The paper does not derive the time complexity.


Hill climbing with noisy data For example

Shoes priced at \$10, 30 of 1000 customers buy for revenue of \$300
Shoes priced at \$20, 25 of 1000 customers buy for revenue of \$500

And then now I have the problem of determining how certain I am the second pricing is yields higher returns

My question is: How can I find out X so that I'm X% confident that the \$20 price yields higher returns than the \$10?

When in doubt, bootstrap.

Make a list with 970 0s (the non-sales at \$10) and 30 10s (the sales at \$10) - customers_10
Make a list with 975 0s (the non-sales at \$20) and 25 20s (the sales at \$20) - customers_20
Repeat lots of times (maybe N = 100,000 times - increase this until the the result settles down)

Randomly take 1000 samples (with replacement) from both lists - samples_10, samples_20
Sum the samples for each list - sum_s10, sum_s20
Record which one generated more revenue (i.e. whether sum_20 > sum_10)

The percentage of times sum_s20 was greater than sum_s10 is your X

(Another approach, which would be equivalent in the limit of N large, would be to model this as a binomial process. But processing is cheap, bootstrapping is robust and easy to implement - getting the binomial model right would be easier to mess up).


How can I use data on customer interactions to drive communication strategy? I have data on my customers (age, location, gender) and number of interactions with customer by channel (#calls, #sms, #letters) and data on whether they have bought a product (yes/no).

I would like to predict what combination of interaction (e.g. 3 calls, 2 letters) is the best approach when contacting different types of customers. What would be an appropriate statistical/ machine learning technique to use to calculate this?

I would do the following:

First, cluster your customers into several groups, based on age, location, gender etc. Second, for each group, use different combination of interaction as features to predict 'yes/no', and select which combination of interaction performs best.