Why Gaussian latent variable (noise) for GAN? When I was reading about GAN, the thing I don't understand is why people often choose the input to a GAN (z) to be samples from a Gaussian? - and then are there also potential problems associated with this?

Why people often choose the input to a GAN (z)
  to be samples from a Gaussian?

Generally, for two reasons: (1) mathematical simplicity, (2) working well enough in practice. However, as we explain, under additional assumptions the choice of Gaussian could be more justified.

Compare to uniform distribution. Gaussian distribution is not as simple as uniform distribution but it is not that far off either. It adds "concentration around the mean" assumption to uniformity, which gives us the benefits of parameter regularization in practical problems.

The least known. Use of Gaussian is best justified for continuous quantities that are the least known to us, e.g. noise $\epsilon$ or latent factor $z$. "The least known" could be formalized as "distribution that maximizes entropy for a given variance". The answer to this optimization is $N(\mu, \sigma^2)$ for arbitrary mean $\mu$. Therefore, in this sense, if we assume that a quantity is the least known to us, the best choice is Gaussian. Of course, if we acquire more knowledge about that quantity, we can do better than "the least known" assumption, as will be illustrated in the following examples.

Central limit theorem. Another commonly used justification is that since many observations are the result (average) of large number of [almost] independent processes, therefore CLT justifies the choice of Gaussian. This is not a good justification because there are also many real-world phenomena that do not obey Normality (e.g. Power-law distribution), and since the variable is the least known to us, we cannot decide which of these real-world analogies are more preferable.

This would be the answer to "why we assume a Gaussian noise in probabilistic regression or Kalman filter?" too.

  Are there also potential problems associated with this?

Yes. When we assume Gaussian, we are simplifying. If our simplification is unjustified, our model will under-perform. At this point, we should search for an alternative assumption. In practice, when we make a new assumption about the least known quantity (based on acquired knowledge or speculation), we could extract that assumption and introduce a new Gaussian one, instead of changing the Gaussian assumption. Here are two examples:

Example in regression (noise). Suppose we have no knowledge about observation $A$ (the least known), thus we assume $A \sim N(\mu, \sigma^2)$. After fitting the model, we may observe that the estimated variance $\hat{\sigma}^2$ is high. After some investigation, we  may assume that $A$ is a linear function of measurement $B$, thus we extract this assumption as $A = \color{blue}{b_1B +c} + \epsilon_1$, where $\epsilon_1 \sim N(0, \sigma_1^2)$ is the new "the least known". Later, we may find out that our linearity assumption is also weak since, after fitting the model, the observed $\hat{\epsilon}_1 = A - \hat{b}_1B -\hat{c}$ also has a high  $\hat{\sigma}_1^2$. Then, we may extract a new assumption as $A = b_1B + \color{blue}{b_2B^2} + c + \epsilon_2$, where $\epsilon_2 \sim N(0, \sigma_2^2)$ is the new "the least known", and so on.
Example in GAN (latent factor). Upon seeing unrealistic outputs from GAN (knowledge) we may add $\color{blue}{\text{more layers}}$ between $z$ and the output (extract assumption), in the hope that the new network (or function) with the new $z_2 \sim N(0, \sigma_2^2)$ would lead to more realistic outputs, and so on.


The differences between SVM and Logistic Regression I am reading about SVM and I've faced to the point that non-kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?

Apparently, SVM chooses the maximum margin classifier and logistic regression is the one that minimizes the cross-entropy loss. Are there situations where SVM  performs better than logistic regression or vice-versa?

If you use logistic regression and the cross-entropy cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find weights that are near to optimal point and not exactly on the optimal point. This means that you can have multiple classifies that reduce the error and maybe set it to zero for the training data but with different weights which are slightly different. This can lead to different decision boundaries. This approach is based on based on statistical methods. As it is illustrated in the following shape, you can have different decision boundaries with slight changes in the weights and all of them have zero error on the training examples.

What SVM does is an attemption to find a decision boundary that reduces the risk of error on the test data. It tries to find a decision boundary that has the same distance from the boundary points of both classes. Consequently, both classes will have a same space for the empty space which there is no data there. SVM is geometrically motivated rather than statistically.

  None kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?

They are linear separators and if you find out that your decision boundary can be a hyperplane, it's better to use an SVM for diminishing the risk of error on test data.

  Apparently SVM chooses the maximum margin classifier and logistic regression the one that minimizes the cross-entropy loss.

Yes, as stated SVM is based on geometrical properties of the data whilst logistic regression is based on statistical approaches.

  In this case, are there situations where SVM would perform better than logistic regression, or vice-versa?

Ostensibly, their results are not very different, but they are. SVMs are better for generalization 1, 2.


TensorFlow: number of channels of conv1d filter I want to apply a ConvNet on my one dimensional data retrieved from 13 sensors. So, each of my samples consists of 13 channels (of 51 values)

I am using 'conv1d' to apply a ConvNet on my data. The network works nicely, but I wonder how 'conv1d' determines the number of channels for it's filters... To my knowledge, a filter should have an equal number of channels as its input data, which makes it a $5x13$ filter. I set the filter to have a width of 5, but don't need to set the number of channels anywhere.

My question is: how does layer 'conv1' determine it's number of channels?

Below is a portion of my code:

# We have 13 1D channels of 51 points each

# Note that we've indicated -1 for batch size, which specifies that this dimension should be dynamically computed
# based on the number of input values in features["x"], holding the size of all other dimensions constant.

input_layer = tf.reshape(features["x"], [-1, 51, 13])

# Convolutional Layer #1
# Shouldn't this filter also need to number of channels?
This should match the input number of channels
conv1 = tf.layers.conv1d(inputs=input_layer, filters=32, kernel_size=5, padding="same", activation=tf.nn.relu)

# Pooling Layer #1
pool1 = tf.layers.max_pooling1d(inputs=conv1, pool_size=2, strides=2)

In TensorFlow there are different convolution layers. Conv1d, Conv2d and Conv3d. the first one is used for one dimensional signals like sounds, the second one is used for images, gray-scale or RGB images and both cases are considered to be two dimensional signals. The last one is used for three dimensional signals like video frames, images as two dimensional signals vary during time. In your case Conv1d is used as one dimensional signal and you can specify the number of filters in the arguments of the method. You can take a look at here and here.


How is the 'feature_importance_' value calculated in sklearn random forest regressor? I have 9000 sample, with five features, and one output variable (all are numerical, continuous values). I used random forest regression method using scikit modules. I got a graph of the feature importance (using the function feature_importances_) values for each of the five features, and their sum is equal to one. I want to understand what these are, and how they are calculated mathematically.
Can someone please explain?

Decision trees usually base feature importance on the impurity reduction achieved by splitting on the features. In classification a usual choice is gini impurity, while regression trees typically use the mean squared error or node sample variance. This is also the case in scikit learn.

For a given (binary) node $m$ with left and right child nodes the impurity reduction $Gain_{m}$ is calculated as

$$ Gain_{m} = impurity_{m} - (weight_{left} \cdot impurity_{left} + weight_{right} \cdot impurity_{right})$$

with the weights being defined as the share of the parents examples in a child node (e.g. $weight_{left} = N_{left} / N_{m}$ where $N$ is the number of examples in a node or leaf).

Now, to derive the total impurity reduction of a given feature $f$ in tree $t$ you need to sum across all nodes $m \in M_f^{(t)} $ which perform a split on that feature $f$ and divide it by the total impurity reduction number of all nodes of that tree:

$$Importance_f^{(t)} = \frac{\sum_{m \in M_f^{(t)}} Gain_m}{\sum_f\sum_{m \in M_f^{(t)}} Gain_m}$$

(Note that due to this normalization step your feature importances sum up to $1$)

Eventually, the total importance of a feature $f$ is calculated across all trees $t$ in your random forest with a total number of trees $T$:
$$Importance_f = \frac{1}{T} \sum_{t=1}^TImportance_f^{(t)}$$


What are some good machine learning techniques for developing rules that govern changes in data? Say I read a set of data in and it looks like so:

[ 1, 2, 3, 4, 5 ]

Now, there's an unknown rule set by my data's source that says if the 3rd element in my list is set to 5, the fifth element must be greater than 10. So, subsequent data reads look like this:

read #2:

[ 2, 2, 3, 4, 5 ]

read #3:

[ 3, 2, 5, 4, 11 ]

read #4:

[ 4, 2, 5, 4, 11 ]

If I'm only seeing the data, is there a way I can extrapolate that rule, so I end up with this:

if (field3 == 5 and field5 < 10)
    return false;
return true;

I know it would be hard for a machine to differentiate the rules without far more data than this example, but in reality, I have thousands of these data records with hundreds of rules that govern their state.

Is there an established method for doing this type of pattern recognition?

Yes. There are different methods to extract some rules from data. It depends on the type of rules that you want to extract. Anyhow, different types of the decision tree (such as fuzzy decision tree) and association rules to find the rules that have strong support and confidence, could help you in this way.


Conditional clustering I have a dataset consisting of addresses (points) that have several attributes; one that distinguishes the "sort" of address and one anntribute that contains a numerical value.

I want to cluster these points based on:
1. their distance to each other
2. the sort of address

However, the summed numerical attribute per cluster cannot exceed a certain
threshold value.

In other words, the systeem needs to form clusters, but needs to stop clustering as soon as the sum of the numerical value attached to each address has been reached.

How do I even go about it? I have R, Python and other geo- applications at my disposal.

It seems that none of the existing clustering algorythms work. For k- means for example I need to know the number of clusters beforehand, which I don't.

It seems rather simple, but I can't find a basic methodology to follow.

Based on your comments, you are looking for agglomerative hierarchical clustering.

You start with one point as its own cluster. Then iterate over pairs of clusters, merging them according to some criterion.

Typically you need to select a "cut point" after which you stop combining clusters. This is not an easy problem in general, and for the most part involves eyeballing your data until it "looks right", much like choosing K in K-means. In your case, however, you can use the external criterion you have in mind. You will need to recompute its value at every step, and then simply stop when its value passes the desired threshold.


Potential problems with expanding training set The problem is a binary classification one. My dataset contains users with activity over multiple days, where they all start with class 0 and can become class 1 after a certain activity (which is not part of the input features). If I have 1000 class 0 users and 100 class 1 users, I will have 1100 training instances.

What I want to do is expand the data such that for each day of activity of each user, there is a row in the training set. So user 1 with 10 days of activity will have 10 rows in the training data, all with class 0, instead of 1 row. User 2 who has 10 days at class 0 and 5 days at class 1 will have 15 rows instead of 1.

This can give the model multiple times more examples to learn from. The only drawback I see is that it will change the ratio of class 0 to class 1 examples (not sure if that is a problem). Are there any unforeseeable disadvantages to this approach?

One problem is that you insert correlated data points. Different days from the same user (may) have some underlying correlations that the days from different users don't. These correlations might be relevant for the classification task, and the model might learn them, instead of learning the relevant features.

This doesn't mean it's a bad thing. Data augmentation is a thing. For example, a common practice when training models such as CNNs on images is to perform data augmentation by rotating/scaling the initial images.

However, make sure you don't touch the validation data. The data augmentation is performed after the train-validation split. The validation data should be as "raw" as possible. You can test it this way, and see how it impacts the performance of your model.


What is the best method for classification of time series data? Should I use LSTM or a different method? I am trying to classify raw accelerometer data x,y,z to its corresponding label.

What is the best architecture for best results?

Or, does anyone have any suggestions on LSTM architectures built on keras with input and output nodes?

I would not use the word "best" but LSTM-RNN are very powerful when dealing with timeseries, simply because they can store information about previous values and exploit the time dependencies between the samples. That said, it is definitely worth going for it.
It has been proven that their performance can be boosted significantly if they are combined with a Convolutional Neural Network (CNN) that can learn the spatial structures in your data, which in this case is one-dimensional.
Take a look at this state-of-the-art method that combines LSTM and CNN, published in 2019.


Models after word2vec outputs I am originally using a bag of word (2-gram) model to approach a classification problem. The one hot encoding of the 2-gram output was sent to a logistic regression or neural network to build a classification model.

Now, I am experimenting the gensim word2vec approach, each word is now a vector from word2vec. That is, if my sentence have 10 words, it would become a 10x30 array (assume word2vec embedding dimension is 30). It's not clear to me how do I send such outputs to a logistic regression or neural network model like before ...

What type of model should I use after the gensim word2vec outputs to solve a classification problem? Thanks!

There exists multiple ways, each consisting on summarizing the embeddings of each word from a document.

The most common ones when using linear models like Logistic Regression are:

max of word vectors
mean of word vectors
concatenation of word vectors (but implies a fixed length)

However, it is important that using such approaches often results in :

either equal or worse performance than using bag-of-words
less interpretability (if applicable)

The reason is the loss of information (especially for long documents).

When using word embeddings (and so called word2vec), the best choice is often to use models that can handle a second dimension in your input.

For that, recurrent and convolutional neural networks have proven to be effective. The former operates at the word level by recurrently feeding the word embeddings, the latter operates at the whole document level (represented by a matrix of $n_{words}\times d_{embedding}$ as you mentioned) by locally convoluting filters with windows of the document.


transition to data scientist for more money Not sure if this is a right forum. Please advise if not.

I am currently employed as a Software Engineer and am fairly happy. Except, I'd like to make more money. I hear data science is the hottest field these days but wasn't sure if it paid more. I was contemplating to get a certificate like this one.

I understand in the beginning it might not be more than what I currently make (~100K) as I am new to the field. However, in few years (let's say about 3 years) is it common to make 150K+?

I have a MS in Computer Science, and was good at math classes. So I feel confident in being able to transition.

Data scientist won't actually make you better paid. You might have seen 150k+ for some data scientists, but those people usually have a Phd and they can get similar salary even they do something else.

Data science is a very big field; you can work in the field as a software engineer. But don't expect training a gradient boosting model; you'll more likely be asked to process Hadloop big data application, optimization Python workflow etc.

In my experience, a data science certificate is useless. If your next position is a software engineering in data science, you don't need a certificate; you'll just need have a good proven record that you can deal with data. If you next position will be mathematical modelling, you'll need something like a Phd.


OpenNLP tutorial or book I am very new to Java and primarily a Python and R user. I want to use OpenNLP for certain NLP tasks that I have been entrusted with. Is there any good tutorial or book on OpenNLP that anyone can recommend?

I've tried this one, and is very straight forward and easy to follow.

Java is very similar to python in some sense so it shouldn't be a problem for you if you have done OO.

If you want to get deeper inside natural language understanding and processing I recommend this book.


Knowing Feature Importance from Sparse Matrix I was working with a dataset which had a textual column as well as numerical columns, so I used tfidf for textual column and created a sparse matrix, similarly for the numerical features I created a sparse matrix using scipy.sparse.csr_matrix and combined them with the text sparse features.

Then I'm feeding the algorithm to a gradient boosting model and doing the rest of the training and prediction. However I want to know, is there any way I can plot the feature importance, of this sparse matrix and will be able to know the important feature column names?

You would have a map of your features from the TFIDF map.

column_names_from_text_features = vectorizer.vocabulary_
rev_dictionary = {v:k for k,v in vectorizer.vocabulary_.items()}
column_names_from_text_features = [v for k,v in rev_dictionary.items()]

Since you know the column names of your other features, the entire list of features you pass to XGBoost (after the scipy.hstack) could be

all_columns = column_names_from_text_features + other columns

(or depending on the order in which you horizontally stacked)

Now, once you run the XGBoost Model, you can use the plot_importance function for feature importance. Your code would look something like this:

from xgboost import XGBClassifier, plot_importance
fig, ax = plt.subplots(figsize=(15, 8))
plot_importance(<xgb-classifier>, max_num_features = 15, xlabel='F-score', ylabel='Features', ax=ax)
plt.show()

These features would be labeled fxxx, fyyy etc where xxx and yyy are the indices of the features passed to xgboost.

Using the all_columns constructed in the first part, you could map the features to in indices in the plot encoding.


Keras save model FailedPreconditionError Model works and fits. After adding model.save('model.h5'). I am receiving
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value dense_11/bias[[Node: _retval_dense_11/bias_0_0 = _RetvalT=DT_FLOAT, index=0, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

I tried model.save_weights('model.h5')  saves the model but after trying to load the model I get:

KeyError: "Unable to open object (object 'dense_12' doesn't exist)"

After saving model to json and then trying to load it i get json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0).

What should I do?
Thanks for help!

The error was caused because after training the model I opened another tensorflow session to save predictions to dataset and put model.save('model.h5') after. So moving this line up, right after the model finished training solved the problem.


Convolution neural network: small dataset affecting accuracy I have dataset of 36 folders 1 image each(total 36 images) the dataset is too small but these are character images which i want to train my val_acc= 0.0229 and y_pred=model.predict(x_test) always predict class=4.

How I can improve this accuracy for such a small dataset? Why does it always predict class 4?

If I understand the question correctly, you have 36 class and 1 sample/class.
Then I believe it's impossible to use Machine Learning for this dataset to predict the label.

Reason: When you split train/test set, there will some classes that only appear in test set and not train set. The model doesn't learn about them at all. A major assumption in Machine Learning problems is that the train set and test set must have the same statistical distribution.


which scoring function for validation_curve (regression)? Is there any thumb of rule which scoring function should be used for e.g. the validation_curve?

Atm I try to study the difference between several optimizers:

validation_curve(grid_best
                                          , X_train
                                          , y_train
                                          , param_name = 'Adam', 'SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adamax', 'Nadam']
                                          , param_range = param_range
                                          , cv=tscv
                                          , scoring="explained_variance"
                                          , verbose = 1
                                          , n_jobs = n_jobs
                )

I use the explained_variance but I think the function has to be minimized cause the values are mostly below zero. That's why I think the explained variance does not make sense here?

edit:

When I use r2 I get the following curve:

Is that normal?

I'd use r2 for validating regression. It'd be much easier to compare different models. Here's wiki, here's sklearn.


MPI, MapReduce, or Spark for complex datasets and processing I have 2 data files: the first one is a database, potentially very large; the second one contains queries I want to answer. My program pipeline is processing the database to get some information first, and then use that same information to answer the queries. Although the number of queries is not big, processing each query takes a long time. So I want to give each worker some queries to answer, then combine all the answers together into one.

This sounds like a MapReduce job. But to answer each query, the worker also needs to use the processed information from the database, and I'm not sure how to do this with MapReduce. I'm new to parallel programming and just heard of MPI and Spark.

Can you help me to choose an appropriate one?

Please ask if the description is not clear.

If I understand your question correctly. What you want to do is

Write a sql query against to a database to get the processed/aggregated results first.
Execute several queries in parallel.

The first thing first, if you want to run MapReduce programs on your data, you need to distribute your data. In addition, MapReduce doesn't run your queries to different nodes separately. If your $query_{A}$ runs on $data_{A}$, what we do here is to divide $data_{A}$ to different nodes and run $query_{A}$ on these nodes on $PartOfData_{A}$ in parallel. By the way, you can use Sqoop to migrate the data from your DB to HDFS(it is used to store data separately).

Here are steps for you:

migrate all/part of your data to HDFS
transform your queries to MapReduce version program


Creating graphs of CNN models: cannot use plot_model from keras.utils.vis_utils I am using keras w/ PyCharm on windows 10. I am trying to print my tensorflow.keras.Model using

from keras.utils.vis_utils import plot_model
...
res_net_model = tf.keras.Model(inputs=inputs,outputs=outputs)
...
plot_model(res_net_model)      #, to_file='cnn_resnet_model.bmp') #, show_shapes=True, show_layer_names=True)

I am for some reason getting

  File "C:\Users\Zack's 2016 Tower\.conda\envs\tensorz\lib\site-packages\keras\utils\vis_utils.py", line 240, in plot_model
    expand_nested, dpi)
  File "C:\Users\Zack's 2016 Tower\.conda\envs\tensorz\lib\site-packages\keras\utils\vis_utils.py", line 79, in model_to_dot
    _check_pydot()
  File "C:\Users\Zack's 2016 Tower\.conda\envs\tensorz\lib\site-packages\keras\utils\vis_utils.py", line 28, in _check_pydot
    pydot.Dot.create(pydot.Dot())
  File "C:\Users\Zack's 2016 Tower\.conda\envs\tensorz\lib\site-packages\pydotplus\graphviz.py", line 2032, in create
    status, stderr_output))
pydotplus.graphviz.InvocationException: Program terminated with status: 1. stderr follows: Format: "ps" not recognized. Use one of:

Process finished with exit code 1

I have confirmed that graphviz & pydotplus has been installed by conda. I've also tried to installing graphviz as an executable that's within my path.

I've looked through similar postings such as this and this with no luck: nothing matches my error. It doesn't even tell me what formats I should use (although this may have the answer). It shouldn't matter because atm I'm not asking it to generate an image. I have no idea where it is getting 'ps' from.

I had exactly the same problem today and stumbled upon your question. As it had no replies, I dug further and eventually found something that helped me. For you it may be too late but anyone else having this problem, maybe it helps if I document my finding.
For the background: my setup is Python 3.7 under Windows 10.
The keras function plot_model uses pydot which itself relies on GraphViz. Moreover, the error message Format: "ps" not recognized. Use one of: you see is caused by dot, one of the tools that belongs to GraphViz, which is called from inside pydot. After playing around with the dot tool, it eventually told me the following:
There is no layout engine support for "dot" Perhaps "dot -c" needs to be run (with installer's privileges) to register the plugins?
So, I fired up a new console with administrator priviliges and ran dot -c. This enabled dot to use its layout engines and this made the error disappear. After this,  plot_model suddenly worked for me.
Hope it helps anyone.


Machine Learning For Keeping Stable Humidity I am working on a project building a system to automate much of the tasks in my home garden.  For a few weeks now I have had the system setup to control a humidifier as well as have a sensor to read the current humidity.  I have been reading the humidity every minute and my program has a min and max humidity set, if the humidity goes below the min it kicks on my humidifier and if it goes above the max it kicks on my exhaust fans.  I have also set the system up to log the temp and humidity reading every time it updates to a database.  What I am looking to do is write an algorithm that will take this data and calculate when it needs to turn the humidifier on/off to keep the humidity in the tent as stable as possible.  Right now if you look at the humidity data it is consistently being brought up to just under the max when the humidifier is running (I have it setup to shut off when it reaches 3/4 of the max humidity) and then dropping back down to the min humidity.  The humidifier is a warm vapour humidifier so it takes a few minutes to heat up every time it kicks on.  I want to write something that would look at past data, look at all these factors and calculate when it should turn the humidifier on or off to try and keep a steady base humidity but, I am not really sure where to start or what to google to work toward an answer.  Any info anyone could provide would be greatly appreciated.  I apologise if I am not giving a great description and feel free to ask me for more details.

This sounds like something you might want to do with a PID or Kalman Filter. It'd probably work better than a complex machine learning model. The humidity is probably cyclical in nature and if you can just tighten the loop a bit, you should be able to achieve your result with a simple PID.

Something like this might be what you're looking for:-

PID example

Application


Back Propagation Vs Learning rate in Neuralnet Optimisation I was doing some research on how backpropagation works? I read that, backpropagation is used to find the optimal weight of each neuron after every iteration using partial derivates and updates the weights of the neuron.
On the other hand, we have hyperparameter called 'learning-rate' used to update the weight of the neuron in each iteration by calculating the direction of the error.
These two cases look like working independently, I mean, while backpropagation algorithm itself finding the optimal weight, we do not need a learning rate parameter itself.
Is my understanding correct? Please correct me if I am wrong.

Using backpropagation is nothing else than performing (stochastic) gradient descent.
It computes the gradient, but it is  not the "optimal" weight. The gradient is used to update the current weight (according to the gradient descent algorithm).
The gradient descent algorithm needs a step size (which is called learning rate in the context of machine learning).
The step size defines how "strong" the current weights are updated by the current gradient.


Reason for Huge Jump in Loss For One Epoch Only? I was wondering what possible reasons there could be for a huge jump in loss for only one epoch during training.  I am getting a result like...

Epoch 1/10
2020-05-13 18:42:19.436235: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 30910 of 40000
2020-05-13 18:42:22.360274: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
200/200 [==============================] - 173s 863ms/step - loss: 0.1844 - val_loss: 0.4250
Epoch 2/10
200/200 [==============================] - 167s 833ms/step - loss: 80890.9766 - val_loss: 0.5157
Epoch 3/10
200/200 [==============================] - 166s 830ms/step - loss: 0.0549 - val_loss: 0.2966
Epoch 4/10
200/200 [==============================] - 170s 849ms/step - loss: 0.0488 - val_loss: 0.2708

Which strikes me as very odd.  This is a 3-lay LSTM network using Keras.

I graphed the data after normalization, but nothing out of the ordinary appears.

Here is the code I used to create the network...

# create new more complicated network
multi_step_model = tf.keras.models.Sequential()
multi_step_model.add(tf.keras.layers.LSTM(32,
                                      return_sequences=True,
                                       input_shape=
[past_history,len(features_considered)]))
multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))
# output layers are same as prediction count
multi_step_model.add(tf.keras.layers.Dense(future_target))
multi_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0), loss='mae')

# run training
multi_step_history = multi_step_model.fit(train_data_multi, epochs=EPOCHS,
                                      steps_per_epoch=EVALUATION_INTERVAL,
                                      validation_data=val_data_multi,
                                      validation_steps=50)

I cannot say for sure, but it seems that it might be an error in the printout. The validation loss doesn't seem to spike, so it seems that the loss may not actually be what is printed.

I can only advise to run training multiple times and see if this happens again. If yes, try toying with the learning rate. There is a small chance that the learning rate could be too high making the loss diverge for one epoch.

Based on your code, it seems that your optimizer clips gradient with an absolute value larger than 1. In a scenario where most of your gradients are larger, this could lead to your model being optimized in the "wrong direction". This could lead to the loss being so large for some epochs.


What types of matrix multiplication are used in Machine Learning? When are they used? I'm looking at equations for neural networks and backpropagation and I see this symbol in the equations, ⊙. I thought matrix multiplication of neural networks always involved matrices that matched dimensions on both sides, such as... [3, 3]@[3, 2]. (This is what is happening in the animated gif).

What part of a neural net uses a Hadamard product and which uses the Kronecker product? Because I see this notation for the Hadamard product (⊙) in papers and deep learning learning materials.

There are two distinct computations in neural networks, feed-forward and backpropagation. Their computations are similar in that they both use regular matrix multiplication, neither a Hadamard product nor a Kronecker product is necessary. However, some implementations can use the Hadamard product to optimize the implementation.

However, in a convolutional neural networks (CNN), the filters do use a variation of the Hadamard product.

Multiplication in Neural Networks

Let's look at a simple neural network with 3 input features $[x_1, x_2, x_3]$ and 2 possible output classes $[y_1, y_2]$.

Feedforward pass

In the feed-forward pass the input features will be multiplied by the weights at each layer to produce the outputs

$\begin{bmatrix}
x_1&x_2&x_3
\end{bmatrix} * \begin{bmatrix}
w_{1,1} & w_{1,2} & w_{1,3} & w_{1,4}\\
w_{2,1} & w_{2,2} & w_{2,3} & w_{2,4}\\
w_{3,1} & w_{3,2} & w_{3,3} & w_{3,4}
\end{bmatrix} = \begin{bmatrix}
h'_1 & h'_2 & h'_3 & h'_4
\end{bmatrix}$

At the hidden layer these will then go through the activation function, if we assume sigmoid then

$\begin{bmatrix}
h_1 & h_2 & h_3 & h_4
\end{bmatrix} = \frac{1}{1+e^{\begin{bmatrix}
-h'_1 & -h'_2 & -h'_3 & -h'_4
\end{bmatrix}}}$

Finally we go through the next set of weights to the output neurons

$\begin{bmatrix}
h_1 & h_2 & h_3 & h_4
\end{bmatrix} * \begin{bmatrix}
v_{1,1} & v_{1,2}\\
v_{2,1} & v_{2,2}\\
v_{3,1} & v_{3,2}\\
v_{4,1} & v_{4,2}
\end{bmatrix} = \begin{bmatrix}
y'_1 & y'_2
\end{bmatrix}$

$\begin{bmatrix}
y_1 & y_2
\end{bmatrix} = \frac{1}{1+e^{\begin{bmatrix}
-y'_1 & -y'_2
\end{bmatrix}}}$

Backpropagation pass

In the backpropagation we will update the weights through gradient descent. Usually derivations will ignore the need for the Hadamard product by just representing the derivatives with indexes, or implying them implicitly. However the Hadamard product can be used to be more explicit in the following places.

$v^{new} = v^{old} - \eta \frac{\partial C}{\partial v}$

$\frac{\partial C}{\partial v} = \frac{\partial C}{\partial \hat{y}} \circ \frac{\partial \hat{y}}{\partial v}$.

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$

$\frac{\partial \hat{y}}{\partial v} = \frac{1}{1+exp(v^Th + b)} \circ (1 - \frac{1}{1+exp(v^Th + b)})$.

Let's look at why we can define this last equation as Hadamard product. The $(v^Th + b)$ is computed as (I ignored the bias term)

$\begin{bmatrix}
v_{1,1} & v_{2,1} & v_{3,1} & v_{4,1}\\
v_{1,2} & v_{2,2} & v_{3,2} & v_{4,2}
\end{bmatrix} * \begin{bmatrix}
h_{1} \\
h_{2} \\
h_{3} \\
h_{4}
\end{bmatrix} = \begin{bmatrix}
\sum v_{i,1} * h_{i} \\
\sum v_{i,2} * h_{i}
\end{bmatrix}$

$\frac{\partial \hat{y}}{\partial v} = \frac{1}{1+exp(v^Th + b)} \circ (1 - \frac{1}{1+exp(v^Th + b)}) = \begin{bmatrix}
\frac{1}{1 + e^{\sum v_{i,1} * h_{i}}} \\
\frac{1}{1 + e^{\sum v_{i,2} * h_{i}}}
\end{bmatrix} \circ \begin{bmatrix}
1 - \frac{1}{1 + e^{\sum v_{i,1} * h_{i}}} \\
1 - \frac{1}{1 + e^{\sum v_{i,2} * h_{i}}}
\end{bmatrix} $.

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y = \begin{bmatrix}
\hat{y}_1\\
\hat{y}_2
\end{bmatrix} - \begin{bmatrix}
y_1\\
y_2
\end{bmatrix} = \begin{bmatrix}
\hat{y}_1 - y_1\\
\hat{y}_2 - y_2
\end{bmatrix}$

$\frac{\partial C}{\partial v} = \frac{\partial C}{\partial \hat{y}} \circ \frac{\partial \hat{y}}{\partial v} = \begin{bmatrix}
\hat{y}_1 - y_1\\
\hat{y}_2 - y_2
\end{bmatrix} \circ \begin{bmatrix}
\frac{1}{1 + e^{\sum v_{i,1} * h_{i}}} \\
\frac{1}{1 + e^{\sum v_{i,2} * h_{i}}}
\end{bmatrix} \circ \begin{bmatrix}
1 - \frac{1}{1 + e^{\sum v_{i,1} * h_{i}}} \\
1 - \frac{1}{1 + e^{\sum v_{i,2} * h_{i}}}
\end{bmatrix}$

As you can see all the matrix multiplications in both these steps are simple matrix multiplication but the Hadamard product can simplify the representation if used.

Convolutional Neural Networks

CNNs add an additional filtering step before passing through the weights. It passes a filter through the matrices to get a value that represents a neighborhood around a value. The filter takes the Hammond product and then sums all the elements of the resulting matrix.

For example, if we have the matrix in green

with the convolution filter

Then the resulting operation is a element-wise multiplication and addition of the terms as shown below. This kernel (orange matrix) $g$ is shifted across the entire function (green matrix) $f$. Performing a Hammond product at each step, and then summing the elements.

Note: The Hammond product is often defined for matrix of exactly the same dimensions, this is relaxed in CNNs due to the filter moving through the image successively. It is also possible to perform the Hammond product on the edges of the image where different padding techniques can be used.


Processing csv file with more than 700K rows of data I have a .csv file (around 400MB in size) which contains 700K rows of structured data.
Table structure is:

+----+----------+-------+-----------+--------+--------+
| Id | Category | List1 |   List2   | Value1 | Value2 |
+----+----------+-------+-----------+--------+--------+
|  1 |        1 | A,B,C | Cat1,Cat2 |    100 |      5 |
|  2 |        1 | D,F   | Cat1,Cat4 |    120 |      4 |
|  3 |        2 | E,A   | Cat3      |    140 |      2 |
|  4 |        2 | E,A   | NULL      |    110 |      3 |
|  5 |        3 | B     | Cat2      |    100 |      6 |
+----+----------+-------+-----------+--------+--------+

For each row (CurrentRow) of the table I want to calculate:
CurrentRow.Value3 = SUM(Value1)/SUM(Value2) of all other rows (OtherRow) in the table where the following conditions are met:

Condition1 = CurrentRow.Category != OtherRow.Category
Condition2 = CurrentRow.List1 intersects OtherRow.List1
Condition3 = CurrentRow.List2 intersects OtherRow.List2 or CurrentRow.List2 = NULL or OtherRow.List2 = NULL

Also, I want to list Ids of rows which where involved into calculation of Value3.

Example for first row:
Condition1: first row has category value 1. As a result, rows 3-4 meet consition #1 because their category values are not equal to 1.
Condition2: column "List1" has values "A,B,C" which intersects with values "E,A" (row #3), "E,A" (row #4), "B" (row #5).
Condition3: Column "List2" has values "Cat1,Cat2" which intersects with values "Cat1,Cat4" (row #2), "Cat2" (row #5) and also we take row #4 as it has "NULL" value.

As a result, we take rows #4 and #5 as they both meet all conditions.
Value3 = (110+100)/(3+6) = 210/9 = 23.33
Ids = "4,5"

The result for table above would be:

+----+----------+-------+-----------+--------+--------+--------+------+
| Id | Category | List1 |   List2   | Value1 | Value2 | Value3 | Ids  |
+----+----------+-------+-----------+--------+--------+--------+------+
|  1 |        1 | A,B,C | Cat1,Cat2 |    100 |      5 | 23.33  | 4,5  |
|  2 |        1 | D,F   | Cat1,Cat4 |    120 |      4 | NULL   | NULL |
|  3 |        2 | E,A   | Cat3      |    140 |      2 | NULL   | NULL |
|  4 |        2 | E,A   | NULL      |    110 |      3 | 20     | 1    |
|  5 |        3 | B     | Cat2      |    100 |      6 | 20     | 1    |
+----+----------+-------+-----------+--------+--------+--------+------+

I have tried to do it using pandas in python:

data = pd.read_pickle('data.df')
for _index, _record in data.iterrows():
    category = _record["Category"]
    list1 = _record["List1"]
    list2 = _record["list2"]
    candidates = data.loc[(data['Category'] != category) & (pd.Series(list1).isin(data["List1"]).any()) & ((pd.Series(list2).isin(data["List2"]).any()) | (data["List2"][0] == 'NULL') | (list2[0] == 'NULL'))]
    value1_sum = candidates["Value1"].sum()
    value2_sum = candidates["Value2"].sum()
    ids = candidates[['Id']].to_numpy()
    if value2_sum > 0:
        data.loc[_index, "Value3"] = value1_sum/ value2_sum
        data.loc[_index, "Ids"] = ids
data.to_csv('result.csv')

It works for small number of rows, but takes forever on 700K rows.
Is there any method to optimize this algorithm?

I think the main time-consumer will be the fact that you are iterating over 700,000 rows, one-by-one.

You could perhaps do a few of your checks/comparisons (used in the line with bit-wise stuff: &, | etc) to begin with, for all rows. So for that row, it looks like this now:

data.loc[
    (data['Category'] != category) &
    (pd.Series(list1).isin(data["List1"]).any()) &
    (pd.Series(list2).isin(data["List2"]).any()) |
    (data["List2"][0] == 'NULL') |
    (list2[0] == 'NULL'))]

Before the loop, you could pre-compute each of those rows, adding the results for each comparison as a new columns:

data["check1"] = data['Category'] != category
data["check2"] = ...
data["check3"] = ...
data["check4"] = ...
data["check5"] = data.List2 == "NULL"

Now use those columns' values in your line with all comparisons.

To gain a better understanding of what it taking up all the time, you could profile the code, e.g. using the CProfile module. You could just run over the first 100 rows of your data for that purpose - the profiling tool will give summary statistics.

If you really want to get stuck into it, check out tha Pandas performance improvement methods, which get into using Numba and Cython. There can be huge gains here, but you need to maybe invest a little more effort. Numba would be the easiest to begin with; unless you are experienced with C-languages.


Estimate the normal distribution of the mean of a normal distribution given a set of samples? Let's say there is a distribution, call it D, for which I don't know details (i.e. mean and variance) but can assume that it's a normal distribution. I now have N samples from D. I cannot take more samples because it's too heavy. Given the N samples, and the assumption that D is a normal distribution, how can I compute the mean and variance of the mean of D? Naturally it's less likely for the mean of D to be far away from the samples (thought still possible), while more likely to be near the samples.

The same can be done through any of the statistical estimation technique such as maximum likelihood or Minimum variance unbiased estimator(MVUE). Generally maximum likelihood is the easiest way to evaluate the mean and variance of the distribution. If you perform Maximum likelihood method the estimated mean of the normal distribution is the sample mean. Given x1,x2.....xn are the samples

Estimated mean u = (x1 + x2 + ... + xn)/n
Similarly the estimated variance is  Σ(xi - u)^2/n

Both of these are obtained by setting the partial derivative of the likelihood function with respect to the mean and the standard deviation to zero.

Hope this helps.


how to convert multiple columns into single columns in pandas? I have a dataframe like this

my desire format is like this

how can i do this?

Let's say you have the following data:

import pandas as pd
import numpy as np
df = pd.DataFrame({'values': ['1', '2', '3', '4', '5', '6'],
                  'month1': ['January', 'March', np.nan, np.nan, np.nan, np.nan],
                  'month2': [np.nan, np.nan, 'February', 'April', np.nan, np.nan],
                  'month3': [np.nan, np.nan, np.nan, np.nan, 'May', 'October']})

  values   month1    month2   month3
0      1  January       NaN      NaN
1      2    March       NaN      NaN
2      3      NaN  February      NaN
3      4      NaN     April      NaN
4      5      NaN       NaN      May
5      6      NaN       NaN  October

You can use the following solution:

df['month']=np.nan
for month in [col for col in df.columns if 'month' in col]:
    df['month'].fillna(df[month],inplace=True)

It first creates an empty column named "month" with NaN values, and you fill the NaN with the values from the "monthX" columns, concretely it gives you:

  values   month1    month2   month3     month
0      1  January       NaN      NaN   January
1      2    March       NaN      NaN     March
2      3      NaN  February      NaN  February
3      4      NaN     April      NaN     April
4      5      NaN       NaN      May       May
5      6      NaN       NaN  October   October


feature importance via random forest and linear regression are different Applied Lasso to rank the features and got the following results:

rank feature prob.
==================================
1       a     0.1825477951589229
2       b     0.07858498115577893
3       c     0.07041793111843796

Note that the data set has 3 labels. The ranking of the features for the different labels are same.

Then applied random forest to the same data set:

rank feature score
===================================
1       b     0.17504808300002753
6       a     0.05132699243632827
8       c     0.041690685195283385

Notice that ranking are very different from the one produced by Lasso.

How to interpret the difference? Does it imply the underlying model is inherently nonlinear?

So your query is a comparison of linear regression vs. random forest's model-derived importance of variables.

The lasso finds linear regression model coefficients by applying regularization. A popular approach to rank a variable's importance in a linear regression model is to decompose $R^2$ into contributions attributed to each variable. But variable importance is not straightforward in linear regression due to correlations between variables. Refer to the document describing the PMD method (Feldman, 2005) in the references below.

Another popular approach is averaging over orderings (LMG, 1980). The LMG works like this:

Find the semi-partial correlation of each predictor in the model, e.g. for variable a we have: $SS_a/SS_{total}$. It implies how much would $R^2$ increase if variable $a$ were added to the model.
Calculate this value for each variable for each order in which the variable gets introduced into the model, i.e. {$a,b,c$} ; {$b,a,c$} ; {$b,c,a$}
Find the average of the semi-partial correlations for each of these orders. This is the average over orderings.

The random forest algorithm fits multiple trees, each tree in the forest is built by randomly selecting different features from the dataset. The nodes of each tree are built up by choosing and splitting to achieve maximum variance reduction. While predicting on the test dataset, the individual trees output is averaged to obtain the final output. Each variable is permuted among all trees and the difference in out of sample error of before and after permutation is calculated. The variables with highest difference are considered most important, and ones with lower values are less important.

The method by which the model is fit on the training data is very different for a linear regression model as compared to random forest model.  But both models don't contain any structural relationships between the variables.

Regarding your query about non-linearity of the dependent variable: The lasso is essentially a linear model which will not be able to give good predictions for an underlying non-linear processes, as compared to tree based models. You should be able to check this by verifying the models performance over a set-aside test set, if the random forest performs better, the underlying process may be non-linear.  Alternatively, you could include variable interaction effects and higher order variables created using a, b, and c in the lasso model and verify if this model performs better as compared to a lasso with only a linear combination of a, b and c. If it does, then the underlying process might be non-linear.

References:

https://en.wikipedia.org/wiki/Decision_tree_learning
https://en.wikipedia.org/wiki/Random_forest
Relative importance of Linear Regressors in R
Relative Importance and Value, Barry Feldman (PMD method)


Metric for label imbalance I'm looking for a metric that can be used to quantify how imbalanced the labels are in a dataset.
I'm not looking for a strategy to solve the imbalance problem, I just want to present how imbalanced my dataset is. I've computed the ratio of the most frequent and least frequent labels which is probably an ok way of doing it but I'm sure there's a more robust way?

You are looking for Entropy. The higher the entropy, the more imbalanced it is. You can use this function for calculating it.


Using Policy Iteration on an automaton I've read many explanation on how do to policy iteration, but I can't find an example, so I'm stuck right now trying to figure out to Policy Iteration.

The numbers next to each state show the reward received for arriving in that state. For example, if the agent started in $S_0$ and took the $Blue$ action, and ended up in state $S_1$, the immediate reward would be $-5$.

The discount value is 0.1, and the initial policy is $\pi(S_0)=Blue$ and $\pi(S_1)=Red$

$S_2 $ state is the terminal state - game over. The two possible actions are Blue and Red as can be seen on the image.

I just need something to help me start, because no explanation really made me understand how do I start the policy iteration till convergence.

Policy Iteration is essentially a two step process:

Evaluate the current policy by calculating $v(s)$ for every non-terminal state.

Internally this requires multiple loops over all the states until the value calculations are accurate enough

Improve the policy by choosing the best action $\pi(s)$ for every non-terminal state

The overall process terminates when it is not possible to improve the policy in the second part. When that happens, then - from the Bellman equations for the optimal value function - the value function and policy must be optimal.

In detail for your example, it might work like this:

Initialise our value function for $[S_0, S_1, S_2]$ as $[0,0,0]$ - note that by definition for the terminal state $v(S_2)=0$ so we never need to recalculate that (this is also consistent with the diagram, which shows the "absorbing" actions) so we could actually calculate it if we want, but it's a waste of time.
Choose a value for accuracy limit $\theta$. To avoid going on forever in the example, let's pick a high value, say $0.01$. However, if this was computer code, I might set it lower e.g. $10^{-6}$
Set our discount factor, $\gamma = 0.1$. Note this is quite low, heavily emphasising immediate rewards over longer-term ones.
Set our current policy $\pi = [B, R]$. We're ready to start.
First we have to do policy evaluation, which is multiple passes through all states, updating $v(s) = \sum_{s',r} p(s',r|s,\pi(s))(r + \gamma v(s'))$. There are other ways of writing this formula, all basically equivalent, the one here is from Sutton & Barto 2nd edition. We also can choose between a "pure" iteration calculating $v_{k+1}(s)$ from $v_{k}(s)$, but I will do the update in place to the same $v$ array, because that is easier and usually faster too.

Note each sum here is over the two possible $(s', r)$ results from each action.
Pass 1 in detail, iterating though states $[S_0, S_1]$:

Set max delta ($\Delta = 0$) to track our accuracy
$v(S_0) = \sum_{s',r} p(s',r|S_0,B)(r + \gamma v(s')) = 0.5(-5 + 0.1 v(S_1)) + 0.5 (-2 + 0.1 v(S_0)) = -3.5$
This sets max delta to 3.5
$v(S_1) = \sum_{s',r} p(s',r|S_1,R)(r + \gamma v(s')) = 0.6(-5 + 0.1 v(S_1)) + 0.4 (0 + 0.1 v(S_2)) = -3$
Max delta is still 3.5
The $v$ table is now $[-3.5, -3, 0]$

Pass 2, because $\Delta \gt 0.01$:

Reset $\Delta = 0$
$v(S_0) = 0.5(-5 + 0.1 v(S_1)) + 0.5 (-2 + 0.1 v(S_0)) = -3.825$
$\Delta = 0.325$
$v(S_1) = 0.6(-5 + 0.1 v(S_1)) + 0.4 (0 + 0.1 v(S_2)) = -3.180$
$\Delta = 0.325 \gt 0.01$
The $v$ table is now $[-3.825, -3.180, 0]$

Pass 3, $v = [-3.850, -3.191, 0]$
Pass 4, $v = [-3.852, -3.191, 0]$, this is small enough change that we'll say evaluation has converged.

Now we need to check whether the policy should be changed. For each non-terminal state we need to work through all possible actions and find the action with the highest expected return, $\pi(s) = \text{argmax}_a \sum_{s',r} p(s',r|s,a)(r + \gamma v(s'))$

For $S_0$:

Action B scores $0.5(-5 + 0.1 v(S_1)) + 0.5 (-2 + 0.1 v(S_0)) = -3.852$ (technically, we already know that because it is the current policy's action)
Action R scores $0.9(-2 + 0.1 v(S_0)) + 0.1 (0 + 0.1 v(S_2)) = -2.147$
R is the best action here, so change $\pi(S_0) = R$

For $S_1$:

Action B scores $0.8(-2 + 0.1 v(S_0)) + 0.2 (-5 + 0.1 v(S_1)) = -2.971$
Action R scores -3.191
B is the best action here, so change $\pi(S_1) = B$

Our policy has changed, so we go around both major stages again, starting with policy evaluation and the current $v = [-3.852, -3.191, 0]$, using the new policy $[R, B]$

I leave it up to you to do the second pass through evaluation and improvement. I suspect it will show that the new policy is optimal, and the policy will remain $[R, B]$ (although $[R,R]$ seems possible without doing the calculation). If the policy stays the same on a complete pass through evaluation and improvement, then you are done.


Image Classification with two very similar and random-ish classes I am trying to build a neural network which takes a picture of a FOREX chart (currency exchange) and outputs an "up" or "down" prediction. I'm classifying up or down based on where the stock is after one hour, and currently I have 400 examples in each class. However, my training attempts are stuck at around 49-51% accuracy. I've come up with a few potential reasons:

1) A bad model. I don't think this is the case - I've tried transfer learning on Inception V3 and two of my own models. However, is there a different type of model that could somehow account for the randomness of markets?

2) Not enough data: I think this is the most likely explanation. From what I've read, you need thousands of images, not a few hundred,  for good image classification.

3) The problem is impossible. I chose FOREX over stocks because FOREX isn't as heavily affected by headlines during the span of a few hours. Most FOREX trading in that time frame is done by technical analysis, which is doing things with charts - trend lines, moving averages etc.

I know that two is the most likely explanation, but before I go out and spend days collecting more data, I would like a bit of feedback to ensure that this is actually possible.

Thanks!

This is likely a combination of 1, 2, and 3. Without any data to back this claim up, 800 data points seems woefully small for the scale of problem you are trying to solve.

Assuming the problem is solvable, why not try to build a model directly onto the time series of the underlying data? What extra information are you getting from the image that you can't get from the time series? I would recommend reading up on some time series models.

Keep in mind that people using way more sophisticated methods are likely trying to build similar models, and likely are not succeeding.


Why does downsampling leads classification to only predict one class? I have a multi-class classification problem. It performs quite well but on the least represented classes it doesn't. Indeed, here is the distribution :

And here are the classification results of my former classification (I took the numbers off the labels):

.

Therefore I tried to improve classificationby downsampling the marjority classes

import os
from sklearn.utils import resample

# rebalance data
#df = resample_data(df)

if True:

    count_class_A, count_class_B,count_class_C, count_class_D,count_class_E, count_class_F, count_class_G = df.grade.value_counts()
    count_df = df.shape[0]
    class_dict = {"A": count_class_A,"B" :count_class_B,"C": count_class_C,"D": count_class_D,"E": count_class_E, "F": count_class_F, "G": count_class_G}
    counts = [count_class_A, count_class_B,count_class_C, count_class_D,count_class_E, count_class_F, count_class_G]
    median = statistics.median(counts)

    for key in class_dict:
        if class_dict[key]>median:
            print(key)
            df[df.grade == key] = df[df.grade == key].sample(int(count_df/7), replace = False)
                                             #replace=False,    # sample without replacement
                                             #n_samples=int(count_df/7),     # to match minority class
                                             #random_state=123)
# Divide the data set into training and test sets
x_train, x_test, y_train, y_test = split_data(df, APPLICANT_NUMERIC + CREDIT_NUMERIC,
                  APPLICANT_CATEGORICAL,
                  TARGET,
                  test_size = 0.2,
                  #row_limit = os.environ.get("sample"))
                  row_limit = 552160)

However it the results where catastrophic. The model accuracy and the model loss looked to have some issues :

And everything was classified in "A" on the test set.

For the looks of it the sampling method you used isnt doing the trick.
The way you use pandas slicing and assignment is a bit off.

Something like this might work:

dfs = []
for key in class_dict:

    if class_dict[key] > median:
        factor = int(count_df/7)
        key_df = df[df.grade == key].sample(factor, replace=False)
    else:
        key_df = df[df.grade == key]

    dfs.append(key_df)
new_train_df = pd.concat(dfs, sort=False)

While this might work, there are more straight forward ways down sampling the data(like setting the exact number of samples expected from each class).
This question has nice answers too.


What should I master better for professional data science in economics and finance? First, excuse me for the noob and long question which is probably doesn’t even belong to here, I know there are several question been answered like this out there, but I think this is going to be up-to-date. Stack Overflow deleted my question and redirected me to here.

I study economics and finance on undergraduate level, and to be honest, I am not really into programming so far. However, I must admit it you can't doing really well nowadays without specific softwares and programming languages on economics/finance related fields.

According to my curriculum, I’ve encountered Matlab, some econometrics softwares, and of course MS Office, especially Excel with VBA. I have some shady framework in my mind, and please feel free to correct me if I am wrong. So as I experienced, for numerical calculations and doing the vast majority of math, Matlab, Octave and Mathematica exists. For econometrics, there are professional softwares like eViews, STATA, SPSS or the open source Gretl and Tableau for data visualization. And last, we can use Excel to manage databases.

Long story short, my basic question would be that, are these above the best tools for doing the job ? Or should I switch to more professional tools – like real programming languages - to being better in solving mathematical problems, numerical calculations, econometrics, data science and exquisite, high-quality data visualization? What are the most desirable skills in the data science industry nowadays in economic/financial areas?

I heard that R is a quite trending statistical programming language in these days, and getting better and better each day - I already wrote some functions and visualizations in Rstudio. I also heard that SQL is also a better option to manage really massive data sets instead of Excel, but is SQL able to do every kind of stuff with data what can be done in Excel ? It seems to me Python is generally the number one language for data analysis, it’s flexible and usable on a broad scale. I find Python libraries - such as matplotlib, numpy, pandas, bokeh - extremely attractive. What about Julia , is this going to be the next R in the future ? To be honest, I am also still confused a little bit by such terms like data science, data analysis, data mining, machine learning, big data – are there any serious difference between these phrases?

From above, which one is that I should really focus on and master it ? Keep practicing on popular softwares, or switch to R , Python, Julia, SQL ? Maybe both of them? Again, we are talking about only graduate and undergraduate level of economics and finance, and related jobs. I don’t want to develop serious and complex softwares/applications, just quantitatively analyze stock prices, corporate and economic data, like annual reports, employments, GDP and so on.

Experienced data analysts, please guide me through the confusing forest of data analysis tools. I appreciate every kind of comment.

welcome to the forum. I‘m a trained economist, I do a lot of econometrics, and I work in research. My opinion (and it is only opinion of course) is that you should focus on R in the first place and consider learning Python. Both is not so hard after all.

Why? R is for free and it offers a lot of support for econometrics (it is well regarded in the community). It also is well regarded in the ML community and you can work with high spec things such as Keras, LightGBM etc. So with R you can‘t be wrong. Many top research papers in economics are done with R. Stata is good for some things (like Panel), but there are so many things you cannot do with Stata. R is more powerful.

Why Python? Python and R are „similar“ wrt what you can do in econometrics/statistics. But Python offers more flexibility in many aspects (some might disagree but this is how I see it). In essence, Python is a thing that offers you a lot of possibilities on top of working with data. So have a look at it. Can be very beneficial for your career.

Data science is an extremely broad field. But if you focus on econometrics/statistics, many other things become relevant as well in the moment. Examples are: working with „big“ data and getting a structure into data (e.g. from online sources etc). I also think that neural nets and tree based things such as boosting will become more relevant in economics in the future. Actually outside the academic world, these methods are used quite a lot in the context of economic problems.

Final note: If you think that you cannot do both (R and Python) it would still be fine to have some knowledge/experience in both. You grow with your problems. Even if you have only little experience, you can advance fast if you know where/how to start. This can also be good wrt getting a good job.

P.S. For Python there are also some good packages/applications for economists: https://lectures.quantecon.org/py/


NLP to detect duplicates for very technical language I have the following scenario, to detect duplicate products based on the description fields. The Description Field contains product technical name, dimensions, characteristics. My model needs to consider that different annotation and abbreviations might have been used for technical names, text errors in data entries,  similar/different dimensions or characteristics might still point to the same product. Therefore, I think that applying a normal fuzzy matching or other NLP text matching will not perform well in my case. I trying to approach this problem as a learning/supervised model, but still not sure how so any suggestion/idea is very appreciated.

If you think fuzzy matching does not work, you basically have two options:

Unsupervised: Try topic modeling to find „similar“ products. The problem is that you will need to pre-define the number of products (groups, aka „topics“), which can be a problem. Also topic modeling will only work well if there are sufficient differences in the descriptions. https://github.com/Bixi81/R-ml/blob/master/NLP_topic_modelling.R

An unsupervised approach seems to be a good option for you, judging from what you wrote about your problem.

Supervised: If you have a good set of descriptions with a label (e.g. product 1, product 2, etc), you can use some NLP model to predict which text belongs to which product. Not sure how much you know about it: You could start with a „Bag of Words“ and try things like Lasso, Boosting, or even Neural Nets on this.

Here is an example on how to apply Neural Nets to text classification. But be aware: in real world applications, text classification can be very tricky, especially if you have many classes (products in your case) and not so many observations per class (descriptions in your case).


How to cluster user historical data? I have transactional-level users data that includes their behaviour, such as reading articles, searching for content, posting, etc. I would like to cluster (most probably K-Means) these users based on aggregated metrics, For eg. SUM(articles read), SUM(searches), etc.
As users sign up at different times across years, they may be at very different cycles of their activity levels (first 30 days? inactive? etc.) So I was wondering what would be an appropriate time window to aggregate relative to each user. Should I just take all user activities in the past 30 days? Or for each user's first 30 days / their peak month? Or just simple averaging per month?
Averaging seems straightforward but for users who are on the site longer the average may be much lower than say, users who has been using the platform for only a month.

If you have a whole year data, use the moving window technique, thus you'll have diffrent clusters each month (since users cycles can change), after that, I don't think you should use averaging because it does not take into consideration the times series aspects, which means clusters from the beginning of the year can be very different from the recent ones, you better attribute different weights to different clusters.
I recommend using Subsequence TimeSeries Clustering since it cluster data and it's based on its temporal aspect.
This research paper might help you out Subsequence Clustering
and this is the source code that you can start with Incremental Subsequence Clustering.


Learning curve using micro F-score and macro F-score I plotted the learning curves using micro and macro F-scores for a Multinomial Naive Bayes classifier.

The first plot is made using micro F-score, and the second using macro F-score. I find it quite difficult to interpret both of them. The explanations I found for learning curves use errors.

I think in the first plot there is variability up to around 300 instances, and then it starts to converge until the point when the lines are parallel. Therefore, even adding more data would not help. But what about bias?

Also can you please explain to me the difference between the plots in terms of micro and macro F-scores?

Micro calculates F score globally by counting the total true positives, false negatives and false positives.

Macro calculates F score for each label and find their unweighted mean. Macro F score does not take label imbalance into account.

Given there is a difference in your performance between the metrics, your data is imbalanced in the base-rate for the class labels.

Adding more data to the minority class(es) will help.


How to utilize dictionary data set for text classification? I have a dataset similar to newsgroup20 for classification. With the training dataset, I have a dictionary data set that explains some jargons in the training dataset. These both are different data set, So how will i utilize the dictionary dataset for improving my model accuracy?

You can try this approach LEARNING TO COMPUTE WORD EMBEDDINGS ON THE FLY. Using additional auxiliary data you can substitute jargons using definitions from dictionary. In this case, jargons  embeddings will become more relevant, in semantic sense .


Can ROC/AUC help model training or just be used for model chosing? if can be used to help for training?how?

Your answer would depend on the model you have in mind. For something like multi-variate linear regression, you would use it for factor reduction. For something like a neural network, you could use it while you test different settings for any number of things like the dropout rate or the learning rate. It really just depends on what you're working on at the time.


Subset of training set produces good results while full training set produces poor results I have an extremely unbalanced data set: around 200 positive samples and 70,000 negative samples. To overcome this problem I have tried to over-sample the minority class as suggested in previous questions here.

Since training the classifier (in my case I am using SVM) takes a long time on my computer I first trained a classifier on a small subset of my data picked randomly, the classifier gave solid results when tested on the test set, yet when I trained a classifier using the full training data the classifier produced very poor results. Does anyone have insight into why this is happening and what should i do? I have also tried using weights but it did not produce a different result.

For the 200 positive samples, are there many different "types"? In other words, will future positive samples look nothing like the current positive samples? If this is the case, you may try anomaly detection in machine learning.

From a lecture by Andrew Ng in Coursera, if you have small number of positive samples (e.g., 20), large number of negative samples (e.g., 10000), and it's hard for any algorithm to learn from the positive samples, you should use anomaly detection, otherwise, you can use supervised learning.


How does strided deconvolution works? I am trying to understand how the shape of the image changes after deconvolution ?
I am trying to understand the example code of convolutional autoencoder from neon.

layers = [Conv((4, 4, 8), init=init_uni, activation=Rectlin()),
          Pooling(2),
          Conv((4, 4, 32), init=init_uni, activation=Rectlin()),
          Pooling(2),
          Deconv(fshape=(3, 3, 8), init=init_uni, strides=2, padding=1),
          Deconv(fshape=(3, 3, 8), init=init_uni, strides=2, padding=1),
          Deconv(fshape=(4, 4, 1), init=init_uni, strides=2, padding=0)]

The input_shapes and output_shapes of each layer are as follows

Convolution Layer 'ConvolutionLayer': 1 x (28x28) inputs, 8 x (25x25) outputs, padding 0, stride 1
Pooling Layer 'PoolingLayer': 8 x (25x25) inputs, 8 x (12x12) outputs
Convolution Layer 'ConvolutionLayer': 8 x (12x12) inputs, 32 x (9x9) outputs, padding 0, stride 1
Pooling Layer 'PoolingLayer': 32 x (9x9) inputs, 32 x (4x4) outputs
Deconvolution Layer 'DeconvolutionLayer': 32 x (4x4) inputs, 8 x (7x7) outputs
Deconvolution Layer 'DeconvolutionLayer': 8 x (7x7) inputs, 8 x (13x13) outputs
Deconvolution Layer 'DeconvolutionLayer': 8 x (13x13) inputs, 1 x (28x28) outputs

I understand how the shapes change after convolution ('valid'). (Thanks to http://cs231n.github.io/convolutional-networks/)
How does the stride affect the size of matrix when deconvolution (full convolution) is used ?

The strided deconvolution is expanding the size of the layer (you can think of it as undo-ing the effect of the pooling without doing unpooling).

Check out slide 18 here. That might help.


image_dataset_from_directory VS flow_from_directory What is the main diffrence between flow_from_directory VS image_dataset_from_directory in keras?
which one should I use?

tf.keras.preprocessing.image_dataset_from_directory
Generates a tf.data.Dataset from image files in a directory.

ImageDataGenerator.flow_from_directory
Takes the path to a directory & generates batches of augmented data.

While their return type also differs but the key difference is that flow_from_directory is a method of ImageDataGenerator while image_dataset_from_directory is a preprocessing function to read image form directory.
image_dataset_from_directory will not facilitate you with augmented image generation capability on-the-fly.

which one should I use?

It's quite common to generate augmented images when working with CNN, so better to use flow_from_directory.
If you do not need augmented image, you may control the same by the parameters of ImageDataGenerator
Reference - Keras docs


How to estimate the transition probabilities for a Markov Chain when time intervals are non-equally spaced I've been given a dataset with a number of observable states. I am trying to apply a Finite State Markov Chain to model the system, but I found that I can't estimate the transition probabilities if the observed states were sampled using different time intervals. How can I find these probabilities?

I will try to make the question a more clear. I have samples collected in random intervals during a 6-month period. This samples represent the "quality" of a system, which is ranked from 0 to 15 in discrete intervals i.e. (0,1,2...15). I need to model de system using a FSMC to mimic the system's behavior. So far, I have estimated the transition probabilities between states using only the frequencies of those transitions using all the samples. I am not interested in modeling the time, however, I am not sure If I can estimate the transition probabilities in such a simple way or if I have to take the time between samples (which in my case is random) into consideration when estimating those probabilities.

With the limited information that you have given, it is not possible to judge the trade-off between different approaches of handling time here. I can propose the following alternatives.

Check if you can discretize the time information and then use transition to same state assumption. For example, let us say that transition from A to B took place at 30 minutes. If you have standard time interval of 10 minutes, you can say that for interval 1 & 2 (10 minutes, 20 minutes), the transition was from state A to A whereas in the third interval the transition was from A to B. There will be problems though when multiple transitions take place in a given interval. You can either make the interval small enough or distribute the transition probability equally in such case.
Is the time information significant to whatever that you are trying to model? Otherwise you can ignore the time information.


Is it possible using tensorflow to create a neural network that maps a certain input to a certain output? I am currently playing with tensorflow, but can't seem to get a hold whether it usefull for my problem?

I need to create a neural network, that is capable of mapping input to output.

The way things are progression now, i haven't a single example in which this has been done, all types of problem tensorflow seem to solve are classification task, and not this kind of mapping problem... Is tensorflow able to do so?, do I have to use a different neural network framework to do so?, And if it is available could somebody show some code (not the MNIST example)

My task:

I am currently trying to make an neural network that takes in samples of audio files, and generates MFCC features from the samples. The MFCC features can be calculated manually, which I have done, so I know what the I output i seek is.  MFCC feature is a feature vector is different real values numbers.  I cannot be classified as class A or Class B, or doing so would greatly reduce the accuracy of the output, as you are fitting the output to predetermined "bins"...

From your description, it seems that you are facing a regression problem, because you want your output to be certain values. This is different from classification problems, that have as output the probability of the input belonging to certain class.

The key to using neural networks for regression is that the output layer should have no activation, that is, it should be a linear layer. As pointed out by @JanvanderVegt, a common loss function for regression problems is the mean square error (MSE) between the current output and the MFCC features you computed.

If you google "tensorflow regression example" you can find dozens of complete examples, like this or this.


Gradient Checking: MeanSquareError. Why huge epsilon improves discrepancy? I am using custom C++ code, and coded a simple "Mean Squared Error" layer.
Temporarily using it for the 'classification task', not a simple regression. ...maybe this causes the issues?

I don't have anything else before this layer - not even a simple Dense layer. It's just MSE on its own.
Its input is a collection of rows of input features. For example, here are 8 rows of input features that will be passed to MSE all at once:

{ a0, a1, a2, a3, a4, a5, a6, a7 }
{ b0, b1, b2, b3, b4, b5, b6, b7 }
{ c0, c1, c2, c3, c4, c5, c6, c7 }
{ d0, d1, d2, d3, d4, d5, d6, d7 }
{ e0, e1, e2, e3, e4, e5, e6, e7 }
{ f0, f1, f2, f3, f4, f5, f6, f7 }
{ g0, g1, g2, g3, g4, g5, g6, g7 }
{ h0, h1, h2, g3, h4, h5, h6, h7 }    //8x8 matrix (contains 64 different values)

Every row of this matrix gets passed into my "Mean Square Error" layer, returning a single scalar for such a row: "Cost".

I then compute a "final error" scalar, which is the average of such Costs.

When doing Gradient Checking, I am looking at how this "final error" quantity changes as I perturb each of the 64 input values, seen above. The idea is that the changes in finalError must correspond to the gradient computed by formula, with respect to my 64 input values. If they match, then I've coded backprop correctly.

Here is the forward prop:

$$finalError =   \frac{1}{r}\sum^r{  \left(  \frac{1}{2n}\sum^n{(input_i-target)^2} \right) }   $$

where $n$ is the number of features per row, and $r$ is the number of rows.

Here is the gradient wrt one of input values, that my backprop is using:

$$\frac{\partial finalError}{\partial input_i} = \frac{1}{rn}(input_i - target)$$

Question:

I peturb each input value 'up', then 'down', running forward prop 64*2 = 128 times. This gives me numerical estimate of gradient for my 64 input values.

However this numerical estimate and the actual analytical gradient become less similar when smaller epsilon is used. This is counterintuitive to me. On the contrary, my vectors match almost exactly, when I use a ridiculously large value for epsilon, such as $1$

Is this expected, or do I have an error in C++ code?

Here is the pseudocode

for every input value i:
   i -= EPSILON
   finalCost_down =  fwdprop( inputMatrix )//very simple - just computes final cost via MSE layer.  finalCost_down is a scalar.
   i += EPSILON
   finalCost_up   =  fwdprop( inputMatrix )
   gradientEstimate[i] = (finalCost_up - finalCost_down) / (2*EPSILON)

//after the loop, some time later, just one invocation of backprop:
trueGradientVec = backprop( vec )

//some time later:

discrepancyScalar =  (gradientEstimate - trueGradientVec).magnitude / gradientEstimate.magnitude + trueGradientVec.magnitude)

//somehow discrepancyScalar decreases the larger the EPSILON was used:
// discrepancy is 0.00275, if EPSILON is 0.0001
// discrepancy is 0.00025, if EPSILON is 0.001
// discrepancy is 2.198e-05, if EPSILON is 0.01
// discrepancy is 3.149e-06, if EPSILON is 0.1
// discrepancy is 2.751e-07, if EPSILON is 1

I would expect discrepancy to decrease when epsilon is decreased, because finer perturbations should give more precise slope estimate...

Andrew NG explanation of GradientChecking

This was caused by numerical precision of floating numbers. It becomes really apparent as we change one of input values, by a tiny $\epsilon$ which (in my example above) emedially affects the cost function.

The tricky thing was - because the cost function is Mean Square Error (MSE), and there are no other layers in my network, we indeed can use any epsilon to estimate the slope. Even ridiculous epsilon will work, and will be numerically more stable, explaining why discrepancy seemed to get better. That's just how $y=x^2$ works.

In practice, when doing gradient-checking, I switched to another cost function, which is linear (instead of MSE). This improved the numerical stability, allowing me to use epsilon that is 10 times smaller:

$$C=\sum^r\sum^n(obtained-expected)$$

the gradient is then simply:

$$\frac{\partial C}{\partial (obtained)_{rn}} = 1$$

I don't use this cost in production, just for purpose of gradient-checking.

So yeah, this explains why large epsilon was improving the discrepancy.

Edit:

Notice that this linear-cost function must not be used if you have a softmax right before it. That's because the sum of softmax is always 1.0

In that case you must use the MSE


Knowledge of statistics To be a data scientist I am an undergraduate student pursuing my BSc in Maths. I have taken statistics classes in first year. I want to know how much statistics do I need to know to get a decent place in the data science world?

Also, please do provide me with suggestions where to begin if I start my preparation now.

Very good question!

Statistics is the main core. Not only theoretical understanding but practical use of statistical concepts are very important (Applied Statistics). Moreover working with statistical Softwares to have some ideas how they work and what are usual input-outputs and visualizations.

More important is the main fields of Data Science namely Data Mining, Machine Learning and Pattern Recognition. They are all built on statistics (try to get a course on Statistical Learning Theory to get some ideas).

After statistics, computer knowledge in general is important. Get to know the concepts of programming (what is a good program and why, which language is proper for which task, what is algorithm and how you should evaluate, produce or read them, etc.). In Data Science, reporting the results are important. Get to know some visualization techniques/libraries which are handy and commonly used.

Then you come to the concept of Database and even more general "What is data in real-world?" Try to get your hands dirty with some SQL-like query languages on some toy examples to learn what the data is and how you load, transform or extract them.

The last but not least are Machine Learning/Data Mining/Pattern Recognition skills. I recommend a MSc. in this field (however it's not necessary but my personal experience as an interviewer shows that usually people with this background stand higher).

These are the main things came to my mind. Please note that

This is just my opinion and based on my experience.
There are more details that I skipped to keep the answer as generally valid as possible.

Good Luck!


What does Theano dimension ordering mean? In this code , line 13 is commented as Theano dimension ordering mode. What does this mean?

Let's say you're working with 128x128 pixel RGB images (that's 128x128 pixels with 3 color channels).

When you put such an image into a numpy array you can either store it with a shape of (128, 128, 3) or with a shape of (3, 128, 128).

The dimension ordering specifies if the color channel comes first (as with theano / "th") or if it comes last (as with tensorflow / "tf").

The code you've posted contains the following line:

inputs = Input((1, img_rows, img_cols))

It specifies the shape of the input as (1, img_rows, img_cols) - i.e. there's one color channel (gray scale) and it comes first. That's why it requires Theano's dimension ordering "th".


Text extraction from documents using NLP or Deep Learning I am looking for references(Papers/github projects) on how to use deep learning in a text extraction task.

Recently I was given a task to extract important information from documents of similar type, say for example legal merger documents. I have thousands of legal merger documents as inputs. A paralegal would go through the entire document and highlight important points from the document. This is the extracted text.

What I want to do: Given a document(say legal merger document) I want to use DL or NLP  to extract the information from the legal document that would be similar to that of the information extracted by paralegal.

I am currently using bag of words model to extract text from the document, calculating sentiment and displaying the sentences with positive or negative sentiments. This yielded very bad results.

My knowledge in DL/NLP is very limited and I am particularly looking for some interesting papers and github projects related to text extraction using these frameworks. Can anyone please provide me with some references and suggestions on how to tackle this issue?

Jurafsky and Martin's NLP textbook has a chapter about information extraction that should be a good starting point. For example, if you want to extract company names it will tell you how to do that.

  A paralegal would go through the entire document and highlight important points from the document.

What you need to do depends heavily on what your definition of "important" is here. It would help if you can give some specific examples.


NAB benchmark (Numenta Anomaly Benchmark) I have doubt on how to use NAB dataset for real time anomaly detection. The available datasets, for example the New York City taxi dataset, contains only 2 columns of timestamp and value, but where are the labels to predict the AUC for the dataset?

The anomallies sometimes are not marked as anomallies, the task consists in checking by yourself which are anomallies, but you have nothing to compare to.


Recommendation System to integrate with an android app I need to build a recommendation system that takes certain parameters as input, computes a score and order suggestions to users based on this score. Well this is what I need to do loosely speaking. I am new to the scene of data science and haven't come across anything that could help me out.

This guy asked a similar question 5 years ago. I basically need something like this:
https://stackoverflow.com/questions/6828013/recommendation-systems-for-a-mobile-market-and-algorithm-suggestion

Apologies for the open-ended and vague question.
Just need to be pointed out in the right direction.

Edit: I do not intend to place the system on the android device. It will be running back-end

I don't want to sound harsh but building a recommender system on a android isn't really Data science.

In any recommender system context, the science on data is usually done in ad-hoc manner.

As for the algorithm implementation, validation and scaling, data science can play a part in that. But this is still not related to the fact of serving the data in a mobile application nor a website. That's mainly computer engineering.

Recommender systems algorithms a quite expensive to compute on any mobile nevertheless. It's best fit to be done by a dedicated server which can also play a role in serving the information when needed. I can't get into the details of this thought as it's quite broad to fit in a single answer on the site.

I advice you to read my answer on the time/space complexity challenge in building this types of application. That may help you getting some insights about the architectural design of a recommender engine in the real world.

I hope that this answers your question.


Why ReLU is better than the other activation functions Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu has a disadvantage and it is its expected value. there is no limitation for the output of the Relu and so its expected value is not zero. I remember the time before the popularity of Relu that tanh was the most popular amongst machine learning experts rather than sigmoid. The reason was that the expected value of the tanh was equal to zero and and it helped learning in deeper layers to be more rapid in a neural net. Relu does not have this characteristic, but why it is working so good if we put its derivative advantage aside. Moreover, I guess the derivative also may be affected. Because the activations (output of Relu) are involved for calculating the update rules.

The biggest advantage of ReLu is indeed non-saturation of its gradient, which greatly accelerates the convergence of stochastic gradient descent compared to the sigmoid / tanh functions (paper by Krizhevsky et al).

But it's not the only advantage. Here is a discussion of sparsity effects of ReLu activations and induced regularization. Another nice property is that compared to tanh / sigmoid neurons that involve expensive operations (exponentials, etc.), the ReLU can be implemented by simply thresholding a matrix of activations at zero.

But I'm not convinced that great success of modern neural networks is due to ReLu alone. New initialization techniques, such as Xavier initialization, dropout and (later) batchnorm also played very important role. For example, famous AlexNet used ReLu and dropout.

So to answer your question: ReLu has very nice properties, though not ideal. But it truly proves itself when combined with other great techniques, which by the way solve non-zero-center problem that you've mentioned.

UPD: ReLu output is not zero-centered indeed and it does hurt the NN performance. But this particular issue can be tackled by other regularization techniques, e.g. batchnorm, which normalizes the signal before activation:

  We add the BN transform immediately before the nonlinearity, by
  normalizing $x = Wu+ b$. ... normalizing it is likely to produce
  activations with a stable distribution.


Classifying encrypted images I know this question is rather broad but hopefully on topic. Are there useful references on classifying encrypted images. For example classifying cat/no cat on encrypted images.
Is there efficient and accurate software for this task?
Thank you!

EDIT for clarification: What I mean is the following: We have an encryption method E. A classifier is trained on encrypted images with their labels. Then at run time an image is sent in encrypted form from the client to the server and classified. This is similar to:

ML Confidential:
Machine Learning on Encrypted Data
https://eprint.iacr.org/2012/323.pdf

Very briefly, homomorphic encryption as used by the linked paper in the question works as follows (excuse ad-hoc notation). Define an encryption method $E(x,k)$ that takes some data $x$ and secret key $k$ that has the following properties:

There is a decryption method $D( E(x,k), k ) = x$ (optionally the two $k$ here can be different)
You can define operations that work on the encrypted data in a self-consistent manner, e.g. $\text{Add}_E(E(x,k),E(y,k)) = E(x+y,k)$ - note that the result is still encrypted, and you need to know $k$ in order to use method $D()$ and find out what $x+y$ is.
You may also need to support operations like $\text{Add}_E(E(x,k),y) = E(x+y,k)$ e.g. where one of the values is normal numeric variable - this might be useful for initialising a model - or using an existing trained model - but the catch is that when training you cannot keep these values unencrypted, once they use updated due to training results, they will also be encrypted.
Note that no operation can output a decrypted value without using the key, otherwise the whole scheme is insecure, it implies some backdoor to get decrypted data.
For ML use, you need an encryption scheme E that supports multiple basic operations, e.g. $\text{Add}_E$, $\text{Subtract}_E$, $\text{Multiply}_E$, $\text{GreaterThan}_E$ etc. You need some minimal set that is enough to build a model.

You can then build variant models that instead of performing arithmetic to sum/multiply etc numeric values, work with the abstract operations on encrypted data. The models would otherwise look just like their unencrypted counterparts - e.g. you could quickly build a linear regression using only a few types of operation.

Given that you are replacing simple fast floating point add, multiply operations that are built into the processor with a more complex custom operation, this significantly affects performance compared to the unencrypted model. How much so depends on the encryption scheme, and how it is implemented. The paper shows in section 5, timings on unencrypted vs encrypted data and the difference is several orders of magnitude slower when encrypted.

The paper was published in 2012, so it is possible that some improvements have been made here. However, on a deeper read of the subject I think that this is still at the stage of research proof-of-concept. I might be wrong, and there are nice workable implementations available that could be used in something as complex as a CNN, but I have not found anything.

You also linked numer.ai from comments. Initially that looked interesting because surely they would have solved efficiency problems. But in fact their main competitions are using data obfuscation techniques, not encryption - a homomorphic encryption paper is linked from their main site, but it does not seem to be what they are using. End users are writing very familiar-looking scripts that perform logistic regression etc using regular Python (no special operators imported)

I think you should take a second look at your wider problem and analyse your threat model. What precisely are you trying to protect against? If it is about keeping your own operations separate from client data, then you might be looking into company process and auditing solutions, rather than purely technical (you may still want to add technical solutions such as disk-level encryption to protect customer's data at rest in you data centre, in case someone gets into the centre physically and just grabs a disk containing all the cat images)

For instance, look at Cloud Security Alliance which as well as having a certification scheme, has analysis showing how their recommendations map to other schemes such as UK's ISO27001. Note that this is typically significant capital investment including 1+ year project to implement, and usually undertaken by mid-sized or larger companies, when they want to work with government or large corporate's data. However, it is probably a more reliable and maybe still cheaper route than trying to research and build a technical solution involving models that process encrypted data at this time.


Gibbs sampling (For inference) vs EM I'm familiar with the Expectation-Maximixation algorithm and, until now, I thought it was the only way to maximize the likelihood of the observed data, assuming a Gaussian mixture model.

In the last days while studying for Dirichlet Process I've discovered the Gibbs Sampling that can fit parameters of a GMM.

Now I'm not familiar at all with the Gibbs sampling, and I would really appreciate if you could give me an overview of it, comparing it to EM. What is the basic idea behind it?

Gibbs sampling is a probabilistic framework which assumes all parameters to be learned are random variables. Here, you assume prior distributions for the parameters and find the full conditionals based on the relations specific to the problem, e.g. if you have three parameters, $\alpha,\beta,\theta$, you derive $p(\alpha|,\beta,\theta)$ ,$p(\beta|\alpha,\theta)$ and $p(\theta|\alpha,\beta)$. After deriving the full conditionals you sample from them and execute many iterations. It's like asking for $\alpha$ if we know $\beta$ and $\theta$, asking for $\beta$ when we know $\alpha$ and $\theta$ etc. After sufficient number of iterations you'll converge to a solution. In GMM, these parameters are mixture proportions, which can have a Dirichlet prior for example, and means and deviations of each gaussian component (you can choose suitable prior distributions for these). An example design would include full conditionals $p(\mu_i|\boldsymbol{\sigma},\boldsymbol{\pi})$, $p(\sigma_i|\boldsymbol{\mu},\boldsymbol{\sigma})$ and $p(\boldsymbol{\pi}|\boldsymbol{\mu},\boldsymbol{\sigma})$. Here choice of your priors are important because when you get formulas for these full conditionals it's better to have common distributions for easy sampling, e.g. $p(\mu_i|..)$ turns out to be gamma distributed etc. If not, you have to design samplers for these probability distributions, which can sometimes cause some pain. It is also an iterative algorithm as EM, but EM is purely an optimization procedure while Gibbs sampling is a Bayesian approach.


Multiclass semantic segmentation with some classes possibly not present in some of the images Let's assume we have a large annotated dataset with 4 classes. In this dataset, there might be annotated images with less than 4 classes, where the remaining classes might or might not be present. As an example, say we want to detect pedestrians/cars/bicycles/roads in images. In our dataset, there are some annotated images with only 3 classes: pedestrians/cars/bicycles, but this does not mean there are no roads in these images. That is, there might be roads in these images that were ignored by the annotator for some reason, or there might no be roads at all. My question is, how do we take this uncertainty into the loss term?
One option is to work with independent networks for each class. But what if we want to train a single network? how do we add something like a "don't care" term for objects not annotated in an image, but might still be present in the image?

That problem is commonly called weak labels and is associated with weak supervision.
Weak labels are often not handled by the loss function. Most conventional loss functions assume perfectly labeled data points and have no mechanism to handle label uncertainty.
An alternative framework for weak labels is Confident Learning which

characterizing and identifying label errors in datasets, based on the principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence.