text:
Is it expected that adding an additional hidden layer to my 3-layer ANN reduces accuracy significantly?
I've been using several resources to implement my own artificial neural network package in C++.
Among some of the resources I've been using are
https://www.anotsorandomwalk.com/backpropagation-example-with-numbers-step-by-step/
https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/
https://cs.stanford.edu/people/karpathy/convnetjs/intro.html,
as well as several others.
My code manages to replicate the results in the first two resources exactly. However, these are fairly simple networks in terms of depth. Hence the following (detailed) question:
For my implementation, I've been working with the MNIST Database of handwritten digits (http://yann.lecun.com/exdb/mnist/).
Using the ANN package I wrote, I have created a simple ANN with 784 input neurons, one hidden layer with 16 neurons, as well as an output layer with ten neurons. I have implemented ReLU on the hidden layer and the ouput layer, as well as a softmax on the output layer to get probabilities.The weights and biases are each individiually initialized to random values in the range [-1,1]
So the network is 784x16x10.
My backpropagation incorporates weight gradient and bias gradient logic.
With this configuration, I repeatedly get about a 90% hit rate with a total average cost of ~0.07 on the MNIST training set comprising 60,000 digits, and a slightly higher hit rate of ~92.5% on the test set comprising 10,000 digits.
For my first implementation of an ANN, I am pretty happy with that. However, my next thought was:
"If I add another hidden layer, I should get even better results...?".
So I created another artificial network with the same configuration, except for the addition of another hidden layer of 16 neurons, which I also run through a reLU. So this network is 784x16x16x10.
On this ANN, I get significantly worse results. The hit rate on the training set repeatedly comes out at ~45% with a total average error of ~0.35, and on the test set I also only get about 45%.
This leads me to either one or both of the following conclusions:
A) My implementation of the ANN in C++ is somehow faulty. If so, my bet would be it is somewhere in the backpropagation, as I am not 100% certain my weight gradient and bias gradient calculation is correct for any layers before the last hidden layer.
B) This is an expected effect. Something about adding another layer makes the ANN not suitable for this (digit classification) kind of problem.
Of course, A, B, or A and B could be true.
Could someone with more experience than me give me some input, especially on whether B) is true or not?
If B) is not true, then I know I have to look at my code again.

Network should learn fine with additional layer, but the problem might not be in the backpropagation code either. There are other details to consider. Are you learning for enough epochs? Deeper models generally need more time to train. Your initial weight range $[-1, 1]$ might be too large, you can try researching about initialization strategies (e.g. Xavier initialization). Are you shuffling your data while learning? There are probably some other things to consider that I didn't mention. You can try using available deep learning software (e.g. Tensorflow) to compare results.

Thanks Brale! I will have a look at those as well. I’m currently not shuffling my data, and I was also wondering about Xavier’s Weight Initialization. I’ll try and see if that improves results.

By the way,@Brale_, if you say B) is not true you could make your comment into an answer and I’ll accept it.

I tried using keras to compare the results and it's in a colab notebook @Chris hope I can help you. https://colab.research.google.com/drive/17kAJ2KJ36grG9sz-KW10fZCQW9i2Tf2c

You probably got the back propagation wrong. I have done a test on the accuracy on adding an extra layer and the accuracy went up from 94% to 96% for me. See this for details:
https://colab.research.google.com/drive/17kAJ2KJ36grG9sz-KW10fZCQW9i2Tf2c
To run the notebook click Open in playground and run the code. There is a commented line which add 1 extra layer. The syntax should be easy to understand even though it is in python.
For back propagation, you can try to see this python implementation of multi layer perceptron backpropagation.
https://github.com/enggen/Deep-Learning-Coursera/blob/master/Neural%20Networks%20and%20Deep%20Learning/Building%20your%20Deep%20Neural%20Network%20-%20Step%20by%20Step.ipynb
A network will not usually decrease it's accuracy by almost a half in normal scenario when you add an extra layer, though it is possible to have the network decrease accuracy when you add an extra layer due to overfitting. Though if this happen the performance drop won't be that dramatic.
Hope I can help you.

Thanks Clement! I’ve accepted this as you are the first to write an answer, and very detailed as well :) I’m looking into my backpropagation code now and will have a look at your code as well… Thanks

Also, I implemented Xavier’s weight initialization  and Kaiming’s weight initialization… And unfortunately neither fix the issue, so it really does seem to be the backpropagation causing trouble

Weight Initialization shouldn't be the cause of the problem, as normal random initialization will already be good, the Xavier initialization should only increase training speed.


text:
Can we calculate mean recall and precision
I'm evaluating the accuracy in detecting objects for my image data set using three deep learning algorithms. I have selected a sample of 30 images. To measure the accuracy, I manually count the number of objects in each image and then calculate recall and precision values for three algorithms. Following is a sample:

Finally to select the best model for my data set, can I calculate the mean Recall and mean Accuracy? For Example:

By precision do you mean IoU? Or any other else?

@Clement Hui : Precision is simply $\frac{TP}{PP} = \frac{TP}{TP+FN}$. I have counted manually the number of objects in each image. Then I obtain the count of objects by the deep learning model.

? What do you mean by TP PP and FN? Thanks

@Clement Hui: It is the number of true positives(TP) over the number of predicted positives (PP). (FN) is false negatives. Thanks.

Oh ok. So isn't calculating the accuracy and precision just counting teh number of all of these and plug it in the function? Can't you do it using a program? Thanks

For the precision metric for example you have:
$$
 Precision = \frac{TP}{TP+FP},
$$
with TP = True Positive and FP = False Positive.
Imagine you have the following values:
Image 1: $TP = 2, FP = 3$
Image 2: $TP = 1, FP = 4$
Image 3: $TP = 3, FP = 0$
The precision scores as you calculated will be:
Image 1: $2/5$
Image 2: $1/5$
Image 3: $1$
Your average will be: $0.533$
On the other hand if you sum them all up and then calculate the precision value you get:
$P = \frac{6}{6+7} = 0.462$
This proves that averaging the precision scores is not the same as calculating the total precision in one go.
Since what you want is to know how precise your algorithm is, independently of the precision for each image, you should sum all the TP and FP and only then calculate the precision for each model. This way you will not have a biased average. The average would give the same weight to an image with a larger number of objects as to another image which had fewer objects.

Thank you very much for the suggestion. I think that works well than taking the mean.

@NilaniAlgiriyage You should also be careful when deciding what metric you are going for. It may depend on how your data-set is distributed (from perfectly balanced to largely imbalanced). Besides, the task at hand and target for the client may differ. F1-score, MCC (Matthews correlation coefficient for binary classification) and ROC (for comparing models) are the ones you need to consider.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Given enough graphical data, could you train an AI to plot a polynomial graph based on the input conditions?
Good day everyone.
I am curious if it is possible for an AI to plot a time-series graph based on a single input.
Using free fall impact as an example.
Assuming we drop a ball from height 100m and record the force it receives relative to time.
We will get a graph that looks something like below.

Now we drop the ball from a height of 120m, record the forces, and we get another graph in addition to our original.

What I am wondering is:
If we have a large set of data on 60m to 140m (20m interval) height drops, would we be able to generate a regression model that plots the responses when given an arbitrary drop height? (i.e plot force response when dropped from 105m)
Thank you all very much for your time and attention.

Yes this is possible, using any machine learning approach that supports regression. You have two main approaches:

Input $h$ the height of the drop, multiple outputs, one per time offset that you want to plot. Each individual output calculates the predicted force at a specific offset time.
Inputs $h$ the height of the drop and $t$ a time offset, one output. The single output calculates the predicted force due to given height and at given time.

The main thing to bear in mind is that statistical learning techniques typically do not generate physics-like models. Test inputs close to training examples should generate reasonable graphs that interpolate between those from training data. Test inputs far away from the training examples (e.g. you train on data of drops from 60m to 140m, but use an input of 10m or 200m) will likely generate wildly incorrect outputs. The main exception to this is if your ML model includes some good guesses at the underlying physics model, in which case it is possible that a regression algorithm will tune the parameters of that model plus filter out terms that should not be part of the model, resulting in a system that extrapolates much better. That is very unlikely happen by chance, it requires up-front design.

Hi there Neil, appreciate you taking the time to respond to this.
I think the second approach you stated is just what I need.
I will thread this modelling with caution based on the concern you raised on physics like models.
Thank you very much.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Optimal RL function approximation for TicTacToe game
I modeled the TicTacToe game as a RL problem - with an environment and an agent.
At first I made an "Exact" agent - using the SARSA algorithm, I saved every unique state, and chose the best (available) action given that state. I made 2 agents learn by competing against each other.
The agents learned fast - it took only 30k games for them to reach a tie stand-off. And the agent clearly knew how to play the game.
I then tried to use function approximation instead of saving the exact state. My function was a FF-NN. My 1st (working) architecture was 9 (inputs) x 36 x 36 x 9 (actions). I used the semi-gradient 1-step SARSA. The agents took much longer time to learn. After about 50k games they were still less good than the exact agent. I then made a stand off between the Exact and the NN agent - the exact agent won 1721 games from 10k, the rest were tied. Which is not bad.
I then tried reducing the number of units in the hidden layers to 12, but didn't get good results (even after playing for 500k+ games total, tweaking stuff). I also tried playing with convolution architectures, again - not getting anywhere.
I am wondering if there's some optimal function approximation solution that can get as-good of results as the exact agent. TicTacToe doesn't seem like such a hard problem for me.
Conceptually I think there should be much less complexity involved in solving it then can be expressed in a 9x36x36x9 network. Am I wrong, and it's just an illusion of simplicity? Or are there better ways? Maybe modeling the problem differently?

I think you can break this problem down into two parts to try and find the solution.
1. Can the neural network model the desired function?
Take the tabular function you have learned in the exact agent, and treat it as training data for the neural network model, using the same loss function and other hyperparameters as you intend to use when the NN is being used online inside the RL inner loops.
You can answer two related questions with this:

Does the loss reduce down to a low value after a suitable number of epochs? If so, then the NN has capacity to learn and can learn fast enough. If not, you need to look at the hyperparameters of the NN.
Does the trained NN play well against the exact agent? Ideally it plays the same, but it is possible that even though the loss is low, one or two key values in the function are compromised, meaning it still loses. I am not entirely sure what you would do in this case, but either try changing the hyperparameters to increase the capacity of the NN, or try augmenting the data so that there are more examples of the "difficult" action values to learn, to see if the issue is something that can be solved in learning.

Probably you will find your NN architecture is good, or only requires minor changes to become useful. The more likely issues are in next section.
2. Is the RL framework set up correctly for function approximation?
It is quite hard to get this right. Bootstrapping value based methods can easily become unstable if converted naively from tabular to function approximation approaches. Some variants are moderately stable - most stable would probably be Monte Carlo approaches.
If you don't want to use Monte Carlo control, then the answer here would be to take ideas from DQN used originally to play Atari games:

Don't learn online. Store transitions in an experience replay table - store $(s, a, r, s', done)$ tuples where $done$ is true if $s'$ is a terminal state - and sample a minibatch from it on every step. Reconstruct the bootstrap estimates of value functions to train from each time you sample, don't store and re-use the estimate from the time the action was taken.
Optionally use two value estimators - the current learning one, used to select plays and which is updated on each step, and a "target" one used to calculate TD targets. Update the target network by cloning the learning network every N steps (e.g. every 100 games).
To avoid figuring out hyperparameters for SARSA epsilon decay, I suggest use one-step Q learning. Also one issue you may be facing with SARSA combined epsilon decay is "catastrophic forgetting" where the agent gets good, starts to train itself only on play examples by good players, and forgets the values of states that it has not seen in the training data for many time steps. With Q learning you can avoid that by having a relatively high minimum epsilon e.g. 0.1.

In fact with TicTacToe learning through self-play, you can get away with $\epsilon = 1$ and Q learning should still work - i.e. it can learn optimal play by observing random self-play. This should apply equally tabular and function approximation agents. It doesn't scale to more complex games where random play would take too long to discover optimal strategies.

This is a good methodology in case you have access to an exact solution. I wonder how can you "debug" your system if you don't?

@DavidRefaeli: For the first part, you don't need the full exact solution for a rough test of NN capacity. A large enough variety of end and near-end states with different vaklues would also do, or even something faked using Monte Carlo play with random agents - the test is not necessarily whether you can represent the optimal function (although that is your end goal), but that the NN function can differentiate between action values that should be different under any policy


text:
Interpretation of feature selection based on the model
The description of feature selection based on a random forest uses trees without pruning.
Do I need to use tree pruning?
The thing is, if I don't cut the trees, the forest will retrain.
Below in the picture is the importance of features based on 500 trees without pruning.

With a depth of 3.

I always use the last four signs 27, 28, 29, 30.
And I try to add to them signs from 0 to 26 by means of cycles, going through possible combinations.
Empirically, I assume that the trait number 0, 26 is significant.
But, on both pictures it is not visible. Although the quality of classification with the addition of 0, 26 has improved.

random forest's feature importances are not reliable and you should probably avoid them. Instead you can use permutation_importance: https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance.html#sphx-glr-auto-examples-inspection-plot-permutation-importance-py

"random forest's feature importances are not reliable and you should probably avoid them" is a very strong statement which does not hold. While random forest variable importance estimates are known to be biased under certain conditions (e.g. gini impurity-based estimates on a mix of continuous and categorical variables), they can provide relatively good estimates in other situations.

@adrin thanks. I want to get the features values in "permutation_importance".
As in "importances = clf.feature_importances_", to build a graph .
If I request"result.importances", I get a lot of items on each index.
It turns out to get only indexes. Which function should I turn to?


text:
Q-learning problem wrong policy
I am coding out a simple 4x4 grid game whereby the agent starts at a particular state and his aim is to reach the terminal state. The agent is supposed to avoid traps along the way and reach the end goal with high reward. The below picture illustrates the environment.

The code that I am running is shown below:
# 4x4 Grid
import random

gamma = 1
grid = [[-0.1 for i in range(4)] for j in range(4)]
episodes = 500000
epsilon = 1 # start greedy
decay = 0.999
min_epsilon = 0.1
alpha = 0.65
# set terminal states
grid[1][0] = -1
grid[2][2] = -1
grid[0][3] = 1

# Set up Q tables
# 0: up, 1: down, 2: left, 3: right
# Q = {(0,0): {0: z, 1: x, 2: c, 3: v}, ... }}
Q = {}

# 4 rows
for row in range(4):
    # 4 columns
    for column in range(4):
        Q[(row,column)] = {}
        # 4 actions
        for k in range(4):
            Q[(row,column)][k] = 0

def isTerminal(state):
    if state == (1,0) or state == (2,2) or state == (0,3):
        return True
    return False

def get_next_state_reward(state, action):

    row = state[0]
    col = state[1]
    #print(row, col)
    if action == 0: # up
        # out of grid
        if (row - 1) < 0:
            return (state, grid[row][col])

    if action == 1: # down

        # out of grid
        if (row + 1) > len(grid) - 1:
            return (state, grid[row][col])

    if action == 2: # left

        if (col - 1 < 0):
            return (state, grid[row][col])

    if action == 3: # right

        if (col + 1 > len(grid[row]) - 1):
            return (state, grid[row][col])

    if action == 0:

        row -= 1
        return ((row,col), grid[row][col])

    if action == 1:

        row += 1
        return ((row,col), grid[row][col])

    if action == 2:

        col -= 1
        return ((row,col), grid[row][col])

    if action == 3:

        col += 1
        return ((row,col), grid[row][col])

state_visit = {}
for row in range(4):
    # 4 columns
    for column in range(4):
        state_visit[(row,column)] = 0

for episode in range(episodes):

    # let agent start at start state
    state = (3,0)

    while not isTerminal(state):

        r = random.uniform(0,1)

        if r < epsilon:
            action = random.randint(0,3)
        else:
            action = max(Q[state], key=lambda key: Q[state][key])

        next_state, reward = get_next_state_reward(state, action)

        TD_error = reward + gamma * max(Q[next_state]) - Q[state][action]

        Q[state][action] = Q[state][action] + alpha * TD_error

        state = next_state

        state_visit[next_state] += 1
        epsilon = max(min_epsilon, epsilon*decay)

        #input()

policy = {}
# get optimal policies for each state
for states in Q:
    policy[states] = max(Q[states], key=lambda key: Q[states][key])

When I finish running the algorithm however, I am unable to achieve the optimal policy no matter how many tweaks I do to the number of episodes, or epsilon decay, or the alpha value.
Particularly, the Q values that I attain for state (2,0), (0,1) and (0,0) have Q values that are equal values for three directions except for the last direction which brings the agent to the terminal state.
For example, these are the Q-values that I get for state (0,0), (0,1) and (2,0) respectively.
(0,0): {0: 2.0, 1: 2.9, 2: 2.9, 3: 2.9}
(0,1): {0: 2.9, 1: 2.0, 2: 2.9, 3: 2.9}
(2,0): {0: 2.9, 1: 2.9, 2: 2.9, 3: 2.9}
I am not sure why the Q-values for the 3 directions should be the same because each extra step that the agent takes incurs a negative reward.
Would anyone be able to help ? Thank you so much !

what does max(Q[next_state]) do ?. Q[next_state] returns a dict with actions as keys and Q values as values. So you are taking max action (3) every time instead of a Q value ?.

yes, Q[next_state] returns a dictionary that maps each action to it's q value for that state. I am taking the highest possible Q value for the next state in the update equation. Why does it return action (3) ?

max considers keys of dict not values. You are taking max action, it returns number 3 every time instead of a Q value.

You have a simple mistake in your TD Error function:
TD_error = reward + gamma * max(Q[next_state]) - Q[state][action]

You have made Q[next_state] a Python dict, so this will take the maximum key which is 3 for all your Q table entries. This is why you end up with values very close to 3 at the end, which is impossible for your problem, the maximum return will be 1.0 when stepping from adjacent grid points to the positive terminal state.
The correct code is:
TD_error = reward + gamma * max(Q[next_state].values()) - Q[state][action]

Alternatively, since your actions are all numeric, you could use a list instead of a dict in your Q table.


text:
How to track performance of your model during experimenting?
During weeks and months of your work, many things may change, for example :

You may modify the loss function
Your training or validation datasets may change
You modify data augmentation

Which tools or processes do you use to track modifications you have made and how did they affected the model ?

Maybe you are looking for a combination of a version control system (like git and Github) and a tool like comet.ml. In the past, I used comet.ml to keep track of different experiments performed with different hyper-parameters or different versions of the code. There are other alternatives to comet.ml, such as sacred, but they may also have different features and may not be as visually pleasing as comet.ml or even free. Personally, I liked comet.ml (even though, at the time, it still lacked some features). In any case, a VCS, like git, is widely used in software development (not just in AI projects) to keep track of different versions of the code, etc. You may also be interested in continuous integration (e.g. Travis CI) and code review (e.g. codacy) tools.

Full disclosure: I work at Dessa, the company that developed this tech.
We built a machine learning experiment management tool, called Atlas. The main feature is experiment management, allowing you to run thousands of experiments concurrently. This might help with your problem above https://github.com/dessa-oss

Hi and welcome to AI SE! Thanks for sharing this info! Atlas looks promising and similar to comet.ml. What about prices? Is it free to use? Maybe you should edit your answer to describe these details.


text:
Is the temperature equal to epsilon in Reinforcement Learning?
This is a piece of code from my homework.
# action policy: implements epsilon greedy and softmax
def select_action(self, state, epsilon):
    qval = self.qtable[state]
    prob = []
    if (self.softmax):
        # use Softmax distribution
        prob = sp.softmax(qval / epsilon)
        #print(prob)
    else:
        # assign equal value to all actions
        prob = np.ones(self.actions) * epsilon / (self.actions -1)
        # the best action is taken with probability 1 - epsilon
        prob[np.argmax(qval)] = 1 - epsilon
    return np.random.choice(range(0, self.actions), p = prob)

This is a method in order to select the best action according to the two polices i think. My question is, why in the softmax computation there is the epsilon parameter used as temperature. Is really the same thing? Are they different? I think they should be two different variables. Should the temperature be a fixed value over time? Because when i use the epsilon-greedy policy my epsilon decrease over time.

Your are correct that epsilon in epsilon-greedy and temperature parameter in the "softmax distribution" are different parameters, although they serve a similar purpose. The original author of the code has taken a small liberty with variable names in the select_action method in order to use just one simple name as a positional argument.

Should the temperature be a fixed value over time?

Not necessarily, if your goal is to converge on an optimal policy you will want to decrease temperature. A slow decay factor applied after each update or episode, as you might use for epsilon (e.g. 0.999 or other value close to 1), can also work for temperature decay. A very high temperature is roughly equivalent to epsilon of 1. As temperature becomes lower, differences in action value estimates become major differences in action selection probabilities, with a sometimes desirable effect of picking "promising" action choices more often, plus with very low probabilities of picking the actions with the worst estimates.
Using a more sophisticated action selection such as the temperature based on in the example code can speed learning in RL. However, this particular approach is only good in some cases - it is a bit fiddly to tune, and can simply not work at all.
The tricky part of using a temperature parameter is choosing good starting point, as well as the decay rate and end values (you have to do the last for epsilon-decay as well). The problem is that the impact of using this distribution depends on the actual differences between action choices. You need a temperature value on roughly the same scale as the Q value differences. This is difficult to figure out in advance. In addition, if the Q value differences are more pronounced in some states than in others, you risk either having next to no exploration or having too much in some parts of the problem.


text:
Autoencoder network for feature selection not converging
I am training an undercomplete autoencoder network for feature selection. I am using one hidden layer in the encoder and decoder networks each. The ELU activation function is used for each layer. For optimization, I am using the ADAM optimizer. Moreover, to improve convergence, I have also introduced learning rate decay. The model shows good convergence initially, but later starts to generate losses (12 digits) in the same range of values for several epochs, and is not converging. How to solve this issue?

The trick was to normalize the input dataset values with the respective mean and standard deviation in each column. This reduced the loss drastically, and my network is training more efficiently now. Moreover, normalizing the data also helps you calculate the weights associated with each input node more easily, especially when trying to find out variable importance.


text:
Why is my model accuracy high in train-test split but actually worse than chance in validation set?
I have trained a XGboost model to predict survival for the Kaggle Titanic ML competition.
As with all Kaggle competitions there is a train dataset with the target variable included and a test dataset without the target variable which is used by Kaggle to compute the final accuracy score that determines your leaderboard ranking.
My problem:
I have build a fairly simple ensemble classifier (based on XGboost) and evaluated it via standard train-test-splits of the train data. The accuracy I get from this validation is ~80% which is good but not amazing by public leaderboard standards (excluding the 100% cheaters).
The results and all the KPIs I looked at of this standard model do not indicate severe overfitting, etc. to me.
However when I submit my predictions for the test set my public score is ~35% which is way below even a random chance model. It is sooo bad I even improved my score by simply reversing all predictions from the model.
Why is my model so much worse on the test?
I know that Kaggle computes their scores a bit differently than I do locally, additionally there is probably some differences between the datasets. Most who join the competition notices at least some difference between their local test scores and the public scores.
However my difference is really drastic and indeed reversing the predictions improves my score. This does not make sense to me because reversing the predictions on my local validations leads to garbage predictions, so this is not a simple problem of generally reversed predictions.
So can you help me understand how those two issues happen at the same time:

Drastic difference between local accuracy and public score
Reversing actually leads to the better public score.

Here is my notebook for the code (please ignore the errors, they are simply because the code does not work on kaggle kernels only locally):
https://www.kaggle.com/fnguyen/titanicrising-test

Looking at your code, one set of data transformations were applied to the train data and a different set of transformations were applied to the test data. Different data transformations could account for different evaluation metric performance.
It is best practices to put all data transformations in a function so they can be applied to all data in a similar way.
Since you are using scikit-learn, sklearn.compose.ColumnTransformer is designed for this purpose. Example code for the Titanic dataset is here.

I will try that but functionally the data transformation for both datasets is the same with the only difference being that the test data does not contain a "Ms" at factor level. Nevertheless you never know what makes a quirk in the data so I will package all transformations in a function and apply the same to both sets to eliminate this factor.

I have cleaned up my code using your answer and a notebook I found on Kaggle as inspiration. By joining test and train dataset first, then doing all necessary transformations and only after splitting them again for modeling and prediction I was able to remove the error. They key to his was actually that I was applying an Imputation Method, I will describe that in an answer in detail.

@BrianSpiering was generally  correct in pointing out that you should always apply the same transformations to your train and test dataset.
This was the key to my solution which was a bit more specific but might actually help others who encounter a similar problem.
Specifically my mistake came about because of Imputation! Some of the factors I used for my model were NA in both the train and test data set. To complete the data I simply imputed these missings using mean and mode respectively. However since I did those transformations separately on both sets the actual mean/mode value that was used differed heavily! By applying the imputation on the full data I also imputed the same data for all missing cases which solved my error.
My resulting accuracy in the public leaderboard is now at 74.2% which is fairly close to my local test score of 79.6%.


text:
Why is Standard Deviation based on L2 Variance and not L1 Variance
Standard deviation and variance are in statistics but the formula for variance is somehow related to the L1 and L2.
Mathematically (L2 in machine learning sense),
$$Variance = \dfrac{(X_1-Mean)^2+..+(X_n-Mean)^2}{N}$$
and,
$$Standard\ Deviation= \sqrt(Variance)$$
Why shouldn't it be (L1 in machine learning sense):
$$Variance = \dfrac{|X_1-Mean|+..+|X_n-Mean|}{N}$$
and,
$$Standard\ Deviation= Variance$$

It's a little bit unclear what you're asking. Do L1 and L2 in your answer refer to the L1 and respectively L2 norms? Please, edit your post to clarify what you're asking and don't just write some formulas.

I've found the answer, the L2 is Standard Deviation, and the L1 is Mean Deviation. Standard deviation describes the variation better and the values are always different on different sets of X while the Mean Deviation gives the same values sometimes.

*Footnote: Why square the differences? If we just add up the differences from the mean ... the negatives cancel the positives:
standard deviation why a       (4 + 4 − 4 − 4) / 4  = 0 So that won't work.
  How about we use absolute values?
standard deviation why a       (|4| + |4| + |−4| + |−4|) / 4  =  (4 + 4 + 4 +
  4) / 4 = 4 That looks good (and is the Mean Deviation), but what about
  this case:
standard deviation why b       (|7| + |1| + |−6| + |−2|) / 4  =  (7 + 1 + 6 +
  2) / 4 = 4 Oh No! It also gives a value of 4, Even though the differences
  are more spread out.
So let us try squaring each difference (and taking the square root at
  the end):
standard deviation why a      √( (42 + 42 + 42 + 42) / 4 ) = √( 644) = 4
  standard deviation why b      √( (72 + 12 + 62 + 22) / 4 ) = √( 904) = 4.74...

Reference:
https://www.mathsisfun.com/data/standard-deviation.html

You can use MathJax to format your answer.


text:
Are PAC learnability and the No Free Lunch theorem contradictory?
I am reading the Understanding Machine Learning book by Shalev-Shwartz and Ben-David and based on the definitions of PAC learnability and No Free Lunch Theorem, and my understanding of them it seems like they contradict themselves. I know this is not the case and I am wrong, but I just don't know what I am missing here.
So, a hypothesis class is (agnostic) PAC learnable if there exists a learner A and a function $m_{H}$ s.t. for every $\epsilon,\delta \in (0,1)$ and for every distribution $D$ over $X \times Y$, if $m \geq m_{H}$ the learner can return a hypothesis $h$, with a probability of at least $1 - \delta$
$$ L_{D}(h) \leq min_{h'\in H} L_{D}(h') + \epsilon $$
But, in layman's terms, the NFL theorem states that for prediction tasks, for every learner there exists a distribution on which the learner fails.
There needs to exists a learner that is successful (defined above) for every distribution $D$ over $X \times Y$ for a hypothesis to be PAC learnable, but according to NFL there exists a distribution for which the learner will fail, aren't these theorems contradicting themselves?
What am I missing or misinterpreting here?

PAC learnability is about learning approximately and probably. In fact, PAC stands for probably approximately correct. It doesn't (necessarily) guarantee that you will learn the correct hypothesis with probability 1. I am not familiar with the details of the NFL theorem(s), so I cannot say more now, but a proper answer to your question probably includes this information I've just mentioned.

I do take that into consideration. The NFL theorem also defines the failure of a learner in terms of a probability $(1- \delta)$ and an error threshold $\epsilon$

There is no contradiction.
First, agnostic PAC learnable doesn't mean that the there is a good hypothesis in the hypothesis class; it just means that there is an algorithm that can probably approximately do as well as the best hypothesis in the hypothesis class.
Also, these NFL theorems have specific mathematical statements, and hypothesis classes for which they apply are often not the same as the hypothesis class for which PAC-learnability holds. For example, in Understanding Machine Learning  by Shalev-Shwartz and Ben-David, a hypothesis class is agnostic PAC learnable if and only if has finite VC dimension (Theorem 6.7). Here, the algorithm is ERM.
On the other hand, the application of the specific version of NFL that this book uses has Corollary 5.2, that the hypothesis class of all classifiers is not PAC learnable, and note that this hypothesis class has infinite VC dimension, so the Fundamental Theorem of PAC learning does not apply.
The main takeaway is that in order to learn, we need some sort of inductive bias (prior information). This can be seen in the form of measuring the complexity of the hypothesis class or using other tools in learning theory.

There is no contradiction between PAC learning and the no-free-lunch theorem as commented in other answers.
But there is indeed a contradiction between the no-free-lunch theorem and its layman's explanation:

for infinite $\mathcal{X}$, whenever $\mathcal A$ is fixed, there is a distribution on which it fails to learn.

This is not true!
Indeed, if $\mathcal X$ is countable and $\mathcal A$ is an algorithm that simply memorizes what it has seen and answers $0$ for unseen samples, then it can be shown that the true error of $\mathcal A$ converges to $0$, so $\mathcal A$ effectively learns. (See page 95)
For uncountable $\mathcal X$, the layman's explanation is correct. In that case, the theorem looks rather obvious, because finite samples provide no information about the behavior of the true mapping $f$, except at a null set.
I personally dislike that part of the book (page 61), because immediately before the no-free-lunch theorem, they say literally "no learner can succeed on all learning task, as formalized in the following theorem:", but they leave the distribution $\mathcal D$ as dependent on $m$ in the theorem statement, which disrupts all the previous definitions of learnability and makes the introductory phrase (the layman's terms) misleading.

PAC-learning is a definition and imposes a condition ("if, for all distribution, and other conditions are met then...") that, if met, then the class is PAC-learnable.
NFL Theorem is a result: There is always a distribution that makes the learner not learn.
The main observation: the statement of the NFL Theorem does not specify which class $\mathcal{H}$ the learner is using, so it manages to prove that the learner fail miserably.
Conclusion: if $\mathcal{H}$ is not restricted in some way, then the NFL theorem is valid. However, if we restrict $\mathcal{H}$, then - maybe - the conditions of the PAC-learning definition will be met.
For example, some successful learners (as presented in the "Understanding Machine Learning" book, right after the NFL statement):

ERM with $\mathcal{H} = \{f\}$
ERM with $\mathcal{H}$ finite that contains $f$ and whose size satisfies $m \ge 8 \log(7|\mathcal{H}|/6)$ (Corollary 2.3)

These learners respect item (1) of the NFL theorem statement ("exists $f: \mathcal{X} \rightarrow \{0,1\}$ with $L_{\mathcal{D}}(f)=0$") because they restrict $\mathcal{H}$.


text:
Trained a regression network and getting EXACT same result on validation set, on every epoch
I trained this network from this github.
The training went well, and returns nice results for new, unseen images.
On training, the loss changed (decreased), thus I must assume the weights changed as well.
On training, I saved a snapshot of the net every epoch.
When trying to run a validation set through each epoch's snapshot, I get the exact same results on every epoch.
How can this be possible? What's causing this?

The answer is I was too tired to see I was using the same model for validation every epoch, instead of that epoch's model.
Write good code, and this won't happen to you.


text:
Choosing Data Augmentation smartly for different application
I'm trying to understand the role of data augmentation and how it can affect the performance/accuracy of a deep model. My target application is fire detection (on video frames), with almost 15K positive and negative samples, and I was using the following data augmentation techniques. Does using ALL the followings always increase the performance? Or we have to choose them somehow smartly given our target application?
rotation_range=20, width_shift_range=0.2, height_shift_range=0.2,zoom_range=0.2, horizontal_flip=True

When I think a bit more, fire is always straight up, so I think rotation or shift might in fact worsen the results, given that it makes the image sides stretch like this, which is irrelevant to fires in video frames. Same with rotation. So I think maybe I should only keep zoom_range=0.2, horizontal_flip=True and remove the first three. Because I see some false positives when we have a scene transition effect in videos.
Is my argument correct? Should I keep them or remove them?

In fact, choosing smartly the values of the image augmentation can help the performance of your system. Where I work we developed an object detector for cars. We had the following image augmentation parameters:

Apect ratio distorsion: it changed the cars dimensions
Additive noise: it blurred the image
Change colorspace: change the cars colors
Saturation change: brightness
Rotation: rotate image axis
Flipping (horizontally): mirror effect
Zooming: zoom in/out of image

At first we just combined all the effects (within a certain range of change per effect) randomly in each augmented image. But then we did an study for our use case and we found that our model performed better with: no aspect ratio deformation and no change in color space. After giving it a bit of a thought it really made sense for our use case because it is a detector for a very specific type of car.
However, although the improvements accomplished by optimizing the parameters of image augmentation boosted the performance of the model, the improvements were not huge, so in the end it did not really worth the effort. To give some numbers it improved $5\%$ in our custom test metrics, and we were expecting something around $10\%$ so from that moment on we left the specific image augmentation parameters but stopped thinking too much about it.
So, in conclussion, does it helps? yes, how much? well, that would depend on the use case. For us, it turned out it did not improve as much as we wanted.
Hope this helps :)

Thanks. Useful thoughts. So after each removal, your trained the model and tested it? What metric do you look to get the best model? Is that val_loss?

Yes, exactly that, we trained the model several times with different parameters. That's how we found the best ones. That is also why we lost so much time and we think it did not worth the effort. The custom metric was an aggregated metric of precission and accuracy for our use case. So after train the model we run some tests on top.


text:
In a neural network, can colors be used for neurons in place of floating points and would there be any benefit in doing so?
Firstly, some context. I have been reading and watching videos on the subject for around 3 years, but I am still very much a beginner in machine learning and artificial intelligence. That said, I might not know what I'm even talking about here. So bear with me.
If I understand correctly, each node in a neural network (neuron) is represented by some floating point number between 0 and 1, that are arranged in layers and have corresponding weights. Right? While a color has RGB values, CMYK values, and HSV values that are all interrelated to each other.
My question is would there be any benefit to having each node represented by a color instead of a single floating point number?
My thinking is that each neuron could select any of the values (r, g, b, c, m, y, k, h, s, or v) contained within the color in some meaningful way, while the Alpha value could possibly represent the weight associated with that neuron.
Thoughts? Would it not work like that? Could you use it to have multiple congruent networks running on 3 different channels? Again, would there be any benefit to doing this than just using a single number? Or would it over-complicate (or even break) the network? Would it be useless?
Although I've also dabbled in Unity3D (which is how I got the idea in the first place), I'm too much of a beginner to know how to even begin an attempt at testing this myself.

As you said, a color is a number (through RGB values). So you want to replace the weight of a neuron, a number, by a number ? It's not going to change anything.
I think your confusion is from how computer and human see colors. For a computer, a color is just a number (or more exactly 3 numbers, R, G, B). The computer just turn on the right pixels based on these numbers for us human to see the color. But it's still just a number. So your proposition change nothing on how neural networks work.

@Astariul Colors being numbers is exactly why I thought they could be used. What would be interesting is being able to "transform" and/or "mix" parallel NNs using various selectors (hue, cyan, red, alpha, etc.) to propagate to the next layer.

@Astariul In other words, say you could have 3 instances running on each channel of your Color NN. For each neuron, you could simply select the same channel and functionality would be no different. However, you could take the hue (for example) of the 3 rgb channels to propagate a new value to the next layer's neurons. Make sense?

@DuttaA I agree. At the very least, this could make for a useful way to represent your NNs visually.

I am currently working on writing a ColorNeuron class in Unity3D I can use to begin building a model/simulation. I'll let you know what I can come up with.

To answer your question, I'll just simplify it and assume you are representing the activation of a neuron (the value it produces) by an RGB value. So a tuple with 3 values ranging from 0 to 1.
It's important to remember that every single machine learning model could be computed by a human on pen and paper using just numbers. So keeping that in mind, this neural network would essentially just be a combination of multiplying tuples of size 3 in a certain order (ignoring activation functions).
I will also assume that each RGB value has it's own unique set of incoming weights, like following:
this essentially amounts to just have 3 unique neurons. There's no difference:

All I did there was get rid of the circle. The calculations are no different. If you changed the calculation, say values are shared between the 3 RGB nodes, then you now just have 3 duplicate nodes. In all cases, you can just represent the equivalent calculation as a standard neural network.
However, in the same vein, in a convolutional neural network, if you create conv layers with an output of 3 channels, you can visualise their outputs as a colour image (assuming you are using an activation function that outputs numbers between 0 and 1), which can be cool to look at.

Thank you! This is what I was thinking. Yes, I understand that when strictly using rgb you get the same functionality. I also understand that using something like cyan could simply be written into your network, taking the "blue neuron" and "green neuron" and running their values through the function to get a "cyan neuron". What I'm suggesting is that having these functions built-in might be useful.
And, yes, I also thought this could be useful for visualization as well.


text:
Binarize ConvNet Feature vector
Given a pre-trained CNN model, I extract feature vector of 3450 reference images FV_R as follows:
FV_R = [       [-8.2, -52.2, 9.07, -1.1, -0.08, -9.1, ........, -4.11],
               [7.8, -3.8, 6.4, -4.27, -2.2, -5.0, ............., 3.6],
               [-1.2, -0.8, 49.3, 1.73, -1.74, -7.1, ..........., 2.41],
               [-1.2, -.8, 49.3, 0.6, -1.24, -1.04, .........., -2.06],
               .
               .
               .
               [-1.2, -.8, 49.3, 12.77. -2.2, -5.0, .........., -51.1]
       ]

and FV_Q for 1200 query images :
FV_Q = [       [-0.13, 2.6, -3.7, -0.5, -1.02, -0.6, ........, -0.11],
               [0.3, -3.8, 6.4, -1.6, -2.2, -5.0, ............., 0.97],
               [-6.4, -0.08, 8.0, 7.3, -8.07, -5.6, ..........., 0.01],
               [-6.09, -.8, 0.5, -8.9, -0.74, -0.08, .........., -8.9],
               .
               .
               .
               [-1.2, -.8, 49.3, 12.77. -2.2, -5.0, .........., -51.1]
       ]

The size info:
>>> FV_R.shape
(3450, 64896)

Query images:
>>> FV_Q.shape
(1200, 64896)

I would like to binarize the CNN feature vectors (descriptors) and calculate Hamming Distance. I am already aware of this answer to probably use np.count_nonzero(a!=b)(if a.shape == b.shape) but does anyone know a  method to binarize a feature vector with different size?
Cheers,

Programming issues are off-topic here. Please, read https://ai.stackexchange.com/help/on-topic for more details and to understand which questions are on-topic here. If you have a programming issue, the best place to ask your question is Stack Overflow or, alternatively, Data Science SE.

I fixed the problem as follows:
import numpy as np

def binarize(FV):
    return np.where(FV > 0, 1, 0).astype(int)

def Hamming_Distance(ref, query):
    b_rf, b_qu = binarize(ref), binarize(query)
    H = np.count_nonzero(b_qu[:, None, :] != b_rf, axis=2)
    return H

If you mean to binarize the vector such that all positive values become 1 and the rest becoming 0, then you can do this.
bin_arr = np.zeros_like(FV_R)
bin_arr[FV_R > 0] = 1.

As an example,
In [7]: arr
Out[7]:
array([[ 0.15, -0.52],
       [ 1.  , -0.43]])

In [8]: bin_arr = np.zeros_like(arr)

In [9]: bin_arr[arr > 0] = 1

In [10]: bin_arr
Out[10]:
array([[1., 0.],
       [1., 0.]])
```


text:
Is there a good ratio between the positive and negative rewards in reinforcement learning?
Is there an ideal ratio in reinforcement learning between the positive and negative rewards?
Suppose I have the scenario of moving a robot across the river. There are two options, walk across the bridge or walk across the river. If it walks across the river then the robot breaks so the idea is to reinforce the robot to walk across the bridge. What would be the best rewards values? Does this ratio vary between cases?
option1:

Bridge: +10
River: -10

Option2:

Bridge: +10
River: -1

Option3:

Bridge: +1
River: -10

There are no hard and fast rules for it. Your reward should be such that it motivates the agent to attain the goal in the most effective way.  In a grid world if you want your agent to reach goal state quicker but award +2 for a move and +5 for reaching goal then your agent might simply wander around and never reach the goal. However, if you set a reward of -1 for each move and  +1 or +10(or even 0) for reaching the goal then your agent will learn to reach the goal state faster.

It usually does not matter, but I'm sure there are situations where it could matter.  In theory, if a reward for good behavior is higher than the rewards for bad behavior, then the neural network will be trained such that the higher rewards are preferred, even if those higher rewards are negative.  For example, if a bad reward is -100, then a relatively good reward could be -50, and the network will then be more likely to choose the action with -50 reward over the action with -100 reward.


text:
Is the negative of the policy loss function in a simple policy gradient algorithm an estimator of expected returns?
Let
$$
\nabla_\theta J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau) \right]
$$
be the expanded expression for a simple policy gradient, where $\theta$ are the parameters of the policy $\pi$, $J$ denotes the expected return function, $\tau$ is a trajectory of states and actions, $t$ is a timestep index, and $R$ gives the sum of rewards for a trajectory.
Let $\mathcal{D}$ be the set of all trajectories used for training. An estimator of the above policy gradient is given by
$$
\hat{g} = \frac{1}{\mathcal{D}} \sum_{\tau \in \mathcal{D}} \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau).
$$
A loss function associated with this estimator, given a single trajectory with $T$ timesteps, is given by
$$
L(\tau) = -\sum_{t = 0}^T \log \pi_\theta (a_t|s_t) R(\tau).
$$
Minimizing $L(\tau)$ by SGD or a similar algorithm will result in a working policy gradient implementation.
My question is what is the proper terminology for this loss function? Is it an (unbiased?) estimator for the expected returns $J(\pi_\theta)$ if summed over all trajectories? If someone is able to provide a proof that minimizing $L$ maximizes $J(\pi_\theta)$, or point me to a reference for this, that would be greatly appreciated.

If I understand your question correctly, you are wondering whether the
policy gradient objective coincides with some real measure of progress.
This is exactly what the Policy Gradient Theorem proves (see Sutton et
al. (2000) or Sutton and Barto (2018), chapter 13). In particular,
policy gradient methods optimize the value of the start state $s_0$ under the
current policy, $v_\pi(s_0)$. Since this value is defined as an
expectation over returns, then your conclusion is correct.
Sutton, Richard S., and Andrew G. Barto. 2018. Reinforcement Learning -
an Introduction. Adaptive Computation and Machine Learning. MIT Press.
http://www.worldcat.org/oclc/37293240.
Sutton, Richard S, David A McAllester, Satinder P Singh, and Yishay
Mansour. 2000. “Policy Gradient Methods for Reinforcement Learning with
Function Approximation.” In Advances in Neural Information Processing
Systems, 1057–63.

Ah, fantastic. I'll take a look at the proof. Thanks for the references!


text:
What is the equation of the separation line for this neuron with identity activation?
I have a single neuron with 2 inputs, and identity activation, where f is activation function and u is output:
$u = f(w_1x_1 + w_2x_2 + b) = w_1x_1 + w_2x_2 + b$
My guessing for the separation line equation:
$u = w_1x_1 + w_2x_2 + b$
$\implies x_2 = \dfrac{u - w_1x_1 - b}{w_2}$
$\implies x_2 = (\dfrac{-w_1}{w_2})x_1 + \dfrac{u-b}{w_2}$
And the questions are:
1) Is the separation line equation above correct?
2) And when f is not identity function, is the separation line equation still the same? or different?

I found the answer, the output u is not related to plotting x2 against x1 here, coz u is like z axis and seeing the separation line is looking perpendicularly to the x1x2 plane, and whatever u value is, it's just a dot.
Put u as zero for the solving steps in the question.
So the separation line equation is this (with any activation function):
$x_2 = \dfrac{-w_1}{w_2}x_1 + \dfrac{-b}{w_2}$


text:
Is it possible to use deeplearning with spark (with a distributed databases as HDFS or Cassandra)?
If it is possible, will it be really useful or the model will end up converging very early(with a typical optimum learning rate) ? Any content on this topic will be helpful for me.

For those not familiar with it, could you explain what "spark" is?

Apache Spark is open source, general-purpose distributed computing engine used for processing and analyzing a large amount of data. Just like Hadoop MapReduce, it also works with the system to distribute data across the cluster and process the data in parallel.

Yes, it is possible to use deep learning architecture with Apache Spark now. Databricks have Spark-deep-learning which is a pipeline based in python and uses tensorflow and keras.
https://github.com/databricks/spark-deep-learning
You can check this. There is also BigDL by Intel analytics.
https://github.com/intel-analytics/BigDL


text:
What are the reasons behind slow YOLO training?
I'm testing out YOLOv3 using the 'darknet' binary, and custom config. It trains rather slow.
My testing out is only with 1 image, 1 class, and using YOLOv3-tiny instead of YOLOv3 full, but the training of yolov3-tiny isn't fast as expected for 1 class/1 image.
The accuracy reached near 100% after like 3000 or 4000 batches, in similarly 3 to 4 hours.
Why is it slow with just 1 class/1 image?

I think you underestimate the size of YOLO. This is the size of one segment of yolo tiny according to the darknet .cfg file:
Convolutional Neural Network structure:
416x416x3                Input image
416x416x16               Convolutional layer: 3x3x16, stride = 1, padding = 1
208x208x16               Max pooling layer: 2x2, stride = 2
208x208x32               Convolutional layer: 3x3x32, stride = 1, padding = 1
104x104x32               Max pooling layer: 2x2, stride = 2
104x104x64               Convolutional layer: 3x3x64, stride = 1, padding = 1
52x52x64                 Max pooling layer: 2x2, stride = 2
52x52x128                Convolutional layer: 3x3x128, stride = 1, padding = 1
26x26x128                Max pooling layer: 2x2, stride = 2
26x26x256                Convolutional layer: 3x3x256, stride = 1, padding = 1
13x13x256                Max pooling layer: 2x2, stride = 2
13x13x512                Convolutional layer: 3x3x512, stride = 1, padding = 1
12x12x512                Max pooling layer: 2x2, stride = 1
12x12x1024               Convolutional layer: 3x3x1024, stride = 1, padding = 1

.cfg file found here: https://github.com/pjreddie/darknet/blob/master/cfg/yolov3-tiny.cfg
EDIT: These networks generally aren't specifically designed to train fast, they're designed to run fast at test time, where it matters

yeah, that tiny network is still huge

It depends upon the factors such as
1. Batch size (GPU memory capacity)
2. CPU speed and number of cores(multi-threading to load the images)
Number of classes increase the number of convolution filters only in the prediction layers of YOLO. It influences only less than 1% speed of the detector to train the model.

u said it right, i train now on gpu and it's 10x to 100x faster


text:
YOLOv3 Model Structure: Why is filters = (classes + coords + 1) * num?
Here's a tutorial about doing custom training of YOLO (Darknet): https://medium.com/@manivannan_data/how-to-train-yolov3-to-detect-custom-objects-ccbcafeb13d2
The tutorial guides how to set values in the .cfg files:

classes = Number of classes, OK
filters = (classes + 5) * 3

Why is it 'plus 5' then 'times 3'?
Some say it's (classes + coords + 1) * num, but I can't guess it out the meaning.

3 boxes per feature map and for every box you predict class probabilities + width + height + x + y + object confidence.

uhm, and what is that 'coords'?

well its x, y, height and width

As said by @brale in the comment below the question:
filters = (classes + 5) * 3
= (classes + width + height + x + y + confidence) * num
= (classes + 1+1+1+1+1) * num
= (classes + 5) * num

YOLOv3 dectects 3 boxes per grid cell, so it is:
filters = (classes + 5) * 3


text:
GANs: Should Generator update weights when Discriminator says false continuously
My GANs is like this:

Train an autoencoder (VAE), get the decoder part and use as Generator
Train Discriminator

After training, do the generation in these steps:

Call Generator to generate an image
Call the Discriminator to classify the image to see whether it's acceptable

The problem is that the Discriminator says 'false' a lot, which means the generated image is not useful.
How should the Generator change (update weights) when Discriminator doesn't accept its generated image?

In general, you should train both discriminator D and generator G simultaneously.
Depending on the metric that you use as the target for your model, you may encounter a Vanishing gradient problem. It can happen when you implement original loss (i.e. JS-divergence). In that case D can become overconfident regarding fake samples and won't provide any useful feedback to the G. To find out if training fell into this problem, you should plot D and G loss. It will look as follows:

The original GAN has a lot of problems, that's why I suggest you to use Wasserstein metric. More information you can find in WGAN paper
Here you can find more information about GAN problems:

https://arxiv.org/pdf/1904.08994.pdf
https://arxiv.org/pdf/1804.00140.pdf


text:
Intutitive explanation of why Experience Replay is used in a Deep Q Network?
I understand that Experience Replay is used for data efficiency reasons and to remove correlations in sequences of data. How exactly do these sequences of correlated data affect the performance of the algorithm?

It is the neural network approximation that suffers, when it attempts to learn from correlated data. Intuitively, this is because the learning algorithm takes gradient steps assuming that the examples it are shown are representative of the dataset as a whole. A neural network update step uses a mini-batch of examples to calculate the gradient of its weights and biases with respect to a cost function. If that mini-batch is not fairly sampled, then the expected value of the loss and of the gradient will not be representative of the population as a whole. They will be biased and can cause an update step to parameters in the wrong direction.
In addition, if you are using a bootstrapping method - any form of TD learning - then the value estimates used to set learning update targets are sensitive to bias in the estimator. This is already something that can cause instability due to positive feedback loops. Adding another source of systemic bias from correlated input data can only make it worse.
You can gain some experience for the effect of this with a simple non-RL experiment.
Goal: To approximate the function $y = x^2$ in the range $-2 < x < 2$. These numbers chosen to make the task simple for a neural network.
Setup: Generate training data in the form $x_i, y_i$ for a few thousand sample points with true values of the function. Optionally add some noise to make the task harder. Keep the data set ordered by $x_i$ values. Create a simple neural network for regression (e.g. 2 hidden layers with 50 neurons and tanh activation), and set it up with a simple optimiser (e.g. SGD)
Run: Train the network twice, using small minibatches (e.g. size 10). Once without shuffling the data on each epoch (or using any shuffling algorithm on the minibatches), and once with standard shuffling and assignment to minibatches. Plot a learning curve of loss vs epoch for each run.
You should find that the ordered, non-shuffled version learns much slower than when using some randomisation to decorrelate the data. Depending on precise hyperparameters, the ordered version may even oscillate with loss quite far away from converging.
It is this effect, or a more subtle version of it, that impacts RL combined with neural networks when learning direct from trajectories. In addition, there is existing starting bias in bootstrap methods, which this effect makes worse.


text:
Is it legal to construct a public image database (for deep learning) with images from the internet?
I am trying to put together a public agricultural image database of corn and soybeans, to train convolutional neural networks. The main method of image collection will be through taking pictures of various fields in the growing season. The images will be uploaded to a public data sharing site which will be accessible by many.
However, I could get much more images compiled if I were to take some off of, say, Google Images. If there anything wrong with this? Would there be any issues with copywright infringements if I find the images on a publicly-available search engine? I need a lot of images, so I thought this would be a good method of increasing my image numbers.

There are many public available stuff that is not exactly legal. Many comic images for example can be found in Google but in many cases they are copyrighted and can't be distributed. In this case however, it is because comics are sold. I don't know about the particular case of fields, but unless you take a picture from a sort of copyrighted book or something, you shouldn't have any issues. The best way is to find a place with many of them and ask owner for permission. If they are images from like imgur uploaded by original owner, those should be fine

This is not really related to AI; there might be a better place to ask this question.

@OliverMason It is related to building a dataset for an AI image classifier, so I do believe that it is related in that way.

I believe this is covered under Section 107 of the Copyright Act states:
the fair use of a copyrighted work, including such use by reproduction in copies or phonorecords or by any other means specified by that section, for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, is not an infringement of copyright.
This section was intended to enable utilization of copyrighted material where that utilization is in the public interest. As long as you stay within the bounds as stated above you should not be considered in violation of a copyright.


text:
Why I have a different number of terms in word2vec and TFIDF? How I can fix it?
I need multiply the weigths of terms in TFIDF matrix by the word-embeddings of word2vec matrix but I can't do it because each matrix have a different number of terms. I am using the same corpus for get both matrix, I don't know why each matrix have a different number of terms
.
My problem is that I have a matrix TFIDF with the shape (56096, 15500) (corresponding to: number of terms, number of documents) and matrix Word2vec with the shape (300, 56184) (corresponding to : number of word-embeddings, number of terms).
And I need the same numbers of terms in both matrix.
I use this code for get the matrix of word-embeddings Word2vec:
def w2vec_gensim(norm_corpus):
    wpt = nltk.WordPunctTokenizer()
    tokenized_corpus = [wpt.tokenize(document) for document in norm_corpus]
    # Set values for various parameters
    feature_size = 300
    # Word vector dimensionality
    window_context = 10
    # Context window size
    min_word_count = 1
    # Minimum word count
    sample = 1e-3
    # Downsample setting for frequent words
    w2v_model = word2vec.Word2Vec(tokenized_corpus, size=feature_size, window=window_context, min_count =  min_word_count, sample=sample, iter=100)
    words = list(w2v_model.wv.vocab)
    vectors=[]
    for w in words:
        vectors.append(w2v_model[w].tolist())
    embedding_matrix= np.array(vectors)
    embedding_matrix= embedding_matrix.T
    print(embedding_matrix.shape)

    return embedding_matrix

And this code for get the TFIDF matrix:
tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True)

def matriz_tf_idf(datos, tv):
    tv_matrix = tv.fit_transform(datos)
    tv_matrix = tv_matrix.toarray()
    tv_matrix = tv_matrix.T
    return tv_matrix

And I need the same number of terms in each matrix. For example, if I have 56096 terms in TFIDF, I need the same number in embeddings matrix, I mean matrix TFIDF with the shape (56096, 1550) and matrix of embeddings Word2vec with the shape (300, 56096). How I can get the same number of terms in both matrix?
Because I can't delete without more data, due to I need the multiplication to make sense because my goal is to get the embeddings from the documents.
Thank you very much in advance.

Your problem is that TFIDF is cutting out around 90 terms.
Since you aren't using the min_df or max_df parameters and as far as I can tell you aren't doing any stemming/lemmatization, the only difference I can see between the two methods is the tokenizer.
There are two things I'd try out if I were you:

Try explicitly converting the word2vec corpus to lowercase. TfidfVectorizer does this by default and I can't see where you're doing it in the word2vec pipeline. Ignore this if your corpus is already lowercased.
Try using the nltk.WordPunctTokenizer() with the TfidfVectorizer. You can do this like this:

wpt = nltk.WordPunctTokenizer()
tv = TfidfVectorizer(min_df=0., max_df=1., norm='l2', use_idf=True, smooth_idf=True,
                     tokenizer=wpt.tokenize)

Thank you very much. I need to do the second thing for fix it.


text:
Oscillating around the saddle point in gradient descent?
I was reading a blog post that talked about the problem of the saddle point in training.

In the post, it says if the loss function is flatter in the direction of x (local minima here) compared to y at the saddle point, gradient descent will oscillate to and from the y direction. This gives an illusion of converging to a minima. Why is this?
Wouldn’t it continue down in the y direction and hence escape the saddle point?

Link to post:
https://blog.paperspace.com/intro-to-optimization-in-deep-learning-gradient-descent/
Please go to Challenges with Gradient Descent #2: Saddle Points.

IIRC, the main issue would be that in high dimensions there would be no "obvious" direction for the gradient descent to move towards, as you would tend to believe in a 3d graph like above. (some analogies fail in high dimensions) im not super certain about this answer (though i think its right), so i'll leave it as a comment instead

Thank you for the reply. If there is no obvious direction then is it to say the saddle points flat area is relatively large in all directions? For a smaller flat area, if the y direction is the local minima in this saddle point then it would make sense to me for it to oscillate in that direction as it continues to overshoot the minima. It never considers the x direction as the gradients are too small. However, if the y direction is the local maxima here, then it should not be stuck in the y direction but x?

yeah tbh i was also wondering about this question when i was first introduced to the problem. i think the point being is that in a pragmatic instance, your error surface won't be strictly-by-definition a saddle point with 0 gradient, but will be a small region with near-0 gradient. because there is small-noise in the gradient, the algo will wander around but cannot escape since there is no clear/strong tangental direction to escape the region with low gradient. actually i think this is the answer, so i'll write it properly

fun fact: a friend told me once that in pragmatic instances (datasets of the "real world"), error surfaces might have an "egg carton shape" ("might", because what even does a high-dim egg carton look like?). we say this because we observe that (obviously: using the same ML architecture and hyper-parameters), we tend to get more or less the same error rates despite learning vastly different weights. the implication is that the error surface has many local minimas but they are more-or-less of "the same depth"

I agree with k c . From my experience in running the same model  many times the local minimum attained seem to result in about the same values for training accuracy with slightly more variance on validation accuracy. The saddle point issue is harder to get a handle on. As noted in other comments while mathematically a saddle point has a 0 value for the derivative in a real world model it might get very small but will not be zero. I would expect that the result would be oscillations in the flat area about the saddle point. Reducing the learning rate might help to converge to a smaller area

It's important to note that in a pragmatic instance of ML on a "real dataset", you likely wouldn't have a "strict" saddle point with precisely zero gradient. Your error surface won't be "smooth", so even though you would have something that resembles a saddle point, what you would obtain would in fact be a small region with near-zero gradient.
So let's say you are in a region with near-zero gradient. Assuming that the gradient is this area is normalized at 0 with small Gaussian distributed noise (thus gradient =  small Gaussian noise). You can then see that the algorithm can't quite escape the region (or at least, will spend a lot of time here) since 1. Gaussian random walks will more-or-less stay in place (unless for a long time) 2. small gradients means there is no obvious direction to leave the region.
In any case, SGD more or less solves this issue, and its usage is standard practice for reasons beyond this problem.


text:
Can you let specific data impact a neural network more than other data?
I have a lot of empty values in my dataset, so I want to let my neural network 'learn more' on the rows that have no empty values because these rows are of higher importance.
Is there a way to do this?

Yes, you can weight the loss function for each example, so that instead of your cost function being
$$J = \sum_i \mathcal{L}(y_i, \hat{y}_i)$$
It will be
$$J = \sum_i w_i\mathcal{L}(y_i, \hat{y}_i)$$
Where $i$ iterates over your data set, $\mathcal{L}$ is the loss function you are using, $y_i$ is ground truth for each example and $\hat{y}_i$ is prediction for each example.
You can set your relevance weighting according to any criteria you like based on the example, and your dataset/goals. So for instance you could set it to $1.0$ for complete examples and $0.1$ for incomplete examples. Depending on your NN framework, it may already offer per-example weighting, and even if it does not, auto-differentiation means that typically you only need to implement the forward logic into the cost function for each minibatch, and the weighting will be applied correctly to gradients with no more work required. If you do need to implement gradient calculations yourself, you simply multiply the initial gradient of each example by the example's weight $w_i$.
Once you make a change like this, you need to take care on how you set up and interpret your test set. When your model is used in production, if there are still inputs with missing details, you probably don't want to weight the metrics (e.g. accuracy rating) in the same way, but to report it as-is against a correctly sampled dataset of unseen examples from the population of expected inputs.


text:
Am I overfitting my GAN model?
I'm training a DCGAN model on a 320x320 dataset of images and after an hour of training the generator started to generate (on the same latent space noise as during training) images that are identical to the dataset. For example, if my dataset is images of cars, I should expect to see unexisting designs of cars, right? Am I understanding this wrong? I know this is a very general question but I was wondering if this is what should happen and if I should try on different latent space values and then see proper results and not just copies of my dataset?

It might be that your dataset of images is to small. Your discriminative network might hardlearn these images at which point your generative network can only produce good images if it copies the same images of your dataset.

Exactly. My dataset consists of only 45 images. What can I do?

I believe that question is answered here.


text:
Is it a sign of overfitting when validation_loss dips and then goes up with increasingly bigger swings?
I am experimenting with a ConvNet to categorize images taken with a depth camera. So far I have 4 sets of 15 images each. So 4 labels. The original images are 680x880 16-bit grayscale. They are scaled down before feeding it to the ImageDataGenerator to 68x88 RGB (each color channel with equal value). I am using the ImageDataGenerator (IDG) to create more variance on the sets. (The IDG does not seem to be able to handle 16-bit grayscale images, nor 8-bit images well, so hence I converted them to RGB).
I estimate the images to be low on features, compared to regular RGB images, because it represents depth. To get a feel for the images, here are a few down scaled examples:

I let it train 4.096 epochs, to see how that would go.
This is the result of the model and validation loss.

You can see that in the early epochs the validation (test / orange line) loss dips, and then goes up and starts to show big swings. Is this a sign of overfitting?
Here is a zoomed in image of the early epochs.

The model loss (train / blue line) reached relatively low values with an accuracy of 1.000. Training again shows repeatedly the same kind of graphs.
Here are the last epochs.
Epoch 4087/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.1137 - accuracy: 0.9286 - val_loss: 216.2349 - val_accuracy: 0.7812
Epoch 4088/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0364 - accuracy: 0.9643 - val_loss: 234.9622 - val_accuracy: 0.7812
Epoch 4089/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0041 - accuracy: 1.0000 - val_loss: 232.9797 - val_accuracy: 0.7812
Epoch 4090/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0091 - accuracy: 1.0000 - val_loss: 238.7082 - val_accuracy: 0.7812
Epoch 4091/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0248 - accuracy: 1.0000 - val_loss: 232.4937 - val_accuracy: 0.7812
Epoch 4092/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0335 - accuracy: 0.9643 - val_loss: 273.6542 - val_accuracy: 0.7812
Epoch 4093/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0196 - accuracy: 1.0000 - val_loss: 258.2848 - val_accuracy: 0.7812
Epoch 4094/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0382 - accuracy: 0.9643 - val_loss: 226.6226 - val_accuracy: 0.7812
Epoch 4095/4096
7/7 [==============================] - 0s 10ms/step - loss: 0.0018 - accuracy: 1.0000 - val_loss: 226.2943 - val_accuracy: 0.7812
Epoch 4096/4096
7/7 [==============================] - 0s 11ms/step - loss: 0.0201 - accuracy: 1.0000 - val_loss: 207.3653 - val_accuracy: 0.7812

Not sure if it is required to know the architecture of the neural network to judge whether this is overfitting on this data set. Anyway, here is the setup.
kernelSize = 3
kernel = (kernelSize, kernelSize)

model = Sequential()
model.add(Conv2D(16, kernel_size=kernel, padding='same', input_shape=inputShape, activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, kernel_size=kernel, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, kernel_size=kernel, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(nr_of_classes, activation='softmax'))

sgd = tf.keras.optimizers.SGD(lr=learning_rate, decay=1e-6, momentum=0.4, nesterov=True)
model.compile(loss='categorical_crossentropy',
            optimizer=sgd,
            metrics=['accuracy'])

Yes this looks a lot like overfitting. The clue is in the low and slowly decreasing training loss compared to the large increases in validation loss.
One simple fix would be to stop training around epoch 50, taking the best cross validation result to select the most general network at that point. However, anything that works to improve stable generalisation could help here - more training data, more regularisation, simpler model.
The tricky part is finding the best combination for generalisation. Typically if you overuse regularisation to make the NN completely stable, it will lose some accuracy, so you need to run multiple experiments and measure things carefully. As you have a small dataset here and seem capable of running 1000's of epochs, I would suggest k-fold cross validation for improved measurement of cross validation loss.

+1 For pointing me to k-fold cross validation which is a new concept for me. Just looked it up and makes intuitive sense. Now I will look for implementing it with Keras. Thanks!

FYI: I implemented the k-fold cross validation. I used a bin size of 5, considered per label. So with 4 labels x 15 examples it resulted in 3 training rounds with each 4 labels x 10 training samples and 4 labels x 5 validation samples. After averaging the training weights and predicting all 60 samples it had 100 accuracy.


text:
How can I use the success and failure data to estimate parameters of a Dirichlet distribution?
I have used Beta function to estimate the performance of the agent. I have failure and success data of the task that runs on the agent.
The parameter $\alpha$ is a number of successful tasks, while $\beta$ is the number of failures. Thus, I can estimate the performance by exploiting the expected value of Beta, as $$\mu = \frac{\alpha} {(\alpha+\beta)}$$
So, I am looking for a similar model, such that its parameter can be estimated from the success and failure data. So far I found Dirichlet distribution.
What is the expected value of Dirichlet distribution? How I can use the success and failure data to estimate parameters of this distribution?
Let's check the following example:
Suppose that we use a Dirichlet prior represented by $Dirichlet(1, 1, 1)$ and observe $13$ results with $8$ Successful, $2$ Missing, and $3$ Failures. Then we get the posterior to be $Dirichlet(1+8, 1+2, 1+3)$. Then if you define the performance value $\alpha$ to be the expectation of $P(x=Successful)$,
then $\alpha$ will be $(1+8)/[(1+8)+(1+2)+(1+3)] = 0.56$
Now
Suppose that we use a Beta prior represented by $Beta(1,1)$ and observe $13$ results with $8$ Successful, and $3$ Failures. Then we get the posterior to be $Beta(1+8, 1+3)$. Then if you define the performance value Pr to be the expectation of $P(x=Successful)$,
then $\alpha = (1+8)/[(1+8)+(1+3)] = 0.69$
Are my calculations and concept right?

Is this problem in the context of reinforcement learning? So, is your agent an RL agent?

Actually no, but thanks for this good idea, seems interesting


text:
Can't solve Towers of Hanoi in PDDL
I'm using PDDL to generate a plan to solve this tower of Hanoi puzzle. I'll give the problem, the rules, the domain and fact sheet for everything.
PDDL is telling me that the goal can be simplified to false; however, I know for a fact that this puzzle is solvable.
Puzzle:

There are 3 posts. Each has rings on it. From bottom to top on each post. The first post has the
  second largest ring. The second post has the smallest ring, with the second smallest ring on top of it.
  The third post has the third largest ring, with the largest ring stacked on top of it.

Rules:

The rules of this game are that you may only stack a ring on top of a larger ring. Your goal is to get all of the rings onto the same post, stacked from largest to smallest.

My Code
Domain
(define (domain hanoi)
  (:requirements :strips)
  (:predicates (clear ?x) (on ?x ?y) (smaller ?x ?y))

  (:action move
    :parameters (?disc ?from ?to)
    :precondition (and (smaller ?to ?disc) (on ?disc ?from)
               (clear ?disc) (clear ?to))
    :effect  (and (clear ?from) (on ?disc ?to) (not (on ?disc ?from))
          (not (clear ?to))))
  )

Problem
(define (problem hanoi5)
  (:domain hanoi)
  (:objects peg1 peg2 peg3 d1 d2 d3 d4 d5)
  (:init
    (smaller peg1 d1) (smaller peg1 d2) (smaller peg1 d3)
    (smaller peg1 d4) (smaller peg1 d5)
    (smaller peg2 d1) (smaller peg2 d2) (smaller peg2 d3)
    (smaller peg2 d4) (smaller peg2 d5)
    (smaller peg3 d1) (smaller peg3 d2) (smaller peg3 d3)
    (smaller peg3 d4) (smaller peg3 d5)

    (smaller d2 d1) (smaller d3 d1) (smaller d3 d2) (smaller d4 d1)
    (smaller d4 d2) (smaller d4 d3) (smaller d5 d1) (smaller d5 d2)
    (smaller d5 d3) (smaller d5 d4)

    ;(clear peg2) (clear peg3) (clear d1)
    ;(on d5 peg1) (on d4 d5) (on d3 d4) (on d2 d3) (on d1 d2))
    (clear d2) (clear d4) (clear d1)
    (on d2 peg1) (on d5 peg2) (on d4 d5) (on d3 peg3) (on d1 d3))

  (:goal (and (on d5 d4) (on d4 d3) (on d3 d2) (on d2 d1)))
)

I'm really at a loss here. Thank you!

Ah hah!
The way I had defined the disks made d5 the LARGEST disk, not the smallest. So, the last few lines of the file should be:
   (clear d4) (clear d2) (clear d5)
    (on d4 peg1) (on d1 peg2) (on d2 d1) (on d3 peg3) (on d5 d3))

  (:goal (and (on d1 d2) (on d2 d3) (on d3 d4) (on d4 d5)))
)

However
If I wanted the opposite to be true, d1 to be the largest, the changes would have been as following:
; Swapped ?disc & ?to in (smaller ...) statement
:precondition (and (smaller ?disc ?to) (on ?disc ?from)

And
; Swapped all d* and peg* objs within (smaller ...) staements
(smaller d1 peg1) (smaller d2 peg1) (smaller d3 peg1)
(smaller d4 peg1) (smaller d5 peg1)
(smaller d1 peg2) (smaller d2 peg2) (smaller d3 peg2)
(smaller d4 peg2) (smaller d5 peg2)
(smaller d1 peg3) (smaller d2 peg3) (smaller d3 peg3)
(smaller d4 peg3) (smaller d5 peg3)

:)


text:
How to set the target for the actor in A2C?
I did a simple Actor-Critic implementation in Keras using 2 networks where the critic learns the Q-Values of every action, and the actor predicts probabilities for choosing each action. In training, the target probabilities for the actor was a one-hot vector with 1.0 in the maximum Q-Value prediction position and 0.0 in all the rest, and simply used fit method on the actor model with mean squared error loss function.
However, I'm not sure what to set as the target when switching to A2C. In all the guides I saw it's mentioned that the critic now learns one value per state, not one value per action in the action space.
This change makes it unclear on how to set the target vector for the actor. The guides/SE questions I went over did not explain this point and simply said that we can calculate the advantage value using the value function (here, here and here) for the current and next state, which is fine, except we can only do that for the specific action taken and not for every action in the action-space because we don't the value for every next state for every action.
In other words, we only know A(s,a) for our memorized a, and we know nothing about the advantage of other actions.
One of my guesses was that you still calculate the Q-Values, because after all, the value function is defined by the Q-Values. The value function is the sum over every action a of Q(s,a)*p(a). So does the critic need to learn the Q-Values and sum their multiplications with the probabilities generated by the policy network (actor), and calculate the advantages of every action?
It's even more confusing because in one of the guides they said that the critic actually learns the advantage values, and not the value function (like all the other guides said), which is strange because you need to use the critic to predict the value function of the state and the next state. Also, the advantage function is per-action and in the implementations I see the critic has one output neuron.
I think that what's being done in the examples I saw was to train the actor to fit a one-hot vector for the selected action (not the best action by the critic), but modify the loss-function value using the advantage value (possibly to influence the gradient). Is that the case?

In short, my last sentence was the correct answer. The "target" is a one-hot with the selected action, but there's a trick.

A2C Loss Function
A very crucial part of A2C implementation that I missed is the custom loss function that takes into account the advantage. The loss function multiplies the advantage with the negative log of the current probability to select the action that was selected.
The trick is that if the advantage is negative, the loss function will switch sign, so the gradients will be applied to the opposite direction.
In one dimension it's easier to understand. Let's say my target prediction is 1 and my actual prediction is 0.6. A simple loss would be defined as target - prediction, or in this case 0.4 and future predictions will be closer to one. If my prediction was 1.4, then the loss would be -0.4. A negative loss would mean predicting a lower result in the future, and a positive result would mean predicting a higher result in the future.
If the sign of the loss function is switched, the prediction will actually move away from 1.
The same thing happens when you multiply the advantage in the loss function. A negative advantage would mean that this action is worse than the value of the state so we need to avoid it, and a positive advantage means that the action is encouraged.

In Keras (Tensorflow 2.0):
Here's the custom loss function:
def custom_actor_loss(y_true, y_prediction, advantage):
    prediction = K.clip(y_prediction, 1e-8, 1 - 1e-8)
    log_probabilities = y_true * K.log(prediction)

    return K.sum(-log_probabilities*advantage)

The values are clipped because log of 0 is undefined.
And the rest of the network building:
input_layer = Input(shape=self._state_size, name='state_in')
advantage = Input(shape=[1], name='advantage')
target_prediction = Input(shape=self._actions_num, name='target')

inner_layer = Dense(units=layer_size, activation='relu')(input_layer)
actor_out = Dense(units=self._actions_num, activation='softmax', name='actor_out')(inner_layer)

self._actor = Model([input_layer, target_prediction, advantage], actor_out, name='actor')
self._actor.add_loss(custom_actor_loss(actor_out, target_prediction, advantage))
self._actor.compile(optimizer=Adam(learning_rate=actor_learning_rate))

And in the training loop (where future_rewards_prediction and critic_prediction are the outputs from the critic for the current and next state (except for the terminal state where the future_rewards_prediction is set to 0):
# Train actor
target_probabilities = np.zeros([1, self._actions_num])
target_probabilities[0][memory[step_idx].action] = 1.0
advantage = memory[step_idx].reward + future_rewards_prediction * self._future_discount - critic_prediction
self._actor.fit((memory[step_idx].state, target_probabilities, advantage), verbose=0)

*Notice how I don't really specify a y in my fit call. This is because of an issue I had when trying to implement a custom loss function in Keras which was solved by this answer.

authors: גרירת אופנועים סימון ציק צאק


text:
Different methods of calculating gradients of cost function(loss function)
We require to find the gradient of loss function(cost function) w.r.t to the weights to use optimization methods such as SGD or gradient descent. So far, I have come across two ways to compute the gradient:

BackPropagation
Calculating gradient of loss function by calculus

I found many resources for understanding backpropagation.
The 2nd method I am referring to is the image below(taken for a specific example, e is the error: difference between target and prediction):

Also, the proof was mentioned in this paper:here
Moreover, I found this method while reading this blog.(You might have to scroll down to see the code: gradient = X.T.dot(error) / X.shape[0]  )
My question is are the two methods of finding gradient of cost function same? It appears different and if yes, which one is more efficient( though one can guess it is backpropagation)
Would be grateful for any help. Thanks for being patient(it's my 1st time learning ML).

But sir, if both are same, why do we use backPropagation, isn't it very lengthy

Backpropagation is a method/process of propagating the loss back to previous layers, whereas gradient descent (and it's variants like SGD,RMSprop etc) is a method to optimize the neural network to reduce the loss. Loss is less when we "descent" to a valley point, which we calculate through gradients or slope (ie. derivatives) and we send this information back using backprop where different channels receive different value of the gradient depending on their contribution. We can also send info other than gradients using backprop, there are also methods which don't use one or other, or either

I'm pretty sure they're the same thing. Both backpropagation and gradient descent's essential ideas are to calculate the partial derivative of the cost with respect to each weight, and subtract the partial derivative times the learning rate.

Yes, I believe both are the same. They are different methods and backpropagation is a faster algorithm. This is mentioned and explained why in [here] (http://neuralnetworksanddeeplearning.com/chap2.html#in_what_sense_is_backpropagation_a_fast_algorithm)


text:
XOR-solving neural network is suffering from local minima
import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
import matplotlib.pyplot as plt
import numpy as np
x_data = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]]).float()
y_data = torch.tensor([0, 1, 1, 0]).float()

class Model(nn.Module):
    def __init__(self, input_size, H1, output_size):
        super().__init__()
        self.linear_input = nn.Linear(input_size, 2)
        self.linear_output = nn.Linear(2, output_size)
        self.sigmoid = torch.nn.Sigmoid()

    def forward(self, x):
        x = self.sigmoid(self.linear_input(x))
        x = self.sigmoid(self.linear_output(x))
        return x

    def predict(self, x):
        return (self.forward(x) >= 0.5).float()

model = Model(2,2,1)
lossfunc = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)
epochs = 2000
losses = []

for i in range(epochs):
    pred = model(x_data).view(-1)
    loss = lossfunc(pred, y_data)
    print("epochs:", i, "loss:", loss.item())
    losses.append(loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

def cal_score(X, y):
    y_pred = model.predict(X)
    score = float(torch.sum(y_pred.squeeze(-1) == y.byte().float())) / y.shape[0]
    return score

print('test score :', cal_score(x_data, y_data))
def plot_decision_boundray(X):
    x_span = np.linspace(min(X[:, 0]), max(X[:, 0]))
    y_span = np.linspace(min(X[:, 1]), max(X[:, 1]))
    xx, yy = np.meshgrid(x_span, y_span)
    grid = torch.tensor(np.c_[xx.ravel(), yy.ravel()]).float()
    pred_func = model.forward(grid)
    z = pred_func.view(xx.shape).detach().numpy()
    plt.contourf(xx, yy, z)
    plt.show()

plot_decision_boundray(x_data)

As you can see, it's a simple neural network which consists of one hidden layer using BCELoss and Adam.

Normally, it results in the correct one like above.

However, it is sometimes stuck in a local minima and a decision boundary becomes awkward.
Because the input data is limited, I guess that preprocessing of those data might not be possible and only initial weights matter in this problem. I tried initializing them with normal distribution but it didn't work. How can I approach this problem?

how do you get the first image?

From the log you can see that the loss value is still big compared to the first image, but at least it decrease on each iteration (even very very small). So I suggest you to:

try to use bigger learning rate, I think you model is underfiting.
try to use another weight initialization method

I think the first solution will solve this problem as I see no problem in the code.


text:
How can I train a neural network if I don't have enough data?
I have created a neural network that is able to recognize images with the numbers 1-5. The issue is that I have a database of 16x5 images which ,unfortunately, is not proving enough as the neural network fails in the test set. Are there ways to improve a neural network's performance without using more data? The ANN has approximately a 90% accuracy on the training sets and a 50% accuracy in the test ones.
Code:
clear
graphics_toolkit("gnuplot")
sigmoid = @(z) 1./(1 + exp(-z));
sig_der = @(y) sigmoid(y).*(1-sigmoid(y));

parse_image;   % This external f(x) loads the images so that they can be read.
%13x14
num=0;
for i=1:166
  if mod(i-1,10)<=5 && mod(i-1,10) > 0
    num=num+1;
    data(:,num) = dlmread(strcat("/tmp/",num2str(i)))(:);
  end
end

function [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i)
  mid_layer(:,1)=sum(w1.*data(:,i));
  mid_layer(:,2)=sigmoid(mid_layer(:,1));
  last_layer(:,1)=sum(mid_layer(:,2).*w2);
  last_layer(:,2)=sigmoid(last_layer(:,1));
  exp_res=rem(i,5);
  if exp_res==0
    exp_res=5;
  end
  exp_result=zeros(5,1); exp_result(exp_res)=1;
  cost = exp_result-last_layer(:,2);
end

function [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i)
  delta(1:5) = cost;
  delta(6:20) = sum(cost' .* w2,2);
  w2 = w2 + 0.05 .* delta(1:5) .* mid_layer(:,2) .* sig_der(last_layer(:,1))';
  w1 = w1 + 0.05 .* delta(6:20) .* sig_der(mid_layer(:,1))' .* data(:,i);
end

w1=rand(182,15)./2.*(rand(182,15).*-2+1);
w2=rand(15,5)./2.*(rand(15,5).*-2+1);

for j=1:10000
  for i=[randperm(85)]
    [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i);
    [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i);
    cost_mem(j,i,:)=cost;
  end
end

You can synthetically increase the number of samples. For example with augmentation or unsupervised adaption (Self-training). With augmentation you grant the system way more robustness so i would really recommend this. For example this github. The problem with such small database sizes is that your test-set is also very small and you cannot test properly if your network generalizes well, or just overfits.
You can try transfer learning with another larger network to adapt those feature extractors and use them on ur problem. That may work better than training a new one from scratch with so less labeled images. Hope i could help at least a little, stay tuned.

Thanks. I was wondering if there was a small tweak you could make to the back propagation algorithm to improve training.

Please if you find that instant contact me haha :D. Mhm maybe consider dropout, or adding noise on your system. Meaning add gaussian noise to the weigths itself, works quite good in self-learning and GAN´s, also stochastic depth is quite nice for the same kind of tweaking

i like the easy keras implementation. Maybe you want to try it here: https://keras.io/layers/noise/

I'm not a fan of python. I usually don't like using libraries either as they aren't very instructive but thanks a lot . I'll still check it out.

In theory, yes, using synthetic data generation. This involves applying transformations to the original images to generate new 'unique' images. Some standard techniques include rotating, flipping, stretching, zooming or brightening. Obviously not all of these make sense depending on the data. In your problem, zooming, stretching and brightening could be used but flipping should not. Rotation could work but only for small angles.
Generally this is implemented by replacing the dataset for each epoch of training. Therefore, the number of images used in each training iteration is the same but the images themselves have been altered.
In practice, it's not a magic bullet. The reason a larger dataset generally yields better models is because the probability of a new feature falling within the feature distribution of the training data is higher. With synthetic data generation the new features are only marginally different to the original so even if the number of images to train on is increased, the feature distributions are not that different. There is a lot of variation in handwritten numbers so it would be very hard to guess how effective this would be without trying it.