text:
What is the time complexity of the forward pass and back-propagation of the sequence-to-sequence model with and without attention?
I keep looking through the literature, but can't seem to find any information regarding the time complexity of the forward pass and back-propagation of the sequence-to-sequence RNN encoder-decoder model, with and without attention.
The paper Attention is All You Need by Vaswani et. al in 2017 states the forward pass cost is $O(n^3)$, which makes sense to me (with 1 hidden layer). In terms of $X$ the input length, and $Y$ the output length, it then looks like $O(X^3 + Y^3)$, which I understand.
However, for training, it seems to me like one back-propagation is at worst $O(X^3 + Y^3)$, and we do $Y$ of them, so $O(Y(X^3  Y^3))$.
This is the following diagram, where the green blocks are the hidden states, the red ones are the input text and the blue ones are output text.

If I were to add global attention, as introduced by Luong et. al in 2015, the attention adds an extra $X^2$ in there due to attention multiplication, to make an overall inference of $O(X^3 + X^2 Y^3)$, and training even worse at $O(XY(X^3 + X^2 Y^3))$ since it needs to learn attention weights too.
The following diagram shows the sequence-to-sequence model with attention,
where $h$'s are the hidden states, $c$ is the context vector and $y$ is the output word, $Y$ of such output words and $X$ such inputs. This setup is described in the paper Effective Approaches to Attention-based Neural Machine Translation, by Luong et al. in 2015.

Is my intuition correct?

I encourage you to focus on one of the models, because your question isn't very simple, because one needs to investigate both models. Furthermore, I suggest you focus either on the forward pass or back-propagation. But these are just suggestions. You can ask different separate questions. One e.g. for the forward pass, one for the model with attention. This will help users to focus on one problem.


text:
How are n-dimensional vectors state vectors represented in Q-learning?
Using this code:
import gym
import numpy as np
import time

"""
SARSA on policy learning python implementation.
This is a python implementation of the SARSA algorithm in the Sutton and Barto's book on
RL. It's called SARSA because - (state, action, reward, state, action). The only difference
between SARSA and Qlearning is that SARSA takes the next action based on the current policy
while qlearning takes the action with maximum utility of next state.
Using the simplest gym environment for brevity: https://gym.openai.com/envs/FrozenLake-v0/
"""

def init_q(s, a, type="ones"):
    """
    @param s the number of states
    @param a the number of actions
    @param type random, ones or zeros for the initialization
    """
    if type == "ones":
        return np.ones((s, a))
    elif type == "random":
        return np.random.random((s, a))
    elif type == "zeros":
        return np.zeros((s, a))

def epsilon_greedy(Q, epsilon, n_actions, s, train=False):
    """
    @param Q Q values state x action -> value
    @param epsilon for exploration
    @param s number of states
    @param train if true then no random actions selected
    """
    if train or np.random.rand() < epsilon:
        action = np.argmax(Q[s, :])
    else:
        action = np.random.randint(0, n_actions)
    return action

def sarsa(alpha, gamma, epsilon, episodes, max_steps, n_tests, render = True, test=False):
    """
    @param alpha learning rate
    @param gamma decay factor
    @param epsilon for exploration
    @param max_steps for max step in each episode
    @param n_tests number of test episodes
    """
    env = gym.make('Taxi-v3')
    n_states, n_actions = env.observation_space.n, env.action_space.n
    Q = init_q(n_states, n_actions, type="ones")
    print('Q shape:' , Q.shape)

    timestep_reward = []
    for episode in range(episodes):
        print(f"Episode: {episode}")
        total_reward = 0
        s = env.reset()
        print('s:' , s)
        a = epsilon_greedy(Q, epsilon, n_actions, s)
        t = 0
        done = False
        while t < max_steps:
            if render:
                env.render()
            t += 1
            s_, reward, done, info = env.step(a)
            total_reward += reward
            a_ = epsilon_greedy(Q, epsilon, n_actions, s_)
            if done:
                Q[s, a] += alpha * ( reward  - Q[s, a] )
            else:
                Q[s, a] += alpha * ( reward + (gamma * Q[s_, a_] ) - Q[s, a] )
            s, a = s_, a_
            if done:
                if render:
                    print(f"This episode took {t} timesteps and reward {total_reward}")
                timestep_reward.append(total_reward)
                break
#             print('Updated Q values:' , Q)
    if render:
        print(f"Here are the Q values:\n{Q}\nTesting now:")
    if test:
        test_agent(Q, env, n_tests, n_actions)
    return timestep_reward

def test_agent(Q, env, n_tests, n_actions, delay=0.1):
    for test in range(n_tests):
        print(f"Test #{test}")
        s = env.reset()
        done = False
        epsilon = 0
        total_reward = 0
        while True:
            time.sleep(delay)
            env.render()
            a = epsilon_greedy(Q, epsilon, n_actions, s, train=True)
            print(f"Chose action {a} for state {s}")
            s, reward, done, info = env.step(a)
            total_reward += reward
            if done:
                print(f"Episode reward: {total_reward}")
                time.sleep(1)
                break

if __name__ =="__main__":
    alpha = 0.4
    gamma = 0.999
    epsilon = 0.9
    episodes = 200
    max_steps = 20
    n_tests = 20
    timestep_reward = sarsa(alpha, gamma, epsilon, episodes, max_steps, n_tests)
    print(timestep_reward)

from :
https://towardsdatascience.com/reinforcement-learning-temporal-difference-sarsa-q-learning-expected-sarsa-on-python-9fecfda7467e
A sample Q table generated is :
[[ 1.          1.          1.          1.          1.          1.        ]
 [ 0.5996      0.5996      0.5996      0.35936     0.5996      1.        ]
 [ 0.19936016  0.35936     0.10336026  0.35936     0.35936    -5.56063984]
 ...
 [ 0.35936     0.5996      0.35936     0.5996      1.          1.        ]
 [ 1.          0.5996      1.          1.          1.          1.        ]
 [ 0.35936     0.5996      1.          1.          1.          1.        ]]

The columns representing the actions and rows representing the corresponding states.
Can the state be represented by a vector? The Q table cells are not contained by vectors of size > 1, so how should these states be represented? For example, if I'm in the state [2], can this be represented as an n-dimensional vector?
Put another way, if Q[1,3] = 4, can the Q state 1 with action 3 be represented as a vector [1, 3, 2, 12, 3]? If so, then is the state_number -> state_attributes mapping stored in a separate lookup table?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Given the coordinates of an object in an image, is it possible to predict the coordinates of the same object in a different perspective?
I am trying to figure out how to approach this.
Given training data of images and the pixel coordinates of the centre of an object in that image, would it be possible to predict the pixel coordinates of the object in the same "scene" in a different perspective, but with the object removed?

When you say "perspective", do you mean the same scene but viewing it from a different viewpoint/angle?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How can I cluster this data frame with several features and observations?
How can I cluster the data frame below with several features and observations? And how would I go about determining the quality of those clusters? Is k-NN appropriate for this?
id     Name             Gender   Dob    Age  Address
1   MUHAMMAD JALIL      Male    1987    33   Chittagong
1   MUHAMMAD JALIL      Male    1987    33   Chittagong
2   MUHAMMAD JALIL      Female  1996    24   Rangpur
2   MRS. JEBA           Female  1996    24   Rangpur
3   MR. A. JALIL        Male    1987    33   Sirajganj
3   MR. A. JALIL        Male    1987    33   Sirajganj
3   MD. A. JALIL        Male    1987    33   Sirajganj
4   MISS. JEBA          Female  1996    24   Rangpur
4   PROF. JEBA          Female  1996    24   Rangpur
1   MD. A. JALIL        Male    1987    33   Chittagong
1   MUHAMMAD A. JALIL   Male    1987    33   Chittagong

If you are looking for code, your question is off-topic here. Are you looking for code or for some guidance on how to do it?

A typical clustering algorithm is k-means (and not k-NN, i.e. k-nearest neighbours, which is primarily used for classification). There are other clustering algorithms, such as hierarchical clustering algorithms. sklearn provides functions that implement k-means (and an example), hierarchical clustering algorithms, and other clustering algorithms.
To assess the quality of the produced clusters, you could use the silhouette method (sklearn provides a function that can be used to compute the silhouette score).
Regarding your specific data frame, note that it contains repetitions, so you may want to remove them before starting the clustering procedure. Also, the IDs are not unique, but you probably don't need the IDs for clustering.

how can I cluster this.give me the code for my data frame please

@t This site is not appropriate for providing code. We are not here to solve your issues, but to help you. Also, note that we are volunteers. Anyway, maybe later I will provide some code to illustrate how you could do that. Meanwhile, I provided many links to useful articles that should help you.

Ok, no problem. I am a beginner that's why I face the problem.I will follow your link

Yes you can use KNN algorithm to cluster (well actually its a classification not a clustering if you use KNN) the data. But, first you need to set one feature as a label because KNN is a supervised learning method, it need a labeled data to train the data first. For example you can use Gender as label to classify the data. To determine the quality of the classification result, you can simply use accuracy.
If you don’t want to use a label, you can use unsupervised learning method like K-Means to do the clusters. Because its unsupervised it doesn’t need label so you can use all of the feature to do the clusters task. For the k-means algorithm you can use a library from scikit-learn or create it from scratch. To evaluate the results you can use silhouette score or elbow method (to find the optimal number of cluster).
And don’t forget to do data exploration because maybe it can increase the quality of the cluster results.
You can learn more about the differentiation between K-Means and KNN in the link below:
https://pythonprogramminglanguage.com/how-is-the-k-nearest-neighbor-algorithm-different-from-k-means-clustering/
I hope this helps :)

KNN can be used in clustering with the data frame. but there are a number of steps that you must take.
1. You must separate the features you want to cluster. for example you can do clustering dob and age.
2. if there is data of type string you have to change it to an integer.
For easier clustering, you can use the Sklearn library. you can access at the following link https://scikit-learn.org/stable/modules/clustering.html

you can clustering the data frame with unsupervised algorithm, for example you can use K-Means method. There are some options you can choose to eliminate some features in your data frame, like del dataFrame['Column Name']. In unsupervised learning, the algorithm not calculate the quality of the clusters, but you can set it up by yourself to make a parameter for calculate the quality for each clusters, for example it depend on sum of data in each clusters. Actually you can use KNN algorithm with your data frame, but you need to add a label in there because KNN is a supervised learning, and its function to make a classification, not clustering. hope it useful.

You can use k-nn clustering but you must convert your dataset to numeric values or you can remove the unrelated features in your dataset.

There are several algorithm for clustering such as: K-means, Mean shift, hierarchical,etc. Based on my experience, actually it's K-means(KNN for classifcication).It is suitable for clustering your dataset, there are several steps for clustering your dataset:

You have to determine which features that you want cluster
Changing your categorical dataset to numerical
This step is optional, You can drop columns that are not related to the features you have chosen before
Try to to code your clustering (like determine centroid from your dataset, calculate the euclidean distance from your centroid,etc) or if you want to use library maybe sklearn is the right place.

And for determine the quality of your clustering, you can measures SSE(sum of the square error from the items of each cluster),Inter cluster distance,Intra cluster distance for each cluster,Maximum Radius,Average Radius.


text:
What are standard datasets for fully connected neural networks?
I am looking for datasets that are used as a testing standard in the fully connected neural networks (FCNN). For example, in the image recognition and CNN, CIFAR datasets are used in most of the papers, but can't find anything like that for the FCNN.

You can use MNIST obviously but I'd also suggest you have a look at UC Irvine's datasets: https://archive.ics.uci.edu/ml/datasets.php

This answer doesn't provide a declaration as to which dataset(s) are used quasi-ubiquitously in research/literature. It simply provides a frame-of-reference for where to look for structured datasets and examples of two structured datasets that could be used in general.

You want to look for structured datasets.
Good examples of this are things like housing price datasets.
Check out Google Datasets (specifically the datasets hosted by Kaggle). Many of these are structured data datasets.
As an answer directly though, try a housing price dataset like the Boston Housing Price Dataset.
You could also use the famous Titanic Dataset on Kaggle.


text:
Is there any way to apply linear transformations on a vector other than matrix multiplication?
I am trying to optimize the cost function calculation in regression analysis using a non-matrix multiplication based approach.
More specifically, I have a point $x = (1, 1, 2, 3)$, to which I want to apply a linear transformation $n$ times. If the transformation is denoted by a $4 \times 4$ matrix $A$, then the final transformation would be given by $A^n * x$.
Given that matrix multiplication can computational expensive, is there a way we can speed up the computation, assuming we would need to run multiple iterations of this simulation?

You can sometimes exploit the structure of your matrix to perform faster matrix multiplication. For example, if your matrix is sparse (or dense), there are algorithms that exploit this fact.
In your case, you can actually compute $A^n$ in less time than $\mathcal{O}(n^3$). For example, have a look at this question at CS SE and this one at Stack Overflow (SO). Note that the provided solutions may not be numerically stable, so I am not suggesting you use them.
Moreover, if you perform your operations on the GPU, they could be faster in practice. See e.g. this question at SO and this one at SciComp SE.


text:
How can I draw a Bayesian network for this problem with birds?
I am working on the following problem to gain an understanding of Bayesian networks and I need help drawing it:

Birds frequently appear in the tree outside of your window in the morning and evening; these include finches, cardinals and robins. Finches appear more frequently than robins, and robins appear more frequently than cardinals (the ratio is 7:4:1). The finches will sing a song when they appear 7 out of every 10 times in the morning, but never in the evening. The cardinals rarely sing songs and only in the evenings (in the evening, they sing 1 of every 10 times they appear). Robins sing once every five times they appear regardless of the time of day. Every tenth cardinal and robin will stay in the tree longer than five minutes. Every fourth finch will stay in the tree longer than five minutes.

I have tried drawing two versions of the network and would love some feedback. Currently, I am leaning more towards the right side network.

Hi and welcome to AI SE! Where did you get that problem from?

i like the second version of the artificial neural network.


text:
Are Q values estimated from a DQN different from a duelling DQN with the same number of layers and filters?
I am confused about the Q values of a duelling deep Q network (DQN). As far as I know, duelling DQNs have 2 outputs

Advantage: how good it is to be in a particular state $s$
Value: the advantage of choosing a particular action $a$

We can make these two outputs into Q values (reward for choosing particular action $a$ when in state $s$) by adding them together.
However, in a DQN, we get Q values from the single output layer of the network.
Now, suppose that I use the same DQN model with the very same weights in my input and hidden layers and changing the output layer which gives us Q values to advantage and value outputs. Then, during training, if I add them together, will it give me the same Q value for a particular state, supposing all the parameters of both my algorithms are the same except for the output layers?

Dueling-DQN has different network architecture comparing to vanilla DQN, so I don't think your version will work as well as the Dueling architecture.

From Wang et al., 2016, Dueling Network Architectures for Deep Reinforcement Learning

On the other hand, since we only have the target Q-value, separating the Q-value into state value and advantage result in the identifiability issue. That is the network might simply learn $V(s)=0$, $A(s,a)=Q(s,a)$ for every state.
To tackle with this issue, we should force an additional constraint on the advantage estimate. We can simply use the equation below as mentioned in the paper, that is, normalize the advantages across actions before combining with the state value.
$$Q(s,a;\theta,\alpha,\beta)=V(s;\theta,\beta)+(A(s,a;\theta,\alpha)-\frac{1}{|A|}\sum\limits_{a'}A(s,a';\theta,\alpha))$$


text:
Why does the growth function need to be polynomial in order for the learning algorithm to be consistent?
Could someone please explain to me why in VC theory, specifically, when calculating the VC dimension, the growth function needs to be polynomial in order for the learning algorithm to be consistent? Why polynomial, and where does the name growth function come from exactly?

I am kinda new to the field, so I am not familiar with such lemma. Yea, agnostic PAC. If I understood correctly, ERM is consistent (meaning all functions in function class F will converge to 0 as number of training samples approaches infinite) if and only if VC dimension of function class F is finite.

@StefanRadonjic Hi and welcome to AI SE :) I suggest you provide a link to the source that states what you're asking and the definition of "consistent" that you're using. I would also suggest you ask the question "where does the name growth function come from exactly?" in a separate post (because that's a distinct question that deserves its own post, in my opinion).

Hello! Thank you for your suggestion, however I've manage to partially answer this question. I although have a new one which I will post shortly with source and everything. Thank you!

@StefanRadonjic Ok, please, then provide a formal answer to your question below. Future readers would benefit from this answer! Btw, you need to tag me with @nbro so that I receive a notification of your message, otherwise, I may not see your messages. It was just luck that I saw your comment above!

@nbro Alright. I will be sure to do that as soon as possible. Once again, thank you for all of the information.


text:
Are there any training datasets using standard font text rather than hand written ones?
Are there any training datasets using standard font text rather than handwritten ones?
I tried using the MNIST handwritten one on font based chars, but it didn't work well.

@DuttaA i guess i was just being lazy. Seems like a lot of work, many different fonts, in the correct format for keras


text:
Is possible to train a robot or AI to prune fruit trees?
I live in a rural area where there is a growing necessity for people with knowledge to prune Pear trees, this process is crucial for the industry, but as people go to the big cities, this skill is being lost, and in a few years there will be no one to do it.
I wanna know if it is possible to train a robot using AI to do this, and want would it take to make this work!
Keep in mind that this would be viable only in an "industrial" way. The trees all have approximately the same size and are disposed in a certain preset way (distance between each other, height, etc).

I am not an expert in robotics (definitely not in pear trees pruning either) but I will try to give some hints to partially answer and also to help reframe the problem a bit. On overall I'll give already an answer: it is most likely possible, but also most likely not convenient.
Problem statement
First things first: in general the rule is that machine learning should be applied when a task can't be automatised otherwise. Before asking if an AI can be trained to solve a task one should always think how to automatise a task using a rule based system. This is important especially because during the process of thinking how to automatise the task you will realise that some steps can be perform without an expert, while others can't be performed without any supervision. Let's brake down your task in subtasks: a system that prune trees should be (at least) capable of:

Moving between trees
Selecting branches to cut
Cut the selected branches

Selecting the branch to cut is probably the step that requires most of the know how and expert supervision and for which a machine learning component might be suitable. Moving instead is a perfect example of a subtasks that could be tackled at different levels. Creating a machine able to anticipate other objects movements and avoid them in real time definitely require to train an AI component, but when you say that the environment is highly structured (trees disposed in a grid) this make me think that maybe some hand coded rules would do the trick without bothering machine learning.
Theoretical tools
Once you have understood which subtasks your machine should be capable of solving, you can start dig on the theoretical feasibility of them. Following the same order as in the previous paragraph:

Self-driving is a widely studied topics, the algorithms used to train robotic vacuum cleaners to move automatically in a house could be applied straight away to the problem of training an agent to move between trees.

Selecting the right branches involves mostly computer vision. This sub-task should actually be also dived again into sub tasks: detecting branches from other objects and select which ones should be cut. Nevertheless, the field is quite huge and training two models able to perform both actions is, in my opinion, doable.

Cutting a branch is probably harder than driving in this situation, because of the small movements that might be required to reach branches positioned in difficult spots. Anyway, it is possible to train robots to perform fine-grained movements (for a funny example see: Robot learn to flip pancakes).

Again it depends also on how high your expectation are about the final machine/system. Should the system have a surgical precision or could it risks to brake few extra branches in hard situations? Obviously the higher the expectations the harder it would be to make everything work harmoniously.
Resources required
Last but not least you also need to understand if what you're trying to do is feasible in reality and not just in theory. A big problem when it comes to use machine leaning is that these models can be trained only with huge amount of data.

Train an artificial agent to move in an environment can be done by reproducing the environment in an artificial simulator, which is good news. A single guy with a laptop could potentially do the job.

Collecting data to train a model able to detect branches on photo and then select which one should be cut will be highly tedious, and also expensive because data need also to be labelled by experts. Which means that some experienced people will have to take photos, in the order of tens of thousands at least, and write for each photo (using a software) which are the branches they will cut if they were working for real on that tree. I strongly doubt that a dataset like this already exists.

Training a robotic arm to cut branches will also be challenging. Despite the fact that also in this case simulators could be leveraged, the task is inherently harder and this comes with bigger difficulties (for example in designing a proper reward function if using reinforcement leaning). Concretely this mean more time to spend in research and testing.

Consider also that the success of the final model trained for each subtask would be not guaranteed at all, reason why I said at the beginning that training a system with AI modules would be probably not convenient, and that the best thing is always to try to create a rule based system first.


text:
After having selected the best model with cross-validation, for how long should I train it?
When using k-fold cross-validation in a deep learning problem, after you have computed your hyper-parameters, how do you decide how long to train your final model? My understanding is that, after the hyperparameters are selected, you train your model one more time on the entire set of data, but it's not clear to me when you decide to stop training.

Short answer: training "duration" or number of epochs/updates should be cross-validated too: you want to early-stop your training to prevent overfitting.
Longer answer:

Think of accuracy on the validation set as an estimate of accuracy on future data, given the value of some hyperparameter. In this case, the hyperparameter of interest is the number of training epochs. So: for each CV fold, train the network (e.g. up to some maximum number of epochs). After each epoch, record accuracy on the validation set. Compute the average validation set accuracy (across CV folds) for each number of epochs. Choose the number of epochs that maximizes this value.

https://stats.stackexchange.com/questions/298084/k-fold-cross-validation-for-choosing-number-of-epochs

I have some problems with this but I'm having a hard time articulating them. I think at its core given the nature of SGD I don't completely trust this to be correct.


text:
Could zero-padding affect learning in a negative way?
I implemented an LSTM with Keras to perform word ordering task (given a syntactically unordered sentence, the goal is to label each word of the sentence with the right position in this one.)
So, my dataset is composed by numerical vectors and each numerical vector represents a word.
I train my model trying to learn the local order of a syntactic subtree composed by words that have syntactic relationships (for example, a subtree could be a set of three words in which the root is the verb and children are subject and object relationship).
I padded each subtree to a length of 20, which is the maximum subtree length that I found in my dataset. With padding introduction, I inserted a lot of vectors composed of only zeros.
My initial dataset shape is (700000, 837), but knowing that Keras wants a 3D dataset, I reshaped it to (35000, 20, 837) and the same for my labels (from 700000 to (35000, 20)).
As loss function, I'm using the ListNet algorithm loss function, which takes a list of words and for each computes the probability of the element to be ranked in the first position (then ranking these scores, I obtain the predicted labels of each word).
The current implementation is the following:
model = tf.keras.Sequential()
model.add(LSTM(units=100, activation='tanh', return_sequences=True, input_shape=(timesteps, features)))
model.add(Dense(1, activation='sigmoid'))

model.summary()

model.compile(loss=listnet_loss, optimizer=keras.optimizers.Adam(learning_rate=0.00005, beta_1=0.9, beta_2=0.999, amsgrad=True), metrics=["accuracy"])

model.fit(training_dataset, training_dataset_labels, batch_size=1, epochs=number_of_epochs, workers=10, verbose=1, callbacks=[SaveModelCallback()])

And SaveModelCallback simply saves each model during training.
At the moment I obtain, at each epoch, very very similar results:
Epoch 21/50
39200/39200 [==============================] - 363s 9ms/step - loss: 2.5483 - accuracy: 0.8246
Epoch 22/50
39200/39200 [==============================] - 359s 9ms/step - loss: 2.5480 - accuracy: 0.8245
Epoch 23/50
39200/39200 [==============================] - 360s 9ms/step - loss: 2.5478 - accuracy: 0.8246

I have to questions:

Could zero-padding affect learning in a negative way? And if yes, how could we not consider this padding?
Is it a good model for what I have to do?


text:
Why can the Bellman equation be turned into an update rule?
In chapter 4.1 of Sutton's book, the Bellman equation is turned into an update rule by simply changing the indices of it. How is it mathematically justified? I didn't quite get the initiation of why we are allowed to do that?
$$v_{\pi}(s) = \mathbb E_{\pi}[G_t|S_t=s]$$
$$ = \mathbb E_{\pi}[R_{t+1} + \gamma G_{t+1}|S_t=s]$$
$$= \mathbb E_{\pi}[R_{t+1} + \gamma v_{\pi}(S_{t+1})|S_t=s]$$
$$ = \sum_a \pi(a|s)\sum_{s',r} p(s',r|s,a)[r+ \gamma v_{\pi}(s')]$$
from which it goes to the update equation:
$$v_{k+1}(s) =  \mathbb E_{\pi}[R_{t+1} + \gamma v_{k}(S_{t+1})|S_t=s]$$
$$=\sum_a \pi(a|s)\sum_{s',r} p(s',r|s,a)[r+ \gamma v_{k}(s')]$$

Check out this proof by Francisco S. Melo.

Why are we allowed to convert the Bellman equations into update rules?

There is a simple reason for this: convergence. The same chapter 4 of the same book mentions it. For example, in the case of policy evaluation, the produced sequence of estimates $\{v_k\}$ is guaranteed to converge to $v_\pi$ as $k$ (i.e. the number of iterations) goes to infinity. There are other RL algorithms that are also guaranteed to converge (e.g. tabular Q-learning).
To conclude, in many cases, the update rules of simple reinforcement learning (or dynamic programming) algorithms are very similar to their mathematical formalization because algorithms based on those update rules are often guaranteed to converge. However, note that many more advanced reinforcement learning algorithms (especially, the ones that use function approximators, such as neural networks, to represent the value functions or policies) are not guaranteed or known to converge.

Thank you for your answer. I understand that since the update rule version of the formula converges it is practical to use it. But my question is how we can use a formula which is not an update rule as an update rule (which converges)?

@SaeidGhafouri That formula is converted to an update rule because WE CAN DO THAT (and nobody prevents us from doing that), it's a NATURAL way of implementing that equation in an algorithm, and it turns out that this is the RIGHT thing to do in many cases (because algorithms based on such update rules converge!). There's no rule that tells you that equations need to be turned or not into update rules. We (humans) just do it because that's a natural way of implementing an algorithm that approximates that function.

Note that this is done in other areas of mathematics too. For example, consider the equation of a line. We can easily represent (or approximate) a line in a computer by expressing it in the same way as the mathematical equation.

To me, Bellman update is simply supervised learning: right hand side (bootstrap) is a sample of the left hand side (conditional expectation).. The Bellman equation simply explains that the right hand side is such a sample.

You're asking why the finite horizon policy evaluation converges to the infinite right?
Since the total reward is bounded(by the discount factor) you know that you can make your finite horizon policy evaluation get arbitrarily close to it in a finite number of steps.
People praise Bartos book but I find it annoying to read as he's not formal enough with mathematics.


text:
What do these numbers represent in this picture of a surface?
The following image is a screenshot from a video tutorial that illustrates the concept of gradient descent algorithm with a 3D animation.
Do the numbers on the top of the balls pointed out by the red arrows represent the gradient?

When you say "Are the numbers on the top of the balls are their gradient descent?", do you mean "gradient" (and not "gradient descent"), right? I think so, thus I edited your post to reflect this. Also, note that you were watching a video that has already 2 dislikes, so that may be a good sign that you shouldn't even watch that video.

@nbro With your reminder I realize that, thank you. Could you explain a bit more about the mistake in that video? For example, gradient usually ranges between (-1.0, 1.0), is my understanding right?

I am not saying that the video is showing the wrong info. The gradient is a vector and the values of the elements of this gradient (i.e. a vector) aren't really restricted to a range. The values depend on the loss function. I think you should investigate a little bit more about derivatives, gradients, etc., then ask a more specific question.

@nbro Thank you. My OP is asking a specific question and your comment to that question is appreciated. I guess I understand the basic idea about derivatives, gradients in general.  The range of (-1.0, 1.0) I use is specifically for the discussion in the context of deep learning, where all features are typically normalized. Please correct my misunderstanding. Sorry for misleading.

Well, yes, typically, you don't want the gradient to take very big or very small numbers, but this not guaranteed, and, as you say, it may also depend on the activation functions that you use. Have a look at the exploding and vanishing gradient problems.

@nbro I also appreciate your reminder for that video.I'd just like to know whether the video conveys something that might be misleading. If yes, which part it is. Neither like nor dislike helps my understanding of the particular concept in my OP, that is, the concept of gradient descent.

That video isn't very useful that's why it was disliked. I encourage you to look for other sources about GD. The gradient is a vector, but that animation doesn't show those points as vectors. Those points could represent the loss at specific values or they could represent the parameters of the loss function that minimize the loss function.

@nbro Got it. You gave me so much helpful information and leads. Thank you so much. Would you mind copy them to answer? I'll accept it.

Ok, I will do that (maybe a little later)!

It represents the value of the loss function J(x1, x2; θ); the valley has value 0 in the video. You can see that the lowest ball with value 3.13 is on a steep point with a high gradient, so it's not the gradient.


text:
How does normalization of the inputs work in the context of PPO?
What does the normalization of the inputs mean in the context of PPO? At each time step of an episode, I only know the values of this time step and of the previous ones, if I take track of them. This means that for each observation and for each reward at each time step I will do:
value = (value - mean) / std

before passing them to the NN, right? Specifically, I compute mean and std by keeping track of the values for the whole episode and at each time step, I add the new values to an array. Is this a valid approach?
Also, how can I handle negative rewards, such that being positive?


text:
What is the gradient of a non-linear SVM with respect to the input?
The objective function of an SVM is the following:
$$J(\mathbf{w}, b)=C \sum_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{w}^{t} \cdot \mathbf{x}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{w}^{t} \cdot \mathbf{w}$$
where

$\mathbf{w}$ is the model's feature weights and $b$ is its bias parameter
$\mathbf{x}^{(i)}$ is the $i^\text{th}$ training instance's feature vector
$y^{(i)}$ is the target class ($-1$ or $1$) for the $i^\text{th}$ instance
$m$ is the number of training instances
$C$ is the regularisation hyper-parameter

And if I was to use a kernel, this would become:
$$J(\mathbf{w}, b)=C \sum_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{u}^{t} \cdot \mathbf{K}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{u}^{t} \cdot \mathbf{K} \cdot \mathbf{u}$$
where the kernel can be the Gaussian kernel:
$$K(\mathbf{u}, \mathbf{v})=e^{-\gamma\|\mathbf{u}-\mathbf{v}\|^{2}}$$
How would I go about finding its gradient with respect to the input?
I need to know this as to then apply this to a larger problem of a CNN with its last layer being this SVM, so I can then find the gradient of this output wrt the input of the CNN.

if by max you mean the actual maximum between two values, you may be in trouble since the gradient of the maximum is not smooth and hence not differentiable. So you can only compute the gradient as long as you have 1-yy_pred !=0.

I found some work done on this here: http://www.ecmlpkdd2013.org/wp-content/uploads/2013/07/527.pdf
They talk about something similar in Section 3.1. Does it hold any weight with what I want to do?

@FeedMeInformation Have you found a solution to this problem?

See also https://ai.stackexchange.com/q/8281/2444.

Have you found the solution/answer to this problem/question? If yes, feel free to write it below. If not, I will consider starting a bounty on this post, if you still need an answer.


text:
To perform a white-box adversarial attack, would the use of a numerical gradient suffice?
I am trying to perform a white-box attack on a model.
Would it be possible to simply use the numerical gradient of the output wrt input directly rather than computing each subgradient of the network analytically? Would this (1) work and (2) actually be a white box attack?
As I would not be using a different model to 'mimic' the results but instead be using the same model to get the outputs, am I right in thinking that this would still be a white box attack.

What exactly do you mean by numerical gradient of the ouput? If you mean using finite differences, this is generally considered a black-box attack. Usually the 'white box attack' setting is assuming that the adversary has full access to the model. The 'black box attack setting' generally assumes either that you only have access to outputs, either in the form of logits or (even more restrictive) just the class prediction (i.e. argmax of logits)


text:
What is the intuition behind the dot product attention?
I am watching the video Attention Is All You Need by Yannic Kilcher.
My question is: what is the intuition behind the dot product attention?
$$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$
becomes:
$$A(Q,K, V) = \text{softmax}(QK^T)V$$

Let's start with a bit of notation and a couple of important clarifications.
$\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word.
$\mathbf{V}$ refers to the values vectors matrix, $v_i$ being a single value vector associated with a single input word.
$\mathbf{K}$ refers to the keys vectors matrix, $k_i$ being a single key vector associated with a single input word.
Where do these matrices come from? Something that is not stressed out enough in a lot of tutorials is that these matrices are the result of a matrix product between the input embeddings and 3 matrices of trained weights: $\mathbf{W_q}$, $\mathbf{W_v}$, $\mathbf{W_k}$.
The fact that these three matrices are learned during training explains why the query, value and key vectors end up being different despite the identical input sequence of embeddings. It also explains why it makes sense to talk about multi-head attention. Performing multiple attention steps on the same sentence produces different results, because, for each attention 'head', new $\mathbf{W_q}$, $\mathbf{W_v}$, $\mathbf{W_k}$ are randomly initialised.
Another important aspect not stressed out enough is that for the encoder and decoder first attention layers, all the three matrices comes from the previous layer (either the input or the previous attention layer) but for the encoder/decoder attention layer, the $\mathbf{Q}$ matrix comes from the previous decoder layer, whereas the $\mathbf{V}$ and $\mathbf{K}$ matrices come from the encoder. And this is a crucial step to explain how the representation of two languages in an encoder is mixed together.
Once computed the three matrices, the transformer moves on to the calculation of the dot product between query and key vectors. The dot product is used to compute a sort of similarity score between the query and key vectors. Indeed, the authors used the names query, key and value to indicate that what they propose is similar to what is done in information retrieval. For example, in question answering, usually, given a query, you want to retrieve the closest sentence in meaning among all possible answers, and this is done by computing the similarity between sentences (question vs possible answers).
Of course, here, the situation is not exactly the same, but the guy who did the video you linked did a great job in explaining what happened during the attention computation (the two equations you wrote are exactly the same in vector and matrix notation and represent these passages):

closer query and key vectors will have higher dot products.
applying the softmax will normalise the dot product scores between 0 and 1.
multiplying the softmax results to the value vectors will push down close to zero all value vectors for words that had a low dot product score between query and key vector.

In the paper, the authors explain the attention mechanisms saying that the purpose is to determine which words of a sentence the transformer should focus on. I personally prefer to think of attention as a sort of coreference resolution step. The reason why I think so is the following image (taken from this presentation by the original authors).

This image shows basically the result of the attention computation (at a specific layer that they don't mention). Bigger lines connecting words mean bigger values in the dot product between the words query and key vectors, which means basically that only those words value vectors will pass for further processing to the next attention layer. But, please, note that some words are actually related even if not similar at all, for example, 'Law' and 'The' are not similar, they are simply related to each other in these specific sentences (that's why I like to think of attention as a coreference resolution). Computing similarities between embeddings would never provide information about this relationship in a sentence, the only reason why transformer learn these relationships is the presences of the trained matrices $\mathbf{W_q}$, $\mathbf{W_v}$, $\mathbf{W_k}$ (plus the presence of positional embeddings).

Ive been searching for how the attention is calculated, for the past 3 days. Your answer provided the closest explanation. Thank you. If you have more clarity on it, please write a blog post or create a Youtube video. It'd be a great help for everyone.

@Nav Hi, sorry but I saw your comment only now. I'm not really planning to write a blog post on this topic, mainly because I think that there are already good tutorials and video around that describe transformers in detail. Also, I saw that new posts are share every month, this one for example is really well made, hope you'll find it useful: http://peterbloem.nl/blog/transformers

@Avatrin The weight matrices Eduardo is talking about here are not the raw dot product softmax wij that Bloem is writing about at the beginning of the article. The weight matrices here are an arbitrary choice of a linear operation that you make BEFORE applying the raw dot product self attention mechanism.
These can technically come from anywhere, sure, but if you look at ANY implementation of the transformer architecture you will find that these are indeed learned parameters. Bloem covers this in entirety actually, so I don't quite understand your implication that Eduardo needs to reread it

@TimSeguine Those linear layers are before the "scaled dot-product attention" as defined in Vaswani (seen in both equation 1 and figure 2 on page 4). They are however in the "multi-head attention". OPs question explicitly asks about equation 1. There are no weights in it.

@Avatrin Yes that's true, the attention function itself is matrix valued and parameter free(And I never disputed that fact), but your original comment is still false: "the three matrices W_q, W_k and W_v are not trained". Neither how they are defined here nor in the referenced blog post is that true.

@TimSeguine Okay, fair enough. I think you are right. I misread the notation Bloem used. While I still believe this answer is misleading, and that attention is better explained through the properties of dot products, my initial comment was wrong as well. I'll delete it.

@Avatrin Yeah, I can understand that. Even still, the additional weight matrices don't really change it that much. You can interpret them as a learnable choice of basis for the query, key and value spaces. But that logic also applies to the weight matrices of many different structures in the right light, so it is fair to say that it distracts more than it helps.

@EdoardoGuerriero. I wish I saw this before. Thank you

@EdoardoGuerriero Do the 3 projection matrices render the embedding (linear) layer redundant? (Performing project (linear) after embedding (linear) is just a single linear operation)

Clearly explanation. But where input embeddings come from?

One more question:  Since 3 W matrices are trained weights, then how can they come from random initialize?


text:
How to produce documents like factset blackline?
Factset blackline reports essentially can compare two 10-Q SEC filings and show you the difference between the two documents. It highlights added items in green and removed items in red + strikethrough (essentially, it's a document difference, but longer-term I would like to run algorithms on the differences).
I don't care to change colors, but what I would like to do is to produce similar extracts that summarize addition and deletions.
Which AI/ML algorithm could do the same?

Financial information companies spend a very large amount of effort to do this kind of thing properly, and their models are generally proprietary, so the real answer is not something you'll be able to get on a public site.
Of course, you can achieve a very simple approximation with a literal diff on the texts, and a more sophisticated one by using out-of-the-box NLP or computer vision techniques, but you're probably not going to get close to the performance of factset or other competitors using those tools.


text:
How to reduce fluctuation of a neural network?
I've modeled an AlexNet neural network, with 50 epochs and a batch size of 64. I used a stochastic gradient descent optimizer with a learning rate of 0.01. I attached the train and validation loss and accuracy plots.

How can I reduce the fluctuation of the first epochs?


text:
How can I reduce combinatorial explosion in an MCTS-like algorithm for program induction?
I'd like to develop an MCTS-like (Monte Carlo Tree Search) algorithm for program induction, i.e. learning programs from examples.
My initial plan is for nodes to represent programs and for the search to expand nodes by revising programs.
Many of these expansions revise a single program: randomly resample a subtree of the program, replace a constant with a variable, etc. It looks straightforward to use these with MCTS.
Some expansions, however, generate a program from scratch (e.g. sample a new program). Others use two or more programs to generate a single output program (e.g. crossover in Genetic Programming).
These latter types of moves seem nonstandard for vanilla MCTS.
One idea I've had is to switch from nodes representing programs to nodes representing tuples of programs. The root node would represent the empty tuple $()$, to which expansions could be applied only if they can generate a program from scratch. The first such expansion would produce some program $p$, so the root would now have child $(p)$. The second expansion would produce $p'$, so the root would now also have child $(p')$ as well as the pair $(p, p')$. Even assuming some reasonable restrictions (e.g. moves can use at most 2 programs, pairs cannot have identical elements, element order doesn't matter), the branching factor will grow combinatorially.
What techniques from the MCTS literature (or other literatures) might reduce the impact of this combinatorial explosion?

I think you may get some inspiration from the work on deterministic environments by Javier Segovia et al. See their paper Computing programs for generalized planning using a classical planner (2019).
If you don't have access to Elsevier's papers, I recommend that you check out the first author's profile on dblp.org. From there you will find links to open access versions of the conference paper that led up to the publication above.

Can you explain why that paper may be useful to answer the second question?

That paper targets a very specific class of programs and shows effective heuristic and search strategies to scale up with program size to some extent.


text:
Weird border artifacts when training a CNN
I've been trying to use this DeepLabv3+ implementation with my dataset (~1000 annotated images of the same box, out of the same video sequence): https://github.com/srihari-humbarwadi/person_segmentation_tf2.0
But I get border artifacts like this:

Any ideas what could be causing it? Note that if I use bigger batches and train for more epochs, the borders tend to get thinner but never disappear. They also appear randomly around the image.
Any clues what could be causing them and how to solve it?


text:
What happens if an agent has two contrasting utility functions?
What would happen if an agent were programmed with two utility functions which are one the opposite of the other? Would one prevail or will they cancel each other out? Are there studies in this direction?


text:
Is there a simple proof of the convergence of TD(0)?
Does anybody know a simple proof of the convergence of the TD(0) value function prediction algorithm?

As far as I know, there is no very simple proof of the convergence of temporal-difference algorithms. The proofs of convergence of TD algorithms are often based on stochastic approximation theory (given that e.g. Q-learning can be viewed as a stochastic process) and the work by Robbins and Monro (in fact, the Robbins-Monro conditions are usually assumed in the theorems and proofs).
The proofs of convergence of Q-learning (a TD(0) algorithm) and SARSA (another TD(0) algorithm), when the value functions are represented in tabular form (as opposed to being approximated with e.g. a neural network), can be found in different research papers.
For example, the proof of convergence of tabular Q-learning can be found in the paper Convergence of Stochastic Iterative Dynamic Programming Algorithms (1994) by Tommi Jaakkola et al. The proof of convergence of tabular SARSA can be found in the paper Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms (2000) by Satinder Singh et al.
See also How to show temporal difference methods converge to MLE?.


text:
Can the degree and minimum remaining values heuristics be used in conjunction?
I am currently studying constraint satisfaction problems and have come across two heuristics for variable selection. The minimum remaining values(MRV) heuristic and the degree heuristic.
The MRV heuristic tells you to choose a variable that has the least legal assignments, while the degree heuristic tells you to choose a variable that has the biggest effect on the remaining unassigned variables.
Can these 2 heuristics for variable selection be used in conjunction with each other? Some books say that degree heuristic can be used as first selection of variables. Which heuristic is better followed after that? The MRV or degree heuristic?

The picture shows the case where degree heuristic is used. If MRV were to be used, then at the 3rd step, the leftmost side of the map would be coloured blue.


text:
What is the difference between an generalised estimating equation and a recurrent neural network?
What is the difference between a generalised estimating equation (GEE) model and a recurrent neural network (RNN) model, in terms of what these two models are doing? Apart from the differences in the structure of these two models, where GEE is an extension of generalised linear model (GLM) and RNN is of a neural network structure, it seems to me that these 2 models are doing the same thing?

Can you provide a link to a reference that describes "generalised estimating equation (GEE)"?

What do you understand each of these to be doing? An RNN is a general function approximator, which can compute sample level targets from input data, whereas a GEE estimates population level effects.


text:
What is the best way to detect and recognize traffic signs in a picture?
I'm working on a project for my college to recognize traffic signs in pictures. I searched a lot but can't find the best method to do it.
Can someone recommend me a paper, article, or even GitHub link that describes the best way to achieve this? It will be helpful.

Here are some articles, the first three include code:

Deep Learning for Traffic Signs Recognition, by Moataz Elmasry, April 2, 2018
Traffic Sign Classification with Keras and Deep Learning, by Adrian Rosebrock, November 4, 2019
Traffic Sign Detection using Convolutional Neural Network, by Sanket Doshi, September 1, 2019
AI to manage road infrastructure via Google Street View, June 19, 2019

Maybe this is helpful:
Recognising Traffic Signs With 98% Accuracy Using Deep Learning, by Eddie Forson.
Greetings
Mario


text:
Which neural network can approximate the function $y = x^2 + b$?
I am new to ANN. I am trying out several 'simple' algorithms to see what ANN can (or cannot) be used for and how. I played around with Conv2d once and had it recognize images successfully. Now I am looking into trend line analyses. I have succeeded in training a network where it solved for linear equations. Now I am trying to see if it can be trained to solve for $y$ in the formula $y = b + x^2$.
No matter what parameters I change, or the number of dense layers, I get high values for loss and validation loss, and the predictions are incorrect.
Is it possible to solve this equation, and with what network? If it is not possible, why not? I am not looking to solve a practical problem, rather build up understanding and intuition about ANNs.
See the code I tried with below
#region Imports
from __future__ import absolute_import, division, print_function, unicode_literals
import math
import numpy as np
import tensorflow as tf
from tensorflow.keras import models, optimizers
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Activation, Dropout, Flatten, Dense, Lambda
import tensorflow.keras.backend as K
#endregion

#region Constants
learningRate = 0.01
epochs: int = 1000
batch_size = None
trainingValidationFactor = 0.75
nrOfSamples = 100
activation = None
#endregion

#region Function definitions
def CreateNetwork(inputDimension):
  model = Sequential()
  model.add(Dense(2, input_dim=2, activation=activation))
  model.add(Dense(64, use_bias=True, activation=activation))
  model.add(Dense(32, use_bias=True, activation=activation))
  model.add(Dense(1))
  adam = optimizers.Adam(learning_rate=learningRate)
  # sgd = optimizers.SGD(lr=learningRate, decay=1e-6, momentum=0.9, nesterov=True)
  # adamax = optimizers.Adamax(learning_rate=learningRate)
  model.compile(loss='mse', optimizer=adam)
  return model

def SplitDataForValidation(factor, data, labels):
  upperBoundary = int(len(data) * factor)

  trainingData = data[:upperBoundary]
  trainingLabels = labels[:upperBoundary]

  validationData = data[upperBoundary:]
  validationLabels = labels[upperBoundary:]
  return ((trainingData, trainingLabels), (validationData, validationLabels))

def Train(network, training, validation):
  trainingData, trainingLabels = training
  history = network.fit(
    trainingData
    ,trainingLabels
    ,validation_data=validation
    ,epochs=epochs
    ,batch_size=batch_size
  )

  return history

def subtractMean(data):
  mean = np.mean(data)
  data -= mean
  return mean

def rescale(data):
  max = np.amax(data)
  factor = 1 / max
  data *= factor
  return factor

def Normalize(data, labels):
  dataScaleFactor = rescale(data)
  dataMean = subtractMean(data)

  labels *= dataScaleFactor
  labelsMean = np.mean(labels)
  labels -= labelsMean

def Randomize(data, labels):
  rng_state = np.random.get_state()
  np.random.shuffle(data)
  np.random.set_state(rng_state)
  np.random.shuffle(labels)

def CreateTestData(nrOfSamples):
  data = np.zeros(shape=(nrOfSamples,2))
  labels = np.zeros(nrOfSamples)

  for i in range(nrOfSamples):
    for j in range(2):
      randomInt = np.random.randint(1, 5)
      data[i, j] = (randomInt * i) + 10
    labels[i] = data[i, 0] + math.pow(data[i, 1], 2)

  Randomize(data, labels)
  return (data, labels)
#endregion

allData, allLabels = CreateTestData(nrOfSamples)
Normalize(allData, allLabels)
training, validation = SplitDataForValidation(trainingValidationFactor, allData, allLabels)

inputDimension = np.size(allData, 1)
network = CreateNetwork(inputDimension)

history = Train(network, training, validation)

prediction = network.predict([
  [2, 2], # Should be 2 + 2 * 2 = 6
  [4, 7], # Should be 4 + 7 * 7 = 53
  [23, 56], # Should be 23 + 56 * 56 = 3159
  [128,256] # Should be 128 + 256 * 256 = 65664
])
print(str(prediction))

$f(x) = x^2 + b$ is a polynomial (more precisely, a parabola) so it is continuous, thus, a neural network (with at least one hidden layer) should be able to approximate that function (given the universal approximation theorem).
After a very quick look at your code, I noticed you aren't using an activation function for your dense layers (i.e. your activation function is None). Try to use e.g. ReLU.

Thank you @nbro for your answer. I will test more, with different activation functions, including ReLU. +1 for pointing out the universal approximation theorem, of which I was not aware yet. That is really helpful to know.

if you know in advance that your data is polynomial and you want to fit the best coefficient, then this is an easier (classical machine learning) problem .
you would just assume a maximum degree e.g $y_{pred} = a_0+a_1.x+a_2x^2+...+a_n.x^n  $
you can just initialize $a_i$s and consider MSE loss $(y_{groundtruth}-y_{pred})^2$.
then derivative with respect to $a_i$s is $2(y_{groundtruth}-y_{pred}).(- d(y_{pred})/d(a_i)) = 2(y_{groundtruth}-y_{pred}).(- x^i)$
so having access to the gradinet, you can easily do gradient descent which is trivial.

So I assume this (the general form of the data generator) is not known.
in this case as you did, we assume a general form and let neural network learn it. it is important to note that you should limit the support (range) of your data by normalization as you did by normalization because it is impossible to find an approximation which holds true every where. but in a small range of [0,1] it is much easier to approximate the desired function.

in general it is possible to approximate this function but on a limited range e.g on [0,1], unless we have the correct general form (in this case polynomial) and then just fit the correct parameters using gradient descent or other methods.
I hope these two cases shed some lights on your question


text:
How to estimate the error during training in deep reinforcement learning
How do I calculate the error during the training phase for deep reinforcement learning models?
Deep reinforcement learning is not supervised learning as far as I know. So how can the model know whether it predicts right or wrong? In literature, I find that the "actual" Q-value is calculated, but that sounds like the whole idea behind deep RL is obsolete. How could I even calculate/know the real Q-value if there is not already a world model existing?

Yes, reinforcement learning is very different from supervised learning, the policy (what you call a model) does not know if its predicting right or wrong, or if its taking the correct action or not. In RL there is no concept of "the right action", everything is evaluated through the reward function.
Also there are no ways to compute the ground truth Q-values, if you had that then there is no need to do RL.
In RL you should not think like in supervised learning, there are no error metrics, everything is evaluated on how much accumulated reward the agent receives over an episode.

Actually, RL and SL aren't so different or unrelated. In fact, every SL problem can be cast as an RL one. However, your suggestions are usually valid. Anyway, I think you should address more directly the question "How could I even calculate/know the real Q-value if there is not already a world model existing?" by showing that you don't need to know the true value e.g. in Q-learning.

@nbro I did not claim that RL and SL were unrelated, but they are very different. Try doing the opposite, casting a RL problem as a supervised learning one, and you will see the difference. In RL you do not have access to "the right answer".

Yes, I agree with you. I just wanted to point out that they aren't completely unrelated, as one may interpret when reading your first sentence.

So, how are the weights updated then? Given a state, when the neural net leads just to a random action, how will the best policy be learned over time?

@MScott That is a question on its own, there are many techniques to do it (Q-Learning, Policy gradients, etc), a neural network is just used as a function approximation inside those frameworks.


text:
Does apprenticeship learning require prospective data?
I am thinking of applying apprenticeship learning on retrospective data. From looking at this paper by Ng https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf which talks about apprenticeship learning, it seems to me that at the 5th step of the algorithm,

Compute (or estimate) $μ^{(i)}$ = $μ(π^{(i)})$,
where $\mu^{(i)}$ = $E[\sum_{t=0}^{∞}\gamma^{t}$$\phi(s_{t})$ | $\pi^{(i)}]$, $\phi(s_{t})$ is the reward feature vector at state $s_t$.

From my understanding, a sequence of $s_0, s_1, s_2 ..$ trajectory would have to be generated at this step, following this policy $\pi^{(i)}$. Hence, applying this algorithm on retrospective data would not work?

This is a relatively old question, but by "retrospective data" or "prospective data" (as in the title), do you mean data generated with previous policies? I suggest that you edit your post and use more common terms to clarify your question.


text:
Why is the loss of one of the outputs of a model with multiple outputs increasing while the others are decreasing?
I'm a newbie in neural networks. I'm trying to fit my neural network that has 3 different outputs:

semantic segmentation,
box mask and
box coordinates.

When my model is training, the loss of semantic segmentation and box coordinates are decreasing, but the loss of the box mask is increasing too much.
My neural network is a CNN and it's based on Chargrid from here. The architecture is this:

For semantic segmentation outputs, it's expected to have 15 classes, for box mask it's expected to have 2 classes (foreground vs background) and for box coordinates it's expected to have 5 classes (1 for each corner of bounding box + 1 for None).
Loss details
Step 1
Here's the loss/accuracy for each of the three outputs at the end of the first step.

Semantic segmentation (Output 1) - 0.0181/0.958
Box mask (Output 2) - 13.88/0.946
Box coordinates (Output 3) - 0.2174/0.0000000867

Last step
Here's the loss/accuracy at the last step.

Semantic segmentation (Output 1) - 0.0157/0.963
Box mask (Output 2) - 73.02/0.935
Box coordinates (Output 3) - 0.06/0.82

Is that normal? How can I interpret these results?
I will leave below, the output of the model fit.

Hi, you ought to be a bit more specific and provide some more details, otherwise nobody will be able to answer your question. Also, the images are pretty much unreadable, so a summary as text would be better.

@OliverMason what do you mean about being more specific?

Give us more detail about your data, architecture, etc

@OliverMason is this better now?


text:
Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working?
I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy)
$$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max_{a'}Q(s',a')-Q(s,a))$$
I am using an epsilon greedy policy.
In this environment, we get a reward only if we succeed. So I explored with 100% until I have 50 successes. Then I saved the data of failures and success in different bins. Then I sampled (with replacement) from these bins and used them to train the Q network. However, no matter how long I train the agent doesn't seem to learn.
The code is available in Colab. I am doing this for a couple of days.
PS: I modified the code for SARSA and Expected SARSA; nothing works.

In general it is quite hard to do code reviews and bug hunts, so you may not get any response. To improve your chances you should give more context on the site, so your question does not rely on people going off site and getting involved in your project. For instance, what diagnostics have you collected, and what are the results? Have you observed more than "the agent doesn't seem to learn" that is worth sharing? What is your state representation for input to the NN, and what its the rough NN architecture - layer sizes, output activation function, loss function etc

Frozen lake is somewhat harder to solve than expected since it's a very stochastic environment. You will have a hard time solving it with DQN. Try solving it with regular Q learning. Epsilon-greedy isn't ideal because you will get stuck in local optima and you won't get out since exploration will diminish. Try using UCB strategy which is generally better than e-greedy for tabular methods. Also, resetting counters completely after certain amount of episodes in UCB helps in this case.

@Brale_ I tried normal Q learning. It did not train. In any case i will also try UCB.

@NeilSlater I  used NN with 2-hidden layers (size = 50)+'relu' activations which outputs 4 values(one for each action). I used Adam with a constant learning rate for minimizing MSE error. I did not use any features to represent the state. I just used the state as the input to the NN.

@kosa If your regular Q learning algorithm does not work, then your DQN has no chance to work either. The neural network in DQN is just a storage device to replace the table for the Q-factors if you run out of memory. My hunch is that your target in test_fun is the real culprit. Currently it's just $Q^{new}$, it should be $(1-\alpha)Q^{old}+\alpha Q^{new}$. Also, frozen lake is a very simple problem (compared to Starcraft or Go), you won't need 2 hidden layers.

@HaiNguyen, I will try with this update rule. I make a wide network with a single layer.

@HaiNguyen "If your regular Q-learning algorithm does not work, then your DQN has no chance to work either." Technically this isn't true. For example, consider Atari 2600 games like Pong that DQN can play but Q-learning cannot.

I see at least 3 issues with your DQN code that need to be fixed:

You should not have separate replay memories for successes/failures. Put all of your experiences in one replay memory and sample from it uniformly.
Your replay memory is extremely small with only 2,000 samples. You need to make it significantly larger; try at least 100,000 up to 1,000,000 samples.
Your batch_target is incorrect. You need to train on returns and not just rewards. In your train function, compute the 1-step return $r + \gamma \cdot max_{a'} Q(s',a')$, remembering to set $max_{a'} Q(s',a') = 0$ if $s'$ is terminal, and then pass it to model.fit() as your prediction target.

The biggest remaining issue is probably this: "I just used the state as the input to the NN". Most neural networks are going to have poor performance handling unless the state values are in a nice range.

@NeilSlater I think is an issue. I will normalize the states.

@Brett I separated them because of two reasons: (1) We get a reward = 1.0 only if we are successful. Until we have enough data on successes, the updates on Q(s, a) are redundant.   (2) Moreover, the distribution of successes vs failures is uneven. However, in the end I just sampled from these bins with replacement, to a new bin, and trained on the Q function using the data in the new bin.  PS: I have tried your way, it was not working at all. So, I came up with this implementation.

@Brett I think I am following the same update rule. Please see the "test_fun" method in the Q_learn class.

@kosa Even though those updates are redundant, they still need to be put into the replay memory because they change the probability of those samples being chosen. It's OK if they're uneven because they reflect the true probabilities of encountering those (s,a,r,s') transitions in the environment. This makes sure your Q-function is an unbiased estimate of the true returns.

@kosa Your "test_fun" method looks correct now. If you make my other proposed changes and it is still not working, then it's possible you need to make additional changes like what NeilSlater suggested, or there could still be other bugs. Also keep in mind that DQN will not be successful on all environments. Maybe you should test your code on CartPole and see if it can solve that to help you debug.

Why should you sample uniformly from all memories? In the Cartpole environment I've experienced catastrophic forgetting every time I did it and the solution I did (which worked) was to only sample from the last x steps from every failing episode.


text:
Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?
Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?
If not, why do they perform as well as networks which use max-pooling?

Convolution is translationally invariant

It's not. The output of a convolutional layer will be shifted.

Neural networks are not invariant to translations, but equivariant,
Invariance vs Equivariance
Suppose we have input $x$ and the output $y=f(x)$ of some map between spaces $X$ and $Y$. We apply transformation $T$ in the input domain. For general map,output will change in some complicated and unpredictable way. However, for certain class of maps, change of the output becomes very tractable.
Invariance means that output doesn't change after application of the map $T$. Namely:
$$
f(T(x)) = f(x)
$$
For CNN example of the map, invariant to translations, is the GlobalPooling operation.
Equivariance means that symmetry transformation $T$ on the input domain leads to the symmetry transformation $T^{'}$ on the output. Here $T^{'}$ can be the same map $T$, identity map - which reduces to invariance, or some other kind of transformation.
This picture is illustration of translational equivariance.

Equivariance of operations in CNN

Convolutions with stride=1:
$$ f(T(x)) = T f(x)
$$
Output feature map is shifted in same direction and number of steps.
Downsampling operations. Convolutions with stride=1, Pooling (non-global):
$$ f(T_{1/s}(x)) = T_{1/s} f(x)
$$
They are equivariant to the subgroup of translations, which involves translations with integer number of strides.
GlobalPooling :
$$ f(T(x)) = f(x)
$$
These are invariant to arbitrary shifts, this property is useful in classification tasks.

Combination of layers
Stacking multiple equivariant layers you obtain equivariant architecture a whole.
For classification layer it makes sense to put GlobalPooling in the end in order to for NN to output the same probabilities for the shifted image.
For segmentation or detection problem architecture should be equivariant with the same map $T$, in order to translate bounding boxes or segmentation masks by the same amount as the transform on the input.
Non-global downsampling operations reduce equivariance to the subgroup with shifts integer multiples of stride.

Nice answer! Why though everyone says that is the pooling layers (MaxPooling) that help the network to achieve the translational invariance? See for example Figure 9.8 of the .

FCNs can and typically have downsampling operations. For example, u-net has downsampling (more precisely, max-pooling) operations. The difference between an FCN and a regular CNN is that the former does not have fully connected layers. See this answer for more info.
Therefore, FCNs inherit the same properties of CNNs. There's nothing that a CNN (with fully connected layers) can do that an FCN cannot do. In fact, you can even simulate a fully connected layer with a convolution (with a kernel that has the same shape as the input volume).

All convolutional networks (with or without max-pooling) are translation-invariant (AKA spatially invariant) because their filters slide over every position in the image. This means that if a pattern that "matches" a filter is present anywhere in the image, then at least one neuron should activate.
Max-pooling, on the other hand, has nothing to do with spatial invariance. It's simply a regularization technique to help reduce the number of parameters later in the network by downsizing activation layers within the network. This can help combat overfitting, although it's not strictly necessary. Alternatively, neural networks can achieve the same effect by using a convolutional layer with a stride of 2 instead of 1.

what? Max pooling specifically creates translation invariance by only outputting the max value over a subset of the input space. Fully convolutional networks does not have this so I'm curious how/if it retains translation invariant properties. There are some architectures which have a bunch of Convolutional layers with strides to reduce output size with a softmax or SVM at the output. It's not clear why this is translation invariant at all.

Your description of max pooling is correct but that is not a significant form of spatial invariance. The convolutional layers themselves induce spatial invariance through the sliding action of their filters, so max pooling is not always needed. I think this article explains the difference well. Specifically, see the section called Getting rid of pooling.

The convolution output is shifted if the input is shifted, it's output is not translation invariant. Max pooling specifically acts to produce translation invariance(at least according to the books I have read).

That is true, but if you shift the input then the output of max pooling can change too. For example, imagine I shift my input image by 1 pixel when I am using 2x2 max pooling with a stride of 2. I am curious if you remember which book says that, so I can check to make sure I understand correctly.

Does this Quora answer help? Looks like the OP is referring to the Deep Learning Book. It appears that both convolutional layers and max pooling provide (slightly different) forms of translation invariance. That might explain why you can remove max pooling from CNNs and still have translation invariance, like I said in my answer.

@BrettDaley Please, make sure you distinguish between translation invariance and equivariance and which parts of the CNN actually provide each of these properties.

@nbro Can you explain why pooling adds the translation invariance?

@FourierFlux As you go deeper into the network the receptive field increases, meaning that the same pattern can be detected even if it is translated at large amount. So translation invariant can be achieved even without pooling by just using convolutions.


text:
Which generative methods are better for generating graphs, while preserving node and edge labels?
I started to dig into the topic of graph generation and I have a question - which out of generative methods (autoregressive, variational autoencoders, GANs, any other?) are better for generating graphs while preserving both node and edge labels? (in other words, I want to generate graphs with node and edge labels, similar to what I have in my dataset)
I checked over some papers, but I didn't find exact clues about how to choose the method for graph generation. If anybody is more familiar with the topic, I'll appreciate any link to papers or any advice/recommendations.


text:
What AI applications exist to solve sustainability issues?
The Sustainable Development Goals of the United Nations describe a normative framework which states what future development until 2030 should strive for. On a more abstract level a basic definition describes sustainable development as

development that meets the needs of the present without compromising the ability of future generations to meet their own needs.

Alone through the consumption of energy, AI technologies already have a (negative) impact on sustainability questions.
What AI applications already exist, are researched or are at least thinkable from which sustainability would benefit?

The paper The role of artificial intelligence in achieving the Sustainable Development Goals (2020, published in Nature) should contain the information you're looking for.
In the introduction, the authors write

Here we present and discuss implications of how AI can either enable or inhibit the delivery of all 17 goals and 169 targets recognized in the 2030 Agenda for Sustainable Development. Relationships were characterized by the methods reported at the end of this study, which can be summarized as a consensus-based expert elicitation process, informed by previous studies aimed at mapping SDGs interlinkages. A summary of the results is given in Fig. 1 and the Supplementary Data 1 provides a complete list of all the SDGs and targets, together with the detailed results from this work.

For example, as stated in the supplementary data, goal 1 is

End poverty in all its forms everywhere

and the first target (1.1.) of goal 1 is

By 2030, eradicate extreme poverty for all people everywhere, currently measured as people living on less than $1.25 a day

Then the authors suggest that, according to their studies, AI may act as an inhibitor or enabler (i.e. it may be used to fight poverty) for this target.

We identified in the literature studies suggesting that AI may be an inhibitor for this target, due to the potential increase in inequalities which would hinder the achievement of this goal (1). Other references however identify AI as an enabler for this goal, in the context of using satellite data analysis to track areas of poverty and to foster international collaboration (2).

Therefore, techniques for satellite data analysis are one of the AI techniques that may be used to tackle suistainability issues.

Thanks for the answer. I am aware of this paper but I am rather disappointed of its vagueness. The poverty example demonstrates this: IMHO tracking down areas of poverty is by far not the crucial point in overcome poverty.

@cknoll "and to foster international collaboration". I gave you one of many examples. You can find other examples in the supplementary data. I doubt you will find something a lot more concrete than their study. Unfortunately, it seems to me that people don't care much about these issues, especially people in the computer science community.


text:
Why am I getting a difference between training accuracy and accuracy calculated with Keras' predict_classes on a subset of the training data?
I'm trying to solve a binary classification problem with AlexNet. I split the original dataset into training and validation datasets using a 70/30 ratio. I have trained my neural network with a dataset of 11200 images, and I obtained a training accuracy of 99% and a validation accuracy was 96%. At the end of the training, I saved my model's weights to a file.
After training, I loaded the saved weights to the same neural network. I chose 738 images out of the 11200 training images, and I tried to predict the class of each of them with my model, and compare them with true labels, then again I calculated the accuracy percentage and it was 74%.
What is the problem here? I guess its accuracy should be about 96% again.
Here's the code that I'm using.
prelist=[]
for i in range(len(x)):
    prediction = model.predict_classes(x[i])
    prelist.append(prediction)
count = 0
for i in range(len(x)):
    if(y[i] == prelist[i]):
        count = count + 1
test_precision = (count/len(x))*100
print (test_precision)

When I use predict_classes on 11200 images that I used to train the neural network and compare its result with true labels and calculated accuracy again its accuracy is 91%.

One problem could be with the selection of the validation set. For your model to work well on data it has not seen as training data is to have a high validation accuracy, but that is not sufficient on its own. The validation set must be large enough and varied enough that its probability distribution is an accurate representation of the probability distribution of all the images. You could have 1000 validation images, but, if they are similar to each other, they would be an inadequate representation of the probability distribution. Therefore, when you run your trained model to make predictions on the test set its performance would be poor.
So the question is: how many validation images did you use and how were they selected (randomly or handpicked)?
Try increasing the number of validation images and use one of the available methods to randomly select images from the training set, remove them from the training set, and use them as validation images.
Keras's flow_from_directory can achieve that or sklearn's train_test_split. I usually have the validation set be selected randomly and have at least 10% as many images as the test set does.
Overtraining is a possibility, but I think unlikely given your validation accuracy is high.
Another thing is how were the test set images selected? Maybe their distribution is skewed. Again, the best thing to do is to select these randomly.
What was your training accuracy? Without a high training accuracy, the validation accuracy may be a meaningless value. Training accuracy should be in the upper 90's for the validation accuracy to be really meaningful.
Finally, is there any possibility your test images were mislabeled? Try switching your test set as the validation set and see what validation accuracy you get.
Here is an example of what I mean. A guy built a CNN to operate on a set of images separated into two classes. One class as "dogs", the other class was "wolves". He trained the network with great results almost 99.99% training accuracy and 99.6% validation accuracy. When he ran it on the test set, his accuracy was about 50%. Why? Well, it turns out all the images of wolves were taken with snow in the background. None of the images of dogs had snow in the background. So, the neural network figures out if snow must be a wolf, if no snow must be a dog. However, in his test set, he had a mixture of wolves in or not in snow, dogs in or not in snow. Great training and validation results but totally useless performance.


text:
Is there a way to ensure that my model is able to recognize an unseen example?
My question is more theoretical than practical. Let's say that I am training my cat classifier with a dataset that I feel is pretty representative of cat images in general. But then a new breed of cat is created that is distinct from other cats and it does not exist in my dataset.
My question is: is there a way to ensure that my model is still able to recognize this unseen breed, even though I didn't know it would come into existence when I originally trained my model?
I have been trying to answer this question by intentionally designing my validation and test sets such that they contain examples that are quite distantly related to those that exist in the training set (think of it like intentionally leaving out specific breeds of cats from the training set).
The results are interesting. For example, slight changes to parameters can dramatically change performance on the distantly related test examples, while not changing performance very much for the more closely related examples. I was wondering if anyone has done a deeper analysis of this phenomenon.

I am tempted to conclude that the best way to ensure that the network still performs well on unseen examples is to regularize it as much as possible (Dropout rate of 0.5 at every layer, for example). Do you think this intuition is correct?

Dropout is just one way of introducing a regularization effect. There are others. Dropout should help to avoid overfitting and thus the network shouldn't memorize the training dataset (i.e. approximate the function that the training set represents), but, instead, approximate a different function (not the one represented by the training set). Clearly, this should help to predict something that isn't in the training set, but this will not ensure that a specific unseen example will be predicted correctly.

So assume the typical train-dev-test paradigm, where all three sets have unique examples but are drawn from the same distribution. With a neural network, you can easily overfit to those particular examples that exist in the training set if you don't regularize the parameters. Regularization allows the model to learn a more generalizable function. But if an unseen example that is not well represented in the train, dev, or test sets is introduced, how can we ensure that the model will be more likely to generalize to this distantly related, new example?

Would regularizing the model even more than seemed necessary given your available dataset help you to generalize to these more dissimilar, unseen examples?

The term regularization is often used ambiguously. For example, in L1 or L2 regularization, the goal is to constrain the parameters to lie in a certain range. In other cases, for example, in the case of dropout, the idea is that you attempt to decorrelate the units of the network (by randomly dropping or killing them during training). In the case of Bayesian neural networks, the regularisation is said to be integrated into the networks because they are based on Bayes rule, so, in a certain way, the uncertainty associated with the learned distributions depends on the available data.

To answer your question directly, regularization may help, but it may depend on the regularization that you use.

The comments already are giving you some good tips about how to improve what your model recognizes, but I think your question goes above that asking if there's a way to ensure that it will always recognize the cats.
The short answer is "no".
The slightly longer answer is "yes, but cheating".
Regardless, there are a lot of steps you might take to improve the generalization aspect of your model.

Long answer:
A drama: cat classification in three acts
Act I: Cat as texts
Let's start with an example. Say that your model is trained with these inputs, and learns to correctly recognize them as a cat or not a cat:
cat → yes!
Cat → yes!
ferret → no
cat. → yes!
Cat! → yes!
Three MC's and one DJ → no

Your goal is to train your model so that every new variation, even unseen ones, will correctly be identified.
With a good level of generalization, your model will correctly classify new inputs that it has never seen before:
skunk → no
cat? → yes!
dog → no
CAT → yes

With this scenario, let's say the model now finds this:
kat → ?

Is that a "cat" misspelled? Is that short for Katherine? What should the model do?
Act II: But surely this doesn't happen in real life
Leaving the analogy for a bit, will your model that's looking at domestic cats properly accommodate for Savannah Cats, or will it consider them out? (They kind of look like cheetahs.) What about Sphinx cats? (They look like raw chicken to me.) Elf cats? (They look like bats.) This is just an example, but you can probably figure out more.
And the reason behind this problem is that the distinction itself between different classifications (in real life) is not binary, but rather a transition between "yes, that's a textbook cat" and "that's a chair". Your model will output binary decisions (maybe accompanied by a confidence interval, but even with it, you'll make the call into deciding if it's a cat or not).
Setting specific boundaries will help. You can define that your model will only detect domestic cats, maybe no bigger than a certain size, only of certain colors, etc... This is limiting what the model will correctly recognize as a cat when we (humans) might disagree. For instance, I would still argue that flourescent cats are still cats.
Going back to the simple text analogy, this is similar to deciding that to be detected as a cat, it has to start with a "c". So now you've discarded ¡Cat!.
In this way, it's not possible to ensure (notice the word) that your model will detect all of these unknown variations. There will be always some room for error that needs to be accepted, as long as the errors are infrequent or rare enough that they can be accepted as a regular part of the model.
Act III: Concept drift, a cautionary tale
Finally, the problem becomes even harder as we might be dealing with concepts that change over time, outside of the knowledge of the model, and outside of the knowledge of the person that supervised the model learning.
As and the breeds of cat changes, your model will have to accommodate by what we (users of the model) consider a valid definition of cat. Which might change in really unexpected ways and not really "look" like a cat. And since your model can only learn from what "looks" like a cat, it's always held in a disadvantaged position.
This will happen with almost any machine learning model that is approximating a result, regardless of the technique/algorithm. Approximations include a level of error because reality is usually complex in ways that we either don't know about or that are too computationally expensive.

I think that explaining by example is usually easier to grasp, so I thought it would make a better answer. However, being an example it doesn't really tackle the underlying issues with model errors in detail, leaving it incomplete. If readers want to expand on it, please, feel free to do so.

Thank you for this answer! Definitely got me thinking, much appreciated.

@mdurrant I'm glad! In your particular case, I would suggest just setting up acceptable boundaries and moving forward. Yes, the model might not be future-proof of 100% accurate... but it still might be useful!


text:
Is the choice of the optimiser relevant when doing object detection?
Suppose that we have 4 types of dogs that we want to detect (Golden Retriever, Black Labrador, Cocker Spaniel, and Pit Bull). The training data consists of png images of a data set of dogs along with their annotations. We want to train a model using YOLOv3.
Does the choice of optimizer really matter in terms of training the model? Would the Adam optimizer be better than the Adadelta optimizer? Or would they all basically be the same?
Would some optimizers be better because they allow most of the weights to achieve their "global" minima?

I have experimented with this to a small degree and have not noticed that much of an impact.
To date, Adam appears to give the best results on a variety of image data sets. I have found that "adjusting" the learning rate during training is an effective means of improving model performance and has more impact than the selection of the optimizer.
Keras has two callbacks that are useful for this purpose. Documentation is at https://keras.io/callbacks/. The ModelCheckpoint callback enables you to save the full model or just the model weights based on monitoring a metric. Typically, you monitor validation loss and set the parameter save_best_only=True to save the results for the lowest validation loss. The other useful callback is ReduceLROnPlateau, which allows you to adjust the learning rate based on monitoring a metric. Again, the metric usually monitored is the validation loss. If the loss fails to reduce after a user-set number of epochs (parameter patience), the learning rate will be adjusted by a user-set factor (parameter factor). You can think of the training process as traveling down a valley. As you near the bottom of the valley, it becomes more and more narrow. If your learning rate does not adjust to the "narrowness" there is no way you will get to the bottom of the valley.
You can also write a custom callback to adjust the learning rate. I have done this and created one which first adjusts the learning rate based on monitoring the training loss until the training accuracy reaches 95%. Then it switches to adjust the learning rate based on monitoring the validation loss. It saves the model weights for the lowest validation loss and loads the model with these weights to make predictions. I have found this approach leads to faster training and higher accuracy.
The fact is you can't tell if your model has converged on a global minimum or a local minimum. This is evidenced by the fact that, unless you take special efforts to inhibit randomization, you can get different results each time you run your model. The loss can be envisioned as a surface in $N$ space, where $N$ is the number of trainable parameters. Lord knows what that surface is like and where your initial parameter weights put you on that surface, plus how other random processes cause you to traverse that surface.
As an example, I ran a model at least 20 times and got resultant losses that were very close to each there. Then I ran it again and got far better results for exactly the same data.

Interesting bio and project goal, but consider potential for abuse - deliberately diagnosing or stigmatizing appearance. An arguably fair medical asset, but unsure a public utility.

Does the choice of optimizer really matter in terms of training the model?

Yes.

Would the Adam optimizer be better than the Adadelta optimizer?

Yes. (But sometimes, adadelta gives better result. Depends upon the dataset and fine-tune mechanism)

Would they all basically be the same?

No. Here is the explanation

Would some optimizers be better because they allow most of the weights to achieve their "global" minima?

It's not possible to check whether the model achieved global minima or not practically.
We can evaluate the model by either over-fitting or under-fitting using the training and validation set, and the generalization of the model with the test set.


text:
How can I constraint the actions with dependent coordinates?
I am working on a customized RL environment where each action is represented as a tuple $a = (a_1,a_2,\cdots,a_n)$ such that certain condition must be satisfied for entries of $a$ (for instance, $a_1+a_2+\cdots+a_n \leq \text{constant}$).
I am using the policy gradient method, but I am having some difficulty modeling the underlying probability distribution of actions.  Is there any work done in this direction?
For the constraint $a_1+a_2+\cdots+a_n \leq \text{constant}$, I was thinking about generating $n+1$ uniform random variables $U_1,U_2,\cdots,U_n, U$, and set $a_i = \text{constant}\times U \times \frac{U_i}{\sum_{j=1}^n U_j}$. Problem is that the joint density is a bit messy to calculate, which is needed to get the negative log likelihood. I am curious about how such issue is handled in practice.

At first glance, I thought this was similar to "continuous-discrete" action selection (https://arxiv.org/pdf/1810.06394.pdf). However, I think your problem is different.
I am assuming that each $a_i$ is continuous and that the action which interacts with your environment is the entire vector $a = (a_1,a_2,\dotso,a_n)$ and not an individual $a_i$. Then you could treat it like a hierarchical problem. If you want $a_1 + a_2 < 2$ for example, then you could sample $a_1 \sim U(0,2)$ and $a_2 | a_1 \sim U(0, 2-a_1)$ and have $p(a) = p(a_2 | a_1)p(a_1)$. The specifics of how you do this depends more finely on how your problem is set up.
Perhaps you can find similar ideas from the paper linked above. Also, other work in the robitics literature studies structured and hybrid action spaces.

Thanks for the answer! Only issue for your method here is $a_1$ and $a_2$ are asymmetrical (e.g. $a_1$ always has higher mean that $a_2$). I kinda would like to treat each entry $a_i$ in a equal manner. I would definitely take a look a the paper you provided. I think policy gradient method can handle hybrid action space as long as the underlying probability distribution is properly specified.

This is very contingent on the structure of your problem. Why not naively set $a_i \sim U(0,\frac{1}{n})$? I can't think of another general and non-hierarchical approach to this.
If you do not want a systematic bias, then perhaps you could randomly sample the order for the indicies as well.

Having $a_i$ being uniform(0,1\n) will impose a 1/n bound on them, in addition, they will be independent. Remember I would like to have dependent entries... I only have one feasible way of doing so which I mention in my post.

Right, I should have added general, non-hierarchical and tractable.
Since you are using policy gradient, don't you need to parameterize the distribution anyway? Perhaps you can simplify the problem by assuming IID while using lagrange multipliers to constrain the parameters.

authors: جوجو تي في Jojo tv -


text:
What's the difference in using multiple convolutional layers and no pooling versus using a single convolutional layer and a single max pooling layer?
I'm currently working on a college project in which I'm designing a Deep Q-Network that takes images/frames as an input.
I've been searching online to see how other people have designed their convolutional stage and I've seen many different implementations.
Some projects, such as DeepMinds Atari 2600 project, use 3 convolutional layers and no pooling (from what I can see).
However, other projects use fewer convolutional layers and add a pooling layer onto the end.
I understand what both layers do, I was just wondering is there a benefit to how DeepMind did it and not use pooling or should I be using a pooling layer and fewer convolutional layers?
Or have I completely missed out on something? Is Deep Mind actually using pooling after each convolutional layer?