text:
Dataset for floating objects detection
I am looking for a dataset, which I could train a model to detect people/boats/surfboards, etc., from a drone view.
Has anyone seen a dataset that could be useful for this purpose?
I have some photos made by me (like this one below), but I need more data. Of course, the best will be if data will be labeled, but, if someone has seen an unlabeled dataset with videos/photos like that below, please share the link to it.
Sample photos I am looking for:

If someone will be looking for a dataset for maritime SAR (Search and Rescue) purposes in the future, we have created the first publicly free to use for academic research dataset of this type:
http://afo-dataset.pl/en/download/

Perhaps you can check this dataset out:
http://www.aiskyeye.com/

The VisDrone2019 dataset is collected by the AISKYEYE team at Lab of
  Machine Learning and Data Mining , Tianjin University, China. The
  benchmark dataset consists of 288 video clips formed by 261,908 frames
  and 10,209 static images, captured by various drone-mounted cameras,
  covering a wide range of aspects including location (taken from 14
  different cities separated by thousands of kilometers in China),
  environment (urban and country), objects (pedestrian, vehicles,
  bicycles, etc.), and density (sparse and crowded scenes). Note that,
  the dataset was collected using various drone platforms (i.e., drones
  with different models), in different scenarios, and under various
  weather and lighting conditions. These frames are manually annotated
  with more than 2.6 million bounding boxes of targets of frequent
  interests, such as pedestrians, cars, bicycles, and tricycles. Some
  important attributes including scene visibility, object class and
  occlusion, are also provided for better data utilization.

It provides many drone view images with bounding boxes. Hope it can help you and have a nice day!


text:
Would either $L_1$ or $L_2$ regularisation lower the MSE on the training and test data?
Consider linear regression. The mean squared error (MSE) is 120.5 for the training dataset. We've reached the minimum for the training data.
Is it possible that by applying Lasso (L1 regularization) we would get a lower MSE for the training data? Would it get lower for the test data? Would this also hold for ridge regression (L2 regularization)?

The answer is largely the same whether we consider $\ell_1$ or $\ell_2$ regularisation, so I will just speak generally about regularisation.
Mean square error for training data
Given some training data $\{(x_i, y_i)\}_{i = 1}^n$, a linear regression line $Y = aX + b$ fit using the least squares method looks for coefficients that minimise the sum of squares, i.e. they are the minimisers given by
$$ \mathrm{arg\,min}_{a, b} \sum_{i = 1}^n \left(y_i - (ax_i + b)\right)^2.$$
This gives the same coefficients as minimising the mean square error
$$ \mathrm{MSE}\left((x_1, y_1), \dots, (x_n, y_n)\right) = \frac{1}{n} \sum_{i = 1}^n \left(y_i - (ax_i + b)\right)^2.$$
So, by definition, the coefficients $(a, b)$ minimise the MSE on the training data. Any regularisation will only increase the MSE on the training data.
Generalisation performance
The main point of regularisation is to prevent overfitting on the data and improve the generalisation performance (i.e. on the test set).
With an appropriate parameter for regularisation, you may obtain a smaller MSE on the test set. This depends on your dataset and the parameters you choose: strong regularisation may lead to underfitting, whereas weak regularisation might not make much difference to the coefficients that you fit.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to improve the reward signal when the rewards are sparse?
In cases where the reward is delayed, this can negatively impact a models ability to do proper credit assignment. In the case of a sparse reward, are there ways in which this can be negated?
In a chess example, there are certain moves that you can take that correlate strongly with winning the game (taking the opponent's queen) but typically agents only receive a reward at the end of the game, so as to not introduce bias. The downside is that training in this sparse reward environment requires lots of data and training episodes to converge to something good.
Are there existing ways to improve the agent's performance without introducing too much bias to the policy?

Andrew Y. Ng (yes, that famous guy!) et al. proved, in the seminal paper Policy invariance under reward transformations: Theory and application to reward shaping (ICML, 1999), which was then part of his PhD thesis, that potential-based reward shaping (PBRS) is the way to shape the natural/correct sparse reward function (RF) without changing the optimal policy, i.e. if you apply PBRS to your sparse RF, the optimal policy associated with the shaped, denser RF is equal to the optimal policy associated with the original unshaped and sparse RF. This means that PBRS creates an equivalence class of RFs associated with the same optimal policy, i.e. there are multiple RFs associated with the same optimal policy. So, PBRS is the first technique that you can use to deal with sparse reward functions.
To give you more details, let $R\left(s, a, s^{\prime}\right)$ be your original (possibly sparse) RF for your MDP $M$, then
$$R\left(s, a, s^{\prime}\right)+F\left(s, a, s^{\prime}\right)\tag{1}\label{1}$$
is the shaped, denser RF for a new MDP $M'$.
In PBRS, we then define $F$ (the shaping function) as follows
$$
F\left(s, a, s^{\prime}\right)=\gamma \Phi\left(s^{\prime}\right)-\Phi(s),
\tag{2}\label{2}
$$
where $\Phi: S \mapsto \mathbb{R}$ is a real-valued function that indicates the desirability of being in a specific state. So, if we define $F$ as defined in \ref{2}, we call it a potential-based shaping function.
The intuition of $F$ in \ref{2} is that it avoids the agent to "go in circles" to get more and more reward. To be more precise, let's say that there's a state $s^* = s_t$ that is desirable but not the goal state. If you shaped the reward function by adding a positive reward (e.g. 5) to the agent whenever it got to that state $s^*$, it could just go back and forth to that state in order to get the reward without ever reaching the goal state (i.e. reward hacking). That's, of course, not desirable. So, if your current state is $s^*$ and your previous state was $s_{t-1}$, and whenever you get to $s_{t-1}$ you just shaped the reward function by adding a zero reward, to avoid going back to $s_{t-1}$ (i.e. to avoid that the next state $s_{t+1} = s_{t-1}$, and then go back to $s^*$ again, $s_{t+2} = s^*$, i.e. going in circles), if you use \ref{2}, you will add to your original sparse reward function the following
$$
F\left(s, a, s^{\prime}\right)=\gamma 0 - 5 = -5,
\tag{3}\label{3}
$$
In other words, going back to $s_{t+1}$ (to later try to go back to $s^*$) is punished because this could again lead you to go back to $s^*$. So, if you define $F$ as in equation \ref{2}, you avoid reward hacking, which can arise if you shape your RF in an ad-hoc manner (i.e. "as it seems good to you").
One of the first papers that reported this "going in circles" behaviour was Learning to Drive a Bicycle using Reinforcement Learning and Shaping

We agree with Mataric [Mataric, 1994] that these heterogeneous reinforcement functions have to be designed with great care. In our first experiments we rewarded the agent for driving towards the goal but did not punish it for driving away from it. Consequently the agent drove in circles with a radius of 20–50 meters around the starting point. Such behavior was actually rewarded by the reinforcement function

You probably also want to watch this video.
Another approach to solving the sparse rewards problem is to learn a reward function from an expert/optimal policy or from demonstrations (inverse reinforcement learning) or to completely avoid using the reward function and simply learn the policy directly from demonstrations (imitation learning).
Note that I'm not saying that any of these solutions fully solves all your problems. They have advantages (e.g. if done correctly, PBRS can speed the learning process) but also disadvantages (e.g., in the case of IRL, if your expert demonstrations are scarce, then your learned RF may also not be good), which I will not discuss further here (also because I don't currently have much practical experience with none of these techniques).

This answers my question and shows how a PBRS function can be designed. But I will say that the conditions for designing a good $\Phi$ are hard to satisfy. In the examples given in that paper it requires knowledge of the goal state to come up with Manhattan distance from goal. In this case the designer is being asked to come up with the goals/subgoals directly. This gets much more complex as the number of absorbing states increase (think chess again where there are countless variations of 'checkmate') or when there is an infinite time horizon. Can you point to any papers that attempt this?

@tryingtolearn I think this would be another great question for the site, i.e. how to come up with a good $\Phi$. I still don't know the answer, but I may investigate it. I only read the paper "Policy invariance under reward transformations: Theory and application to reward shaping", and a few other related ones, but none of the papers that I've read so far go into the direction of what you're asking. So, I suggest that you ask another question, maybe someone already has a good answer for it.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is it possible to use k-nearest neighbour for classification with more than two attributes?
If I were to have a dataset of 9 attributes of different types that describe current weather, such as temperature, humidity, etc., and want to classify the current weather by use of a k-NN algorithm, is this possible?
From what I understand, k-NN has two different attributes that are plotted, and, wherever a point is drawn, its nearest neighbors will classify it.
Could I do the same thing but each data point is placed based on its 9 attributes?

The number of features is not important to use K-NN algotihm. You have to decide distance measure to detect neighbors. I share with you some links that you can check to see which kinds of distance measures that you can use. Just decide the meause and use your feature vectors in the measure.
https://www.kdnuggets.com/2020/11/most-popular-distance-metrics-knn.html
https://medium.com/@luigi.fiori.lf0303/distance-metrics-and-k-nearest-neighbor-knn-1b840969c0f4

Maybe I'm misunderstanding, but I thought that the amount of features is limited to 3, since the only way to calculate the distance between points would be in 3 dimensional space. How could I map the value that take all 9 values into account?

You can/should think your features in high dimensional vector space. So, for example if you have two vectors x and y in 9 dimensional space, just substract x from y and take absolute value of the result and sum the residual vector elements to get distance between two vectors. This is the simplest method. You do not have to live in a 3 dimensions in "the mathematical world". You can have infinite dimensions in math.

Okay, I think I have a general idea of what that means, but I’m missing the x and y part and how each point would get a unique one if it’s just a vector. Is there any reading that you could link that would explain it better for a beginner? Perhaps we could communicate via email or something so I could get some more guidance if possible?

I hope this helps you better https://math.stackexchange.com/questions/1047649/distance-and-angle-between-vectors-in-multiple-dimensions


text:
In Q-learning, wouldn't it be better to simply iterate through all possible states?
In Q-learning, all resources I've found seem to say that the algorithm to update the Q-table should start at some initial state, and pick actions (which are sometimes random) to explore the state space.
However, wouldn't it be better/faster/more thorough to simply iterate through all possible states? This would ensure that the entire table is updated, instead of just the states we happen to visit. Something like this (for each epoch):
for state in range(NUM_STATES):
  for action in range(NUM_ACTIONS):
    next_state, reward = env.step(state, action)
    update_q_table(state, action, next_state, reward)

Is this a viable option? The only drawback I can think of is that it wouldn't be efficient for huge state spaces.

If your algorithm is executed multiple (or enough) times using an outer loop, it would converge to similar results as Q-learning would with $\gamma = 0$ (as you don't look what is the expected future reward).
In this case, the difference is that you would pass as much time to explore each possible couple of (state, action) while Q-learning would pass more time on the pair which seem more promising and, as you've said this wouldn't be efficient for a problem with a huge number of pair (state, action).
If the algorithm is executed only once, then, even for a problem with a few pairs (state, action), you need to assume that an action effected on a state will always bear the same result for your method to work.
In most cases, it isn't true either because there is some sort of randomness in the reward system or in the action (your agent can fail to make an action) or because the state of your agent is limited to its knowledge and so doesn't represent perfectly the world (and so the consequence of its action can vary just like if the reward had some randomness).
Finally, your algorithm doesn't look at the expected future reward, so it would be equivalent to having $\gamma = 0$. This could be fixed by adding a new loop updating the table after your current loops if you execute your algorithm only one time or by adding the expected future reward directly to your Q-table if there is an outer loop.
So, in conclusion, without the outer loop, your idea would work for a system with few pairs of (state, action), where your agent has a perfect and complete knowledge of its world, the reward doesn't vary, and where an agent can't fail to accomplish an action.
While these kinds of systems indeed exist, I don't think that it's an environment where one should use Q-learning (or another form of reinforcement learning), except if it's for educational purposes.
With an outer loop, your idea would work if you are willing to pass more time training to have a more precise Q-table on the least promising pair of (state, action).

I indeed assumed that there is no outer loop and that the algo was limited to what was shown. I will edit the answer to make it clear.
Assuming that there is no outer loop and a deterministic environment the algo is indeed equivalent to having γ = 0, I will also make it clear in my answer but it's a problem that could be easily fixed so I don't think that it's the core of the problem.

By "outer loop" I assume you mean training epochs? I added that to my question - indeed a single pass would just fill in initial values for the table, but it would require further passes for those values to converge.

An "outer loop" would repeat the algo. It could be a fixed number of time which is equivalent to trainings epoch or until convergence (ie, the Q-table isn't updated anymore)

Indeed, in my case the state space is continuous, but I'm using a discretized representation - and so since there is a range of possible "real" states represented by a given discrete state, the result of a given action is not necessarily deterministic.

In short, yes, provided that you have a small number of states.
In pretty much any real system, the number of states is much higher than you could ever hope to explore exhaustively in any reasonable time. This is why you need to set some sort of exploration/exploitation policy to make sure that you mostly visit promising states while also checking states that might look initially poor but may lead to better states as you explore further.
As a few minutes thought will convince you, determining the exact nature of that exploration/exploitation trade-off is probably the most important aspect of effective Q-Learning (and pretty much any other search algorithm for that matter).

The issue is trading off reward vs learning, Q-learning attempts to learn and do things which produce reward(basically, operate suboptimally).
I'm not sure if Q-Learning is actually any different performance wise, short-term Q-Learning will produce more reward(probably) but will also miss states.


text:
Can the rewards be matrices when using DQN?
I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working with images, but I'd like to observe the agent doing it in real-time. Just a fun side project.
Anyway, to encourage an RL agent to craft a specific image I'm crafting a reward function that returns a $N \times N$ dimensional matrix. Which represents the distance between the state of the target image (RGB values for each pixel location) and the image the agent crafted.
Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?

How would you adapt DQN to work with vectorized rewards? Note that I changed the title to be what I think is your actual question. If that's not your actual question, let us know what it is.

See also this and this.

Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?

Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and reinforcement learning (RL) methods.
Although it is possible to accumulate matrices in various ways, by e.g. simple matrix addition, and come up with an analog for expected return which would be a weighted sum of matrices, you then get stuck. There is no fixed way to rank matrices and decide whether one is a better result than another. This is a requirement for any learning process that aims to improve at a task - it needs feedback that changes it makes are better or worse related to some reference. As a result, most objective functions and metrics in optimisation use real-valued scalars, which can always be placed into order to decide a highest or lowest value.
This does not prevent you using a matrix representation for your project, if it is a natural fit. To turn it into a usable reward, you will need to convert that matrix into a real-valued metric. Perhaps the L2 norm or other standard measure that summarises the matrix will be good for your task.
It is possible to process multiple scalar rewards at once with single learner, using multi-objective reinforcement learning. Applied to your problem, this would give you access to a matrix of policies, each of which maximised the reward value of one cell within the matrix. It also allows for switching between objectives in a hierarchical manner using a "policy of policies" if some preference for what to achieve changes. It is not 100% clear, but I do not think this is what you want to do.

It might be worthing explicitly mentioning in your answer "multi-objective reinforcement learning" and the "reward hypothesis". This survey also seems a good reference for further details on MORL (i.e. it formulates the MDPs, value functions, etc., in the case of vector-valued rewards).

@nbro: Thanks, I'll have athink about that. In this case MORL is a bit of a red herring as OP appears to have a singular task. If I have it right, the vector of rewards in MORL represents multiple separate goals to be processed at once, such that a higher order goal selector policy can be used in a hierarchical fashion. It is not a way to avoid needing scalar reward values for optimality, but a way to parallel process multiple scalar reward schemes in an otherwise identical environment.


text:
Why is the update in-place faster than the out-of-place one in dynamic programming?
In Barto and Sutton's book, it's written that we have two types of updates in dynamic programming

Update out-of-place
Update in-place

The update in-place is the faster one. Why is that the case?
This is the pseudocode that I used to test it.
if in_place:
    state_values = new_state_values
else:
    state_values = new_state_values.copy()
old_state_values = state_values.copy()

for i in range(WORLD_SIZE):
    for j in range(WORLD_SIZE):
        value = 0
        for action in ACTIONS:
            (next_i, next_j), reward = step([i, j], action)
            value += ACTION_PROB * (reward + discount * state_values[next_i, next_j])
        new_state_values[i, j] = value

max_delta_value = abs(old_state_values - new_state_values).max()
if max_delta_value < 1e-4:
    break

Why is the in-place version faster, and what is the difference? What I think is that it is only better for storage usage, I don't understand the speed increase part.

When you make updates in-place, then some of the entries in state_values[next_i, next_j] that you are referencing here
value += ACTION_PROB * (reward + discount * state_values[next_i, next_j])

will already be updated earlier in the same loop. Which means you get to use the latest and likely more accurate values earlier.
The strength of this effect varies depending on the order that states and actions are visited. If you can manage to visit them in reverse order that they would appear in natural trajectories, then the speed improvement will be very noticeable.


text:
Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?
I am currently experimenting with the U-Net. I am doing semantic segmentation on the 2018 Data Science Bowl dataset from Kaggle without any data augmentation.
In my experiments, I am trying different hyper-parameters, like using Adam, mini-batch GD (MBGD), and batch normalization. Interestingly, all models with BN and/or Adam improve, while models without BN and with MBGD do not.
How could this be explained? If it is due to the internal covariate shift, the Adam models without BN should not improve either, right?
In the image below is the binary CE (BCE) train loss of my three models where the basic U-Net is blue, the basic U-Net with BN after every convolution is green, and the basic U-Net with Adam instead of MBGD is orange. The learning rate used in all models is 0.0001. I have also used other learning rates with worse results.

Well, some time ago I also faced the same issue in the semantic segmentation task. Batch normalization is expected to improve convergence, because the normalization of activations prevents the explosion of the gradients magnitude and leads to more steady convergence.
Adam is an adaptive optimizer with momentum and division by the weighted sum of gradients on previous iterations squared. https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c.
The loss surfaces of the neural networks is a difficult and poorly understood topic in the present. I suppose, that the poor convergence of SGD is caused by the roughness of loss surface, where the gradient makes big leaps, and jumps over the mimima. The adaptive learing strategy of the Adam, on the other hand, allows to dive into the valleys.


text:
What is the target output for updating a Deep Q Network
I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.
The algorithm for table-based Q-Learning updates a single entry of the Q table - i.e. a single $Q(s, a)$. However, a neural network outputs an entire row of the table - i.e. the Q-values for every possible action for a given state. So, what should the target output vector be for the network?
I've been trying to get it to work with something like the following:
q_values = model(state)
action = argmax(q_values)
next_state = env.step(state, action)
next_q_values = model(next_state)
max_next_q = max(next_q_values)

target_q_values = q_values
target_q_values[action] = reward(next_state) + gamma * max_next_q

The result is that my model tends to converge on some set of fixed values for every possible action - in other words, I get the same Q-values no matter what the input state is. (My guess is that this is because, since only 1 Q-value is updated, the training is teaching my model that most of its output is already fine.)
What should I be using for the target output vector for training? Should I calculate the target Q value for every action, instead of just one?

As you say, the output of a $Q$ network is typically a value for all actions of the given state. Let us call this output $\mathbf{x} \in \mathbb{R}^{|\mathcal{A}|}$. To train your network using the squared bellman error you need first calculate the scalar target $y = r(s, a) + \max_a Q(s', a)$. Then, to train the network we take a vector $\mathbf{x'} = \mathbf{x}$ and change the $a$th element of it to be equal to $y$, where $a$ is the action you took in state $s$; call this modified vector $\mathbf{x'}_a$. We calculate the loss $\mathcal{L}(\mathbf{x}, \mathbf{x'}_a)$ and back propagate through this to update the parameters of our network.
Note that when we use $Q$ to calculate $y$ we typically use some form of target network; this can be a copy of $Q$ where the parameters are only updated every $i$th update or a network whose weights are updated using a polyak average with the main networks weights after every update.
Judging by your code it looks as though your action selection is what might be causing you some problems. As far as I can tell you're always acting greedily with respect to your $Q$-function. You should be looking to act $\epsilon$-greedily, i.e. with probability $\epsilon$ take a random action and act greedily otherwise. Typically you start with $\epsilon=1$ and decay it each time a random action is taken down to some small value such as 0.05.

Comments are not for extended discussion; this conversation has been moved to chat.

There are a couple ways you can define the architecture of a DQN. The most common way of doing it is by taking in the states and outputting the value function of all possible actions - this leads to a DQN with multiple outputs. The other, less efficient way, includes taking in an state-action as input and outputting a single real value - this approach is typically avoided since we need to run the model multiple times to get estimates for different actions.
The replay buffer is used to store $(S,A,R,S')$ transitions as encountered using your $\epsilon$-soft policy. We sample one of these transitions from the replay buffer and calculate an estimate of the value function for $(S,A)$ i.e $\hat Q(S,A,\theta)$ and then we calculate a target as follows. $$target =R+\max_\limits{a'}\hat Q(S',a',\theta^-)$$
Assuming you use the first model, you can then use a Squared error loss function, defined as follows, and modify your parameter as a function of that
$$L(\theta) = (target-\hat Q(S,A,\theta))^2$$
Assuming for now the target is fixed (I'll explain this in a minute), only $Q(S,A,\theta)$ is a function of $\theta$ in the loss function. $Q(S,A,\theta)$ corresponds to one output node of your DQN and therefore, as you've already highlighted, when carrying out EBP the parameters are updated such that we make the value of this one node tend to the specified target.
This is just how Q-learning works, we use samples generated by the behaviour policy to create $L(\theta)$ and then tweak the parameters to minimise the cost. As we do this for more and more samples the network hopefully figures out a way that accommodates for every sample it's been trained on so far (with more emphasis on the most recent samples).
As to your issue, are you sure you're training on multiple different samples and not just a specific one? it may just be a bug you've overseen.

Explaining  $\theta^-$
I used a slightly different notation, $\theta^-$, for the parameters used to generate the bootstrapped estimate, $\max_\limits{a'}\hat Q(S',a,\theta^-)$. $\theta^-$ is only matched to $\theta$ every $n^{th}$ step because we want to keep the target constant as much as possible. The reason for this is because Q-learning does not necessarily converge when using neural networks partly due to bootstrapping which can cause a divergence of optimisation because of state generalisation. By using this $\theta^-$ we help prevent things like this from happening.
Ultimately the idea of the replay buffer and the fixed parameter for bootstrapping are to try to convert the RL problem into a supervised learning problem because we know much more about how to deal with supervised learning problems when using DNNs.


text:
How UCT in MCTS selection phase avoids starvation?
The first step of MCTS is to keep choosing nodes based on Upper Confidence Bound applied to trees (UCT) until it reaches a leaf node where UCT is defined as
$$\frac{w_i}{n_i}+c\sqrt{\frac{ln(t)}{n_i}},$$
where

$w_i$= number of wins after i-th move
$n_i$ = number of simulations after the i-th move
$c$ = exploration parameter (theoretically equal to $\sqrt{2}$)
$t$ = total number of simulations for the parent node

I don't really understand how this equation avoids sibling nodes being starved, aka not explored. Because, let's say you have 3 nodes, and 1 we'll call it node A is chosen randomly to be explored, and just so happens to simulate a win. So, node A's UCT$=1+\sqrt(2)\sqrt{\frac{ln(1)}{1}}$, while the other 2 nodes UCT = 0, because they are unexplored and the game just started, so by UCT the other 2 nodes will never be explored no? Because after this it'll go into the expansion phase and expansion only happens it reaches a leaf node in the graph. So because node A is the only one with a UCT $> 0$ it'll choose a child of node A and it will keep going down that node cause all the siblings of node A have a UCT of 0 so they never get explored.

Note that for those 2 unvisited nodes in your example, we do not have UCT = $0$. It's technically undefined (due to a division by $n_i = 0$), but if we want to pretend that division by $0$ isn't undefined, the most "sensible" value to assign to it would probably be $\infty$ (infinity). And if you assume that all unvisited nodes have a value of $\infty$, you basically arrive at Cohensius' answer

First explore the nodes A,B,C once.
For reference see this paper by David Silver and Sylvain Gelly, Combining Online and Offline Knowledge in UCT

If any action from the current state $s$ is not represented in the tree, $\exists a \in \mathcal{A}(s),(s, a) \notin \mathcal{T},$ then the uniform random policy $\pi_{\text {random }}$ is used to select an action from all unrepresented actions, $\tilde{\mathcal{A}}(s)=\{a \mid(s, a) \notin \mathcal{T}\}$.


text:
Is there a machine learning model that can be trained with labels that only say how "right" or "wrong" it was?
I'm trying to find the name for a model that is used to output a decision (maybe something like right, left, or do nothing = -1, 0,1) but that can be trained with labels that contain how "correct" or "incorrect" it was. I've tried to google around and ask some friends in my machine learning class, but no one seems to have an answer.
The classic example I seem to always see is the models used in the snake game. We don't know what the right decision was per se, but we can say that if it ran into the wall, that was really wrong. Or if it got an apple and gained 50 points, then it was correct and if it got 2 apples and gained 100 points then it was even more correct, etc.
I'm looking for a network where the exact labels don't exist, but where we can penalize or reward its decisions.
I'm assuming this requires some kind of modified cost function, but I would imagine this type of network already exists. I'm hoping someone can provide me with the name for this type of network and whether or not there is a Keras frontend for something like this.

What you are looking for is called "reinforcement learning".
A reinforcement learning algorithm will try to maximize a reward function. This reward represents how "good" or "bad" an action is in the actual context. For example, in the snake game, your reward will be positive for eating an apple and negative when the snake hits a wall.
The interesting thing is that, with reinforcement learning, you can learn without having a reward at each step. In the case of the snake game, your agent can learn that going in the direction of the apple is better than going in the direction of the wall, even if none of this action will directly give a reward (positive or negative).
If you want to use a neural network as your post seem to imply then you should look at deep Q-learning, a reinforcement learning algorithm, which use a neural network to learn to predict the expected reward of a couple (state, action).

I agree and also think the OP is looking for RL. It is worth noting in the answer that the neural network can only be part of the RL algorithm. You cannot write a neural network that "does RL" using e.g. Keras. The OP will want to learn the general approach and structure for RL first. A good starting method might be Q-learning and DQN for how neural networks fit it.

Perfect! Thank you. The terms I was googling were just not giving me "reinforcement learning" haha. I've heard the name many times but just didn't realize that is what it was called.

@NeilSlater I'm starting to write a DQN following the Keras examples for an actor and critic now. So far it's pretty unique compared to what I'm used to but this seems like a great topic to learn next.


text:
What is the relation between the context in contextual bandits and the state in reinforcement learning?
Conceptually, in general, how is the context being handled in contextual bandits (CB), compared to states in reinforcement learning (RL)?
Specifically, in RL, we can use a function approximator (e.g. a neural network) to generalize to other states. Would that also be possible or desirable in the CB setting?
In general, what is the relation between the context in CB and the state in RL?

The notion of a state in reinforcement learning is (more or less) the same as the notion of a context in contextual bandits. The main difference is that, in reinforcement learning, an action $a_t$ in state $s_t$ not only affects the reward $r_r$ that the agent will get but it will also affect the next state $s_{t+1}$ the agent will end up in, while, in contextual bandits (aka associative search problems), an action $a_t$ in the state $s_t$ only affects the reward $r_r$ that you will get, but it doesn't affect the next state the agent will end up in. The typical problem that can be formulated as a contextual bandit problem is a recommender system.
In CBs, like in RL, the agent also needs to learn a policy, i.e. a function from states to actions, but actions that you take in a certain state are independent of the actions you take in other states.
So, as Sutton and Barto put it (2nd edition, section 2.9, page 41), contextual bandits are an intermediate problem between (context-free) bandits (where there is only one state or, equivalently, no state at all) and the full reinforcement learning problem.
Another important characteristic of many RL algorithms, such as Q-learning, is that they assume that the state is Markov, i.e. it contains all necessary info to take the optimal action, but, of course, RL is not just applicable to fully observable MDPs. In fact, even Q-learning has been applied to POMDPs, with some approximations and tricks.
Regarding the use of neural networks to approximate $q(s, a)$ or a policy in CBs, in principle, this is possible. However, given that the optimal action in a state $s$ is independent of the optimal action in another state $s'$, this is probably not useful, but I cannot guarantee you that this has not been successfully done, because I've not yet read the relevant literature (maybe someone else will provide another answer to address this aspect).

thanks! regarding the last paragraph, perhaps you do know whether this approximation task can be decoupled from the rest of CB or is it an integral part of the algorithm, and in the latter case what algorithms are those? the only CB algorithms i encountered this far are linear UCB and Thompson sampling CB.

@MaximVolgin To be honest, I don't know. I suspect that you could do both, but maybe you will need to have a separate neural network for each state, but that doesn't look like a good approach. Like you, I'm only familiar with the typical bandit algorithms, so I can't say much about function approximation in CBs. I am also curious to see what people have been trying. Later, maybe, I will search for a few relevant papers and maybe I will update this answer to address your questions.

@MaximVolgin See page 238 of this book (section 19.2 Stochastic Linear Bandits), which mentions the use of a neural network as a feature map in the context of bandits. Here is another potentially relevant paper, but I'm still not sure whether the neural networks here are used for the same typical purpose as in RL, as suspect that this is not the case, from what I've read there. As I said, I will update my answer later with some more relevant info.

the first link is about linear UCB i mentioned before, but the other one (about NeuralUCB) is potentially what i need, or at least the article most probably will give me some insights. thanks!

@MaximVolgin Once you have read that paper (or, in general, you know more about the application of function approximation to CBs), feel free to write a formal answer below that answers your other question.

Conceptually, in general, how is the context being handled in CB, compared to states in RL?

In terms of its place in the description of Contextual Bandits and Reinforcement Learning, context in CB is an exact analog for state in RL. The framework for RL is a strict generalisation of CB, and can be made similar or the same in a few separate ways:

If the agent is be optimised for immediate reward only (discount fatcor $\gamma=0$), then optimal action choice depends only on current state without considering consequences. However, the environment may not behave much like a contextual bandit over multiple time steps, so it would be hard to think in terms of the kind of optimisations that apply for CB (such as minimising regret).

If state progression in RL is unrelated to the action chosen, then optimal action choices depend only on the current state. There still might be some benefit from understanding the expected state progression in order to predict future rewards, and ability to learn about different states may be limited by the progression, so this in not full equivalence, but it is very close.

If state progression in RL is unrelated to any previous history (of states, actions, rewards), and state is drawn from the same population at any time step, then the full MDP description is not necessary, each time step is expected to be like the last. A contextual bandit model could well be more appropriate.

Another thing to consider is what your objectives are for studying the environment or applying an agent within it. Bandit solvers are usually applied to environments where the agent is expected to learn strictly online, and the goal of the developer is to write a learner that uses a minimal amount of information to decide on optimal or near optimal choices. One common metric for this is to minimise regret, or the expected difference in reward between the agent's action choices and the ideal choice summed over time.
If you have offline data to work from, then predictions for an optimal agent in a CB enviroment devolve to supervised learning of a regression task. There is no simple equivalent for this in RL, because actions have consequences that create links between states. As a result, offline RL methods are very similar to online RL ones - the state, action, reward data is processed much the same way.


text:
How to keep track of the subject/entity in a sentence?
I'm working on Sentiment Analysis, using HuggingFace to perform sentiment analysis on articles
 classifier = pipeline('sentiment-analysis', model="nlptown/bert-base-multilingual-uncased-sentiment")
 classifier(['We are very happy to show you the  Transformers library.',  "We hope you don't hate it."])

This returns

label: POSITIVE, with score: 0.9998

label: NEGATIVE, with score: 0.5309

Now I'm trying to understand how to keep track of a subject when performing the sentiment analysis.
Suppose I'm given a sentence like this.

StackExchange is a great website. It helps users answer questions.  Hopefully, someone will help answer this question.

I would like to keep track of the subject when performing sentiment analysis. In the example above, in the 2nd sentence 'it' refers to 'StackExchange'. I would like to be able to do track a subject between sentences.
Now, I could try to manually try to parse this by finding the verb and trying to figure find the phrase that comes before it. However, it doesn't sound like a very safe or accurate way to find the subject.
Alternatively, I could train similar to a Named Entity Recognition. However, finding a dataset for this is very hard, and training it would be very time-consuming.
How can I keep track of an entity within an article?

What you're describing is known as coreference resolution. More specifically, this example is anaphora resolution. The short answer is that this is an open research question and there is no well-established solution.
You mentioned Hugging Face in your question. The neuralcoref module in spaCy is itself from Hugging Face (note the reflexive anaphor used for emphasis in this sentence). If you're not a spaCy kind of person, then there's also Stanford's CoreNLP in Java that has coreference resolution. A Python wrapper is also available.
I also wanted to address a couple other topics you mentioned. You are right in that they're all somewhat connected. But you need to scope down your goal/research question because what you're aiming to achieve is too difficult for a first task. Named entity recognition, constituency parsing, and sentiment analysis. Pick just one to focus on.

Thanks for your answer.  I'm new to this field so this information is very useful for my research.  I saw CoreNLP and was debating on using it.  I think I'll need to make some hybrid solution, which uses CoreNLP to analyize the subject, and some savvy rules to maintain track of the subject

You're welcome! Part of the difficulty in getting start with NLP is understanding the correct terms to call things. One you know the right term you can often find useful answers.


text:
Is there a graph neural network algorithm that can deal with a different number of input and output nodes?
I am new to graph neural networks and their applications. I have an input graph $G = \{V, E\}$ and an output graph $G' = \{V', E'\}$ where the number of nodes $V$ and $V'$ are different. I am trying to learn the function where $f(G) = G'$ and $V > V'$, thus, the function is mapping many-to-one ($n$ number of nodes map to one). The Graph Convolution Network (GCN) seems to have the same number of nodes in input and output with the function being learnt. Could I utilize the GCN for my task?

I suggest you look into link prediction. I have had good luck with the StellarGraph library. They have several algorithms implemented, including GCN.
Link prediction is a binary classification problem. Given two nodes, $v_i$ and $v_j$, does there exist a link between them? Using a library like StellarGraph will also produce node embeddings while performing link prediction.
For you scenario I'm picturing a three step process:

Link prediction and node embeddings on $G$.
Link prediction and node embeddings on $G'$.
Link prediction reusing existing embeddings where each link is between the two graphs. So each link is a tuple of the form: $(v_i, v_j')$ where $v_i \in V$ and $v_j' \in V'$. If there were no links predicted from $v_i$ to $v_j'$ then that might suggest to remove $v_j'$.

In the link prediction tasks that I've outlined, you can use GCN with StellarGraph. So there should be no problems in terms of the number of nodes.

Thanks. So in case, where none of the edges exist, the node is removed? Also wouldnt that lead to lot of computation if the V' is much smaller than V.


text:
Would it be possible to determine the dataset a neural network was trained on?
Let's say we have a neural network that was trained with a dataset $D$ to solve some task. Would it be possible to "reverse-engineer" this neural network and get a vague idea of the dataset $D$ it was trained on?

You can already do this with some neural networks, such as GANs and VAEs, which are generative models that learn a probability distribution over the inputs, so they learn how to produce e.g. images that are similar to the images they were trained with.
Now, if you're interested in whether there is a black-box method, i.e. a method that, for every possible neural network, would tell you the dataset a neural network was trained with, that seems to be a harder task and definitely an ill-posed problem, but I suspect that people working on adversarial machine learning have already attempted or will attempt to do something similar.


text:
Relationship between Rewards and Q Value (Graph between Q(s, a) vs episodes)
I'm employing the Actor-Critic algorithm. The critic network approximates the action-value function, i.e. $Q(s, a)$, which determines how good a particular state is, when provided with an action.
$Q(s, a)$ is approximated using the backpropagation of the temporal difference error (TD error). We can understand that $Q(s, a)$ has been approximated properly when TD error is minimized, i.e. when it is saturated at lower values.
My question is, when exactly can you say that $Q(s, a)$ is approximated properly, if you don't have TD error, i.e. if you have to plot the graph between $Q(s, a)$ vs episodes, then what would be the optimal behaviour?
Will it be increasing exponential with saturation value around reward values, or increasing exponential with saturation (around any value)?
Follow up: What can be the possible mistake, if the output of Q-value function is around 5x the rewards, and not saturating?

Which actor-critic algorithm have you implemented exactly? From which paper or book? I suppose you're using a neural network to approximate $Q(s, a)$, given your description. Moreover, when you say "if you have to plot the graph between Q(s, a) vs episodes", what graph are you exactly talking about? Can you show an example of such a graph that you have plotted? Also, why don't you have the TD error?

I'm using the standard actor critic, where critic network is optimized using TD Error. I wanted to plot the values of $Q(s, a)$ (my network has 1 output neuron) vs training episodes, to know what are the possible values around which $Q(s, a)$ get saturated. However, my question is answered by little more experimentation and research. I'll be adding the answer soon.

So, are you using function approximation or not? In any case, I would suggest that you put the title in the form of a question, your specific question. Right now, the title is not very clear. Also, what you wanted to plot, according to your answer, was the mean Q(s, a) value, but that alone doesn't mean much. The Q-value is a function of the state and action, so if you average the Q-values across all states and actions, that's probably meaningless.

Yes, I'm using function approximation. I'll take care of the title as well. @nbro, can elaborate on why averaging Q-values across all the states and actions is probably meaningless? My state is basically the channel coefficient H, and action is the amount of power transmitted in that H.

Because, for example, in one state $s_1$, the average value across all possible actions in that state may be 1, while the average value across all actions in state $s_2$ may be 1000. However, maybe, in both cases, the relative value of the actions in the specific state is the same, i.e. the optimal action is the same, the second best optimal action is also the same and so on. Of course, the values of the states depend on 1. how you initialize the Q-network, 2. how you explore the environment, 3. the learning update.

If the action space for a particular state $s_i$ is about a mean $\mu_{i}$, and $\mu_{i+1}$ is comparable to the previous $\mu_{i}$ (which is my case), and $\mu_{i+1} - \mu_{i} \in [0, 5]$, will taking mean of $Q(s_i, a)$ over all $i$'s still be useless?

I didn't yet read your answer, but what did you want to get from the mean of the Q-values across all states and actions? What do you think that represents? Wait, are you computing the mean across all states for a fixed $a$? If this is this last case, it probably makes some sense (but this depends on what you wanted to achieve with this).

The mean of Q-values across all the states and actions will denote the stability of the algorithm as used in DQN Paper. The figure 2c and 2d. There is another question posing similar concerns over averaging all the q-values.

TLDR;
The output of Q-value function will eventually saturate. Can't say when, but it will surely do.
Detailed Answer

if you don't have TD error

It meant, if I don't have logs of the error.

$Q(s, a)$ vs Episodes Graph

To understand, how $Q(s, a)$ behaves as the episodes increase.
I was under a wrong impression that, the $Q(s, a)$ will saturate around the values given by the reward function.
As evident from Loss vs Training Episodes Curve, we can see that loss (TD Error) is almost saturated.
However, $Q(s, a)$ vs training episodes curve is not saturated yet.

The only explanation for the above two graphs could be given as follows:
The target estimate ($r + \gamma Q(s', a')$) and $Q(s, a)$ are almost similar due to which, the error is quite low. But, $Q(s, a)$ is still nowhere near optimal value $Q^{*}(s, a)$.
Hence, I gave it another shot and made it run for twice training episodes i.e. 20000, and below are the results.
A nearly saturated loss

and a nearly saturating $Q(s, a)$.
.
Note that, the value of $Q(s, a)$ saturating is around 250 - 300 (will run it for more iterations) and it is nowhere around the reward values $ \in [-100, 35]$.
Hence, the $Q(s, a)$ vs episodes will saturate.


text:
Is case-based reasoning a machine learning technique?
A few years ago when I was in university, I had implemented (for my final year project) an Itinerary Planning System, which incorporates an AI technique called "case-based reasoning".
Is case-based reasoning a machine learning technique or an AI technique (that is not machine learning)?

The book Machine Learning (1997) by Tom Mitchell covers case-based reasoning (CBR), a form of instance-based learning (nearest neighbor is the typical example of IBL) in chapter 8 (p. 230).
T. Mitchell  writes

Instance-based methods such as $k$-NEAREST NEIGHBOR and locally weighted regression share three key properties. First, they are lazy learning methods in that they defer the decision of how to generalize beyond the training data until a new query instance is observed. Second, they classify new query instances by analyzing similar instances while ignoring instances that are very different from the query. Third, they represent instances as real-valued points in an $n$-dimensional Euclidean space. Case-based reasoning (CBR) is a learning paradigm based on the first two of these principles, but not the third. In CBR, instances are typically represented using more rich symbolic descriptions, and the methods used to retrieve similar instances are correspondingly more elaborate. CBR has been applied to problems such as conceptual design of mechanical devices based on a stored library of previous designs (Sycara et al. 1992), reasoning about new legal cases based on previous rulings (Ashley 1990), and solving planning and scheduling problems by reusing and combining portions of previous solutions to
similar problems (Veloso 1992).

He then goes on and gives the example of a CBR system: the CADET system. He also formulates CBR as a learning problem and uses the term "learn" to refer to a search process that CADET goes through, which is similar to what k-NN does.
He then writes

To summarize, case-based reasoning is an instance-based learning method in which instances (cases) may be rich relational descriptions and in which the retrieval and combination of cases to solve the current query may rely on knowledge-based reasoning and search-intensive problem-solving methods.

To conclude, yes, CBR can be considered a machine learning technique (if you also consider k-NN a learning algorithm, which people often do), even though it may rely on knowledge-based reasoning and search-intensive problem-solving methods.
You may also be interested in the paper Representation in case-based reasoning (2005) by Ralph Bergmann et al. Moreover, the famous AIMA book (3rd edition) mentions case-based reasoning in chapter 19 (p. 799), which is dedicated to knowledge and learning.


text:
In this implementation of pix2pix, why are the weights for the discriminator and generator losses set to 1 and 100 respectively?
I am working on a pix2pix GAN model that was inspired by the code in this Github repository. The original code is working and I have already customized most of the code for my needs. However, there is one part I am unable to understand.
The pix2pix GAN is a conditional GAN network that takes an image as a condition and outputs a modified image - such as blurry to clear, facades to buildings, filling up cut out part of an image, etc. The combined model thus takes as input a conditional image, the discriminator compares it with the dummy matrix named valid or fake, containing 0s or 1s according to validity (0 for generated samples, 1 for real samples). The generator loss is according to similarity with real sample + discriminator. The following code corresponds to what I told:
self.combined = Model(inputs=[img_A, img_B], outputs=[valid, fake_A])
self.combined.compile(loss=['mse', 'mae'],
                      loss_weights=[1, 100],
                      optimizer=optimizer)

The losses are thus set as MSE for discriminator output and MAE for generator. That seems to be OK, but I can not understand why the implementation uses 1 and 100 for the weights of the discriminator and generator losses, respectively, which seems to imply that the discriminator loss is 100 times lower than the loss of the generator. I couldn't find the reason in the original article. Are my understandings of the GAN incorrect?
Disclaimer: I have posted this question on Stats SE, but have no luck with answers. Maybe it is more suitable for AI.

Maybe you can take a look at the original/official implementation of pix2pix and see if they are doing something similar: https://github.com/phillipi/pix2pix.

After further research, I have found the answer. nbro was of course right, the weighting is not implementation dependant, it was already introduced in the paper (arXiv). However, there is minimal information about that, only it is mentioned within the optimization function:
$\arg \min_G \max_D \mathcal{L}_{cGAN}(G,D) + \lambda \mathcal{L}_{L1}(G)$
In fact, the parameter lambda stands for weight of $L_1$ loss. There should be a second parameter $\epsilon$ for $\mathcal{L}_{cGAN}(G,D)$ because in real implementation you can actually change that as well. In theory, however, only modulation of $L_1$ loss is needed. The paper does not state why the $\lambda$ parameter is needed, only mentions that setting it to 0 will lead to pure cGAN implementation. David Brownlee in his blog states:

The adversarial loss influences whether the generator model can output
images that are plausible in the target domain, whereas the L1 loss
regularizes the generator model to output images that are a plausible
translation of the source image. As such, the combination of the L1
loss to the adversarial loss is controlled by a new hyperparameter
lambda, which is set to 10, e.g. giving 10 times the importance of the
L1 loss than the adversarial loss to the generator during training.

The loss weights thus are hyperparameters that tell the network how much plausible translation of the source image do we need.

In addition, I would like to point out that I've seen this weighting of one component of the loss also being used in the context of variational auto-encoders (VAEs) and variational Bayesian neural networks, where the loss function is usually the ELBO (Evidence Lower BOund), which is composed of 2 terms, which can be weighted. There's actually a little bit of debate about what the correct weighting should be (at least in the context of variational BNNs).


text:
Why are tree-based models more widely used in Medical Diagnosis?
In Chapter 14.4 (p. 664) of the book Pattern Recognition and Machine Learning by Bishop, it is mentioned that tree-based models are more widely used in Medical Diagnosis.
Apart from giving better performance, is there a human-centric reason for this trade-off as medical diagnosis is mainly performed by a human?

Either you missed it or I don't fully understand why you were confused when you asked this question, but the same Bishop argues (in the same sentence where he says what you are wondering about) why tree-based models are popular in fields such as medical diagnosis.

A key property of tree-based models, which makes them popular in fields such as medical diagnosis, for example, is that they are readily interpretable by humans because they correspond to a sequence of binary decisions applied to the individual input variables. For instance, to predict a patient's disease, we might first ask "is their temperature greater than some threshold?". If the answer is yes, then we might next ask “is their blood pressure less than some threshold?". Each leaf of the tree is then associated with a specific diagnosis.

Nowadays, with the successes of neural networks (for example, in Go, Atari, image classification and segmentation, and even machine translation), which are not easily interpretable (so they are known as black-box models), there are always more studies/research on interpretable models or techniques to interpret black-box models, such as neural networks. You can take a look at this answer for a list of explainable/interpretable AI approaches that have been developed. This post contains many answers that further motivate the need for explainble AI.

One possible reason may have something to do with the scrutability of models, as described in the first few paragraphs of this article.  It presents a case study of a hospital whose policy was to send asthma sufferers to an intensive care unit; the intensive care meant they were less likely to develop pneumonia and therefore the data showed that people with asthma were less likely to have pneumonia.
Essentially, since machine learning models learn false relationships if the data are in any way flawed, it is beneficial to be able to "debug" them.  The processes by which decision trees make their decisions, and the reasons for making them, are more readily visible than in other models - particularly neural networks - which makes errors such as the example given in the article more likely to be picked up and corrected.

I myself working on similar product for Medical Diagnosis. The reason for it is we as ML engineer, generally try to replicate, how human approach any problem into mathematical model and build library on top of it to use it in application.
So, how Doctor approach to any specific decision of diagnosis, based on symptoms.
Mainly symptoms are in two form True or False, you have it or not.
Now we just need to replicate doctors approach to make decision, But yes to extract those symptoms from raw data in the form of text, image of sound we need to use other classifier and clustering models.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What are the biggest barriers to get RL in production?
I am studying the state of the art of Reinforcement Learning, and my point is that we see so many applications in the real world using Supervised and Unsupervised learning algorithms in production, but I don't see the same thing with Reinforcement Learning algorithms.
What are the biggest barriers to get RL in production?

There is a relatively recent paper that tackles this issue: Challenges of real-world reinforcement learning (2019) by Gabriel Dulac-Arnold et al., which presents all the challenges that need to be addressed to productionize RL to real world problems, the current approaches/solutions to solve the challenges, and metrics to evaluate them. I will only list them (based on the notes I had taken a few weeks ago). You should read the paper for more details. In any case, for people that are familiar with RL, they will be quite obvious.

Batch off-line and off-policy training

One current solution is importance sampling

Learning on the real system from limited samples (sample inefficiency)

Solutions: MAML, use expert demonstrations to bootstrap the agent, model-based approaches

High dimensional continuous state and action spaces

Solutions: AE-DQN, DRRN

Satisfying safety constraints

Solutions: constrained MDP, safe exploration strategies, etc.

Partial observability and non-stationarity

Solutions to partial observability: incorporate history in the observation, recurrent neural networks, etc.
Solutions to non-stationarity: domain randomization or system identification

Unspecified and multi-objective reward functions

Solutions: CVaR, Distributional DQN

Explainability
Real-time inference
System delays (see also this and this answers)

There's also a more recent and related paper An empirical investigation of the challenges of real-world reinforcement learning (2020) by Gabriel Dulac-Arnold et al, and here you have the associated code with the experiments.
However, note that RL (in particular, bandits) is already being used to solve at least one real-world problem [1, 2]. See also this answer.

Technical barriers: There should be at least these common sense big barriers:

Trial-and-error technique makes the model hard to learn (too many), compared to ready-to-use supervised data
Number of time-steps (which usually equals the number of actions of the agent in the trajectory) is large, thus brute-force exploration won't work as the number of trials to find errors is exponential, although negative rewards may help cut short the brute-force tree.
Real-life RL takes unlimited number of episodes (for each episode, a sequence of actions should be learnt), and the incremental training is harder and harder in time with more explored data, unless some past and no-longer-related data are removed, just like humans, we forget some of the past to learn more, remember more the present.

The technical barriers are at first the barriers to applying them to business. People may produce some supervised data manually rather quick, and thus supervised learning is usually opted first, nobody wish to try RL.
Harder to find human resources: AI engineers with experiences in supervised learning are more popular and easier to find some; fewer work with RL, thus business projects are not carried out easily if using RL.
However, from my point of view, RL is very much promising in future as AI entities are now more and more on their own.


text:
Does DQN generalise to unseen states in the case of discrete state-spaces?
In my understanding, DQN is useful because it utilises a neural network as a q-value function approximator, which, after the training, can generalise to unseen states.
I understand how that would work when the input is a vector of continuous values, however, I don't understand why DQN would be used with discrete state-spaces. If the input to the neural network is just an integer with no clear structure, how is this supposed to generalise?
If, instead of feeding to the network just an integer, we fed a vector of integers, in which each element represents a characteristic of the state (separating things like speed, position, etc.) instead of collapsing everything in a single integer, would that generalise better?

An environment is said to have a discrete state-space, when the number of all possible states of the environment is finite. For example, $3\times3$ Tic-tac-toe game has a discrete state-space, since there are 9 cells on the board and only so many different ways to arrange Os and Xs.
A state-space can be discrete regardless of whether integers or non-integers are used to describe it. For example, consider an environment where a state is represented with a single number. If the set of all possible states is $ \{0, 0.3, 0.5, 1\}$, your state-space is discrete, because there are only $4$ states. However, if all possible states is the set of real numbers from $0$ to $1$, than it's not discrete anymore - because there are infinitely many of them. State-space can still be discrete, even if possible states are represented with multiple numbers. For example, our environment could be $10\times10\times10$ cube, where the agent is only allowed to stand on integer coordinates. In this scenario, there are $1000$ different places where the agent can be and hence the state-space is discrete.
The Deep Q-Network can be designed to accept any type of input; just like a regular ANN, it's not restricted to only one integer. An input to DQN is the state of the environment, regardless of how it's represented. For the previous example, you can setup the DQN to have an input layer with $3$ neurons, each one accepting an integer that describes agent's position along the $x, y, z$ axes.
One downfall of Q-learning is that when the environment has a very large number of states and actions, representing each state-action pair becomes impractical in terms of memory. Think of chess, where there are so many different possible positions, and multiple available moves in each of them. Moreover, for an agent to learn properly, every state-action pair value must be visited (agent needs to determine Q-values), which can impractical in terms of training time.
DQN algorithm takes care of these problems: it only needs to store the neural network (also few other things if you use a variation of DQN), and it doesn't need to visit every state-action pair to learn. They way it learns is by adjusting the weights and biases in the network in order to approximate the optimal policy. Given the algorithm is implemented correctly, the agent should be able to pick up some useful patterns (or solve the environment).
I used this paper as a reference for one of my projects. It implements the DQN algorithm to learn to play Sungka (a game similar to Mancala), which has finite number of possible states and actions.


text:
Are there any active areas of research in machine learning that do not involve neural networks at all?
So far, I have not been able to find many papers that do not involve neural networks, so I was hoping I can gain some insight here. Any references would be greatly appreciated.

Numenta and Vicarious are using techniques different from neural networks. Look into their publications.

If you look into the top conferences on machine learning and neural networks, such as NeurIPS, ICLR, and ICML, you will find many papers related to neural networks and deep learning, given that these are still very hot/promising topics. However, occasionally, you will find accepted papers that do not involve neural networks. Here's a small list of them that I've found after a quick search.

Near-Tight Margin-Based Generalization Bounds for Support Vector Machines (ICML, 2020, pdf) by Allan Grønlund et al.
Implicit Regularization of Random Feature Models (ICML, 2020, pdf) by Arthur Jacot et al.
Quantum Expectation-Maximization for Gaussian mixture models (ICML, 2020, pdf) by Alessandro Luongo et al.
From PAC to Instance-Optimal Sample Complexity in the Plackett-Luce Model (ICML, 2020, pdf) by Aadirupa Saha et al.

So, yes, research in machine learning is not exclusively devoted to neural networks and deep learning. You can probably find more papers here or in the proceedings of similar conferences or journals.


text:
What is the meaning of the terms in this evaluation function for chess?
I'm trying to improve my evaluation and I saw this here
materialScore = kingWt  * (wK-bK)
              + queenWt * (wQ-bQ)
              + rookWt  * (wR-bR)
              + knightWt* (wN-bN)
              + bishopWt* (wB-bB)
              + pawnWt  * (wP-bP)

How do I get the value, let's say wK? Do I get the position of the king and score it relative to the board? For example, wK is more safe than the bK, so let's say wK - bK = 1 - 0.5. So, the result will be 90 * (0.5). Is this really how it works?

Is this really how it works?

Yes and no. An evaluation function based on pure material advantage is a perfectly legal function. It's certainly better than nothing. However, it's too simple in practice.
The state-of-the-art methods involve neural network. Google "Stockfish NNUE" for details.

The function you are showing (from this website) only calculates the score of a position based on how many pieces are on the board. It does not take into account where the pieces are located. So wK, bK, wQ, bQ, etc. are simply the number of the specific pieces that are on the board. The weights rank the pieces according to their importance.
So for example if white has 8 pawns and black has 6 pawns, the last term of the score would be pawnWt * (wP-bP) = 1 * (8-6) = 2. (assuming the value of a pawn to be 1) That leads to a relative advantage for white only based on pawns, but according to how many of the other pieces are on the board, black might still have an advantage.


text:
How to incorporate a symmetry constraint in the loss function to train a CNN?
I have a task of extremely sparse binary segmentation, i.e. the segmentation mask contains either 0 or 1, and there are ~95% zeros and only ~5% ones. I use the focal loss to address the sparseness (which is equivalent in my case to imbalances). I have another piece of information that I want to incorporate in the loss term.
The desired output is always symmetric over the diagonal. I was searching for a way to use this information in the loss, but I couldn't find a solution. How would I do this?
For some example of the symmetry in the segmentation maps, I added an arrow to show the axis of symmetry:

You could add a term that sums (or averages depending on your exact preference) the difference between both triangle parts of the matrix (self-regulating loss, as it does not need a true prediction).

If you know it is symmetric, then you could do a couple things.

Zero out a half.

Don't bother learning both halves of the image. Just put a zero mask over the upper or lower half of the output matrix and just have the network regress the other half. Just don't make the network do more work than it needs to do.

Learn both, but add symmetric loss

In your case, it looks like you could create two loss functions added together.
$focal(x, y) + focal(x^T, y)$
This will help the network learn both halves equally.

L1 between $x$ and $x^T$

This might be silly but adding a Huber loss between $x$ and $x^T$ might help promote symmetry, but I'm not as much if a fan of it. Personally I'm more partial to (1).

Combine (1) and (2)

You could take the loss function of (2) at train time but at test time just use whatever half had better metrics and copy that to the bottom half.

Post Processing

Add a post processing custom layer that takes the average of the two halves so that you guarantee symmetry.
$x' = \frac{1}{2}(x + x^T)$
Then do your normal focal loss. Personally I like this the best since it always guarantees a symmetric output and is a pretty easy custom layer.


text:
Do AlphaZero/MuZero learn faster in terms of number of games played than humans?
I don't know much about AI and am just curious.
From what I read, AlphaZero/MuZero outperform any human chess player after a few hours of training. I have no idea how many chess games a very talented human chess player on average has played before he/she reaches the grandmaster level, but I would imagine it is a number that can roughly be estimated. Of course, playing entire games is not the only training for human chess players.
Nonetheless, how does this compare to AI? How many games do AI engines play before reaching the grandmaster level? Do (gifted) humans or AI learn chess faster?

A simple Google search such as https://www.quora.com/How-many-games-did-Alpha-Zero-played-against-itself-during-its-four-hours-training will tell you about AlphaZero took about 44 million games to achieve in just hours. No human will of course can do it.

According to Table S3 of the AlphaZero paper (p. 15)

AlphaZero was trained for 9 hours and, during these 9 hours, it played 44 million games of chess.
According to this Wikipedia article, the longest human lifespan is that of Jeanne Calment, who lived to age 122 years and 164 days.
Let's assume that humans cannot live more than 123 years (which is a reasonable assumption, although this record could eventually be broken). Let's also assume that a chess game lasts at least 10 minutes, which means that you can play at most 6 games in 1 hour, which means that you can play at most $6*24 = 144$ games in one day (assuming that you never sleep, which is, of course, impractical, but I'm just trying to show you an upper bound). Let's say that a year has 365 days. So, here is roughly the maximum number of games that a human could play
$$
6*24*365*123 = 6464880 \tag{1}\label{1},
$$
which is smaller than 44 million games by a factor of more than 6, i.e. any human could at most play 1/6 of the games that AlphaZero played, and \ref{1} is a very loose upper bound that doesn't take into account that humans need to sleep, eat, and do many other things.
So, humans learn to play chess a lot slower than AlphaZero. This is not surprising at all, given that computers can perform calculations a lot faster than us (that's why they are called computers), and this has been the case for many years. We (humans) just made computers make the right calculations for them to approximately play chess better than us. That's it.

Thank you for the answer! So we learn a lot slower in absolute time, but the best chess players do very well in comparison in number of games played. It's frightening to think that one day AI may learn faster even in number of games played.

@220284 Good observation. We'll probably need to develop some new insights in transfer learning for that to happen, but I agree with you that, when AIs can learn from fewer examples/experience than us, that will definitely be a good sign of "intelligence" (or, more precisely, the ability to "generalize" or "transfer knowledge" that we often associate with intelligent systems).


text:
Reinforcement Learning algorithm with rewards dependent both on previous action and current action
Problem description:
Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way:

if current action == previous action, you get reward = $R(a,s)$
if current action != previous action, you get reward = $R(a,s) - \text{penalty}$

In this environment, switching actions bears a significant cost. We would like the RL algorithm to learn optimal actions under the constraint that switching action is costly, i.e. we would like to stay in selected action as long as possible.
The penalty is significantly higher than an immediate reward, so if we do not take it into account, the model evaluation will have a negative total reward with almost 100% probability, since the agent will be constantly switching and extracting rewards from environment that are smaller than the cost of switching actions.
Action space is small (2 actions: left, right). I'm trying to beat this game with PPO (Proximal Policy Optimization)
Questions

How one might address this constraint: i.e. explicitly make the agent learn that switching is costly and it's worth sitting in one action even if immediate rewards are negative?

How can you make the RL algorithm learn that it's not the reward term $R(a_t|s_t)$ that is negative, and thus decreasing $Q(a_t|s_t)$ and $V(s_t)$, but it's the penalty term (taking the action that is different from the previous action at step $t-1$) that is pushing total reward down?

The answer to both your concerns is:

Add the previous action choice to the state representation.

It is all you need to do. It gives the agent the data it needs to learn the association of negative reward from not matching the previous action.
By making this data part of the state, you re-establish the Markov property in the MDP model of the environment, which you had otherwise lost by making the reward dependent on a variable that was both systematically changing and hidden from the agent.
The state in a MDP is often not just the current observations that the environment provides, but can include any relevant knowledge that the agent has. At the extreme that can include a complete history of all observations and actions taken to date. It is common practice to derive the state as a summary of recent history of observations and actions taken so far. In your case, all you need do is concatenate the previous action to the observation, because you know about the constraint and how it affects optimisation.

Thanks! That was my initial idea too. You are absolutely right, that MDP property has to be restored. My concern is that adding previous action to the state representation might be not enough, as it will take some time for the agent to learn to associate this state feature with reward. Maybe there is a more explicit way of influencing learning that switching actions bears a penalty?

@FQT: There is no specific clever way that is normally used in RL to solve this issue as far as I know. However, the new MDP with the previous action might be solved faster using a different algorithm, as that part of the behaviour is deterministic. You could try to re-structure the agent in other ways too, because you know specific things aboutthis penalty that you might not know about the rest of the environment (for instance reward prediction models could focus only on the other part of the environment with action matching hard-coded), but modifying state is simpler and IMO best.

As said, you will have to include previous input state(s) in your training input patterns.
Suppose we use straightforward NN backpropagation learning..
You would expand the input layer with additional neurons and weights connected to the past. The time window neurons should introduce additional weights into the first hidden layer. A neural net architecture doing this is called Time Delay neural net (TDNN)
https://stats.stackexchange.com/questions/160070/difference-between-time-delayed-neural-networks-and-recurrent-neural-networks
I used TDNN in the past for signal processing. My TDNN includes the past of N inputs, supporting M steps in the past. A flexible TDNN can also be configured to connect some hidden layer past output to the next hidden layer.
https://neuron.eng.wayne.edu/tarek/MITbook/chap5/5_4.html
https://www.mathworks.com/help/deeplearning/ref/timedelaynet.html


text:
Aren't scores in the Wasserstein GAN probabilities?
I am quite new to GAN and I am reading about WGAN vs DCGAN.
Relating to the Wasserstein GAN (WGAN), I read here

Instead of using a discriminator to classify or predict the probability of generated images as being real or fake, the WGAN changes or replaces the discriminator model with a critic that scores the realness or fakeness of a given image.

In practice, I don't understand what the difference is between a score of the realness or fakeness of a given image and a probability that the generated images are real or fake.
Aren't scores probabilities?

I didn't read that paper and I don't have time now to read it, but the term score in statistics (and machine learning) sometimes refers to a specific concept. Take a look at this. If nobody provides an answer, you may want to read the WGAN paper (to know how they defined the score there) and the linked Wikipedia article, and then provide a formal answer to your own question below ;)

Figure 3 in the original WGAN paper is actually quite helpful to understand the difference between the score in WGAN and the probability in GAN (see screenshot below). The blue distribution are real samples, and the green one are fake samples. The Vanilla GAN trained in this example identifies the real samples as '100% real' (red curve) and the fake samples as '100% fake'. This leads to the problem of vanishing gradients and the well-known mode collapse of original GANs.
The Wasserstein GAN, on the other hand, gives each sample a score. The benefit of the score is that we can now identify samples that are more likely real than others, or more likely fake. For example, the further to the left a distribution is located, the more negative the WGAN score will be. We have therefore a continuum that doesn't end in 0 and 1, but can compare between samples that are 'good' and those which are 'better'. A normal GAN would identify both as 'good', making further improvement difficult.

Thanks! There is an aspect which is not clear to me, if a binary cross entropy is used as activation for the last layer of the discriminator you will obtain for each sample a probability for belonging to a class. So "the further to the left a distribution is located" the lower will be the probability for the samples to belong to the fake class. The only problem that could come to mind is that the classifier is so sure of the choice, since in this case the samples are even linearly separable, which will always return probability about 1 for one of the two classes, is it correct?

Yes! That's at least one of the reasons that lead to mode collapse in traditional GANs, i.e. the inability to recover a multi-modal distribution (see example in Figure 2 of the WGAN paper). If the generator outputs one of the modes, but the samples are already judged as "very real looking" (that is, "very far on the left"), there is no motivation for it to improve as the gradients are vanishing and don't give useful information. If instead the Wasserstein distance is used, which can be identified as a score, we don't have this problem but a monotonous curve with useful gradients.


text:
How to use DQN when the action space can be different at different time steps?
I would like to employ DQN to solve a constrained MDP problem. The problem has constraints on action space. At different time steps till the end, the available actions are different. It has different possibilities as below.

0, 1, 2, 3, 4
0, 2, 3, 4
0, 3, 4
0, 4

Does this mean I need to learn 4 different Q networks for these possibilities? Also, correct me if I am wrong, it looks like if I specify the action size is 3, then it automatically assumes the actions are 0, 1, 2, but, in my case, it should be 0, 3, 4. How shall I implement this?

There are two relevant neural network designs for DQN:

Model q function directly $Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$, so neural network has concatenated input of state and action, and outputs a single real value. This is arguably the more natural fit to Q learning, but can be inefficient.

Model all q values for given state $Q(s,\cdot): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$, so neural network takes input of current state and outputs all action values related to that state as a vector.

For the first architecture, you can decide which actions to evaluate by how you construct the minibatch. You pre-filter to the allowed actions for each state.
For the second architecture, you must post-filter the action values to those allowed by the state.
There are other possibilities for constructing variable-length inputs and outputs to neural networks - e.g. using RNNs. However, these are normally not worth the extra effort. A pre- or post- filter on the actions for a NN that can process the whole action space (including impossible actions) is all you usually need. Don't worry that the neural network may calculate some non-needed or nonsense values.

I am using the second architecture, can you elaborate more on how to post-filter the action values?

@TracyYang: That will depend on your software library. The point is to ensure that they are not considered when choosing an action to take (or looking at the maximum action value from a state for updates). So for example, if you are using a max or argmax function to select, then set the unwanted values to -inf before running it.

So, the action size is still 5, but when it should be 0,3,4 only, then set values corresponding to 1 and 2 to -inf first before using argmax. Is this correct?

@TracyYang Yes that is correct


text:
What does the parameter $y$ stand for in function $g(y,\mu,\sigma)$ related to REINFORCE algorithm?
I am wondering what the parameter $y$ in the function $g(y,\mu,\sigma)=\frac{1}{(2\pi)^{1/2}\sigma}e^{-(y-\mu)^{2/2\sigma^2}}$ stands for in Section 6 (page 14) of the paper introducing the REINFORCE family of algorithms.
Drawing an analogy to Equation 4 of the same paper, I would guess that it refers to the outcome (i.e. sample) of sampling from a probability distribution parameterized by the parameters $\mu$ and $\sigma$. However, I am not sure whether that is correct or not.

As far as I can tell, it is the point at which you would evaluate the density function of the random variable (in this case the normal distn). In a policy gradient setting I imagine that $y$ here is the action. Does this sound correct?

If you take a look at the Wikipedia page related to the normal distribution, you will see the definition of the Gaussian density
$$
{\displaystyle f(x)={\frac {1}{\sigma {\sqrt {2\pi }}}}e^{-{\frac {1}{2}}\left({\frac {x-\mu }{\sigma }}\right)^{2}}} \label{1}\tag{1}
$$
and you will see that the $y$ in your formula corresponds to the $x$ in equation \ref{1}.
I've seen this notation in the context of computer vision and image processing, where the Gaussian kernel is used to blur images.
So, as pointed out by someone in a comment, $y$ should indeed be the point where you evaluate the density.
Maybe the confusing part is that all parameters are treated equally in terms of their purpose, while $\mu$ and $\sigma$ are clearly the parameters that define the specific density, so they are not the inputs to the specific density.
After having read the relevant section of the paper, I now understand why you're confused. The author refers to $y$ as the output (not yet sure why: maybe it's the output of another unit that feeds this Gaussian unit?), but I think that this explanation still applies. The output of the Gaussian density $g$ is not $y$, but the density that corresponds to $y$. In fact, in appendix $B$ of the paper, the author says that $Y$ is the support of $g$ and $y$ is an element of $Y$.

So, in other words, $f(x)$ simply returns the probability of input value $x$ with respect to a Gaussian distribution parameterized by $\mu$ and $\sigma$. Is that correct?

$f(x)$ is not a probability but a density value, i.e. it's not necessarily a number in the range $[0, 1]$. Remember that, for continuous r.v.s., the probability that it takes a specific value is zero. So, with continuous r.v.s, you typically want to calculate the probability that the r.v. lies in some range. Note also that when people use the term "probability distribution", they may refer to the pdf, if the r.v. is continuous. Maybe check this answer.

Ok, thanks a lot! The referred answer helps quite a lot as well!

A crucial bit I just overlooked from the paper (page 14) was "density function $g$ determining the output $y$". So, $y$ is indeed a sample from a Gaussian parameterized by $\mu$ and $\sigma$. I suspect that the term $y$ is only shown in that definition of the density function $g$, since $g$'s partial derivative (in this way including $y$ as well) is used to compute the gradient of the policy network's output nodes having produced the parameterization which eventually led to the sampling of $y$ (where those nodes shall be updated also considering the "quality" of the drawn sample $y$).

@DanielB. Yes, I think you're right, $y$ can be a sample from the Gaussian distribution (so, in a way, an "output" from the Gaussian distribution), but it can also denote an input to the Gaussian density. So, the samples from the Gaussian distribution should indeed be values that you can evaluate the Gaussian density at. For example, let's say that you have a Gaussian with $\mu = 0$ and $\sigma = 1$. Then you sample $y=0$ (i.e. a number from the Gaussian, which, in this case, happens to be the mean), then you can evaluate the density of $0$ (it would be the height of this Gaussian at zero).

He's probably using the letter $y$ rather than $x$ to emphasize that this is a sample (output) from a Gaussian distribution rather than the value you want to evaluate the Gaussian density at, given that $y$, in the context of neural networks and stuff, is usually used to denote an output.

I think important is also to consider that the variable $x$ had already been reserved to denote inputs to NN layers constituting the $\mu$ and $\sigma$ generating NN. So, I think the author really intends to adhere to the standard Machine Learning way of referring to inputs and outputs as $x$ and $y$ respectively. And after all that's also pretty accurate; except for the fact that the NN (i.e. the policy network) is not a vanilla NN, but simply one whose regular output layer is appended by an additional layer of random generators generating outputs given the outputs of the previous layer.

It states that "To simplify notation, we focus on one single unit and omit the usual unit index subscript throughout"
So they are simply removing the i-th index from the equation for simplicity.  So g is a function of a given instance "y" and the parameters μ and σ.

But what, in particular, does the 'given instance "y"' refer to?

"y" is just some measurement or data instance.  So if you have a dataset of temperature measurements in the day, "y" could be one single temperature measurement in your dataset, e.g. 20


text:
How can I find words in a string that are related to a given word, then associate a sentiment to that found word?
I came up with an NLP-related problem where I have a list of words and a string. My goal is to find any word in the list of words that is related to the given string.
Here is an example.
Suppose a word from the list is healthy. If the string has any of the following words: healthy, healthier, healthiest, not healthy, more healthy, zero healthy, etc., it will be extracted from the string.
Also, I want to judge whether the extracted word/s is/are bearing positive/negative sentiment.
Let me further explain what I mean by using the previous example.
Our word was healthy. So, for instance, if the word found in the string was healthier, then we can say it is bearing positive sentiment with respect to the word healthy. If we find the word not healthy, it is negative with respect to the word healthy.

There are many ways to solve this problem.  One way is to apply stemming or lemmatization to reduce your words. Using NLTK's Porter stemmer for example on healthy, healthier, healthiest, not healthy, more healthy, and zero healthy gives:
healthi , healthier , healthiest , not healthi , more healthi , zero healthi

This can help make word comparisons easier.
Sentiment analysis on the phrases will provide positive, neutral, and negative scores.  There are a lot of algorithms for doing this but a common one is Valence Aware Dictionary and sEntiment Reasoner (VADER).  Here is a recent article with code using NLTK and the VADER lexicon:

How To Run Sentiment Analysis In Python Using VADER by George Pipis

The following article also does sentiment analysis using NLTK and includes stemming and lemmatization.  Instead of VADER they use a Naive Bayes classifier on a labeled data set of tweets: How To Perform Sentiment Analysis in Python 3 Using the Natural Language Toolkit (NLTK) by Saumik Daityari.

Maybe you can also describe how words should be compared for similarity.


text:
What is $ \nabla_{\theta_{k-1}} \theta_{k}$ in the context of MAML?
I am attempting to fully understand the explicit derivation and computation of the Hessian and how it is used in MAML.  I came across this blog: https://lilianweng.github.io/lil-log/2018/11/30/meta-learning.html.
Specifically, could someone help to clarify this for me: is this term in the red box literally interpreted as the gradient at $\theta_{k-1}$ multiplied by the $\theta_k$?

Here is a related question.

$\nabla_{\theta_{k-1}} \theta_k$ is gradient of $\theta_k$ with respect to $\theta_{k-1}$, it follows chain rule as noted in the side comment in the image. $\nabla_{\theta} \mathcal L(\theta_k)$ is also not a Hessian but a gradient vector.


text:
Why is neural networks being a deterministic mapping not always considered a good thing?
Why is neural networks being a deterministic mapping not always considered a good thing?
So I'm excluding models like VAEs since those aren't entirely deterministic. I keep thinking about this and my conclusion is that often times neural networks are used to model things in reality, which often time do have some stochasticity and since neural networks are deterministic if they are not trained on enough examples of the possible variance inputs in relation to outputs can have they cannot generalize well. Are there other reasons this is not a good thing?

Your intuition is right. The main reason why a deterministic function can be undesirable (or even dangerous, as I will explain below with an example) is that we may not have enough data to learn the correct function, so we may end up learning the incorrect one. Right now, no other reason, from a theoretical point of view, comes to my mind, but below I will mention a few applications/cases where a deterministic function may not be desirable.
If we had all data pairs $\{(x_i, y_i)\}$, where $x_i \in \mathcal{X}$ and $y_i = \mathcal{Y}$ are, respectively, an input and output from the unknown function that you want to learn $f: \mathcal{X} \rightarrow \mathcal{Y}$, i.e. $f(x_i) = y_i$, then you could reconstruct $f$: whenever $x_i$ is given, you just need to return $y_i$.
Of course, in reality, we almost never have a large enough (training) dataset to approximate out desired (but usually unknown) function. If we learn only one (deterministic) function, then, in principle, you can catastrophically fail, i.e. your approximation of $f$, denoted as $f_\theta$ (where $\theta$ are the parameters of the neural network or any other model), can produce outputs that are completely wrong.
Let me try to give you a simple example. Let's say that $f$ is defined as follows
$$f: \mathbb{N} \rightarrow \{0, 1\}$$
You are given a training labelled dataset $$D = \{(4, 1), (11, 0), (8, 1), (31, 0), (16, 1), (7, 0) \}.$$
Apparently, our unknown function is defined as
\begin{align}
h_1(x)=
\begin{cases}
1, &x \text{ mod } 2 \equiv 0\\
0, &\text{otherwise}
\end{cases}\tag{1}\label{1}
\end{align}
Given that $D$ is small, your neural network, $f_\theta$, can easily overfit $D$, i.e. learn to output $1$ when $x$ is even and $0$ otherwise.
However, what if $f$ is not that function in equation \ref{1} and we collected just a dataset that doesn't represent $f$ well enough? If you look at $D$ more carefully, you will see that another possible hypothesis for $f$ is the following
\begin{align}
h_2(x)=
\begin{cases}
1, &x \text{ mod } 4 \equiv 0\\
0, &\text{otherwise}
\end{cases}\tag{2}\label{2}
\end{align}
However, given that your neural network can only compute one of these functions at a time, it could compute the wrong one. Let's say that $f_\theta \approx h_1$, then it should produce $1$ when $x = 6$ (an even number). If the correct unknown function was $h_2$, i.e. $f = h_2$, then $f_\theta(6) = 1$ would be wrong (because $6$ is not a multiple of $4$).
Of course, this is just a toy example. However, there are many other cases where this can happen, which may not be desirable, such as healthcare, medicine or self-driving cars, where the wrong prediction can lead to catastrophic consequences, such as the death of a person.
If we maintain a probability distribution over the possible functions that are consistent with the observed data so far, we can (partially) avoid this issue. So, continuing with the example above, this probability distribution over functions should be highly uncertain about $x = 6$, whether it produces $0$ or $1$, because it has never seen the label for $x=6$, so a medical doctor or the human driver could intervene in the case of (high) uncertainty.
For this reason, in the last decade, people have started to incorporate uncertainty estimation in neural networks. Neural networks that model uncertainty (to some degree) are often called Bayesian neural networks (BNNs), and there are different approaches (such as variational BNNs, MC dropout or Monte Carlo-based approaches). If you are interested in this topic, the paper Weight Uncertainty in Neural Network (2015) is a good start, especially if you are already familiar with VAEs. Given that this is a very new research area, the current solutions are still not very satisfactory. For example, you can find examples in the literature that report that MC dropout can produce very bad estimates of uncertainty (even in my master's thesis I have observed and thus concluded that this is the case), i.e. they can be highly certain when they should be highly uncertain.