text:
Can Q-learning be used in a POMDP?
Can Q-learning (and SARSA) be directly used in a Partially Observable Markov Decision Process (POMDP)? If not, why not? My intuition is that the policies learned will be terrible because of partial observability. Are there ways to transform these algorithms so that they can be easily used in a POMDP?

I think answers so far are missing the fact that classifying a problem as a POMDP is not a simple binary choice. There are both types and degrees of POMDP. For instance, Open AI Gym's LunarLander-v2 is strictly a POMDP as the agent does not observe the mountains which vary on each episode, and can have a strong effect on the outcome. However, a straightforward DQN can solve it well. Do you have a specific problem in mind when asking this question? Are you interested in these degrees of POMDP, or should we assume a difficult one, like Poker?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DQN (an extension of Q-learning) is further extended to handle the partially observed nature of a single frame of an Atari 2600 game.

@NeilSlater What do you mean by "degrees"? Do you mean that full information about states are sometimes available even in the context of POMDPs? I don't get how this is strictly related to the question.

@nbro: I mean there is more than one way for a system to be a POMDP. Philip linked a paper above that I was going to refer as well - one way for an environment to be a POMDP is to have states that do not summarise history fully, but where actual state is relatively easy to infer from history - and you can see that DQN can be extended using a RNN in place of regular NN, without many other changes for that kind of environment. There are other ways that a system can be a POMDP, for example in some games, the environment will use additional knowledge against the agent, which is harder to solve.

@NeilSlater Yes, I am aware of this. I added a paragraph to my answer that is related to this.

@nbro: That's fine. It would be hard to cover all the ways a system can be a POMDP (and would require a different sort of answer). In quite a few cases, extensions to DQN may not even be required, or can be straightforward. Which is why I am asking OP for clarification.

@NeilSlater I don't think the question needs more clarification. In general, Q-learning assumes that the current state is Markov and accessible, but this is not the case in POMDP, so Q-learning (tabular one, at least) shouldn't be directly applicable to POMDP. Nowadays, people use NNs to represent the Q functions, so a lot of approximations can be done, and Q-learning or variations of it are applied to partially observable problems/environments (where some engineering is made in order to build/represent artificial states).

@nbro: There's more to it than that, and I think clarification would help.

@NeilSlater I think you brought up a good point that I wasn’t not aware of. Then I hope to know how applicable Q learning and SARSA are to each type and degree of POMDP.

The usual (as presented in Reinforcement Learning: An Introduction) $Q$-learning and SARSA algorithms use (and update) a function of a state $s$ and action $a$, $Q(s, a)$. These algorithms assume that the current state $s$ is known. However, in POMDP, at each time step, the agent does not know the current state, but it maintains a "belief" (which, mathematically, is represented as a probability distribution) in what the current state might be, so it cannot maintain (an approximation of) the function $Q(s, a)$. Hence, the usual Q-learning and SARSA algorithms shouldn't be directly applicable to a POMDP.
However, $Q$-learning is often used in the contexts where observations emitted by the environment (or transformations of the raw observations) are used to build the current state (which is assumed to be Markov, even if it is not). For example, in the original DQN, the action taken at the current step and the raw observation and the reward emitted by the environment (after this action is taken) are combined to produce the current (Markov) state. It might not be the case that the way they combine the action, the reward and the observation is sufficient to fully describe the current state (which might not even be Markov).
In this report, Deep Reinforcement Learning with POMDPs, the author attempts to use Q-learning in a POMDP setting. He suggests to represent a function, either $Q(b, a)$ or $Q(h, a)$, where $b$ is the "belief" over the states and $h$ the history of previously executed actions, using neural networks. So, the resulting parameterized functions would be denoted by $Q(b, a; \theta)$ or $Q(h, a; \theta)$, where $\theta$ is a vector representing the parameters of the corresponding neural network. Essentially, the author uses a DQN (with an experience replay buffer and target network), but the results are not great: $Q$ values converge, but policies do not and they are not robust (in that they are sensitive to small perturbations).

Another approach (can't recall paper) is to factorize state space into observability classes, states of the same class have the same observation, assign reward and transition distributions to those classes, after that work with them as observable MDP

@mirror2image any reference to that?

@drerD google POMDP infosets (also infosets determinization)

@mirror2image You can provide a formal answer! I encourage you to do it.


text:
Understanding the configuration of replay memory and epsilon in deep reinforcement learning
I am tentatively reusing a codebase of pacman to train my own deep reinforcement learning model. While most of the components seems reasonable and understandable to me, there are two things that seem obscure to me:

How to decide the size of the replay memory? Currently, since I set the total step of learning as 4000 (note that in the referred codebase this value is set as 4000000), so I just proportionally decrease the replay_memory_size as 400. Would that make sense?
What is the return value epsilon when calling function PiecewiseSchedule? I also proportionally decrease its parameters as follows:

        epsilon = PiecewiseSchedule([(0, 1.0),
                                     (40, 1.0), # since we start training at 10000 steps
                                     (80, 0.4),
                                     (200, 0.2),
                                     (400, 0.1),
                                     (2000, 0.05)], outside_value=0.01)
        replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)

where the original function call is like this:
        epsilon = PiecewiseSchedule([(0, 1.0),
                                     (10000, 1.0), # since we start training at 10000 steps
                                     (20000, 0.4),
                                     (50000, 0.2),
                                     (100000, 0.1),
                                     (500000, 0.05)], outside_value=0.01)
        replay_memory = PrioritizedReplayBuffer(replay_memory_size, replay_alpha)

And in general, what is the principle (guideline) behind setting a good size of "replay memory" and calling function PiecewiseSchedule? Thank you!

Just to correct you, it is not reply memory, it is replay memory

Replay memory is a recording of the games that the agents has played. It is this data that is used to train the neural network (or whatever machine learning method you are using).
The training is performed by looking at the games played and the resulting reward signal (in Pacman, this might be your score), and then learning which game strategies worked well and which did not.
It is difficult to say exactly how big the replay memory should be, this is something that you will need to experiment with. The larger it is, then the more examples will be available to be learned from during a particular training period, but of course it takes longer to acquire them.
Epsilon is a parameter that governs the trade-off between exploration and exploitation. By this, I mean that for each action that your agent takes, it needs to decide whether to do the best thing that it has already discovered to do (exploitation), or try something new (exploration). Usually, a value of epsilon near to 1 means that the agent will choose to explore more, and a value closer to 0 will mean that the agent will exploit more.
You can imagine therefore that at the beginning of an agent's training you might want epsilon ~1, because it has lots of new strategies to try to discover, but as it gets better you would reduce it to ~0 so that it can explore strategies that are only reachable after a long sequence of game steps.
Naturally, when the agent is fully trained and ready to be deployed then epsilon would = 0.
In the example that you posted, you can see that Pacman starts off doing nothing but exploring, but after 500000 games it exploits 95% of the time.

To augment the last paragraph I would add that the exploitation vs exploration mix usually needs to keep the agent close to optimal behaviour towards the end, even for off-policy learners (e.g. Q-learning) because the agent needs to learn about what happens close to the optimal path in order to make minor switches of action. In an environment that is challenging to complete (e.g. many steps before top rewards), then a high epsilon won't ever reach states that are interesting for optimal control.

@NeilSlater Yes, that is a good explanation. I'll edit to include.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What model to use for fully unbalanced data?
I am working on an anti-fraud project. In the project, we are trying to predict the fraud user in the out time data set. But the fraud user has a very low ratio, only 3%. We expect a model with a precision more than 15%.
I tried Logistic Regression, GBDT+LR, xgboost. All models are not good enough. Step wise Logistic Regression performs best, which has a precision of 9% with recall rate 6%.
Is there any other models that I can use for this problem or any other advise ?

@DuttaA This should probably be upgraded to an answer, since the OP's problem is exactly what Anomaly Detection is for.

You can balance your data-set.
Many models work with batches of samples. If you have a very unbalanced dataset, you can simply split it and ensure your batches are balanced (for example, for a Neural Network, using minibatches of 32 samples, you could draw 16 from your fraud users, and 16 from non-fraud users).
During the learning phase, this ensures the model doesn't just output the most common class, but instead tries to learn to distinguish both.

Ive used it before for a skewed movie ratings dataset without issues. Keep in mind ypu still use the same amount of samples. You just change their relative frequency, such that both classes are equally likely to appear

Heavily imbalanced classification tasks do not need a certain type of model, you can get different ones to work.
You have two options: either use class weights (for example setting them to 'balanced' in the ScikitLearn SVM) in order to indicate that samples from a class are more important (the underrepresented one) or rebalance your dataset. For rebalancing purposes, and assuming you are using Python, I recommend Imbalanced Learn. There you have algorithms for over-sampling, under-sampling, over-sampling followed by under-sampling or ensemble sampling. If you use them, please check the plausibility of the synthetic samples you created by reducing dimensionality first and then plotting them for example in two dimensions. Are the synthetic samples similar to the true class?
I would also recommend you to think about relevant metrics for (heavily) imbalanced problems and consider the no-information rate. That is another question though.

I tried under-sampling. It performs quite unstable between each run, some times better and most of the time it performs worse.

What I would do is not try one of the techniques randomly, but reduce your data to 2D and plot it before and after the transformation. Then check for plausibility. Try also for example the Smote-Tomek technique, which combines Over- and Undersampling. If both methods do not work to your satisfaction simply use class weights.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What is Imagination Learning and Imagination machines?
I recently came across a Quora post, where I saw the term "Imagination Learning". It seems to be based on something called "Imagination Machines" (the link is based on a guy's work profile as of now; subject to change).
The only thing that I could find on Internet about it is this paper: Imagination-Augmented Agents for Deep Reinforcement Learning. (But I'm not sure if it's related to that concept.)
Any ideas on this would be appreciated.

The whole idea of Imagination Learning seems to be in its infancy. The author of the response you linked wrote a paper on the subject, but as he notes the paper is more of an outline for "a new overarching challenge for AI." The second link you posted is not directly related to the ideas referenced in the paper on Imagination Learning, other than that both use the idea of imagination in humans as inspiration. In short, there does not seem to be much information on this topic and I suspect it will stay that way until we learn some of the underlying processes that go along with imagination in humans and then maybe Imagination Machines may come to fruition.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Recognition of lines in a chalkboard
I'm trying to develop a real-time application that, from the sequence of chalkboard images captured by a webcam, recognizes the lines being draw on it.
It must be able of recognize the lines from the chalkboard background, filter the presence in the image of the teacher, and translate these lines to some representation, something as a list of basic events like "start of line at xxx,xxx", "continue line at xxx,xxx", ...
After several days looking for references and bibliography, none is found. The most similar are the character recognition applications, in particular when they have a stroke recognition stage.
Any hint ?
Input will be a sequence as this one,  this one or this one (just without the presence of the students). I've expect the teacher not hidding his hand. We could imagine a start with an empty chalkboard.
Thanks.
Note: I am looking for more than an answer which says only something similar to "you can use a deep learning training it with two classes", without details or references.

I would look into Hough transform https://en.wikipedia.org/wiki/Hough_transform

@GeorgeWhite: thanks for your comment. As you know, Hough transform is mainly to detect straight lines. It can be generalized to some other parametric curves, but I do not see promising use it in the case of free forms.

From ""start of line at xxx,xxx", "continue line at xxx,xxx"" I thought you were assuming straight lines.

I will assume that the camera is stable (no change in position, zoom or other settings during the video recording), otherwise the task becomes markedly more complicated.
Let's say that your dataset is an array of rasters (images in array format). You mention that you want to detect events "start line" and "end line".
One way of doing this would be to compute an approximate time derivative of your image series. For instance, take the image raster at index idx and the one right after, at index idx+1 (captured at instants $t$ and $t + \Delta t$, where $\Delta t$ is the sampling interval).
At coordinates $(i,j)$, this derivative could look something like: timeDerivative = (images[idx][j][i] - images[idx+1][j][i])/DeltaT. This is a crude estimate, and there are better ways of computing an approximate discrete derivative, but you get the idea.
Following this, you could declare a state of the recording: drawing line or not drawing line. The states, we assume are always alternating, as a teacher has to take his hand off the blackboard to draw a new line.
When a derivative with large values is detected (a region in the image goes from being black to white suddenly) and the state is "not drawing", the event "start line" is recorded and the state switches to "drawing". While a derivative with large values continues to be detected as time goes on in the vicinity of the previous spot with a large derivative, nothing changes. Once this is no longer true, the state changes to "not drawing" and the event "stop line" is recorded at the last location with a large derivative.
This is the main idea, which can be improved with:

Defining the area of the blackboard, either by hand or automatically
Thresholding the images to better isolate the chalk trace from the blackboard
Using a tracker such as a Kalman filter to know where to look next for the chalk

Thanks for your answer. I think movement of the teacher in front of the chalkboard will cause hard interferences to your proposal of algorithm.

Indeed. To cut out the teacher, you can threshold then apply a blob extraction algorithm, such as the two pass algorithm. This can identify the teacher if done correctly. You can then cut the teacher out of the image and color his pixels like the blackboard. This comes of course before the start of what I wrote above.

Or maybe just divide the shalkboard in regions and update them only if the teacher isn't present. Just tracking how much the green part is changed in a region with a threshold could be a simple and effective way to identify the presence of a big object and reject the update. A heavily pixelated version of the video can work too as "region" selector

Maybe Just use simple Convnets (Pre-trained perhaps) and train it on the images of the teacher on the blackboard. You could use a GAN to remove the teacher and complete the rest of the image (https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwj9lKKSh43uAhWbfH0KHeaDC1UQFjAAegQIAxAC&url=http%3A%2F%2Fstanford.edu%2Fclass%2Fee367%2FWinter2018%2Ffu_guan_yang_ee367_win18_report.pdf&usg=AOvVaw2tG3bStRlys_NPLX9-XPep) But that would be too troublesome.
The best way would be to take real-time video chunks, use a Convolutional Network to detect the shapes you want, and return bounding boxes for the appropriate shapes (for the location purposes) and their 'classification' - whether a straight line, curved line, or some other user defined shape. You could also choose to use some other YOLO (You Only Look Once) technique. You can check out with: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjAJegQIARAC&url=https%3A%2F%2Ftowardsdatascience.com%2Fobject-detection-using-deep-learning-approaches-an-end-to-end-theoretical-perspective-4ca27eee8a9a&usg=AOvVaw2TkW094_O5Q7-mcVMJ5SEN ;
You also won't have to deal with the pesky teacher with the above method (assuming he doesn't stand in one place and constantly block a part of the BB). These methods are near SOTA and would be extremely effective than conventional algos. Not to mention that using Keras is a piece of cake with a giant community and endless resources to help you in case you get stuck on some problem. Since it is easy to use, you can setup a prototype in almost no time.
Beginner's guide and Introduction-> https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjALegQIGhAC&url=https%3A%2F%2Fmachinelearningmastery.com%2Fobject-recognition-with-deep-learning%2F&usg=AOvVaw3M-b1gYnbTdzzwPahTTWWR ;
A research paper from ArXiv--> https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjAPegQIHhAC&url=https%3A%2F%2Farxiv.org%2Fpdf%2F1807.05511&usg=AOvVaw33CWoOX4LlrA3T_f75zVZu
Training with YOLO: https://machinelearningmastery.com/how-to-perform-object-detection-with-yolov3-in-keras/

please, recall note at the end of the question

@pasabaporaqui You want someone to implement the code for you? Apart from that, the only details I can add in my answer is an explanation on CNN's, if you want.

What I do not want, as the note says, is a empty "use a GN" or "use a CNN" followed by a useless list of 3 links explaining the generics of Deep Learning or object detection. Instead, it is expected a reference to a paper, algorithm or method, something as "in xxx work xxx shows how he/she has combined the Hough transform with a CNN to obtain the xxx result in a scenario similar to the one of the question".

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What are the most common fault prediction algorithms?
I have to predict a fault (automotive related) as much in advance as possible.
Right now I have found a solution that is somewhat satisfactory (a good number of true positives and a low number of false positives) but it doesn't feel optimal.
I know that usually this fault appears when a variable X drops (although it's not always a fault when X drops).
For each fault (time t0) of each vehicle in the training dataset I have found the moment (t-k) when the drop of X starts. I have set the target variable y to 1 from t-k to t0. y is 0 anywhere else.
Using XGBoost I then predict y.
The problem now is that I have multiple predictions  for each fault (one for each record) and I need to group them somehow.
To send the alert I could simply take the first prediction equal to 1 for each fault of each vehicle, but by doing so I would have too many false positives.
The solution I am using right now groups predictions using a "manual" logic: there must be a minimum number of predictions equal to 1 in the last hours, there must be a minimum ratio of predictions equal to 1 compared to predictions equal to 0, variable X cannot be higher than a certain threshold (because if it's not dropping it shouldn't be a fault), etc.
But by doing so I have to find the optimal combination of these parameters (with a grid search) and it doesn't feel like it's the right way to do it.
Can anyone share a better way to do this?


text:
Should the training data be the same in each epoch?
Should the training data be the same in each epoch?
If the training data is generated on the fly, for example, is there a difference between training 1000 samples with 1 epoch or training 1000 epochs with 1 sample each?
To elaborate further, samples do not need to be saved or stay in memory if they are never used again. However, if training performs best by training over the same samples repeatedly, then data would have to be stored to be reused in each epoch.
More samples is generally considered advantageous. Is there a disadvantage to never seeing the same sample twice in training?

Can you please clarify what you mean by "data is generated on the fly"? Do you mean that at each epoch new data is sampled from the data-generating process?

An epoch refers to running over the entire training set. So for an epoch to actually be an epoch, the data must be the same. If the data changes each epoch, you aren't running epochs, but rather iterations. I'm confused as to why there are answers suggesting otherwise.

@nbro that is correct. Every sample could be unique for a virtually unlimited data stream. But if training requires that a specific set of samples must be seen repeatedly, then that data must be saved to be used again.

Let's quickly get out our copies of Deep Learning by Goodfellow et al. (2016). More specifically, I'm referring to page 276.
On this page, the authors argue for a relatively small minibatch size, since there are less than linear returns for estimating the gradient when increasing the minibatch size. Returns here refer to the reduction of the standard error of the mean (gradient per weight) computed over a minibatch.
So, yes. In theory, having unlimited resources, you will get the best performance when averaging the loss over all samples in your dataset. In practice, however, the larger the size of minibatches, the slower the training procedure, and consequently the less the total number of weight updates that can be afforded. Reversely, in practice, the cheaper the weight updates, the quicker the training procedure can converge to a (subjectively) satisfactory result.
Eventually, also Goodfellow et al. state that rapidly computing gradients leads to much faster convergence (in terms of total computations) for most optimization algorithms than when training them more slowly on exact gradients.
So, to summarize: If the main concern is to get to a specific level of accuracy at all, go for rather low minibatch sizes, whereas you could go up to a few hundreds (as the Goodfellow et al. state as a reasonable upper bound on page 148) if you are interested in more accurate gradients for your weight updates.

Thank you for the reference to this book. I will read it

This would be more suitable as a comment but I don't have enough points; but here's my opinion.
Optimisation algorithms like gradient descent are iterative algorithms. So it is rarely possible that they arrive at the minima in 1 epoch. A single epoch means that all data points have been visited once or a certain number of data samples have been taken from a distribution. However more passes might be necessary.

generated on the fly

I am assuming that the data is being generated as a part of a fixed distribution. Hence multiple epochs of multiple samples is still the ideal scenario.
1000 samples 1 epoch: Not enough training.
1 sample 1000 epoch: Overfitting or possibly not enough training.

How does this answer the question?

My apologies @nbro. I think now the answer is more suitable.

I think you need to explain a bit more.... Answer seems off, although, seems correct to me.


text:
Embedding Layer into Convolution Layer
I'm looking to encode PDF documents for deep learning such that an image representation of the PDF refers to word embeddings instead of graphic data
So I've indexed a relatively small vocabulary (88 words). I've generated images that replace graphic data with word indexed (1=cat, 2=dog, etc) data. Now I'm going to my NN model
right_input = Input((width, height, 1), name='right_input')
right = Flatten()(right_input)
right = Embedding(wordspaceCount, embeddingDepth)(right)
right = Reshape((width, height, embeddingDepth))(right)
right = vgg16_model((width, height, embeddingDepth))(right)

Image data is positive-only and embedding outputs negative values though so I'm wondering if it is necessary to normalize the embedding layer with something like this after the Embedding layer
right = Lambda(lambda x: (x + 1.)/2.)(right)

The word indexed image looks like this:

Also, is this a problematic concept generally?

Thus far, I am finding that the Embedding does work. Lambda normalization doesn't seem to make a difference, but a OneHot (custom) layer seems to work better than the Embedding layer. Might be a little bit far in the stack to train properly


text:
How do randomly initialized neural networks behave?
I am wondering how the output of randomly initialized MLPs and ConvNets behave with respect to their inputs.  Can anyone point to some analysis or explanation of this?
I am curious about this because in the Random Network Distillation work from OpenAI, they use the output of randomly initialized network to generate intrinsic reward for exploration.  It seems that this assumes that similar states will produce similar outputs of the random network.  Is this generally the case?
Do small changes in input yield small changes in output, or is it more chaotic?  Do they have other interesting properties?

The output of randomly initialised neural networks will be random. I am not sure if I understand your question correctly but I would like to try and explain what OpenAI is trying to achieve with their RND Distillation approach.
Any re-enforcement or supervised "learning" algorithm such as a neural network (NN) with back-propagation, genetic algorithm, partial swarm etc requires a fitness function. The fitness function "directs" the algorithm to a solution.
The basic idea is that the algorithm is randomly initialised. The algorithm is run, and then parts of the algorithm are changed. In this case the weights of the NN.
A number of algorithms exist to update the weights, the most common back propagation / gradient decent as used in the paper. To update the weights the output of the NN is compared to the optimum output. Sometimes the optimum is known. Other times it is not. In the case of a game the optimum output is not known. It is assumed that the more points gained the better the NN does. In this case the fitness function or policy is the sum of points gained when playing the game. But gradient descent needs an error (i.e difference between output and expected output), which does not exist when playing a game, because you don't know what the points should be. However the NN could play one round of the game score some points, then run the NN again on the next move to determine a prediction of future points. The difference between the points gained and the prediction is then the error, which gradient descent can use. [Hope this makes sense]
This is fine for games like pong, a NN could play the game pong randomly and score point at each prediction, which will direct gradient descent into updating the weights to maximise points.
However, games such as Montezuma’s Revenge, requires a lot of complex actions before points are gained. If the points stay 0, how does gradient descent determine if the weights should be decreased or increased?
Instead, the authors initialise a second random neural network which remains the same for the duration of training. As mentioned above the output of a randomly initialised neural network is random.
The output of the actual NN, and world state is then passed through the random NN (RND), which will return a random value. The NN is then required to not only determine a game action but also determine the result of the RND.
The policy function is extended from just points to also predict the result of the RND. The greater the prediction error the more bonus points are allocated to the policy function. This encourages gradient descent to change the weights of the NN to explore unseen parts of the game as this increases the prediction error of the RND which increases points. At the same time gradient descent must update the weights of the NN to predict the RND.
I hope this answers your question.
In summary randomly initialised NN include multilayered perceptrons or convolutional networks will behave randomly with respect to their inputs. The RND network is randomly set once at the star of training, and used to basically remind the NN undergoing training that it has done this before do not do this again - i.e. rewards the NN for doing something new (exploration).

Here is someone that explained it better than me :) https://towardsdatascience.com/curiosity-in-deep-reinforcement-learning-understanding-random-network-distillation-747b322e2403

My question is more about how chaotic they are: like do small changes in inputs in the input yield small changes in outputs? (I updated the question to clarify this)
I would like to see some experiments that show this, or some theoretical analysis.
What about them makes them good for curiosity?  Like why not just use a single layer of random weights that you pass the input through?  My guess is that they have some interesting properties.

Oh I found this Twitter thread that I was thinking about: https://twitter.com/LiamFedus/status/1020324447348318208 when asking this question

Ah :) ok that is an interesting question. I don't know of any experiments that have done this. But I suspect the effect input has on output would be dependent on the architecture. In a small NN with few inputs, a small change in one input may make a huge difference on the output. In a large network I think the weights would make  a bigger difference. I don't really know the answer to this. It is an interesting question!

In the Large-Scale Study of Curiosity-Driven Learning paper (the prequel to the Random Network Distillation work), in their discussion of Random Features, they reference 3 papers that discuss this:

K. Jarrett, K. Kavukcuoglu, Y. LeCun, et al. What is the Best Multi-Stage Architecture for Object Recognition?
A. M. Saxe, P. W. Koh, Z. Chen, M. Bhand, B. Suresh, and A. Y. Ng. On Random Weights and Unsupervised Feature Learning
Z. Yang, M. Moczulski, M. Denil, N. de Freitas, A. Smola, L. Song, and Z. Wang. Deep Fried Convnets

I just briefly glanced over these.  For now, one interesting idea from [2] is to use randomly initialized networks for architecture search.  To evaluate the architecture for the task, you don't have to train it; you can just randomly initialize it and measure its performance.


text:
What are suitable predictive analytics models for data from multiple sensors?
I am a newbie in the field of AI/ML. I am trying to implement predictive analytics model on the data generated and collected every minute from a device with sensors.
I have two questions:

What are various ML algorithms I can use to predict the number of
people in a room given temperature, humidity, luminosity, and motion
{ 0 | 1 }?
What other things can we predict using the above data from the
sensor that is deployed in a closed room?

Context:
The device sends temperature, humidity, luminosity, and motion(yes/no) in real-time. I deployed this device in a closed room and started collecting data. Now I want to use this data to predict the number of people in the room using the data collected. I believe a multiple (linear/poly) regression model will help me in achieving this but, wanted to know if there are any other algorithms or any other use cases I can look into.

I expect there will be a lot of error margin (as I know people who can't sit still, and other who are extremely sedentary; some sweat a lot and other's not so much) but it's an extraordinarily interesting problem.  Welcome to AI!

What about sound?  Humans make a lot of sound (breathing/heartbeat vibrations etc) depending on how good your sensors are and where they are located...?  Just an idea...

@DukeZhou one here, is trying to broaden the scope of the question.

@DenizC That's a great idea.  Dogs will generally pant, or their collars give a little jingle.  Wonder if the sensors can pick up things like heartbeat though...

Google remote heartbeat sensors - military has been using them for years to detect heartbeats through walls.


text:
Could a CNN hear the difference between sound of a pet moving, and a person?
"Deep Learning" neural networks are now successful at image-recognition tasks that I would not have expected say 10 years ago.  I wonder if the current state of the art in machine learning could generally tell the difference between the sound of a dog or cat moving around a house, and a person walking in the same area, taking as input only the sound captured by a microphone. I think I could generally tell the difference, but it is hard to explain exactly how.  But this is also true of some tasks that deep learning is now succeeding at.  So, I suspect it is possible but it's not clear how you would go about it.
I have found algorithms to detect human speech (wikipedia:"Voice activity detection") but separating animal and human footsteps seems more subtle.

Welcome to AI!  This sounds entirely possible.  While it would be easy to identify a dog, say, on a hard floor by the scrabbling of their nails, I'm guessing that relative weight of the subjects would be a reliable dataset, even on carpet.  Humans weigh more than most dogs, certainly than most cats.  Patterns of footfalls would likely be distinct for all three.

I think it should be possible. The main difficulty will be in getting enough labelled data, deep learning approaches are very data hungry. Audio tasks such as speech typically require hours of labelled data. It's hard to tell from the problem description whether you would need that much - the classification is simpler (just two classes, not identifying phonemes), but I think the perceptual difference between the classes is harder, as you say yourself you are not quite sure how you are sensing the difference.
You might have success with a relatively low amount of data if you can find a pre-trained deep network for some other audio task (with a similar domain e.g. activity classification), and fine tuning the last few layers using your own data. One such pre-trained model is called VGGish and has been trained on 100s of hours of short video soudtracks.

It is an interesting application. It is possible. You can interpret sound as histogram (2D image) and apply same image processing techniques (CNN) to extract information. Alternatively, you can keep them as phase / intensity values and train a network on top of them (RNN). That is a great idea. Go for it!


text:
Does gradient descent in deep learning assume a smooth fitness landscape?
I've come across the concept of fitness landscape before and, in my understanding, a smooth fitness landscape is one where the algorithm can converge on the global optimum through incremental movements or iterations across the landscape.
My question is: Does deep learning assume that the fitness landscape on which the gradient descent occurs is a smooth one? If so, is it a valid assumption?
Most of the graphical representations I have seen of gradient descent show a smooth landscape.
This Wikipedia page describes the fitness landscape.

What is the fitness landscape? Is it the surface where gradient descent occurs? To compute the derivatives we need a smooth manifold. So, I think the landscape is smooth.

I am not sure about what fitness landscape mean but if it is the landscape where gradient descent occur. Then I think smooth means there is not discontinuty or sudden jump in the surface, so we can find out the derivatives.

i dont believe it's smooth landscape theoretically, since gradient descent in neural net does partial derivation thus other weights may get to wrong values when a certain weight is optimised; this results oscillation sometimes

You say "I've come across the concept of fitness landscape before". Can you please provide the link to the research papers or books that use this term "fitness landscape". I think this term occurs more in the context of evolutionary algorithms and not deep learning.

@nbro: You are right. It does occur mainly in the evolutionary context. I just thought the concept might still be applicable.

Please, provide the link to the papers where you found this term. People are giving answers but maybe based on wrong assumptions, and that makes those answers basically useless, because you didn't provide enough context.

https://en.m.wikipedia.org/wiki/Fitness_landscape

Just edit your post to include the relevant links and context, and maybe explain that you though that "fitness landscape" would also be applicable in the context of DL.

The Wikipedia page gives a good overview

So, what do you mean by "fitness landscape" in the context of deep learning? Maybe it would be better to change "fitness landscape" to a more appropriate term. Maybe "loss landscape"?

Main answer
To answer your question as directly as possible: No, deep learning does not make that "assumption".
But you're close. Just swap the word "assumption" with "imposition".
Deep learning sets things up such that the landscape is (mostly) smooth and always continuous*, and therefore it is possible to do some sort of optimization via gradient descent.
* quick footnotes on that bit:

Smoothness is a stronger condition than continuity, that's why I mention them both.
My statement is not authoritative, so take it with a grain of salt, especially the "always" bit. Maybe someone will debunk this in the comments.
The reason that I say "(mostly) smooth" is because I can think of a counter example to smoothness, which is the ReLU activation function. ReLU is still continuous though.

Further elaboration
In deep learning we have linear layers which we know are differentiable. We also have non-linear activations, and a loss function which for the intents of this discussion can be bundled with non-linear activations. If you look at papers which focus specifically on crafting new types of non-linear activations and loss functions you will usually find a discussion section that goes something like "and we designed it this way such that it's differentiable. Here's how you differentiate it. Here are the properties of the derivative". For instance, just check out this paper on ELU, a refinement on ReLU.
We don't need to "assume" anything really, as we are the ones who designate the building blocks of the deep learning network. And the building blocks are not all that complicated in themselves, so we can know that they are differentiable (or piecewise differentiable like ReLU). And for rigor, I should also remind you that the composition of multiple differentiable functions is also differentiable.
So hopefully that helps you see what I mean when I say deep learning architects "impose" differentiability, rather than "assume" it. After all, we are the architects!

I'm going to take the fitness landscape to be the graph of the loss function, $\mathcal{G} = \{\left(\theta, L(\theta)\right) : \theta \in \mathbb{R}^n\}$, where $\theta$ parameterises the network (i.e. it is the weights and biases) and $L$ is a given loss function; in other words, the surface you would get by plotting the loss function against its parameters.
We always assume the loss function is differentiable in order to do backpropagation, which means at the very least the loss function is smooth enough to be continuous, but in principle it may not be infinitely differentiable1.
You talk about using gradient descent to find the global minimiser. In general this is not possible: many functions have local minimisers which are not global minimisers. For an example, you could plot $y = x^2 \sin(1/x^2)$: of course the situation is similar, if harder to visualise, in higher dimensions. A certain class of functions known as convex functions satisfy the property that any local minimiser is a global minimiser. Unfortunately, the loss function of a neural network is rarely convex.
For some interesting pictures, see Visualizing the Loss Landscape of Neural Nets by Li et al.

1 For a more detailed discussion on continuity and differentiability, any good text on mathematical analysis will do, for example Rudin's Principles of Mathematical Analysis. In general, any function $f$ that is differentiable on some interval is also continuous, but it need not be twice differentiable, i.e. $f''$ need not exist.

This answer seems reasonable to me, i.e. you need differentiability to apply back-propagation, but note that not all neural networks are trained with back-prop (although this may be an assumption, when you use the term "deep learning"). But why do you say that $G$ is a graph? Maybe you mean a space (https://en.wikipedia.org/wiki/Space_(mathematics))? In any case, it seems that you're using the term "graph" not as defined in graph theory.

@nbro Thanks for the question, I mean graph as in the literal sense of plotting a graph of a function (you can see an example of this usage here, for example). My choice of notation has clearly added to the confusion as it is common to denote a graph (in the network sense) by $G$, but I don't mean to say $G$ is a network. Hopefully the latest revision clarifies this.

Does deep learning assume that the fitness landscape on which the gradient descent occurs is a smooth one?
One can interpret this question from a formal-mathematical standpoint and from a more "intuitively-practical" standpoint.
From the formal point of view, smoothness is the requirement that the function is continuous with continuous first derivatives. And this assumption is quite often not true in lots of applications - mostly because of the widespread use of ReLU activation function - it is not differentiable at zero.
From the practical point of view, though, by "smoothness" we mean that the function's "landscape" does not have a lot of sharp jumps and edges like that:

Practically, there's not much difference between having a discontinuous derivative and having derivatives making very sharp jumps.
And again, the answer is no - the loss function landscape is extremely spiky with lots of sharp edges - the picture above is an example of an actual loss function landscape.
But... why the gradient descent works then?
As far as I know, this is a subject of an ongoing discussion in the community. There are different takes and some conflicting viewpoints that are still subject of a debate.
My opinion is that, fundamentally, the idea that we need it to converge to the global optimum is a flawed one. Neural networks was shown to have enough capacity to completely remember the training dataset. A neural network, that completely remembered the training data. has reached the global optimization minimum (given only the training data). We are not interested in such overtrained models - we want models that generalize well.
As far as I know, there is no conclusive results on which properties of the minimum are linked to ability to generalize. People argued that these should be the "flat" minima, but then it was refuted. After that a "wide optimium" term was introduced and gave rise of an interesting technique of Stochastic Weight Averaging.

Why would we be looking for the "widest" minimum? Moreover, although I think what you mean, you should explain what you mean by "widest minimum" (maybe a more common term is "flat minimum"). I really don't understand why we would not be looking for the global minimum. The global minimum/maximum is the optimal solution. Not sure what you're talking about here and where your conclusions come from.

@nbro Would you agree that a neural network, that completely remembered the training data, has reached the global optimization minimum?

Mathematics is not about agreeing on something or not. It's about making assumptions or starting from premises and follow the rules to reach some conclusions. In any case, that would be the global optimum but for the training dataset. In reality, as you know, we're not trying to overfit the training data but to find a function that generalizes well to unseen data too. The thing is: I don't know where your conclusion that we're trying to find the "widest minimum" comes from and why would that mean "better generalization", which is what you seem to mean/imply now.

To be honest, the question is not even clear. The author is talking about the "fitness landscape", but they didn't even clarify what that would mean in the context of deep learning. They are probably talking about "loss landscape", but it's also not clear. I'm not an expert on the topic, but I don't think that your conclusion (last part) is well motivated. The first paragraphs seem reasonable to me, though, although, again, I am not a mathematician and not an expert on this specific topic of "loss landscapes", whatever the definition is.

@nbro I thought that my three last sentences unambiguously stated that I'm talking about generalization... If you have a suggestion on how to rephrase that - I'd happily do that. I didn't want to go into formal details of it because it was my slightly opinionated aside on the subject. Also, I do agree that the question is not well-phrased, but demanding mathematically precise formulation from beginners is not very productive, wouldn't you say?

In your last paragraph, you indeed say "we want generalization". That's correct. But where does the "widest minimum" come from and what does it even mean? It's not clear. That's what I'm questioning.

Furthermore, I think that giving an answer with many mathematical details to such a poorly formulated question may not be easy to follow for the beginner, but it definitely would clarify certain doubts, if you understand the math. What I think, generally, is that "no information is better than misleading information", so opinions are typically not very useful and this site is not about opinions, but more about facts. If the question is "Do you think that X is Y?", then, ok, you could say your opinion, but these questions are not suitable for our site.

@nbro I just used "wide minimum" as an intuitive analog of "minimum that generalizes well". I can remove "wide" if you insist. But I don't feel like using "flat" in that context, because there's some research that shown that minimum curvature is not necessarily linked to its generalization performance (as well lts "globalness").

I think you should just define what you mean by that. What kind of minimum are we exactly looking for and, more importantly, why do you say that we're not looking for the global minimum and clarify that you mean the global minimum given only the training data.

@nbro Done. Actually I did a refresher on literature and "wide" is a pretty widely (heh) used term in publications on the subject...

@Kostya, if the loss function is not smooth, how can we compute derivatives?

@Kostya, yes RELU is not differentiable at $x=0$, but we assume the derivate is zero at $x=0$.

@Kostya, I don't understand how you create link between gradient descent on the function that is not smooth and finding the local minima or better minima(may be your saying it as wider minima). Since we are standing on a function that has many local minima it is natural to stuck in a local minima using gradient descent. I don't see how non smooth fuction and getting local minima linked together.


text:
Real time image processing for object recognition using security cameras
I was thinking, what if we could combine Artificial Intelligence (Neural network for image recognition), computer hardware and a security camera for identify any breaking into our backyard at 12:00am - 8:00am? Of course my current knowledge leads me to only a simple question. So, in order to have a general idea:

¿Have been this already solved using a commercial or free software?
¿Can this be done using TensorFlow?
¿Is there any free set of images with millions of them to teach any AI distinguish between a man and another moving object?
¿Approximate hardware requirements for doing this?

If this question could be silly please mark it as off-topic. I based this idea on autonomous driving car, they can both recognize images and drive at the same time. Unless they have within a super computer I guess maybe the previous idea can be fulfill.
Update 1: I found this Can ConvNets be used for real-time object recognition from video feed? but I guess it could be outdated. Right now I'm in the land of "maybe" (lack of knowledge).

Welcome to AI!  From what I've been gleaning off of the DL folks, while you might need robust computing to train the algorithm, once it is trained, it would be able to run on a much less robust system.   I'd think the real problem with training in this regard would be the dataset.  I'm guessing frequency of break-in attempts is fairly low, and these types of systems need a huge sample.

So basically the problem lays in a large images dataset (specific for recognizing human shapes) and a powerful hardware to train the neural network.

in term of data sample problems, I was more meaning images/videos of actual break-in and trespassing attempts vs. lawful presence on the property (by the owners, neighbors, friends, etc., even the kids sneaking out at night;)  Lawful presence would be easy, but break-ins are probably statistically rare, even in high-crime areas. So getting a sufficient sample size of break-in attempts sufficient to train a Deep Learning system would probably require a monumental, coordinated effort.

By contrast, simply recognizing a human being, their status & intentions irrelevant, would probably be doable.  So if you're just looking for a system that tells you there is a person in the yard, that's significantly less difficult.

Sorry, I should have started with that. I was thinking about night, probably 3:00 am and with no kids playing around XD. The only presence of a human being in your backyard at 3:00 am it's at least an emergency. Changing the conditions then what do I need now? Of course, if the AI gets a picture (frame) and recognize a human shape the problem then is solved. I guess there is no need for learning about videos of breaking in cases.

In terms of applied ML, that's currently outside my skill set. (I try to keep up on what's going on in the field, but my own AI endeavors are in different areas.)  But I'm pretty sure human-shape recognition is an area of great focus in the field. Autonomous vehicles, for instance, ideally need to make this distinction.

but, would you really need this approach to solve your problem?

@k.c.sayz'k.csayz' An Artificial Intelligence approach do you mean?

@Ophion as in, depending on your use case / data source / real world concerns, you could pursuit simpler methods.

Here is some general info on how NNs work in relation to this specific problem. Hopefully it will provide some insight:
To identify target object, you can just train an NN to perform classification of images. Now, you will run into potential problems where this image can be located anywhere in given frame of input from the camera.
Let's say; you train a human NN that's able to act on 60px x 120px (so you will have 10800 input in your NN for the input layer). Under the assumption you can train the NN to identify whether or not the target object is showing in the image provided, you can use this trained NN to locate the object on the input frame.
Let's say the input frame from the camera is 640px x 480px. You will run into several potential problems
1) The object that's closer to the camera will be larger while the object that's further away from the camera will be smaller
2) The object may be located anywhere in the frame.
To overcome this problem with your already trained NN, I would start with a different mask size, for example, the first one I will probably have my first mask convolute all pixels with 240px x 480px. The first step is, scale it to 60px x 120px (which you can tell it is pretty much 4x) and capture the subset of pixels from pixels located in the area of (0,0) and (240, 480) and run it through the NN, If it yields true, then I will typically draw a box around the region to indicate the target object has been found.
Next, you shift over to 1-3 pixel and rerun it. For example, if we choose to move by 3px, then the region we are testing will be (3, 0) and (243, 480).
Once you finish this scale, you will want to choose a smaller mask, maybe something like 200px x 400px and do the same thing. When finished, go even smaller until you believe there's no point to search for an even smaller region because it is not going to be sufficient resolution for NN.
That's just my thought; maybe there's an even more efficient algorithm. Ultimately, I am sure there's a lot more can be optimized than what I've mentioned above!

Your approach looks good, but can you tell me an approximated hardware, not the required for training but the required for "run" the NN in real time. Could you also tell me if the process can be simplified using TensorFlow?

It depends on your input layer. I believe forward-prop isn't too heavy to perform. What's really heavy is the back-prop. It will definitely help if you have a NVidia GPU which does matrix-multiplication fast. I say for "non-mission-critical", if you exam 2-3 frames per seconds, a regular CUDA support graphic card will be enough for forward-prop.

(continue)
Just to put runtime in perspective (please correct me if I am wrong), depending on the number of layers you have, since the separation of concave regions may exist if it is greater than or equal to 3 layers, we can assume 3 layers for the sake of simplifying the calculation.

(continue)
We can also assume that each neuron is a pretty simple function (i.e. sigmoid function), that can be considered as constant O(K) and given that for each step we have n neurons O(n), we can roughly estimate the running time is O(n^2 + K) for this forward prop to run. Therefore, based on the big O rules, we pretty much can assume it is O(n^2).

So, consider you do it in GPU that it runs matrix multiplication in parallel, it is going to be relatively fast! In theory, your CPU shouldn't even spike much because we don't use CPU to process calculation.

I'm taking your answer as the answer because my question was theoretical and you knew how to clear some doubts I had.


text:
How can I send vectors as a chat context?
Since the context/memory of a chat or question for LLMs more precisely GPT is limited to a token length I struggle about how to provide own data that the model got not trained on.
A very common approach looks like embeddings are the way to.
OpenAI provided an article https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb
how to create an embedding of a user query, match it against a local vector database ans provide the closest results as text to the context/memory.
Here I do struggle, since it might be very well possible that even we find the most matching documents locally in a vector database, context might still be too small if we would like to provide multiple matches.
The question to me is, how could I send all the relevant embedding vectors rather than the relevant texts which got matched to the vectors?
These vectors are highly condensed and would save a lot of tokens. GPT would anyhow be able to understand the vector since they created it from their embeddings API, right?
Or is it just not possible to convert the vector back to text at their end?

Do you mean sending the vector of embedded text as the context, instead of the text itself?
If you think it through that might mean sending in [2.12234, 12.134123, 3.132412, ....., 1.123124] to chatGPT instead of Hi, can you tell me what is the grandfather paradox?. Do you spot the problem here?
GPT models are designed in a way to only take textual inputs and spit out textual inputs. So, in short, what you are proposing is not possible with the API endpoints currently provided by OpenAI.
Theoretically, taking in an embedding vector and producing the relevant text is a possibility but considering the transformer architecture, there is a logical fallacy in your proposed argument.

Embeddings are produced from the text itself, so to infer a sequence from a generated embedding would mean that you have already used up a certain toke length, i.e. at this point it really doesn't matter what is the max allowed token length.

Not just that, but the ADA embedding model is different and is fine-tuned to generate better similarity rather than aid in generating CHATesque text. So, even if OpenAI offered that possibility (which it doesn't) the generated text wouldn't be as good as sending in text as context to GPT4 api endpoint.

Vectors are not provided as context - you need to provide the original text. Embeddings are used with e.g. vector databased to provide similarity queries so you can reduce the context you provide to GPT by only selecting relevant parts (e.g. 1-2 documents from a library of 1000s of documents).

Pleast break the answer into smaller bits for unexperienced questioners. It will increase your answer's effectiveness.


text:
Confusion in the "goal" of multi arm bandit problem
Say, we have $n$ actions to choose from, assume that after an action has been chosen, the state of the system stays the same, only our knowledge gets updated. What exactly is the goal of the multi arm bandit problem?

Is it that we want to find the action that has highest expected mean reward, so that after sufficiently long amount of time, we have gotten one ideal action? Here it seems that one should always go for exploration, for each state perform the experiment very high number of times, to get an accurate estimate of the expected reward and then we are guaranteed to find the best action to always go for in the future.

Is it that I want to maximize the total reward accumulated over time due to repetition of the experiment? Here, I will have to be careful about my strategy and hence I need to ponder over exploitation vs exploration.

Background
I am reading chapter 2 of Sutton's Reinforcement learning. But it seems very verbose, confusing and convoluted. Please provide any general suggestions that you might have, should I stick to the book or try something else? I think I might be too much used to definition, theorem, proof style of presentation.

The specific goal for an instance of a  bandit problem will depend on why you want to use the framework. S&B is being cicumspect about it, because they are offering more broad analysis.
In practice, you would often use the bandit approach on "live" systems where you are not able to simulate or have pre-existing data for the choices being made. It is a common approach in web advertising to optimise click-through of adverts that have not been seen before.
In that scenario, you care about the explore/exploit trade-off and maximising returns whilst learning (there is often a financial impact to optimise here). This is often phrased as minimising regret which is usually measured as total of expected return from actions taken compared to selecting the (in hindsight) best action on every attempt.
S&B don't go into bandit problems in any depth. In that book, bandit problems are being used to illustrate the explore vs exploit tradeoff for learning from experience, in isolation, before then diving into the full reinforcement learning problem.
Out of the options in your question:

Is it that I want to maximize the total reward accumulated over time due to repetition of the experiment?

This is the more common use case and goal for applying bandit problem theory. The theory is slightly broader than that, as it attempts to model a group of problems, and you can explore aspects of these models and associated algorithms.


text:
Does this article make use of the chain rule? And where?
References:

Chain rule in Wikipedia: https://en.wikipedia.org/wiki/Chain_rule
Chain rule in Towards Data Science: https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd#:~:text=The%20algorithm%20is%20used%20to,parameters%20(weights%20and%20biases).

Searching the internet in some articles, I learned that the chain rule is used so that the network can identify the contribution of each weight to the error, and how much each weight needs to be adjusted.
This article I mentioned in the question: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/, they teaches you how to program a multilayer perceptron neural network from scratch in Python. During the calculations in the backpropagation phase, he uses some different terms such "delta", and
it calculates the "delta" layer by layer. I know that the explanation in the article must make use of the chain rule, but I couldn't understand where.
I couldn't see where the chain rule is. Does this article make use of the chain rule? And where?

The chain rule is spread in the 3 core functions of your reference: forward_propagate(network, row), backward_propagate_error(network, expected), update_weights(network, row, l_rate) which are linked in the train_network(network, train, l_rate, n_epoch, n_outputs) function. This is mostly a programming question.

Can you please don't use links in the title and don't use some much bold?

@nbro Okay.  Today I will edit the question and remove the title link and remove the bold text.

Unless the derivation has been added in a comment, when you read code implementations of backpropagation, then the chain rule has already been used to derive the update rules. As it is basically a rule about multiplying one simpler derivative by another to resolve the derivative of a more complex function, then typically some of the terms in each statement represent $\frac{\partial f}{\partial x}$ and some represent $\frac{\partial g}{\partial f}$. When those are multiplied together you find the numerical value of $\frac{\partial g}{\partial x}$ from the composite $g(f(x))$.
The "deltas" being stored in variables are components of the gradient of the loss function with respect to the network's internal states and free parameters, at its current value (due to current item, or an aggregate across the current batch).

Some are temporary calculations that don't need to be stored longer term because they are not gradient components of changeable parameters.

Some represent components of $\nabla_{W,b} J$, the gradient of the cost function $J$ with respect to the weights and biases of the
neural network - these are the values that you can change as part of
training, so they are usually stored in a structure the same shape as the weights and biases so that they can be aggregated across multiple examples and subtracted from $W,b$ later.

Thanks. Please, I have a question: the article uses the Sigmoid, and it shows its derivative.  So according to your explanation in "[...] when you read code implementations of backpropagation, then the chain rule has already been used to derive the update rules.", this means that if I have an activation function complex, and I don't know the derivative of it, so that's where I would apply the derivation to find the derivative?  But in article this derivation has already been done (the derivative of the Sigmoid), I don't need to do the derivation because it was done. Did I understand correctly?

in other words, do I just use the chain rule to find the derivative of the activation function when I don't know what the derivative ?

@willTheJ If you are being shown working code for training, you should not need to do any extra maths on the code or data in order to use it. Quoting the chain rule is explaining how it works, and how you might build your own NN from scratch. If you are building your own NN from scratch, using e.g. NumPy, and stuck, then please as a new question about that.

Okay, I will create new questions for other questions.. Thanks for explaining

Your last article is mainly focused on programming the classic full GD BP algo from scratch using pure Python under the default MSE cost function but never mentioned or emphasized in the main body except somewhere in the extremely long comment section, thus it may cause you cannot link with chain rule.
The first outer chain under MSE is obviously just the error signal from the final output compared with its true target, the 2nd chain is the derivative of the activation function which is the classic sigmoid function which you're already aware, and the final inner chain is the output of the originating neuron of the updated weight edge in the previous layer or just one of the input signals if there's none. These correspond to the three usual multiplicative terms in a delta rule.

Thanks a lot for explaining better the chain rule!


text:
Unclear paragraph in Sutton-Barto on exploration/exploitation relating to bandit like decision tasks
This is a text from Sutton-Barto, second edition, page 30:

Even if the underlying task is stationary and deterministic,the learner faces a set of bandit like decision tasks each of which changes over time as learning proceeds and the agent’s decision-making policy changes. Reinforcement learning requires a balance between exploration and exploitation.

Could someone explain (and if possible give some examples) this paragraph?

What is unclear exactly? What do you know or not know about what is written here?

The quoted text is from the end of a paragraph that is explaining some aspects of bandit algorithms, and looking forward to how knowledge of them is applicable in a reinforcement learning context. Just before your quote, the relevant text is:

For example, suppose the bandit task were nonstationary, that is, the true values of the actions changed over time. In this case exploration is needed even in the deterministic case to make sure one of the nongreedy actions has not changed to become better than the greedy one. As we shall see in the next few chapters, nonstationarity is the case most commonly encountered in reinforcement learning. Even if the underlying task [ . . . ]

In particular, your quote is explaining that deterministic RL environments present a bit like a series of non-stationary bandit environments, because when a policy changes, it impacts the expected values of earlier actions, maybe to the point that an earlier "best" action choice needs to be changed.
We can illustrate this with a simple deterministic RL environment. In the diagram, states are squares, actions circles, and rewards shown as labels on the next state transition.

You can look at the choices of action in states $s_0$, $s_1$ and $s_2$  as simple bandits. They have a choice between action $a_1$ or $a_2$. There is some extra complexity, because the results from the choice in $s_0$ depend on the later choice in state $s_1$ or $s_2$. This means that the optimal choice for the "bandit" at state $s_0$ depends on what the current choices will be in state $s_1$ and $s_2$ - but a bandit algorithm doesn't look ahead, or have ways to ask other bandit algorithms what they will do*, it only gets to know the end result, i.e. what the eventual reward is.
At the start of learning, you may decide to set a simple equiprobable policy in all states i.e. $p(a_1) = 0.5, p(a_2) = 0.5$ and the same for each separate "bandit". This would be a normal starting policy when you know nothing about rewards. In that case, the expected return from actions in state $s_0$ would be $q(s_0, a_1) = 1.5$ and $q(s_0, a_2) = 0$, making the maximising choice from $s_0$ the action $a_1$ - if you only trained the first $s_0$ bandit, as it experienced the results from randomly trying $a_1$ and $a_2$, and did not update the others from their starting policies, then the first bandit would learn to take action $a_1$.
Later on during learning, the policies in the later states $s_1$ and $s_2$ may have converged towards the maximising choices there. For the purposes of this discussion, it doesn't matter how you do that, only that the policies have changed.
Let's say for the sake of argument, learning has progressed in states $s_1$ and $s_2$ to the point that the policy chooses the maximising reward 90% of the time - action $a_2$ for the "bandit" in $s_1$ (getting +2 reward) and action $a_1$ for the "bandit" in $s_2$ (getting +3 reward).
At that stage of learning, the expected return from actions in state $s_0$ would be $q(s_0, a_1) = 1.9$ and $q(s_0, a_2) = 2.4$, making the maximising choice from $s_0$ the action $a_2$.
From the perspective of a bandit-like choice in state $s_0$, the expected rewards and best action choice have changed over time, even though the environment is deterministic and has not changed in any way. That's because the policy is not static during the learning process, and this has an impact on earlier decisions due to how later decisions navigate the same environment.

* This may immediately make you ask "why not look ahead and link the states/action choices that way?". The answer is that you can, this is called planning, or searching (a tree search, typically). That is not relevant to understanding the bandit algorithm and related RL issues in early chapters of Sutton & Barto, but will come up in various ways later in the book.

Thank you for the effort but the explanations are very confusing.  Let me go step by step. In first paragraph, you say "At the start of learning, you may decide to set a simple equiprobable policy in all states.".  What does this mean? Is this the policy you want to learn? If yes, then how can you set it yourself?

@DSPinfinity when you want to learn an optimal policy, you typically start with some arbitrary policy and improve it. There is no way to gain experience without some kind of current policy - in order to learn from experience you must make decisions so you can observe what happens after you choose a certain action. So yes, when you start a learning process you do set a policy

@DSPinfinity The idea of starting with an arbitrary policy, observing the results and making changes to that policy is the main focus of the chapter on bandit problems. After introducing the bandit concept, the chapter shows several variants on how to setup and learn policies

Amazing explanation after update. You should write an RL book and I believe that it will be the best seller.


text:
Should I use RL for this allocation problem and, if yes, which RL approach?
I want to use RL to solve the problem of Unit Allocation.
SETTING: So for example in a game you can use units of type A and units of type B and you know that your opposing player has units of type C and type D. Now given the knowledge of the opposing units the neural net should give me the composition of my units to deploy against them. Let's say x of type A and y of type B.
PROBLEM Since I have an environment to play and score the outcome my idea was to use Reinforcement Learning. Resulting in my first question, would this be the right choice of method ?
I searched for resource allocation problems and all I found was some Communication Problems, where I can't fit my problem.
What type of RL would be the best choice here? DQL/Q-Learning or PPO?

I was just reading Sutton and Barto last night and a passage stuck with me which I will paraphrase.  RL is marked by the evaluation of an action, not the instruction of an action.  So I would say if the correct mixture of units is not known before hand and must be learned by evaluation then I would say that RL is correct.
The first decision point I'd consider on determining which would be reasonable choice in methods would be the input type/range.
I assume the enemy must deploy a whole, positive number of units (ie integer).  In this case Q-table might be okay.  The decision point would be how many values these inputs can take on.  A Q-Table stores the Q value for every state, so as the number of states grow, so does memory requirements.
If the range of the inputs gets significantly large (or the inputs are real valued), one should consider a function approximator (chiefly NN in DQN/PPO but there are alternatives).  Which exact function approximator and how to train it will be along ideological/practical lines.

Great deduction, especially on the DQN/PPO part. What I can't  get my head around is the time aspect, which I associate with the training. In my problem the time would solely pass in the environment (for each composition). Does this still fit the RL frame ?

Are you saying that the composition is chosen at the start of the round, time passes until the end of the round and a winner is chosen?  But the agent can only act/perceive between rounds?

Yes exactly, kinda a one shot round.

Sure, rock, paper, scissors except you must determine how much of each to deploy.  Might start there, I think Kaggle has an  exercise for rock, paper scissors.

You might look at portfolio allocation examples for chatter on how to do this mixture of resources thing.

Thank you for the pointer mate, will have a look.


text:
Is there any example of a Markov Decision Process (MDP) with infinite number of states?
I was learning fundamentals of reinforcement learning from various sources like coursera, Udacity, Reinforcement Learning: An Introduction Book by Andrew Barto and Richard S. Sutton.
All of them consider Finite MDPs as reinforcement learning problem and try to solve that.
In real life, is there any example of an infinite MDP (MDP with infinite number of states) and if yes, can it be described and solved as a reinforcement learning problem?
Can anyone suggest some sources to read more about this problem statement?

Could you define what you mean by infinite MDP? There are a few traits of finite MDPs that can be changed such that they are not finite/discrete and they can be solved using different approaches. For instance, there is "inifnite horizon MDP", i.e. MDP which never terminates, and you want to consider all future results. Are you asking about that?

Yes, exactly. I was talking about MDPs which never terminate.

Finite MDPs are not defined as MDPs with infinite-horizon. Finite MDPs are defined as MDPs with a finite set of states and actions (and maybe rewards too). This is the definition adopted by Barto and Sutton and many other sources I've come across. Also, the same book covers infinite-horizon MDPs, assuming finite MDPs. So, please, edit your post to clarify what you're really asking here. I'd recommend that, before doing that, you read the sections of this book that define finite-MDPs and infinite-horizon settings.

I have updated the question with more clarification. I will definitely read upon the concept of horizon.

Not really an answer, but it will lead you to it, - Yes, look at continuous state and action spaces, they are both infinite in size and a common RL problem.

Many real-world problems have an infinite number of states, but, in practice, digital computers cannot represent an infinite number of states or numbers anyway, so you will always need to discretize the problems (or you will do that implicitly).
Now, think of an image, where each pixel can be a number between 0 and 1.0. If we used real numbers, there could be an infinite number of images, even if we restrict our images to finite shapes. So, in theory, we could have an infinite number of states. In practice, when we use real computers, we will never have an infinite number of states.
Having said this, yes, RL has been applied to problems where, in theory, the number of states could be infinite. For example, DQN was trained and used to play video games given a sequence of images (observations) as states. You could argue that pixels can also just have integer values between 0 and 255. In that case, you can think of problems like the pendulum or the mountain car.


text:
Negative KL-divergence RLHF implementation
I am struggling to understand one part of the FAQ of the transformer reinforcement learning library from HuggingFace:

What Is the Concern with Negative KL Divergence?
If you generate text by purely sampling from the model distribution things work fine in general. But when you use the generate method there are a few caveats because it does not always purely sample depending on the settings which can cause KL-divergence to go negative. Essentially when the active model achieves log_p_token_active < log_p_token_ref we get negative KL-div. This can happen in a several cases:

top-k sampling: the model can smooth out the probability distribution causing the top-k tokens having a smaller probability than those of the reference model but they still are selected
min_length: this ignores the EOS token until min_length is reached. thus the model can assign a very high log prob to the EOS token and very low prob to all others until min_length is reached
batched generation: finished sequences in a batch are padded until all generations are finished. The model can learn to assign very low probabilities to the padding tokens unless they are properly masked or removed.

These are just a few examples. Why is negative KL an issue? The total reward R is computed R = r - beta * KL so if the model can learn how to drive KL-divergence negative it effectively gets a positive reward. In many cases it can be much easier to exploit such a bug in the generation than actually learning the reward function. In addition the KL can become arbitrarily small thus the actual reward can be very small compared to it.

I understand why the KL-divergence that is computed here is an approximation that can be negative as opposed to the real one. However, I cannot wrap my head around the details of why these specific sampling parameters would lead to negative KL-divergence. Could someone elaborate on these points?

well KL is a distance, pretty sure that it cannot be negative by definition, so they're doing something sketchy


text:
Clarification on Formulation of a DP Problem
I am self-teaching reinforcement learning and for now I am trying to solve the following DP problem:

A driver is looking for inexpensive parking on the way to his destination. The parking
area contains $N$ spaces, and a garage at the end. The driver starts at space 0 and
traverses the parking spaces sequentially, i.e., from space $k$ he goes next to space $k + 1$,
etc. Each parking space $k$ costs $c(k)$ and is free with probability $p(k)$ independently
of whether other parking spaces are free or not. If the driver reaches the last parking
space $N − 1$ and does not park there, he must park at the garage, which costs $C$. The
driver can observe whether a parking space is free only when he reaches it, and then, if
it is free, he makes a decision to park in that space or not to park and check the next
space. The problem is to find the minimum expected cost parking policy.

As a first step, I defined the state space at stage $i$ to be $S_i = \{1, ...,\min\{i, N\}\}$ (because $\min\{i, N\}$ is the farthest that the driver can reach), and the action space to be $A_i = \{0, 1\}$, where $0$ stands for the action of staying at the current space, and $1$ denotes moving on to the next space.
How am I supposed to define the state-reward distribution? At stage $i$, is it that
$$
\begin{cases}
p_i(s+1, c(s+1)\mid s, 1) = p(s+1)\\
p_i(s, c(s)\mid s, 0) = p(s)
\end{cases}\;(s < N),\quad p_i(s, C\mid s, 0) = 1\quad (s = N)?
$$

https://dspace.mit.edu/bitstream/handle/1721.1/75813/6-231-fall-2008/contents/exams/midterm_solution.pdf ?

Would you add your current idea of state and policy? You mentioned state of parking space. How about state of driver?

@CloudCho Oh I first wanted to define the driver's state via his location - but I ignored that his target was to park the car instead of reaching a certain parking space...

If the driver rewarded by parking his car in a parking spot. Is his goal the max rewarding parking spot (cheapest one) or park, pull out and park in different spots to increase rewards?

@CloudCho The driver simply stops at where he decides to park.


text:
What does "fully visible belief network" stand for?
In CS231n, I saw the terminology "fully visible belief network", one category of explicit density generative model such as PixelCNN and NADE.
Although I can understand what this terminology wants to say, but I need more explanations in detail:

What exactly does "fully visible" (or "visible") mean?
I have searched about deep belief net, and I believe that it is just a model in category of deep neural network but trained in unsupervised way with energy function. Am I right?

deep belief net has little to do with neural network, if not sharing the concept of "neurons" but they are defined in 2 very different ways, though, i have no idea what they mean with fully visible belief network... maybe having no "hidden/latent variables", but it's a pretty loose definition

It's easier to understand in the context of the lecture. I'm assuming you're referencing this slide in Lecture 12:

What distinguishes the FVBNs from the other explicit density models are that they have a tractable density: that is, that you can exactly calculate the probability of samples of your dataset (or at least, that's the assumption made by the model). This is what's meant by "fully visible".
Take PixelRNN/CNN, for example. Here, you model images autoregressively, where a given pixel only depends on previous pixels. This allows you to decompose the distribution over many pixels using the chain rule. Look at slide 23 in the lecture for more info. The main idea is that this decomposition allows you to calculate the probability of a sample exactly, making it "fully visible".
The connection to deep belief networks is a bit more iffy in my opinion. The NADE paper explains how the NADE forward pass is similar to an approximation of a RBM. I'm unsure if this extends to other models, however.


text:
What is the best Text-to-speech model available open-source?
I tried a couple of different websites and libraries. Also found this topic from 3.5 years ago - What are the current open source text-to-audio libraries?
It looks like nobody published anything in the last couple of years and most solutions are really really not good. Even Amazon sounds like some weird robot.
One of the best I tried is Coqui, but their best models were never published to GitHub. So TTS on their website sounds perfect, but you have to pay.
Also, ElevenLabs is amazing, but it is not open-source.
I cannot believe that there are no published models that sound good. I need it to generate a huge amount of text, so it will be very expensive to pay for subscriptions. They all charge about $5/20-40 min of speech and it sounds a little bit too much for me.
Could you recommend anything I can use for free? Preferably open-source, but not mandatory.

For me, the go-to source for answering the question "What's the state of the art for [task]?" is Papers with Code. They compile tasks, benchmarks, papers (and their associated official/unofficial implementations) into a series of leaderboards.
Consider, the text-to-speech leaderboard, for example. At the top, you see a series of benchmarks which you can click on to see the top performing papers.
Many popular leaderboards (like the one for TTS) also has a libraries section. These may not be the state of the art, but it'll probably be the easiest to get up and running with.
There are somethings to keep in mind, however.

Take benchmarks with a grain of salt. Especially for something like text-to-speech, where it's difficult to quantify performance into a single number.

Research repositories can be very finnicky. Make sure to look at if data/checkpoints are available. I'd recommend starting with existing, maintained libraries, but if you're looking for the state-of-the-art, this might not be an option. You may be able to email paper authors to request checkpoints/data, if they aren't publically available.

Check facebook's Massively Multilingual Speech (MMS) : Text-to-Speech Models. You can find it in the HuggingFace repo.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
How to generate quality synthetic images of human subjects to be used for training stable diffusion
I'm trying to generate some synthetic images of humans, that are "good enough" to be used to fine-tune a stable diffusion model, but i'm not sure if this is possible.
I have experimented with some methods, but none gave satisfactory results. I tried using stable diffusion itself and some hacks which can be found here , but the outputs had minor differences that might degrade the quality of the model. I also tried using GANs, specifically InterFaceGAN, to modify the pose and smile of the characters, but the images were 95% similar and too low quality and for training.
I wonder if there is a better method to do so. Is it even feasible?

Can you explain the scope of what you want to fine tune and generate? Some synthetic image approaches may lack certain features, but those may not be necessary for your project. E.g. what variations, if any, do you need for backgrounds, lighting, framing, poses/activity, character diversity, clothing and accessories? Is there any rendering styles you need (photographic, illustration, cartoon?) Are any of those fixed, but would be specific to your project (e.g. they must all be wearing historical costumes)?

Are you trying to create a LoRA (or similar) for some recurring characters? So one constraint is "all the same character" for any specific set?

Hello @NeilSlater Im trying to fine-tune an sd model on human images, except that I don't want to use real training data for legal reasons, im looking for synthetic images that are neutral (neutral facial expressions, background, and clothing and wearing a simple bikini would be my perfect training data, and of course to look as realistic as possible. I actually got some consistent results using this model using a very detailed prompt, but I am afraid there might be subtle differences that cannot detected by eye but sd wil detect them.

You still aren't saying what the scope is of your fine tuning? E.g. Single character, or preferred image style, or better general human anatomy? These would each require different training data sets, that might suit different approaches to generation. I'm getting the sense that you want to have fine-tuned images for some named characters? E.g. you want to describe "Bob", have enough training examples of "Bob" to fine-tune in Stable Diffusion and then be able to generate more images of "Bob" doing stuff?

The perfect flow would be generating a random real human with random physical appearance if the random human is approved by some hypothetical user, i want to be able to generate a set of consistent and similar images that can be used to train SD on that random human.

So you want a series of random characters. But for training purposes, multiple images of each character?

yes @NeilSlater exactly to give a follow up in my progress, i found this experiments that does so using textual inversion https://github.com/BelieveDiffusion/tutorials/blob/main/consistent_character_embedding/README.md but it includes a little of manual selection however im trying to make all fully automated


text:
Applicability of Holland's Schema Theorem to Genetic Algorithms with Non-Binary Individual Representations
I'm currently working on a problem formulation that requires non-binary individual representations in a genetic algorithm (GA). I've been exploring Holland's Schema Theorem as a theoretical basis for understanding the effectiveness of GAs. However, the theorem is typically discussed in the context of a "canonical GA" with binary representation, fixed-length individuals, fitness-proportional selection, single-point crossover, and gene-wise mutation.
In my case, each individual in the population can be represented by a string of integers within a certain range, making it a non-binary representation. I'm wondering whether the Schema Theorem still holds true for such a scenario. Can I apply the principles of the Schema Theorem to this type of GA with non-binary individual representations? If not, are there any alternative theorems or theoretical frameworks that can help demonstrate the effectiveness of GAs using non-binary representations? Specifically, I'm interested in understanding whether schemas with fitness greater than the population average are still likely to appear more frequently in the next generation, and how this might impact the convergence and optimization properties of the algorithm.
Any insights, references, or resources that could shed light on this topic?

It's been a while since I've been active in the research community, and it's possible that this is a controversial opinion, but I suspect it's still true that the Schema Theorem is not really used that much to inform our modern understanding of GAs. That doesn't mean that it's not true or useful as a tool to think about the dynamics, but I think we struggle to use it to answer interesting questions, so it's just not all that relevant today. The fact that it is independent of problem instance is a pretty large limitation, because GAs are great on some problems and terrible on some problems, so if your explanatory tool doesn't have "problem" as a variable in the model, you're obviously missing something really important. Specifically, what it really describes is the dynamics of the population over time, whether or not those dynamics are performing anything we'd describe as a useful search. It's similar to the idea of a hill-climber that thinks it's doing great because it's climbing the tiny hill that's nearby and completely missing the much higher peak over there. All you have is a model of how the climbing looks, and not a model of what climbing that particular peak means in the broader sense of how well you're doing on that optimization problem.
That said, you can kind of look at Holland's schema theorem as two things. It's (1) a qualitative statement that GAs work by recombining short pieces of solutions into longer components in a way that progressively improves overall fitness, and (2) a quantitative and precise inequality that puts bounds on the probabilities for the mechanisms that are responsible for that progressive improvement. The first part of that is pretty much a given to be true for any GA that'd we'd recognize as a GA. It's just the kind of statement that you squint a little and go, "Yeah, that seems obviously true I guess". The second part only works on the "canonical GA", because the computation of the probabilities assumes a specific structure and set of operators.
If you want a similar schema theorem for some custom representation and set of operators, there's no real alternative to just trying to do the same math Holland did. What does a "building block" look like in my non-binary GA? What is the probability that my crossover operator disrupts such a building block? What is the probability that it creates such a block? Same for the mutation operator. Coming up with those probabilities tends to be pretty hard to do for any complex representation, and if you do it successfully, you don't get a lot out of the effort (because that inequality is difficult to put to a pragmatic use).


text:
What exactly is meant by variational distribution?
What specifically does the term "variational distribution" refer to?

The encoder of Variational autoencoder?
Forward process of denoising diffusion probabilistic models?
Images or latent variables?

The variational distribution is the distribution (or set of distributions) that you use to approximate the distribution you are looking for. It's often denoted by $q$, $q_\phi$ or $q_\phi(z \mid x)$, where $\phi$ are the variational parameters. For example, in the case of a Gaussian distribution, $\phi = \{ \mu, \sigma \}$. In the case of the VAE, $q_\phi(z \mid x)$ approximates $p(z \mid x)$ and the encoder outputs $\phi$, which can be used to sample $z$, a latent vector, which encodes the original input $x$ (e.g. an image).
So, the variational distribution is not the latent space or vector. However, you can speak of variational parameters, which would be $\phi$, and the latent vector is sampled from the variational distribution.
The VAEs use a general approach to learn distributions, which is called variational inference. I think the term variational comes from calculus of variations, which studies how to optimize functionals, like the ELBO, which is optimized in the VAE. You can read about it in this blog post (I didn't really read it but it looks ok).
This is the simplest implementation of the VAE that I know of, which I recommend you check out.
Right now, I don't know anything about diffusion models, so I don't know if they adopt these techniques.
(I did my master's thesis, a few years ago, on the application of Bayesian neural networks, some of which are based on variational inference, to a specific context)

Thank you very much. I get it in the case of VAE.

@diffusionstable If the diffusion models use variational inference, then the concept of the variational distribution that I describe here should also apply to them. But maybe someone will provide an answer that addresses that specific case.


text:
Is there a way to design neural networks with symmetric Jacobians?
Is there a way to design neural networks with symmetric Jacobians-the Jacobian of the output with respect to the input? Could you point me to any relevant literature in this area?

While Im not sure to have any pointers to the specific literatures on the topic, it sounds rather challenging. You can take a look at the document here  https://princeton-introml.github.io/files/ch11.pdf for general information on back propagation. The document touches on the jacobians also, so it might be useful.


text:
Looking for an audio classification approach
I am working on a deepfake audio classification project with a dataset consisting of only 3000 samples. I have made several attempts to address this challenge. Firstly, I extracted melspectrograms and utilized them as input for CNN and BiLSTM models. Also, I extracted various features such as chroma_stft, rms, spectral_centroid, rolloff, zero_crossing_rate, and mfcc, and passed them in different machine learning models including SVM, RF, XGBoost, and others.However, I have concerns regarding the generalizability of my model. I am unsure if it will perform well on unseen data. Hence, I would greatly appreciate any suggestions for alternative approaches or recommendations for pretrained models that might be suitable for this particular case.


text:
What is the Input to the agent in the paper "Model Based Reinforcement Learning for Atari", and why does the world model run at inference time?
I am currently reading the paper Model based Reinforcement Learning for Atari. However, they do not specify what exactly they use as input for the agent. I believe it to be the observation space - meaning the stacked frames. During training time, these would be provided by the world model, while at inference time the real environment would provide them - right? However, they also specify under Section 4 under "Stochastic Models" that this stochastic model - which is part of the world model in a sense as far as i understood - is running at inference time. But this would only make sense if we would use any of the outputs of the world model during that time, which we dont as far as i understood. Could somebody clarify this?