text:
Which Reinforcement Learning algorithms are efficient for episodic problems?
I have some episodic datasets extracted from a turn-based RTS game in which the current actions leading to the next state doesn’t determine the final solution/outcome of the episode.
The learning is expected to terminate at a final state/termination condition (when it wins or loses) for each episode and then move on to the next number of episodes in the dataset.
I have being looking into Q learning, Monte Carlo and SARSA but I am confused about which one is most applicable.
If any of the above algorithms are implemented, can a reward of zero be given in preliminary states before the termination state, at which it will be rewarded with a positive/negative (win/loss) value?

When applying techniques like SARSA(which are on-policy), one needs to have control over a simulator. If one is able to access only the episodic dataset, then the only choice is to opt for Q-learning or Off-policy Monte-Carlo(or off-policy methods in general).

Can a reward of zero
  be given in preliminary states before termination state of each
  episodes at which it will be rewarded with a positive/negative
  (win/loss) value?

With regards to the above question, the answer is yes. The task would be a sparse reward task with a reward occurring only at the last transition. The issue that one faces in a sparse reward task is that of slow convergence(or even lack of convergence).
Some guidelines for tackling sparse reward tasks are as follows :

Monte-Carlo, and n-step Q-learning are preferred over Q-learning/SARSA.
Consider the 10-step chain MDP, where the only reward is +1 when the transition from s10 to END occurs.

Let us consider the first episode in training to be start->s1->s2->s3->....->s10->end. The Q-learning update would result in no sane updates to states s1, s2, s3, s9 because the Q-value of the next state is random-value. The only state with a sane update is s10
However, if we use n-step Q-learning or a Monte-Carlo based update. The Q-values of all the states are updated in a logical manner since the reward from the end of the episode propagates to all states in the episode.
n-step Q-learning would be ideal since by adjusting the value of n, one can trade-off the benefits of Monte-Carlo methods(described above) and Q-learning(low variance).
The use of pseudo/auxiliary rewards.
This is not something necessarily recommended since the addition of new reward-structures can cause unintended behaviour. On the flip-side, it could lead to faster convergence.
A simple example is as follows: Consider a game of chess, where the only reward is at the end of the game. Since the game of chess has very long episodes, one can introduce the following reward-structure instead :

+100 for winning the game
+1 for capturing any piece on the board

Hence, the pseudo reward may provide some direction for learning. Note the different scales in the two rewards(100:1). This is necessary because the primary goal of the task should not shift to capturing pieces, but to win the match.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Does MSE loss function work in NN training for predicting values between 0-1?
In a NN regression problem, considering that MSE is squaring the error and the error is between 0 and 1 would it be pointless to use MSE as our loss function during model training?
For example:
MSE = (y_pred - y_true) ^ 2

@ Expected model output range [0, 1]:
MSE = (0.1 - 0.01) ^ 2 = 0.0081

// Significantly larger error is less pronounced in the MSE output
MSE = (0.1 - 0.0001) ^ 2 = 0.00998001

@ Expected model output range [10, 20]:
MSE = (10 - 12) ^ 2 = 4

// Significantly larger error is more pronounced in the MSE output
MSE = (10 - 20) ^ 2 = 100

If it’s indeed useless for that range, would using RMSE allow us to use this loss function at 0-1 range to benefit from its outlier sensitivity during training or is there another loss that would mimic the effect of MSE for values between 0 and 1?

@user253751 under the assumption that since squaring error values that are <1 has the opposite effect (i.e 0.2^2 = 0.04 but 2^2 = 4) since this is being used in a loss function from which the neural network is computing gradients the outliers move the gradient less than values within the common range of errors.

Depending on what you want to do, there are advantages to other loss functions (crossentropy) and other regression models (beta regression), but there is not necessarily a reason to dislike MSE as a loss function when the target is between $0$ and $1$, no. For instance, it might be that you know your outcome has a Gaussian distribution with a mean between $0$ and $1$. Then what you’re doing by minimizing MSE is equivalent to maximum likelihood estimation, which has many nice statistical properties.
From calculus, $f(x)$ and $\sqrt{f(x)}$ are minimized by the same $x$. Thus, RMSE and MSE are equivalent as loss functions in the sense that whatever minimized one minimizes the other.

Using a cross entropy loss vs. a regression loss amount to doing two greatly different tasks. From my understand OP has output values in $[0, 1]$, whereas a cross entropy loss is used for classifications where your values(/labels) are in ${0, 1}$.

It might be that the true targets span the entire interval. It also could be that the targets are just the integers and the OP wants to estimate the probability. That is why it depends on the task.

Yes just to clarify the expected output is a range between 0-1 not binary. It’s a regression problem not a classification.

When aggregated over a batch of data, MSE and RMSE have different minima. And I think it is perfectly fine to use cross entropy for non-binary labels, they just represent uncertainty of the correct answer. I do this regularly with binary image masks with blurred edges.

To summarize all the comments and what my original mistake was:
The sensitivity effect is still there even though yes the squared of the error value less than 0 is less that itself however:
Example:

For two error values of 0.9 and 0.2

MAE: 0.9
MSE: 0.9 ^ 2 = 0.81

MAE: 0.2
MSE: 0.2 ^ 2 = 0.04

MAE: 0.9 / 0.2 = 4.5 # less sensitive in MAE
MSE: 0.81 / 0.04 = 20.25 # still significantly sensitive in MSE

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Can PPO be applied when the environment is "input-driven"?
I'm reimplementing an RL paper about learning a job scheduling policy that acts so as to minimize average job completion time. They claim that this is an "input-driven" problem, i.e. much of the variance in rewards is due to the randomness in job arrival sequences, rather than the policy's actions. They deal with this by computing advantages using "input-dependent" baselines. During each training iteration, they have $N$ rollout workers running in parallel on the same job arrival sequence, and after all the rollouts are collected, time-based baselines are computed as follows:
$$b(t) = \frac{1}{N}\sum_{i=1}^N \hat{r}^i(t) \quad \forall t\geq0$$
where $\hat{r}^i(t)$ is some continuous interpolation (e.g. piecewise linear) of the discounted returns $\{r^i_t\}_{t\in\mathcal{T}^i}$ at time $t$ from rollout $i$, where $\mathcal{T}^i \subset \mathbb{R}$ is the set of simulator wall-times of each step in rollout $i$. Then, advantages are computed for each rollout as
$$A^i_t = r^i_t - b(t) \quad \forall t\in\mathcal{T}^i, \forall i\in[N].$$
Since the baselines are computed only as a function of the current training iteration's job arrival sequence, they claim that the variance problem is eliminated. I think this is an interesting approach, and I haven't seen it elsewhere, though I admit I am very new to RL.
Once the advantages are computed, they learn from all the data collected in rollouts $i=1,\ldots,N$ in a single step using REINFORCE, i.e.
$$\theta \gets \theta + \alpha \sum_{i=1}^N \sum_{t\in \mathcal{T}^i} A^i_t\nabla_\theta \log \pi_\theta(a^i_t | s^i_t).$$
I am wondering: is PPO suitable for this problem? Could it work well using these advantage calculations instead of using a critic network? If not, could a critic network be configured so as to reduce variance in a similar way, i.e. taking into account only the current job sequence? Thanks!

Can you please put your specific question in the title?

@nbro I've updated the title

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to use a trained neural network to find optimal function inputs?
I have a deep neural network with 4 input nodes, 4 hidden layers with 4 nodes each and 1 output layer with one node (TRUE, FALSE). I have already trained the NN using backpropagation because I have labelled data. That four input parameter values trained my model on test data to get the output. My model is trained now with optimal weights.
I have already trained my model using that labelled training data. Now using that model I want to get the new values of that 4 input parameters which give me accurate results. After that, I want to calculate the delta between the actual 4 input parameters and tunned-input parameters.
In summary, my question is how I can manipulate(tune) my input parameter, based on that model which I created using labbeled data.

Hi Ibrar, I think you misunderstand the neural network concept. Can you give more clarificatoin, for example what is the result in one output node you expect from the model?

In simple words, can I tune input parameter not the weights. Mean finding grading decent using loss function on input parameter

Unfortunately "tune input parameter" is still broad, may be you can edit your question and tell us your case and explain what the actual problem you want to solve

Could you explain what you are attempting to tune your input parameters to achieve? I suspect you are hoping to use a neural network to learn a function and then find some optimal inputs to it that should minimise or maximise the output of that function? You can definitley do that mechanically and discover inputs that seem important to your network (this is similar to how Deep Dream works for example), but there are limitations. If this is what you want to do, please add a comment or edit about it. If not, please tell me how my guess is wrong.

@NeilSlater yes that i want to do that

I have tried to re-write your question based on my guess, and using as much of your original words as possible. Please say if I have got it right - is this the question you want to ask?

@NeilSlater I modified the question a bit to be more specific.

I no longer understand the question unfortunately. With the binary output, you don't appear to be optimising anything, but you are trying to discover some other characteristic of the inputs. I cannot figure out what that is from your question though - what is your goal for tuning the input parameter? What do you hope the tuned input parameter will do to the system that you are modelling, different from the labelled data that you have already collected?

You can try to optimize the 4 input parameters to maximize or minimize the output of the neural network with a mathematical optimizer. The scipy.optimize package has some methods you can use.
It is possible that the optimization result looks good but performs badly, so you should test with the real system and add it as an additional data point if the results aren't good, and repeat until they become better.

Perhaps you could add a warning about limitation of neural networks when extrapolating. They will want to constrain the input parameters to be within ranges seen in the training data

I already have labelled data, that's mean for test data I know what are the values of 4 input values what are its result (TRUE, FALSE).  I want to train my model first, then using that model I want to get the new values that 4 parameters which give me accurate results. After that, I want to calculate the delta between the actual 4 input paraters and tunned-input parameters.


text:
How to solve a reinforcement learning problem with changing rewards?
I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for the same $(s, a)$ pair can change with time too.
In my problem, a UAV flies and visits several devices that would go through non-stationary environments. Each visit of the UAV can result in a performance improvement for the visited device. This happens gradually until a new environment appears for this device.
So, the state is the time elapsed since the last time it visited each device. The action is the chosen device. The reward is the summation of the average cost of the current visited device and estimated rewards of other unvisited devices. This results in the fact that the UAV's reward can change. Most of the time, it increases gradually. Occasionally, it drops because of a new environment. Like a staircase going up, up, up, and suddenly down. Then it goes up and repeats. The time frequency of a new environment for each device is unique.
I tried the above state and reward settings. When I use Actor-Critic to solve it and only use the reward of the current visited device, the UAV tends to visit one device only and ignores other devices. But when I add the reward of other unvisited devices, no convergence can be seen. In this case, the UAV tends to visit only one device occasionally, even when the reward is extremely low.
My questions are:

Am I setting the state and reward in the right way?

What algorithm should I use here?

Sounds like it settles down too early.  It needs broader exploration and experimentation to find truth.  There is a decent Q-learning example that allows the agent to start at many initial conditions, just click the topics-links near the bottom  (https://people.revoledu.com/kardi/tutorial/ReinforcementLearning/).

Please, put your specific question in the title. "reinforcement learning with changing rewards" is not a question.


text:
How to identify and diferentiate several edge lines of an object?
I want to create an AI to detect and identify certain edge lines on my image. The input image is a locker key, and I want to know the exact position of certain edges.
Sample input image:

Sample output image. As output i have each red line position.

I was thinking in finetuning some kind of instace segmentation or even yolo network but maybe there is a better approach. Ideally, I would like the software to be light enoguh to run in a mobilephone.
I also considered two "simpler" approaches:

Using corner detection
Using Unet to get the contour + opencv approxpoly to get straight lines coordinates

Nevertheless, I believe I would obtain a more consistent result with more advanced AI. (I do have a lot of training images to use)

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

Please, put your specific question in the title. "Precision edge identification" is not a question and it's also not specific. A question is something like "What is 2+2?".

I don't think that more advanced AI would necessarily produce more consistent results. Check something as simple as the Prewitt operator, which is pretty damn good at edge detection. I would suggest starting with something simple, looking at what are the failure edge cases (pun unintended), and only use a more complicated network if the failure cases are plenty and addressable by some more advanced technique.

The problem is that when I detect edges is hard to identify and diferentiate the actual edge lines. For example, in this image first and second from the left are really close, how can I actualy obtain the points that define the lines? Also consider some images have more shadows or lights...

I suppose this boils down to how you define edges. My previous suggestion would basically identify the boundaries of the object that's on the picture, not just the top "edges" as you call them. Your definition of edges is thus different, more nuanced than the usual ML edge-detection tasks which try to identify outlines of the objects in the picture.

I think your second proposed approach is worth trying. Get the whole outline of the object using whatever edge-detection CNN you have at your disposal and create a function that identifies straight lines that lie on the pixels that correspond to the outline.

thanks, I edited the post to clarify my definition of edges


text:
Machine Learning book for fundamentals - Simon Haykin vs. Christopher M. Bishop
Since I started studying Machine Learning, I was torn between two books in this area, and I could never decide which one is the best to follow.
The first book is widely used and known: Pattern Recognition and Machine Learning, by Christopher Bishop. I've never head about him before, probably because he is a Computer Scientist and I am an Engineer, and he doesn't have many books, AFAIK. But I know that this one I like a bible for machine learning fundamentals.
However, I came across another book: Neural Networks and Learning Machine, by Simon Haykin. I am very familiar with this author since I already read many of his books throughout my graduation on other topics, such as communication systems, signals and systems, adaptive filters, etc... Then I decided to read it and I like it a lot!
However, I realized that Bishop is much more referenced than Simon Haykin regarding machine learning. Furthermore, both books differ a lot with regard to terms. For instance, the term "induced local field", which is largely adopted in the Simon Haykin's book, is not even mentioned once by Bishop (at least, I didn't find it when I looked it up).
All of these aspects make me fell unconformable and doubtful about my book adoption since I could use expressions and terms that I not so used by other authors.
It may be a taste matter, but I do need suggestions whether the Bishop's book is much better than anything else to get the foundations or it is ok to adopt Simon Haykin.
Thank you in advance.

I don't know much about the Haykin's book.
The Bishop book seem to be the standard text for a graduate-level course in CS departments in top research universities.
I remember back in the day when I took the ML course, the professor used the Bishop book and was very fond of it.
He said he had read the book three times, once as a undergrad, once as a PhD student, and once when he taught from the book as a professor. And he said after three reads, he finally got every detail in the book.
Basically, he was saying the book was quite dense and written in a way that is not easily penetrable, but once you truly get it, you would appreciate the beauty of exposition style of the book.


text:
How to instruct Mask RCNN to identify objects too close to each other?
I have been trying to train a Mask RCNN model to identify individual poker chips in a stack. No matter what property I change, the end results look like the following image. I was guessing the issue is that the objects are too close to each other for the proper detection. Is there any alternative model or property of mask RCCN or my training model I could possibly try to change?

Could you please reformulate the title in the form of a question? Thanks.

Is there some NMS (non maximum suppression) step ? That may remove overlapping objects too greedily.

Mask R-CNN should work well at this task.  Before training, make sure to draw a mask on every chip in the image (put each mask in a different file).  If you mask out only a small subset of the chips in an image, the NN will observe more negative examples than positives, and thus may be unable to identify positive chips consistently (since many chips present in the image should be detected, but have not been identified by you as positives, thus effectively marking them as negatives).

There could be many reasons why your Mask RCNN model is not performing well on your poker chip dataset:

One potential issue with training a Mask RCNN model to detect poker chips in a stack could be that the chips are very similar in appearance and are close together, making it difficult for the model to accurately identify each individual chip.

Another issue could be that the Background Clutter is similar to the color of the poker chips, making it difficult for the model to accurately segment the chips.

The objects in your dataset are too small or too close together for the model to properly detect them.

The images in your dataset are of poor quality, making it difficult for the model to learn from them.

Your dataset is too small for the model to accurately learn the features of the poker chips.

There is too much variation in the appearance of the poker chips in your dataset for the model to accurately learn their features.

If you are able to improve the quality of your dataset and/or increase its size, you may see improved results from your Mask RCNN.

There are a number of potential alternative models or properties that you could try changing in order to improve the performance of your Mask RCNN model. Some potential options include:

Increasing the overlap between the masks of adjacent objects, and using a higher threshold for the IoU (intersection over union) metric when evaluating detections.
Try to experiment with different hyperparameter values or data augmentation techniques to see if you can improve the performance of your model.
Use a larger mask, such as $512$ pixels or $1024$ pixels. This will increase the size and accuracy of the mask.
Try training a different object detection model on your dataset, such as a Faster R-CNN or YOLO model.
Adding or removing layers from the model.
Modifying the loss function.
Changing the optimizer.

There is a way to define an object and its interior instances (parts of the object) in the labeling. You can use a technique called bounding box regression, which involves finding the best bounding box for each instance in the image and then assigning it a label based on the location and size of the box.


text:
which method are able to approximate the following sequence 1 0 1 0 0 1 0 0 0 1 0 0 0 0 1 5x0 1 6x0
I'm learning deep learning and machine learning techniques, but it looks like neural networks are not capable of predicting the next element in the sequence 1 0 1 0 0 1 0 0 0 1 0 0 0 0 1 5x0 1 6x0
Is any other method from machine learning sphere, that can predict next value in such sequences?
Code implementation for the sequence
def sequence() {
    i = 0
    while (true) {
       yield 1
       for j in range(i):
            yield 0
       i += 1
    }

}

Hi, welcome to AI Stack Exchange. I think this question might be interesting, but it is not really clear what you are aiming for. If that was the only sequence, it would be easy to predict. Could you explain what your set of all possible sequences that might need to be predicted is pulled from - maybe in the form of a few Q&A examples. I can see there is some "logic" to the sequence, and it reminds me a little of non-verbal reasoning IQ tests, but it is possible you are looking for a very general AI that can apply some form of Occam's Razor?

For an example of a sequence that is non-obvious, but that would fit your question: https://oeis.org/A269303 - and OEIS has over 1000 different sequences that contain 1, 2, 3, 4, 5, 6 but do not have 7 next . . . and these are all notable sequences that follow some kind of consistent internal logic.

Hi, assume this is the only  sequence, so it can be implemented as in the the code(I've just attached to the question)
I’m actually not so profficient in AI, I’ve tried some simple Neural Networks models,
but it looks like they are unable to predict this sequence. So I wonder if exist any method that can predict next value in this sequence?
by the way, I’ve also tried this problem in GPT3(not sure if it’s suitable for this), but it also failed to predict it

Regarding problem you provided, it look interesting,  but I think it is too complex for any known method to approximate it, I’m not sure if even human can do this

Yes I thought of GPT-3, tried it on your problem unsuccessfully, although it can solve shorter versions of this and other simple logic tests.


text:
How to justify the chosen neural architecture?
I had a task to implement a neural network that would carry out multiclass classification of traffic by several parameters. On the advice of colleagues, I chose the "Multilayer Perceptron" architecture. One of these days I will have a defense of my work, but I absolutely do not understand how to answer the question: "Why did you choose this type of architecture?". Please tell me if there are any theses why the "multilayer perceptron" architecture is better than other neural network architectures for solving problems of multi-class traffic classification?

That's hard to answer without more details about the problem. And when you say MLP, do you mean a network with threshold activation functions, or just a "vanilla" fully connected network?

Hello! Thank you for your answer. The activation function is selected by ReLU. The neural network has the following structure: several neurons of input values, 2 fully connected layers of neurons and several resulting outputs for multiclass classification. It's just interesting to find out what is the advantage of MLP over other architectures in the tasks of multiclass traffic classification

This is a very general question, so I'll just point to a reference that should be a good starting point. Deep Learning for Encrypted Traffic Classification: An Overview seems to contain exactly what you're looking for:

Several factors affect the choice of deep learning models for network traffic classification. The most important one is the choice of features. ...

Table II summarizes features, the corresponding models, and their properties.

Thank you so much! I will read this article.


text:
Finetuning Mistral or MistralForSequenceClassification for text classification
I need to do text classification and have a dataset of 10K entries. I am considering using mistral and following a tutorial like https://huggingface.co/docs/transformers/training and replace model name by Mistral. But I have noticed there is also MistralForSequenceClassification.
Could you please help me with the following questions:
Is using the above mentioned tutorial and replacing the model name with mistralai/Mistral-7B-v0.1 the correct approach?
Do I need to use MistralForSequenceClassification?
Many thanks.


text:
Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?
I am currently using Proximal Policy Optimization (PPO) to solve my RL task. However, after reading about Soft Actor-Critic (SAC) now I am unsure whether I should stick to PPO or switch to SAC. Moreover, from this post, it seems that much of the performance in the original PPO paper comes from code optimizations and not the novel clipped objective.
The main characteristics of my RL task are the following:

The action space is discrete. SAC was originally designed for continuous action spaces but, if I'm not wrong, it can be adapted to discrete action spaces with no problem.
I am trying to learn a policy for generating synthetic data (i.e., generating novel graphs), so diversity is key. For this reason, I want to learn a policy with as much entropy as possible (while still solving the task). Both PPO and SAC try to maximize the policy entropy.
Obtaining trajectories to train the policy is very expensive. My algorithm spends much more time obtaining the trajectories than training the deep neural network of the policy. Here, I think SAC is the clear winner, as it is off-policy whereas PPO is on-policy. Still, PPO is supposed to be very sample-efficient.

Given my current needs, do you think it is worth it to switch to SAC instead of PPO?

Rather than writing a very generic title like "PPO vs SAC for discrete action spaces", please, put your specific question in the title! Thanks.

Thanks for the comment. Do you think something like Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces? would do or is it too long?

I don't think it's too long, but I don't know it's accurate because I didn't fully read your post. If it's accurate, it seems good ;)

Okay, just changed the title ^^

PPO does not solve a maximum entropy objective as far as I know.

First, both SAC and PPO are usable for continuous and discrete action spaces. However, in the case of discrete action spaces, SAC cost functions must be previously adapted. As explained in this Stable Baselines3 issue, its efficient implementation is not an easy task.
Contrary to your hypotheses, off-policy algorithms as SAC are generally more sample-efficient than on-policy algorithms (i.e. PPO), which are generally sample-inefficient due to the data loss that occurs when updating its policy. However, if you're looking for faster training, PPO is your option.
Moreover, as explained by Haarnoja et al. (2018), SAC allows to have a stochastic actor while being more optimal and sample efficient than on-policy methods such as A3C or PPO. It is also less sensible to hyperparameters than all these methods.
Depending on the problem you are dealing with, this Reddit thread may provide you further indications. Finally, here you have a post with some additional information about DRL algorithms comparison.
In conclusion: for a generic problem I would recommend using SAC as it has shown better performance in most problems. However, if in your case performance is not an aspect to evaluate and you value more the speed of training and lightness of implementation, I recommend PPO. Another option could be to use both and compare their performance on your specific problem, which could also be quite interesting. Finally, if you are open to more alternatives, don't rule out TD3 if none of the above convinces you!

Actually, the SAC cost functions do not need to be adapted as per the linked article. You can obtain differentiator samples from a discrete distribution using the Gumbell Softmax trick.

@Antonio I have read both the Medium post you linked and the article https://arxiv.org/pdf/1910.07207.pdf, explaining how to adapt SAC to discrete action settings and it seems quite straightforward. The only modification to the policy, actor and temperature cost functions is that, for the expectation over policy actions, we can now calculate it exactly as we have a discrete probability distribution (while before we had to approximate it with samples). So I don't know why in the Stable Baselines3 issue says the discrete SAC implementation is not trivial...

@DavidIreland It's true that the Gumbell-softmax trick allows the gradients to flow across stochastic nodes (i.e., a node where a discrete probability distribution is sampled) but, related to my previous comment, it seems like there is no point in doing that. Instead, if we simply modify the cost functions (for Discrete SAC) we are now able to exactly calculate the expectation over policy actions. This reduces the variance in the estimation of the policy objective. This is explained in arxiv.org/pdf/1910.07207.pdf (paragraph on top of Equation 10).

There is a point. For instance, if you have both continuous and discrete actions then it is natural to sample from your discrete actions using the Gumbell-softmax trick without making any of the aforementioned modifications. If you have only discrete actions then it is not advisable to use SAC when the DQN and it's variants perform so well.

@DavidIreland I understand your point. However, even if only discrete actions are used, I think SAC is still a sensible alternative to DQN. The authors of the Discrete SAC paper (arxiv.org/pdf/1910.07207.pdf) show that the performance of discrete-SAC is very similar to that of Rainbow, even without tuning the SAC hyperparameters. That along with the fact that implementing discrete-SAC seems simpler than implementing Rainbow or other complex DQN variants seems like a good reason for using SAC over DQN (at least in some scenarios).

@Aeryan , I agree, as you pointed out it might be the necessary modification of the entropy estimation in the implementation + some details which make the adaptation not straightforward but still quite intuitive?


text:
What is the best way to create a vector representation (with fasttext) of a list of words?
Basically what I want to do is to create a single vector representation of a list of skills belonging to employees at a company (one list per employee). The embedding will be a representation of an employee's "profile". The motivation behind this is (among other reasons) that I want to be able to identify clusters among the employees.
Assume I already have a trained FastText model (or Word2vec) that can generate good representations of the individual words in the list.
My current solution is simply to add all the word embeddings in an employee's list together (without any form of normalization). But I'm very unsure about whether this is the best approach to generating a good representation of an employee's profile.
The dimensions of the vectors are 300 and there are usually around 10 to 30 skills in a single list.
Any help would be greatly appreciated!
Example:
Let's say we have an it-consulting firm where each employee has their own set of skills. Some consultants are more experienced or versatile, thus having more skills listed in their profiles. eg we have:
 alex_skills = ['microsoft azure', 'machine learning', 'data science', 'python', 'sklearn', 'xgboost', 'nginx', 'flask', 'SHAP', 'git', 'word2vec', 'statistics', 'deep learning', 'linux','docker compose', 'pandas']

carla_skills = ['devops', 'machine learning', 'deep learning', 'continuous integration', 'kubernetes', 'python','git', 'speech recognition', 'github', 'bitbucket', 'scikit-learn', 'natural language processing', 'pandas']

adam_skills = ['automation', 'robotic process automation', 'banking and finance', 'process mapping', 'IAM', 'väsentlighetsanalys', 'business intelligence', 'auditor', 'requirements handling', 'risk management', 'coordinator', 'project manager', 'data visualization']

As you can see Alex and Carla are more similar and should possibly be in the same cluster, while Adam might not be.
So I wan't to make a vector representation of the entire list of skills. And then I will use these vector representations in some clustering algorithm (eg HDBscan) and by some distance metric (eg. cosine distance), capture the relation between Alex and Carla.
I suspect the fact that the lists have different lengths might cause problems, therefore maybe divide by the length of the list after adding?

Hi @Love Book and welcome to AI Stack Exchange! For clarification, could you define your notion of 'best' and 'good' in this question? For example, are you wondering about a specific use case after the representation/embedding is created, or are you wondering about the most standard/general way of creating the representation (or something else)? We hope to see more of you on this site!

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

Something that might work is a Universal Sentence Encoder. It's not quite what you asked for, as it generates encodings directly from text, not from other word vectors, but it is designed for matching related text:

The researchers who developed it claim the generated "sentence embeddings can be trivially used to compute sentence level semantic similarity scores that achieve excellent performance on the semantic textual similarity (STS) Benchmark". (It may not work as well on your task, of course.)
It's available on the Tensorflow Hub, with a demo on Colab, or you can read the research paper here.

Welcome to the community.
So basically you would like to come up with a single vector representation for each employer considering different skills that particular employer has?
If that is the case, I would say it depends on what would be your goal after doing this analysis. I have seen papers where they simply take the average of these different vectors or just sum them up. Try different approaches and see if it works in your downstream analysis.
You mentioned that one motivation is to identify clusters among employers. I suggest you try averaging and summing them up and analyzing your downstream task (clustering) to see how each approach works.
I will give you an example: In the language of life domain (DNA, RNA, amino acid sequences), there is a paper which describes SeqVec, a tool that utilizes ELMo language model to train and provide vector representations for protein sequences.
Say you have a protein sequence as follows: HMGHGHHMGKGAAKKHMALLKGGAGLCMLK
Now tokenize it in 3-mers: HMG MGH GHG HGH ...
Then imagine we get a vector representation of real numbers for each of these 3-mers.
Then if you want to come up with a representation for the whole protein sequence, would you get the sum of all 3-mers or get the average of it or do something else? I would say, it depends on what problem you are solving and trying different approaches to see which one works in your case.
Hope my answer helped you a little bit. Cheers.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

I agree about the averaging! I realize it might be necessary given that the lists are of different lengths.


text:
For simple weight constraints: Add constraint directly or use parameterization without constraint
I am wondering if it makes sense to parameterize simple weight inequalities, for example if the weights should be $w\geq 0$, one cound train $\exp w$ over the unconstrained set instead. Also, if $\sum w_i=1$ one could parameterize $\frac{e^{w_i}}{\sum e^{w_i}}$ and optimize over the unconstrained set.
While the solutions should be the same, I wonder if it makes numerically a difference, and what way is the right one. I guess the values of $\exp$ might explode and it becomes numerically unstable.
Question: should I use something like tf.keras.constraints.NonNeg() or should I replace each weight $w$, on which I want to apply the constraint with $\exp w$ ("reparameterization") (since exp>0)or is there no general rule?

Can you please reformulate the title in the form of a specific question, so that it's clear what you're really asking here? I don't see any question in your post!

Sorry if it is not clear. I want to use a non-negative weight constraint in one layer. Question: should I use something like tf.keras.constraints.NonNeg() or should I replace each weight $w$ on which I want to apply the constraint with $\exp w$ ("reparameterization") (since $\exp >0$)or is there no general rule?


text:
Is it possible to create an AI that produces output without giving it input?
AFAIK an AI is first trained using a data set of input and output values.
After the training proccess you can give the AI input and it will produce output.
For example when you write a sentence to an AI chatbot it will write an answer.
But is it possible to create an AI that produces output without giving it input?
For example that would mean there is a chatbot saying things without being asked.
Edit: In the training proccess I do not care if input data is used. I want to know if after the training proccess it is possible to create a chatbot (for example) which says things without giving input.
For example when you are on a discord channel it is normal that people start conversations without being asked something.

Yes, and in fact this is exactly what all of the GPT models (including ChatGPT, I think) do!

I upvoted the question since i like the topic. I think the question 'is it possible' is not so much the right question as 'is it useful'. yes it is definitely possible to create a machine that just randomly outputs stuff without giving it input. This would be a satisfactory machine for your question. However, whether it is possible to create a useful machine that outputs stuff without giving it any input is a much more difficult question. You hence need to specify your question further. First question would be: do you allow it to train on input data? Or is that input also not allowed?

@RobinvanHoorn See my edit.

@RobinvanHoorn Do you know why I don't get any results when googeling for "satisfactory machine"?

Okay, with the edit, it is even more obvious to me that the logical question to ask is 'is it useful'. Like you can indeed just let GPT-3 or some large language model predict the next word without giving it any input. But the output is probably not very useful. So 'is it possible' is definitely yes

'Satisfactory machine' is not a term. You stated some set of requirements for your 'algorithm/machine', and i proposed some 'machine' which adhered to those requirements, making it a 'satisfactory machine' for your requirements.

For your discord example you may search keyword like "discursive meaning/semantics".


text:
Why clip the PPO objective on only one side?
In PPO with clipped surrogate objective (see the paper here), we have the following objective:

The shape of the function is shown in the image below, and depends on whether the advantage is positive or negative.

The min() operator makes $L^{CLIP}(\theta)$ a lower bound to the original surrogate objective.
But why do we want this lower bound? In other words, why clip only at $1+\epsilon$ when $A > 0$ ?
Isn't it important to keep the new policy in the neighborhood of the old policy, so that even $r_t(\theta) < 1-\epsilon$ should be undesired?

In the PPO loss in PyTorch, they clip from both sides (i.e without considering the sign of Advantage), but it still works.

Yes, the idea of PPO is to keep the updates small so that the new policy is not too far from the old policy. If you look at the left figure, this is the case as the absolute magnitude of L^clip is capped. The only region where this absolute magnitude is uncapped is on the right hand portion of the right figure. In this region, r is > 1. Since r = new prob / old prob, it means the previous update has increased the probability of an action that resulted in a worse than expected outcome (hence the negative advantage). Therefore, we want to unroll that update and not capping the ratio will achieve that goal better.

Okay, so if I understand correctly, the fact that the right plot is not capped at 1+e, means that we can unroll that update at the next update step because then the gradient contribution at that specific time-step is non-zero. However, if it would be capped at 1+e, then the gradient would be zero, and unrolling would not occur. Do you agree?

Yes, that is my understanding. If it were capped at 1+e, unrolling should still occur if the ratio is below 1+e as you would still have non-zero gradients passing through. But if the ratio were large, which means the probability of those actions have changed a lot in the wrong direction, you end up not unrolling when it is most needed.

A positive advantage increases the probability of taking that action, hence $A_t > 0$ means that the gradient update makes $r_t(\theta)$ larger. We don't want to take too big of a step, hence we only let $r_t(\theta)$ increase to $1 + \epsilon$ before we start ignoring that advantage.
If $A_t > 0$ but $r_t(\theta) < 1 - \epsilon$ it must mean that there are many other gradient samples in the training batch that are pushing down $r_t(\theta)$, because if we only had $A_t$ it would increase $r_t(\theta)$. In this case you can see $A_t$ is actually pushing in the opposite direction of the gradient update. If $A_t > 0$ and $r_t(\theta) > 1 + \epsilon$ then $A_t$ is going in the same direction of the gradient update.

Quote from https://huggingface.co/learn/deep-rl-course/unit8/visualize :

We update our policy only if:

Our ratio is in the range
Our ratio is outside the range, but the advantage leads to getting    closer to the range

Being below the ratio but the advantage is > 0
Being above the ratio but the advantage is < 0

And that's exactly what the image you've attached is showing. Keep updating while we are climbing towards the range. It's also fine to update within the Trust Region. Ignore if we overshoot this region and wanted to go even further.

There are six different situations:

Case 1 and 2: the ratio is between the range
In situations 1 and 2, the clipping does not apply since the ratio is between the range $[1 - \epsilon, 1 + \epsilon]$
In situation 1, we have a positive advantage: the action is better than the average of all the actions in that state. Therefore, we should encourage our current policy to increase the probability of taking that action in that state.
Since the ratio is between intervals, we can increase our policy’s probability of taking that action at that state.
In situation 2, we have a negative advantage: the action is worse than the average of all actions at that state. Therefore, we should discourage our current policy from taking that action in that state.
Since the ratio is between intervals, ** the probability that our policy takes that action at that state**
Case 3 and 4: the ratio is below the range
If the probability ratio is lower than $[1 - \epsilon]$, the probability of taking that action at that state is much lower than with the old policy.
If, like in situation 3, the advantage estimate is positive ($A>0$), then you want to increase the probability of taking that action at that state.
But if, like situation 4, the advantage estimate is negative, we don’t want to decrease further the probability of taking that action at that state. Therefore, the gradient is = 0 (since we’re on a flat line), so we don’t update our weights.
Case 5 and 6: the ratio is above the range
If the probability ratio is higher than $[1 + \epsilon]$, the probability of taking that action at that state in the current policy is much higher than in the former policy.
If, like in situation 5, the advantage is positive, we don’t want to get too greedy. We already have a higher probability of taking that action at that state than the former policy. Therefore, the gradient is = 0 (since we’re on a flat line), so we don’t update our weights.
If, like in situation 6, the advantage is negative, we want to decrease the probability of taking that action at that state.
So if we recap, we only update the policy with the unclipped objective part. When the minimum is the clipped objective part, we don’t update our policy weights since the gradient will equal 0.
Source: DEEP RL Course - Visualize the Clipped Surrogate Objective Function


text:
Higher validation loss after using Dropout
I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use nn.CrossEntropyLoss() as my loss function. I noticed that my model was overfitting (validation loss increased after some epochs) and hence I used a dropout (p=0.4) in the last FCN layer. I’m now seeing a higher loss (for both train and val)
The green line denotes training without dropout

From left(Iteration-wise loss (train loss recorded after every batch), train loss (recorded after every train epoch), validation loss (recorded after every train epoch))
Is this behavior justified or am I doing something wrong?

What is the plot all the way on the left?  Can you display curves for training and validation on the same plot (one for with and one for without dropout)?  As an aside, I have never seen anything like the validation loss with dropout that you are showing.  Very bizarre pattern!

Thanks for clarifying that the iter loss is the loss recorded after every batch, but you should specify that it is the train loss (recorded after every batch).

Sorry for the confusion. I've updated my question. The logs were rotated. I'll re-run the experiments and update them with the plots you've asked

I'm stumped as to what is happening with your learning curves, but you may try moving the dropout layer around and tuning the dropout rate to see how the behavior changes.  Dropout is supposed to reduce / slow down overfitting.  They can slow down learning altogether, but this is very bizarre, especially the validation loss after adding dropout.


text:
How to embed quantitative variables?
I have 3 data types that I want to feed into a neural net.
One, is a time series, which I am going to feed into the neural net directly. The second, are categorical variables that I am going to embed into vectors of equal length as the time series. The third, is quantitative numbers that I also want to embed like my categorical variables, but I do not know the best way to do this.
My initial thought is that I could input the numeric data into a vector of all 0's.
For instance, if a numeric data point is $1.32$, then:
$[1.32, 0, 0, ..., 0]$.
Or I could do something similar like: $[1.32, 1.32, 1.32, ..., 1.32]$.
Is there a common best practice for what I am trying to do?

Is there a common best practice for what I am trying to do?
No, there are no best practices for what you are trying to do.  Your categorical variables, time series, and numeric variables don't have to be the same size.  Therefore, there is no need to vectorize your single numeric input, nor should you.  You just need to have that variable as its own input.  If it is only one number, then so be it.
On a related note, fusing multiple data types is an exciting emerging area.  This review article will give you ideas on how you can perform integration:  "Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines".

So you're saying that I can take the time series data (let's say it's 5 time steps), the embedded categorical feature (let's says it's of length 3), and my numerical feature of length 1, and concatenate all of them into a single vector of length 9, and then put that final vector into the neural net?

Hi, @Mattpats, that is very close to what I'm suggesting, but remember to one-hot encode your categorical variables.


text:
Why my best fit line is not having a single straight line | Multiple Linear Regression
I am working on Multiple Linear Regression (Multiple variables). I am been able to predict and get a good r2 score. But I am not sure that I understood the part of plotting the best fit line, I can't understand the visualization of such a model.
As I am using 50 startups dataset, which have 3 features and 1 target variable (Profit).

The three features and other dataset info from Kaggle(Advertising
dataset) :
https://www.kaggle.com/datasets/yasserh/advertising-sales-dataset

I will show some of my main code snippets to better illustrate my model.
main.py

df =  pd.read_csv('ad_sales.csv')

# Perform feature scaling
# Separate features and target variable
features = df[['TV Ad Budget ($)', 'Radio Ad Budget ($)', 'Newspaper Ad Budget ($)']]
target = df['Sales ($)']

# Perform feature scaling
scaler = MinMaxScaler()
scaled_features = scaler.fit_transform(features)

# Create a new DataFrame with scaled features
features = pd.DataFrame(scaled_features, columns=features.columns)

features = np.array(features)
labels = np.array(target)
## all features are Min-Max Scaled

model = MultipleLinearRegression()

model.train(x_train = features, y_train = labels, epochs=150)

My model :

    def predict(self,x):
        y_pred = [self.b0 + ( (self.b1 * xi[0]) + (self.b2 * xi[1]) + (self.b3 * xi[2]) ) for xi in x]
        return y_pred

    def train(self, x_train, y_train, epochs):
        for epoch in range(epochs):

            y_pred = self.predict(x_train)
            loss = mse(y_pred, y_train)
            self.optimize(x_train, y_pred, y_train, learning_rate=0.001)

        r2 = r2_score(y_train, y_pred)
        print("R2 Score:", r2)

        # # Scatter plot - Target vs Each Feature
        plt.scatter(x_train[:, 0], y_train, color="blue", label="Actual feature 1")
        plt.scatter(x_train[:, 0], y_pred, color="red", label="Predicted feature 1")
        # Line of best fit
        plt.plot(x_train[:, 0], y_pred, color="green", linewidth=2, label="Best fit line")
        plt.xlabel("Feature 1")
        plt.ylabel("Target")
        plt.legend()
        plt.show()

        plt.scatter(x_train[:, 1], y_train, color="blue", label="Actual feature 2")
        plt.scatter(x_train[:, 1], y_pred, color="red", label="Predicted feature 2")
        # Line of best fit
        plt.plot(x_train[:, 1], y_pred, color="green", linewidth=2, label="Best fit line")
        plt.xlabel("Feature 2")
        plt.ylabel("Target")
        plt.legend()
        plt.show()

        plt.scatter(x_train[:, 2], y_train, color="blue", label="Actual fetaure 3")
        plt.scatter(x_train[:, 2], y_pred, color="red", label="Predicted feature 3")
        # Line of best fit
        plt.plot(x_train[:, 2], y_pred, color="green", linewidth=2, label="Best fit line")
        plt.xlabel("Feature 3")
        plt.ylabel("Target")
        plt.legend()
        plt.show()

  ## As it have 3 features, instead of 3d plot, I am plotting one feature with true labels and y_predicted. the plot screenshot is shared in attachment.

As you can the plot is not showing a straight line, it have a many straight lines combined to match all data points, how ? Isn't the best fit line supposed to be a single line in between the data points . Or it is because of some visualization techniques I used wrong ?
|| Edited : I want to visualize my Features and if possible want to fit a bit fit line, or curve line if in Poly Model. Like in this Figure :

https://www.megatrend.com/wp-content/uploads/2022/10/2-1.jpg

Can you upload your 3 feature input raw files? Apparently not even needing to dig deeper into your own optimize() algo details, your pyplot is plotting based on raw (unsorted) input feature slice zipped with their corresponding target labels (noticing the red-only  connecting lines).

Yeah sure, I have edited and added the features link from Kaggle, all those first three are my features ['TV Ad Budget ($)', 'Radio Ad Budget ($)', 'Newspaper Ad Budget ($)'] . @mohottnad

So indeed you are just plotting the raw unordered zipped $(x_i, y)$ tuples, while you seem to want to draw the resultant regression lines for the 3 pairs. Multivariable (homoscedastic) linear regression is a straightforward model with analytics closed form solution for all the coefficients as $\hat{\beta}=(X^TX)^{-1}X^Ty$, or you can use simple gradient descent and even sciki-learn's LinearRegression model.

If you want to plot some fitted line, you have to project the hyperplane (your overall fitted model) onto different slices (2D planes), which will indeed produce lines.
To find the equation of the line, you just need to inject the constants values for each feature into the global model
$$ y = {a_1}{x_1}+{a_2}{x_2}+{a_3}{x_3} + ... + b $$
For example, if you want to observe the best fitted line on a plot $y$ vs $x_1$, you can choose the slice $x_2=x_3=...=x_n=0$ . This corresponds to the line equation $y=a_1x_1+b$.
Also if you look at a different slice $x_2=x_3=...=x_n=1$, then the new line equation is $y=a_1x_1+b+\sum_{i=2}^na_i$ .
This also shows that the overall slope will not change (here it's still $a_1$ in both examples) for slices corresponding to the same feature but the intercept will be affected by the slice choice.
However you need to be cautious about how you interpret these plots as they only show you a partial view of the full model !

Hyperplanes works best till 3 or 4 features maybe. i don't know how does this plot and best fit line works when it comes to Multiple variables against y_pred. So , are you saying that for 3 features f = x[:0], x[:1], x[:2], i need to use those slices in this equation = y = mx + b , where x is each x[:0] etc. right ? I have somehow plotted x[:0], y_pred. Iam unable to draw conclusions out of it.

You just cant visualise the full model if you have more than two features.

Isn't the best fit line supposed to be a single line in between the data points

Not here. The linear relationship is between prediction y and each feature $x_i$ respectively, while keeping other features constant. So there is no guarantee that the relationship is linear between predictions.
Recall for simple linear regression: $y = ax+b$, so $x$ and $y$ hold a linear relationship.
But for multivariate linear regression: $y = {a_1}{x_1}+{a_2}{x_2}+{a_3}{x_3} + ... + b$
If you think about the relationship between $y$ and each of the $x_i$, it is clear that in general their relationship is linear only when the other $x_j$ are constant, which is usually not the case.
Under the same argument, the outputted predictions $y$ are in general nonlinear too.

Can you explain a bit further. But I am trying to fit best fit line through features and y_pred, right ? I understand that the regressed points are good and close, but i want to see best fit line in single plot or multiple like i plotted (x_train[:, 0], (x_train[:, 1] ...... like so on. Can you explain further a bit more.

@NiranjanadasM No, the linear plot you want does not make sense. For 2+ variables it is not a line, but a hyperplane. To visually inspect the prediction result, check e.g. the partial residual plot.

Guess I have to study more on hyperplane and how does it effects the plotting. plotting more than 1 feature won't get me a straight best fit line against y_pred. I have to look onto it. Thanks tho for the explanation


text:
How to normalize input data to Reinforcement learning platform (Gym and stable-baselines)
I created a custom environment with Gym and trained it with stable baseline 3 algorithms.
The observation and space action are both continues.
The observation space includes 10 values and action space has 2.
The action space is [0,1] and I know it's not symmetric.
I want to normalize my inputs to the NN and I Don't Know how to do it.
The environment observes Temperature and Power (input values) and takes action (continuous between [0,1]) and returns state (10 values not normalized)
I believe I have to normalized all inputs to get better results but I don't know how can I normalize it bf feeding to neuronal network and unwrapped it afterwards.
I appreciate any help specially if you can link some dummy code or a practical case.

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.


text:
Neural Network output for the game of Checkers
I'm trying to train a RL agent to play the game of checkers (AlphaZero style) and so far I've managed a proof of concept training a connect 4 agent up until perfection. However, unlike connect 4, checkers moves pieces rather than placing them and sometimes even multiple times. I think I understand how I would do this for chess: I have an output size of 80 (16+8*8) and have the first 16 outputs represent the piece that will move and the other 64 represent the position it will move to. I'm not sure if this is a valid solution though. The real problem arises when considering checkers with multiple jumps. Is there any solution to this and am I thinking about it the right way? I've pondered not changing the player turn whenever a double jump is available but I feel like this will screw with the MCTS.

Correction regarding policy encoding
The policy encoding you propose for Chess is not the one used by the original AlphaZero and would not work. The network needs to output a policy distribution over the possible moves, so there needs to be a dedicated output value for each move. Your encoding only allows the network to express a single move, or at best some low-rank approximation of the full distribution.
The AlphaZero paper (free access) has a section Representation, which explains the policy encoding they used. It has shape 73x8x8. Of course in a chess position there are never actually that many moves, so invalid moves are masked out.
Complex move representations in AlphaZero
As you also note there are two possible solutions to representing these types of compound moves:

Figure out all possible moves and some reasonable way to encode them in a policy head. AlphaZero can still train quite well with a large, sparsely used policy head as shown by their chess results.

Represent the game differently, with moves split up into submoves that don't always change whose turn it is, and then find an encoding for this smaller set of moves. AlphaZero and more generally MCTS doesn't require alternating players, for example the followup work MuZero (free access) works just fine on the Atari games with only a single player.

Practical advice for checkers
I think both solutions would work fine for checkers. Even including compound moves I think checkers still has less than 73*8*8 = 4672 possible moves, although I don't know for sure. If it's in the same ballpark or lower the first method should work. The first solution also has the advantage that it's simpler to implement and that you can achieve deeper search with a fixed number of neural network evaluations.
The second solution is in some sense more general and scalable, so it could be interesting to experiment with as well.

I think I will end up separating multi moves into sub moves since trying to encode multi jumps in several different directions and including king moves jumping will be unworkable. I appreciate your detailed response

@AnikPatel You could also go for a hybrid where you encode some compound moves as a single move. (and disallow the sequence of single moves that would lead to the compound move to prevent duplication). I've seen this work for the game Arimaa for example.


text:
Machine Translation Transformers: Why Mask in Decoder?
I am trying to understand the purpose of masking in the decoder in the "Attention is All you Need" paper. Why wouldn't we want to rely on bidirectional context when translating? What goes wrong if masking isn't used in the decoder? Are there successful models in which the decoder doesn't use masking? Thank you!

Masking
The purpose of masking in the decoder is to prevent the model from having access to future tokens when predicting the current token in the output sequence. During training, the model is given the entire target sequence as input, but it should not be allowed to use information from future tokens to predict the current token, as this would violate the autoregressive nature of the decoding process.
The decoder should only have access to the leftward (previous) context when generating a translation. If masking were not used, the model would be able to "cheat" by leveraging information from future tokens, which would not be available during inference or when generating translations in a real-world scenario. This would lead to a model that performs well on training data but fails to generalize to new, unseen data.
Alternative Approaches
As for models that do not use masking in the decoder, most sequence-to-sequence models rely on some form of autoregressive decoding, either through masking or through other mechanisms. For example, traditional RNN-based encoder-decoder models do not use explicit masking, as they generate the output sequence one token at a time, inherently relying only on the previous context. However, there are non-autoregressive models that predict the entire output sequence in parallel, which do not use masking in the same way as autoregressive models. These non-autoregressive models can be faster but often sacrifice some accuracy compared to their autoregressive counterparts.


text:
Multiplying tensors element-wise along one dimension
I'm new to TensorFlow and I want to perform the following operation on two tensors. Let $A_{\mu_1,\dots,\mu_M}$ be a tensor of rank $M$ and let $B_{\nu_1,\dots,\nu_N}$ be a tensor of rank $N$. The indices $\mu$ and $\nu$ all run over $1,\dots,d$.
Now let's pick a dimension $i$ for tensor $A$ and a dimension $j$ for tensor $B$. I want to construct the tensor
$A_{\mu_1,\dots,\rho_i,\dots,\mu_M} B_{\nu_1, \dots,\rho_j, \dots, \nu_N}\;\textrm{(no Einstein summation)}$.
That is, I want to multiply the tensors element-wise along two specifically chosen dimensions. Note that I don't want to contract over $\rho$. I want a tensor of dimension $M + N - 1$. The final ordering of the indices doesn't matter as long as I can keep track of them.
EDIT: For clarity, let's just multiply the last index of $A$ and the first index of $B$ element-wise.
EDIT 2: The solution previously posted here as an edit has been posted as a solution.

Can you give an example using real numbers of what you are looking to do?

I added working code. I don't know if there is a simpler way to do this with native broadcasting. Suggestions are welcome.

Thanks for posting an example.  This does help me visualize what you are trying to do.  This is a viable solution, so it is fair to post it as an answer.  As far as I know, there is no built-in TF2.0 functionality that allows you to do this more easily.  I do think this is more of a programming question, so StackOverflow will be a better place to post this (and you will probably get more input).

I'm getting the hang of TensorFlow and found a solution. I managed to write the following example code to multiply the last index of $A$ and the first index of $B$:
import tensorflow as tf

def stitch(a, b):
    rank_a = int(tf.rank(a))
    shape_a = list(tf.shape(a))
    rank_b = int(tf.rank(b))
    shape_b = list(tf.shape(b))
    rank_c = rank_a + rank_b - 1

    num_pad_right = rank_c - rank_a
    num_pad_left = rank_c - rank_b
    shape_padded_a = shape_a + [1] * num_pad_right
    shape_padded_b = [1] * num_pad_left + shape_b
    shape_c = shape_a[:-1] + shape_b

    padded_a = tf.reshape(a, shape_padded_a)
    padded_b = tf.reshape(b, shape_padded_b)
    padded_c = tf.math.multiply(padded_a, padded_b)
    c = tf.reshape(padded_c, shape_c)
    return c

x = tf.constant([[[1.0, 1.0], [2.0, 2.0]], [[3.0, 3.0], [4.0, 4.0]]])
y = tf.constant([[2.0, 2.0], [2.0, 2.0]])

print("=== Left tensor ===")
print(x)
print("\n")
print("=== Right tensor ===")
print(y)
print("\n")
print("=== Stitched tensor ===")
print(stitch(x, y))
print("\n")

This outputs
=== Left tensor ===
tf.Tensor(
[[[1. 1.]
  [2. 2.]]

 [[3. 3.]
  [4. 4.]]], shape=(2, 2, 2), dtype=float32)

=== Right tensor ===
tf.Tensor(
[[2. 2.]
 [2. 2.]], shape=(2, 2), dtype=float32)

=== Stitched tensor ===
tf.Tensor(
[[[[2. 2.]
   [2. 2.]]

  [[4. 4.]
   [4. 4.]]]

 [[[6. 6.]
   [6. 6.]]

  [[8. 8.]
   [8. 8.]]]], shape=(2, 2, 2, 2), dtype=float32)

I don't know if there is a way to do this in a less cumbersome way with native TensorFlow broadcasting rules. If you are aware, let me know. As Snehal said in the comments, I might try my luck on a more programming oriented forum.


text:
ML-based algorithm/software for solving a sudoku puzzle a human way
I am a new contributor and have no experience in ML, so this first question is a general one.
I've developed a sudoku solving app and since then I wonder whether it would be feasible to design a ML-based algorithm/software which would mimic the thinking process of a human for solving a sudoku puzzle.
Maybe there exists already such a software?
I don’t mean any existing brute-force algorithm a human cannot apply!
I see many challenges in the design of such an algorithm, e.g. determine that the solution is not unique, uncover the highly sophisticated solving techniques that only talented players know and can apply to hard puzzles.
My thoughts even go so far as to imagine that such an algorithm would "discover" a solving technique that we humans have not yet discovered.

Update
I posted on kaggle a notebook titled Artificial Sudoku Player. I
intended to launch a code competition on the subject but I face a
problem of specifying a proper metrics. All metrics proposed on kaggle
are statistical metrics for ML models, but the goal of this
competition is to design a learnable RL model of the deterministic
sudoku solving process. Any constructive suggestion is welcome.
https://www.kaggle.com/code/sudokoach/artificial-sudoku-player.
Check out for a description of the competition: https://github.com/Sudokoach/Artificial-Sudoku-Player/blob/main/Artificial%20Sudoku%20Player.pdf

Related: https://ai.stackexchange.com/q/33994/32722

maybe imitation learning, or inverse reinforcement learning

@maxy Thank you for this reading. Since I posted the question I put my thoughts on paper that I'd like sharing. My conclusion is that's an RL application. However the implementation is far beyond my competencies because I know nothing about ML/RL and above all it's a team work! That's why I'm looking for a community which would be interested in such a development. May be you know one?

Your problem here is not the application of ML techniques to solving Sudoku. Reinforcement learning might be a reasonable search method, and you could augment it with results from other search methods, to enable it to learn in reasonable time.
Your problem is trying to train AI to behave "like a human" whilst having little to no definition for what that is. Supervised learning will require a large dataset of how humans solve the puzzles to copy. Reinforcement learning will require you to provide a reward signal that gives better rewards for when it behaves like a human.
What professional puzzle setters do is write variations of traditional AI search methods that they tune to score difficulties of different search steps, depending on how they define "easy" or "hard", and take some aggregate score to rate the puzzle as a whole. As an example, you could take the search depth of testing different initial guesses to resolve a single number and look at the consequences. Higher search depth required, and larger number of initial guesses available make finding a number harder.
Some Sudoku setters keep their "human-like" search methods proprietary because publishing puzzles is a business. You may be able to find some examples online though.

Its feasible to do so, not sure about `new techniques'. one can use RL by desigining your reward function (state space is just the current state of sudoku puzzle, action would be putting a particular number at some square), and solve using available methods. One can combine this with ml (perhaps approximating the reward function using past data).
however this seems easier to solve using methods from more conventional AI at first glance - i am not fully aware of the nuances of 'hard puzzles'.
Either way, you shoudl read more about standard AI, ml and then rl if interested.


text:
Segmentation of x-ray images to detect Covid-19
I’m currently working on covid detection project using x-rays. I applied K -means clustering algorithm (https://www.kaggle.com/code/naim99/image-classification-clustering-step-by-step?scriptVersionId=48171200) used in this link, on x-ray image (The following is the image result).

However, while training a machine learning classifier (RandomForestClassifier) on the segmented images using k-means, the classifier performs a bit worse (performance result in below image on segmented images with accuracy= 0.889… on the left, performance on unsegmented images with accuracy = 0.9..image on the right.

Could an image segmentation technique make the prediction worse? Or does it mean that the segmentation technique of k-means isn't efficient on image x-rays and i should try another way?

"Could an image segmentation technique make the prediction worse?"
Yes, it is entirely possible that a classifier trained on the segmented image performs worse than a classifier trained on unsegmented images.  One possible reason is that there is information elsewhere in the image that was revealing of COVID status that is removed in the segmented image.  For example, it has been suggested that COVID has impact on more than just the lungs.  Second, the two models that you are training are different, and they can absolutely have different performances.  Finally, it is worth noting that the differences that you are seeing are small and may be within a margin of error.  In other words, by sampling a different batch of images, you may see a reversal in the ranking of model performances.


text:
OpeanAI Gym. Train problem: invalid values
I have a problem with my reinforcement learning model.
I am trying to simulate an electric battery storage. To keep it as simple as possible, the efficiency of charge, storage and discharge are 100%. No depletion either.
The battery charges when the electricity prices are low and discharge ONLY to the user at fixed hours during the day, every day.
Therefore, the only cost for the user is power of charge * electricity price at the hour.
1. Define a dataframe where to store electricity prices for 500 days
df=pd.DataFrame(np.random.randint(0,500,size=(24, 100)))

2. Define the main parameters
Lookback_window_size=7
Current_day=Lookback_window_size
P_charge=2 #MW
P_discharge=3 #MW

3. Define the class Battery(Env)
class Battery(Env):

    metadata = {'render.modes': ['human']}

    def __init__(self, df):

        #Import the dataframe

        self.df = df

        # The action space is an array of shape (24,). Since we are simulating day-ahead market, the action space returns
        # the overall daily charge / no charge scenario
        # action = 1 means that we charge our battery, action = 0 means that we don't charge

        self.action_space= spaces.MultiBinary(24)

        # The observation space is a matrix with 24 rows and N columns. The first N-1 columns represent the electricity
        # prices for the current day + all the days before included in the lookback window size. The last column is all
        # zeros except the first two rows: they store SOC (state of charge) at the end of the day and overall cost
        # (how much we paid for electricity).
        # Example: if lookback window is 7, then we have a matrix of 24 rows and 9 columns. The first 8 columns are just
        # used to store the electricity prices, the 9th column store SOC and costs.

        self.observation_space = spaces.Box(
            low=0, high=np.inf, shape=(24, Lookback_window_size + 2), dtype=np.float16)

    def _next_observation(self):

        # Add the prices of the last days to the monitor matrix

        prices=self.df[[i for i in range(self.Current_day - Lookback_window_size,self.Current_day + 1)]][0:].to_numpy()

        extra_values=np.zeros(24)

        # Add extra values to monitor such as SOC, cost

        extra_values[[0,1]] = self.SOC, self.Cost

        obs=np.c_[prices,extra_values]

        return obs

    def _take_action(self, action):

        # Being the action space an array, the for loop will check the action at every hour (action[i]) and update the
        # cost and the state of charge

        i=0

        for x in action:

            #When action = 1 then we charge our battery, if action = 0 then we don't charge

            if x == 1:

                # The cost increase based on the price of the electricity at that hour

                self.Cost+=self.df[self.Current_day][i]*P_charge

                # If we charge, then the state of charge (SOC) increases as well

                self.SOC+=P_charge

            # Everyday we discharge the battery always at the same hours

            if (i in range(8,14)):

                self.SOC-=P_discharge

            # if the battery is depleted, then we directly buy electricity from the grid

                if self.SOC<0:

                    self.Cost+=self.df[self.Current_day][i+1]*(-self.SOC)

                    self.SOC=0

            i+=1

    def step(self, action):

        self._take_action(action)

        self.Current_day += 1

        # Maximizing the reward means minimize the costs

        reward =  - self.Cost

        # Stop at the end of the dataframe

        done = self.Current_day >= len(self.df.columns)-1

        obs = self._next_observation()

        return obs, reward, done, {}

    def render(self, mode='human', close=False):

        print(f'Day: {self.Current_day}')

        print(f'SOC: {self.SOC}')

        print(f'Cost: {self.Cost}')

        print(f'Actions: {action}')

    def reset(self):

        self.Current_day = Lookback_window_size

        # Give an initial SOC value

        self.SOC = 50

        # Cost at day 0 is null

        self.Cost = 0

        return self._next_observation()

4. Train
env = Battery(df)
model = PPO("MlpPolicy", env, verbose=1)

Once I run these lines, I get the following error:
5. VALUE ERROR
ValueError: Expected parameter logits (Tensor of shape (64, 24)) of distribution Bernoulli(logits: torch.Size([64, 24])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<AddmmBackward0>)

In more details:
| time/              |      |
|    fps             | 784  |
|    iterations      | 1    |
|    time_elapsed    | 2    |
|    total_timesteps | 2048 |
-----------------------------
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [31], in <cell line: 1>()
----> 1 model.learn(total_timesteps=100000)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\ppo\ppo.py:317, in PPO.learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps, progress_bar)
    303 def learn(
    304     self: PPOSelf,
    305     total_timesteps: int,
   (...)
    314     progress_bar: bool = False,
    315 ) -> PPOSelf:
--> 317     return super().learn(
    318         total_timesteps=total_timesteps,
    319         callback=callback,
    320         log_interval=log_interval,
    321         eval_env=eval_env,
    322         eval_freq=eval_freq,
    323         n_eval_episodes=n_eval_episodes,
    324         tb_log_name=tb_log_name,
    325         eval_log_path=eval_log_path,
    326         reset_num_timesteps=reset_num_timesteps,
    327         progress_bar=progress_bar,
    328     )

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\on_policy_algorithm.py:283, in OnPolicyAlgorithm.learn(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps, progress_bar)
    280         self.logger.record("time/total_timesteps", self.num_timesteps, exclude="tensorboard")
    281         self.logger.dump(step=self.num_timesteps)
--> 283     self.train()
    285 callback.on_training_end()
    287 return self

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\ppo\ppo.py:214, in PPO.train(self)
    211 if self.use_sde:
    212     self.policy.reset_noise(self.batch_size)
--> 214 values, log_prob, entropy = self.policy.evaluate_actions(rollout_data.observations, actions)
    215 values = values.flatten()
    216 # Normalize advantage

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\policies.py:644, in ActorCriticPolicy.evaluate_actions(self, obs, actions)
    642 features = self.extract_features(obs)
    643 latent_pi, latent_vf = self.mlp_extractor(features)
--> 644 distribution = self._get_action_dist_from_latent(latent_pi)
    645 log_prob = distribution.log_prob(actions)
    646 values = self.value_net(latent_vf)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\policies.py:615, in ActorCriticPolicy._get_action_dist_from_latent(self, latent_pi)
    612     return self.action_dist.proba_distribution(action_logits=mean_actions)
    613 elif isinstance(self.action_dist, BernoulliDistribution):
    614     # Here mean_actions are the logits (before rounding to get the binary actions)
--> 615     return self.action_dist.proba_distribution(action_logits=mean_actions)
    616 elif isinstance(self.action_dist, StateDependentNoiseDistribution):
    617     return self.action_dist.proba_distribution(mean_actions, self.log_std, latent_pi)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\stable_baselines3\common\distributions.py:380, in BernoulliDistribution.proba_distribution(self, action_logits)
    379 def proba_distribution(self, action_logits: th.Tensor) -> "BernoulliDistribution":
--> 380     self.distribution = Bernoulli(logits=action_logits)
    381     return self

File ~\anaconda3\envs\tf-gpu\lib\site-packages\torch\distributions\bernoulli.py:51, in Bernoulli.__init__(self, probs, logits, validate_args)
     49 else:
     50     batch_shape = self._param.size()
---> 51 super(Bernoulli, self).__init__(batch_shape, validate_args=validate_args)

File ~\anaconda3\envs\tf-gpu\lib\site-packages\torch\distributions\distribution.py:56, in Distribution.__init__(self, batch_shape, event_shape, validate_args)
     54         valid = constraint.check(value)
     55         if not valid.all():
---> 56             raise ValueError(

     57                 f"Expected parameter {param} "
     58                 f"({type(value).__name__} of shape {tuple(value.shape)}) "
     59                 f"of distribution {repr(self)} "
     60                 f"to satisfy the constraint {repr(constraint)}, "
     61                 f"but found invalid values:\n{value}"
     62             )
     63 super(Distribution, self).__init__()

ValueError: Expected parameter logits (Tensor of shape (64, 24)) of distribution Bernoulli(logits: torch.Size([64, 24])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<AddmmBackward0>)

What I am doing wrong? Am I defining some variables in a wrong way?

Your self.Cost variable aggregates rewards over time instead of giving the cost for the current time step as a reward. This causes the gradients to grow larger and everything diverges.

Isn't a reward connected to the daily cost more greedy than one that consider the cumulative cost? For example: if a day has really low price of electricity, I would like to charge as much as possible so that I can "save" some energy to shield myself from higher prices in the future. Having a reward coupled only to the daily cost seems to go against this logic.

The problem is that it is integrated in the algorithm itself to consider and optimize for the cumulative cost. If you look in the RolloutBuffer class which is used by the PPO implementation you can see that it computes the return which is the discounted sum of rewards. This is then used as an optimization target.


text:
Would the optimal policy remain same, if I replace R with V*?
In the context of RL, say I'm performing Value Iteration on a reward function R1. And the converged optimal policy is P1 and values are V1.
Then, let's say I set rewards to be R2=V1 and perform value iteration from scratch on these rewards and get a value V2 and policy P2.
Is P2 == P1 ?
Rewards can be just dependent on just the states as well, for example, R(s), it doesn't necessarily have to depend on the action being taken in that state.
For MDP = $<S, T, A, R>$, S are set of states, T is transition function as $S \times A \times S \rightarrow [0,1]$, A is set of actions, R is the reward function $S \rightarrow \mathbb{R}$.
Formally, $MDP1 = <S, T(s,a,s'), A, R(s) = R1(s)>$ yields optimal value $v*(s) = v_1$ and policy $\pi_1$,
and $MDP2 = <S, T(s,a,s'), A, R(s) = v_1(s)>$ yields optimal policy $\pi_2$.
Would $\pi_2(s) = \pi_1(s)$? when $\gamma$ = 0, 0 < $\gamma$ < 1, $\gamma$ = 1.

The reward function is usually dependent on the state $s$ and action $a$, and possibly the next state $s'$, that is either $R(s,a)$ or $R(s',a,s)$. The value function of any policy $\pi$ is a function of the state, that is $V^{\pi}(s)$. How do you set the second reward function to equal first value function in your case?

You could say that reward function $R_1$ is dependent only on either $s$ or $s'$, and that $R_2 = V^*(s)$ (or same function of $s'$, matching how you have defined $R_1$). This would answer mikkola's question, but it does show that any result cannot be general to all MDPs.

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

Hi @mikkola I have updated my question to be more formal. Just to reiterate on your point, R(s) is perfectly valid as you can marginalize the effect of actions from R(s,a) or marginalize a, s' in R(s,a,s'). Therefore, R and V are both defined on $S \rightarrow \mathbb{R}$.

@mikkkola I may have misunderstood the conversion from R(s,a) to R(s), but regardless the question I want to ask is with respect to R(s).

Setting $R_2(s)=v_1^(s)$ amounts to saying per-state reward in the second MDP will equal the expected long-term optimal reward starting from said state in the first MDP. My gut feeling says probably $\pi_2^ \equiv \pi_1^*$, but I have not yet been able to prove this, nor to come up with a counterexample.


text:
Why does KNN Model return 99% accuracy on dataset with default parameters?

I am building a model that predicts if a user will like a stock or not based on different features, such as Market Cap, Current Ratio, Sector, Trailing PE, etc. I am going to implement this model in a website, where the model is able to adapt over time to user preferences. I made an example dataset that generally represents how the data that gets passed to the model will look. I have build a KNN model based on that sample dataset, and I get 99% accuracy on the training split and test split. I also get 98%-99% on my classification report even though I have class imbalance, about 500 of one class, and 10500 of another class. My macro average and weighted average are both 99%. I was planning on optimizing this model by hyper tuning, but does anyone know why I got 99% accuracy immediately? This is how a sample of my data looks like above.
Here is some of my code:
X = data.drop(["Stocks", "target"], axis = 1)
y = data["target"]
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

categorical_features = ["sector", "quoteType"]
one_hot = OneHotEncoder()
transformer = ColumnTransformer([("one_hot",
                                  one_hot,
                                  categorical_features)],
                               remainder = "passthrough")

X = transformer.fit_transform(X)

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X = scaler.fit_transform(X)
X = pd.DataFrame(X)
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

np.random.seed(42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
knn = KNeighborsClassifier(metric = "minkowski")
knn.fit(X_train, y_train)
knn.score(X_test, y_test)

y_preds = knn.predict(X_test)

from sklearn.metrics import classification_report
print(classification_report(y_test, y_preds))

This produces:
precision    recall  f1-score   support

           0       0.98      1.00      0.99      1227
           1       1.00      0.98      0.99      1012

    accuracy                           0.99      2239
   macro avg       0.99      0.99      0.99      2239
weighted avg       0.99      0.99      0.99      2239

Comments are not for extended discussion; this conversation has been moved to chat.

First assess whether the accuracy is unreasonably high.  You as the domain expert are the best arbiter of that.  Keep in mind that a high accuracy is not always impossible.  For example, 99% accuracy can be seen on MNIST digits dataset.  It is also possible that you have data leakage, so assess whether this may be at play. The second thing you can do is perform an n-fold cross-validation.  This will provide you with a better estimate (and error bounds) for your model's performance.  If your accuracy remains high on all folds, then you can exclude a spurious split on your holdout validation.
Another issue is that you claim that you have an imbalanced dataset.  However, your print out shows roughly equal support for 0 and 1 classes.
Finally, you are performing preprocessing of your features on the total dataset.  Instead, you should apply fit_transform on your training data, and the just apply the transform on your test data.

I have an imbalanced dataset but the data I show above is just a sample of what my data looks like. Also, why should I should apply fit_transform on my training data, and just apply the transform on my test data?

Ok, makes sense.  In that case, I would say that it is customary to perform evaluation on the total test set, not just a subset that you selected, regardless of whether your dataset is balanced or not.  Regarding the second issue of fit / fit_transform, the suggestion has to do with data leakage.  see article

I ran a classification report and it returned a score of 0.99 as shown above, so does that say that the model was able to generalize well?


text:
Can I sample finite or infinite images with AutoRegressive Models?
I'm learning about AutoRegressive Models used on images, and I've studied the training phase, where you model each pixel on the basis of the previous ones using a certain model architecture.
My question is about generating new images (sampling).
I've seen that the sampling is usually done setting manually the value of the first pixel and calculating the following pixels using the model, i.e. for every pixel you want to generate, you take the previous n pixels and give them to the model which outputs the most likely value, where here "likely" is to be intended as the value which is output given the parameters fitted on the training dataset.
But since the model parameters are fixed, and since you set only the first pixel, does this mean that all pixels except the first one are deterministic and hence you can only generate 256 images (256 is the number of possible values of the first pixel in grayscale)?
Thank you!

if you just take the maximum likelihood color, then probably yes, but you should not...
In other words, assume an distribution, for example a gaussian centered at the predicted color with some variance, and sample from it... this way, you will likely sample a color that is near the one that the model has predicted, but with a bit of noise, which should cause little to no difference visually but will drive the model in different areas of the distribution that it has learned
(not to mention that potentially, you could just sample a floating point number as first pixel, like $128.3$, and the model won't complain)

Thanks. What do you mean with "if you take the ML color"? All the pixel following the first one are conditioned on the previous ones, so you choose only the first. Also, the model parameters shouldn't be set so that a slight change in value of a pixel causes the following pixels to change drastically, so even with floating point numbers the range of images you can obtain shouldn't be so wide, it should be approximately like the range of images used for training. Am I missing something?

@SuperFluo say you have to pick the first pixel, and you setup you net with a single linear unit, which says that it should be the color "124.8", now, you can interpret this as the net predicting $\mu$ of a certain Normal Distribution, thus $o \sim N(\mu^\theta;\sigma | x)$... in other words, you can sample a color from $N(124.8, \sigma)$ where sigma can be whatever... if it's 0, then you get a Dirac distribution which implies you just pick the ML color, which is the predicted one... in poor words, you introduce some noise in the process so that you have infinite samples


text:
Model-based RL algorithms for continuous state space and finite action space
At the beginning, if I have a complete model $p(s' \mid s, a)$ (an assumed true model that describes the environment well enough) and the reward function $r(s,a,s')$. How can I exploit the model and learn a good policy in this situation? Assume that the state space is continuous, and the action space is finite.
Traditional dynamic programming/planning methods like policy iteration or value iteration cannot be directly applied since there are infinitely many states. If I try to get samples from the model and apply an algorithm like DQN or any non-linear function approximation, it looks like that I use the model-free approach and do not get the full advantages of the model.
May I ask if there are any reinforcement learning/planning methods that use the explicit model given at the beginning to solve the MDP? (in case of continuous state space and finite action space)

Are you asking which model-based RL algorithms exist (assuming that you don't consider value/policy iteration an RL algorithm but a planning algorithm)?

@nbro Yes, kind of. I understand that policy/value iteration are planning algorithms. But in Sutton's book it's only for finite state space. Assuming that the state space are continuous, and the transition function $p$ and reward function $r$ are explicitly given to the agent at the beginning, is there any method/algorithm to use the model to find a good or optimal policy?

Please, edit your post to include this info and change the title to be more specific and have your question

@nbro thank you for your editing, I've also added some more information to the post.

In optimal control field to minimize certain well-defined costs especially in process industries, continuous state space model-based planning methods such as model predictive control (MPC) is a common decision time planning/control method which can handle both linear and nonlinear models, and unlike training data intensive model-free RL methods it's much more data efficient, faster, online, and good (though may not optimal):

MPC is based on iterative, finite-horizon optimization of a plant model. At time
$t$ the current plant state is sampled and a cost minimizing control strategy is computed (via a numerical minimization algorithm) for a relatively short time horizon in the future:$[t,t+T]$. Specifically, an online or on-the-fly calculation is used to explore state trajectories that emanate from the current state and find (via the solution of Euler–Lagrange equations) a cost-minimizing control strategy until time $t+T$. Although this approach is not optimal, in practice it has given very good results. Much academic research has been done to find fast methods of solution of Euler–Lagrange type equations, to understand the global stability properties of MPC's local optimization, and in general to improve the MPC method.

If your model is linear (most mechanical systems) and cost is quadratic, LQR is also a common optimal feedback control method.

The main differences between MPC and LQR are that LQR optimizes across the entire time window (horizon) whereas MPC optimizes in a receding time window,[4] and that with MPC a new solution is computed often whereas LQR uses the same single (optimal) solution for the whole time horizon.

Finally similar to Dyna-Q learning architecture with background learning & planning introduced in Sutton and Barto's Reinforcement Learning, An Introduction, especially if your initial known sample model is not accurate or keeps evolving, one can combine model-based with model-free RL methods, you just replace tabular Q-planning there with your linear or nonlinear function approximations planning version such as described in Jong and Stone's paper Model-Based Exploration in Continuous State Spaces:

This paper develops a method
for approximating continuous models by fitting data to a finite sample of states, leading to finite representations compatible with existing
model-based exploration mechanisms. Experiments with the resulting
family of fitted-model reinforcement learning algorithms reveals the critical importance of how the continuous model is generalized from finite
data. This paper demonstrates instantiations of fitted-model algorithms
that lead to faster learning on benchmark problems than contemporary
model-free RL algorithms that only apply generalization in estimating
action values. Finally, the paper concludes that in continuous problems,
the exploration-exploitation tradeoff is better construed as a balance between exploration and generalization.