text:
What is the difference between artificial intelligence and machine learning?
These two terms seem to be related, especially in their application in computer science and software engineering.

Is one a subset of another?
Is one a tool used to build a system for the other?
What are their differences and why are they significant?

You should accept an answer, so that the best answer is on top of the others. In my view, the best answer is also the most upvoted one, i.e. this one, which I think you should accept, as it's objectively correct and simple. However, I've changed this answer to make it even more valuable too, so you may consider it too. Take a look at https://ai.stackexchange.com/help/someone-answers for more info about how to act when you get answers to your own question.

You can also read the related questions What is machine learning? and What is artificial intelligence?.

Machine learning has been defined by many people in multiple (often similar) ways [1, 2]. One definition says that machine learning (ML) is the field of study that gives computers the ability to learn without being explicitly programmed.
Given the above definition, we might say that machine learning is geared towards problems for which we have (lots of) data (experience), from which a program can learn and can get better at a task.
Artificial intelligence has many more aspects, where machines may not get better at tasks by learning from data, but may exhibit intelligence through rules (e.g. expert systems like Mycin), logic or algorithms, e.g. path-finding).
The book Artificial Intelligence: A Modern Approach shows more research fields of AI, like Constraint Satisfaction Problems, Probabilistic Reasoning or Philosophical Foundations.

Machine learning is a subset of artificial intelligence. Roughly speaking, it corresponds to its learning side. There is no "official" definitions, boundaries are a bit fuzzy.

Artificial intelligence
According to the book Artificial Intelligence: A Modern Approach (section 1.1), artificial intelligence (AI) has been defined in multiple ways, which can be organized into 4 categories.

Thinking Humanly
Thinking Rationally
Acting Humanly
Acting Rationally

Figure 1.1 (of the same book) contains 8 definitions (by renowned people like Bellman, Winston or Kurzweil).

Each box contains 2 similar definitions (i.e. both fall into the same category). These definitions vary along 2 dimensions. The definitions in the top row are concerned with thought-processes and reasoning, while the ones in the bottom are concerned with behaviour. The definitions on the left are associated with human intelligence, while the ones on the right with an idealized version of intelligence, which the authors of the AIMA book call rationality. So, for example, the definitions in the top-left corner are based on thinking humanly, while the definitions on the bottom-right corner are based on acting rationally.
There is also a definition of AI by John McCarthy, who is one of the official founders of the AI field in 1956.

It is the science and engineering of making intelligent machines, especially intelligent computer programs. It is related to the similar task of using computers to understand human intelligence, but AI does not have to confine itself to methods that are biologically observable.

Machine learning
There have been also multiple (similar) definitions of machine learning (ML). For example, Tom Mitchell, in section 1.1. of his book Machine Learning, defines machine learning as follows.

A computer program is said to learn from experience $E$ with respect
to some class of tasks $T$ and performance measure $P$, if its performance at tasks in $T$, as measured by $P$, improves with experience $E$.

What is the difference between AI and ML?
ML is a subfield of AI, which is data-oriented (or experience-driven). AI is not just ML, but it's also composed of Natural Language Processing, and other subfields.

The machine learning is a sub-set of artificial intelligence which is only a small part of its potential. It's a specific way to implement AI largely focused on statistical/probabilistic techniques and evolutionary techniques.Q
Artificial intelligence
Artificial intelligence is 'the theory and development of computer systems able to perform tasks normally requiring human intelligence' (such as visual perception, speech recognition, decision-making, and translation between languages).
We can think of AI as the concept of non-human decision makingQ which aims to simulate cognitive human-like functions such as problem-solving, decision making or language communication.
Machine learning
Machine learning (ML) is basically a learning through doing by the implementation of build models which can predict and identify patterns from data.
According to Prof. Stephanie R. Taylor of Computer Science and her lecture paper, and also Wikipedia page, 'machine learning is a branch of artificial intelligence and it's about construction and study of systems that can learn from data' (like based on the existing email messages to learn how to distinguish between spam and non-spam).
According to Oxford Dictionaries, the machine learning is 'the capacity of a computer to learn from experience' (e.g. modify its processing on the basis of newly acquired information).
We can think ML as computerized pattern detection in the existing data to predict patterns in future data.Q

In other words, machine learning involves the development of self-learning algorithms and artificial intelligence involves developing systems or software to mimic human to respond and behave in a circumstance.Quora

Machine learning is a subfield of artificial intelligence, as the following diagram (taken from this blog post) illustrates.

Many terms have 'mostly' the same meanings, and so the differences are just in emphasis, perspective, or historical descent. People disagree as to which label refers to the superset or the subset; there are people who will call AI a branch of ML and people who will call ML a branch of AI.
I typically hear Machine Learning used as a form of 'applied statistics' where we specify a learning problem in enough detail that we can just feed training data into it and get a useful model out the other side.
I typically hear Artificial Intelligence as a catch-all term to refer to any sort of intelligence embedded in the environment or in code. This is a very expansive definition, and others use narrower ones (such as focusing on artificial general intelligence, which is not domain-specific). (Taken to an extreme, my version includes thermostats.)

In simple words, Artificial intelligence is a field of science that is trying to mimic humans or other animals behavior.
Machine Learning is one of the key tool/technology behind Artificial intelligence.

Artificial Intelligence (AI) and Machine Learning (ML) are two very hot buzzwords right now, and often seem to be used interchangeably. They are not quite the same thing, but the perception that they are can sometimes lead to some confusion. So I thought it would be worth writing a piece to explain the difference.
Machine learning is a core sub-area of artificial intelligence; it enables computers to get into a mode of self-learning without being explicitly programmed. When exposed to new data, these computer programs are enabled to learn, grow, change, and develop by themselves.

First of all, I encountered the term MachineLearning much more in my Business Intelligence classes than in my AI classes.
My AI Professor Rolf Pfeifer would have put it that way: (after having a long speech about what intelligence is, how it can be defined, different types of intelligence, etc.). ML is more static and "dumb", unaware of its physical environment and not made to interact with it, or only on an abstract basis. AI has a certain awareness of its environment and interacts with it autonomously, making thereby autonomous decisions with feedback loops.
From that point of view, Ugnes Answer would be probably the closest.
Besides that, of course, ML is a subset of AI.
Machine Learning is not real intelligence (imho), it's mostly human intelligence reflected in logical algorithms, and as my Business Intelligence Prof would put it: about data and its analysis. Machine Learning has a lot of supervised algorithms which actually do need humans to support the learning process by telling what's right and what's wrong, so they're not independent. And once they're applied, algorithms are mostly static until humans readjust them.
In ML you mostly have black boxes designs and the main aspect is data. Data comes in, Data gets analyzed ("Intelligently"), Data goes out, and Learning most times applies to a pre-implementation/Learning fase. In most cases ML doesn't care about the environment a machine is in, it's about data.
AI instead is about mimicking human or animal intelligence. Following my Prof's approach, AI is not necessarily about self-consciousness but about interaction with the environment, so to build AI you need to give the machine sensors to perceive the environment, a sort of intelligence able to keep on learning, and elements to interact with the environment (arms, etc.). The interaction should happen in an autonomous way and ideally, as in humans, learning should be an autonomous, ongoing process.
So a drone that scans fields in a logical scheme for colour patterns to find weeds within crops would be more ML. Especially if the data is later analyzed and verified by humans or the algorithm used is that a static algorithm with built-in "intelligence" but not capable of rearranging or adapting to its environment.
A drone that flies autonomously, charges itself up when the battery's down, scans for weeds, learns to detect unknown ones and rips them out by itself and brings them back for verification, would be AI...


text:
How can I handle overfitting in reinforcement learning problems?
So this is my current result (loss and score per episode) of my RL model in a simple two players game:

I use DQN with CNN as a policy and target networks. I train my model using Adam optimizer and calculate the loss using Smooth L1 Loss.
In a normal "Supervised Learning" situation, I can deduce that my model is overfitting. And I can imagine some methods to tackle this problem (e.g. Dropout layer, Regularization, Smaller Learning Rate, Early Stopping).

But would that solution will also work in RL problem?
Or are there any better solutions to handle overfitting in RL?

Overfitting refers to a model being stuck in a local minimum while trying to minimise a loss function. In Reinforcement Learning the aim is to learn an optimal policy by maximising or minimising a non-stationary objective-function which depends on the action policy, so overfitting is not exactly like in the supervised scenario, but you can definitely talk about sub-optimal policies.
If we think of a specific task like avoiding stationary objects, a simple sub-optimal policy would be to just stay still without moving at all, or moving in circles if the reward function was designed to penalise lack of movements.
The way to avoid an agent to learn sub-optimal policies is to find a good compromise between exploitation, i.e. the constant selection of the next action to take based on the maximum expected reward possible, and exploration, i.e. a random selection of the next action to take regardless of the rewards. Here's a link to an introduction to the topic:
Exploration and Exploitation in Reinforcement Learning
It is worth mentioning that sometimes an agent can actually outsmart humans though, some examples are reported in this paper The Surprising Creativity of Digital Evolution. I particularly like the story of the insect agent trained to learn to walk while minimising the contact with the floor surface. The agent surprisingly managed to learn to walk without touching the ground at all. When the authors checked what was going on they discovered that the insect leaned to flip itself and then walk using its fake 'elbows' (fig7 in the linked paper). I add this story just to point out that most of the time the design of the reward function is itself even more important than exploration and exploitation tuning.

I wouldn't describe overfitting as the situation where a model gets stuck in a local minimum, but as the "gap between training and test performances". This definition should also make it clearer why it may not make sense to talk about overfitting in RL. If we use your description of overfitting, one could argue that "sub-optimal policies" correspond to local minima, and they wouldn't be wrong. I think it would also be useful you if commented the plots that the asker provided. What do thee plots represent and can we understand whether the RL agent is "overfitting" from them?

ah... I see.. so can you help me to understand why my Agent's Score decreasing after 400 episodes? it keeps decreasing until almost 0 after 1000 episodes. Because in this game the reward is the score (0 to 12), so I think the problem is not a sub-optimal policy, right?

@malioboro Could you explain a bit more in details the game and what reward function you're using? Honestly the only thing I can tell from your graphs is that after 200 iterations your neural net starts overfitting, affecting the performances of the agent in selecting the next action. You could just stop training the deep net when the loss start increase, and keep using the same model to train the policy of your agent. Also, consider that sometimes it can take more than 1k, even 10k iterations to reach an optimal policy, the drop you see might be just a random fluctuation.

sorry about the last sentence, I read the comment quickly, if after 1k iterations the reward approach zero then I think the problem is most definitely related to the performance of the neural net.

@EdoardoGuerriero thank you so much for your input, I solve this problem by tune the RL model to use more steps before update the target network, and also tune the NN parameters as you said

and that's true that RL resuls is very fluctuative, some times the score is decreasing but after few episodes it starts increasing again

In Reinforcement Learning you do want to minimise/maximise a loss function!

@RémyHosseinkhanBoucher not necessarily. Classic q learning for example require only a reward function, which is much different to a loss function conceptually.

The accepted answer does not provide a good definition of over-fitting, which actually exists and is a defined concept in reinforcement learning too. For example, the paper Quantifying Generalization in Reinforcement Learning completely focuses on this issue. Let me give you more details.
Over-fitting in supervised learning
In supervised learning (SL), over-fitting is defined as the difference (or gap) in the performance of the ML model (such as a neural network) on the training and test datasets. If the model performs significantly better on the training dataset than on the test dataset, then the ML model has over-fitted the training data. Consequently, it has not generalized (well enough) to other data other than the training data (i.e. the test data). The relationship between over-fitting and generalization should now be clearer.
Over-fitting in reinforcement learning
In reinforcement learning (RL) (you can find a brief recap of what RL is here), you want to find an optimal policy or value function (from which the policy can be derived), which can be represented by a neural network (or another model). A policy $\pi$ is optimal in environment $E$ if it leads to the highest cumulative reward in the long run in that environment $E$, which is often mathematically modelled as a (partially or fully observable) Markov decision process.
In some cases, you are also interested in knowing whether your policy $\pi$ can also be used in a different environment than the environment it has been trained in, i.e. you're interested in knowing if the knowledge acquired in that training environment $E$ can be transferred to a different (but typically related) environment (or task) $E'$. For example, you may only be able to train your policy in a simulated environment (because of resource/safety constraints), then you want to transfer this learned policy to the real world. In those cases, you can define the concept of over-fitting in a similar way to the way we define over-fitting in SL. The only difference may be that you may say that the learned policy has over-fitted the training environment (rather than saying that the ML model has over-fitted the training dataset), but, given that the environment provides the data, then you could even say in RL that your policy has over-fitted the training data.
Catastrophic forgetting
There is also the issue of catastrophic forgetting (CF) in RL, i.e., while learning, your RL agent may forget what it's previously learned, and this can even happen in the same environment. Why am I talking about CF? Because what it's happening to you is probably CF, i.e., while learning, the agent performs well for a while, then its performance drops (although I have read a paper that strangely defines CF differently in RL). You could also say that over-fitting is happening in your case, but, if you are continuously training and the performance changes, then CF is probably what you need to investigate. So, you should reserve the word over-fitting in RL when you're interested in transfer learning (i.e. the training and test environments do not coincide).

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the most suitable measure of the distance between two VAE's latent spaces?
The problem I'm trying to solve is as follows.
I have two separate domains, where inputs do not have the same dimensions. However, I want to create a common feature space between both domains using paired inputs (similar inputs from both domains).
My solution is to encode pairs of inputs into a shared latent space using two VAE encoders (one for each domain). To ensure that the latent space is shared, I want to define a similarity metric between the output of both probabilistic encoders.
Let's define the first encoder as $q_\phi$ and the second as $p_\theta$. As for now, I have two main candidates for this role:

KL-divergence : $\text{KL}(p || q)$ (or $\text{KL}(q||p)$), but, since it is not symmetrical, I don't really know which direction is the best.

JS-divergence: symmetrical and normalized, which is nice for a distance metric, but, since it is not as common as KL, I'm not sure.

Other candidates include adversarial loss (a discriminator is tasked to guess from which VAE the latent code is, the goal of both VAE being to maximally confuse it) or mutual information (seen more and more in recent works, but I still don't fully understand it).
My question is: according to you, which loss could work best for my use case? KL or JS? Or other candidates I didn't think of?
-- More context --
My ultimate goal is to use transfer learning between morphologically distinct robots e.g a quadrupedal robot and a bipedal robot. The first step in my current approach is to record trajectories on both robots executing the same task (walking for example). From said trajectories I create paires of similar states (to simplify the problem, I suppose that both robot achieve the same task at the same speed so temporaly aligned states for both robots are paired). Then my goal is to encode these paired states (that doesn't have the same dimension due to the difference in number of joints) into two latents spaces (one for each VAE) such that similar pair of inputs are close in the latents spaces. If I was working with simple autoencoders I would simply minimize the distance in the latent space between paires of inputs such that similar states on both robots maps to the same point in the latent space. But I need the generative capabilities of VAE so instead I would like to make the distributions outputed by the VAEs as close as possible. Make sense ?

Let me try to understand if I understand your current approach. You want to train two VAEs, but you want to keep the latent distributions as close as possible to each other? But then which latent distribution will you use, and for what? In other words, it doesn't seem that your current approach would learn only one "shared feature representation", but you would actually learn two hidden distributions that are close, so, in a way, your two distributions would try to be similar, but they wouldn't be shared. Anyway, it's not clear to me what your ultimate goal is. Why do you want to do this?

Are you building this as a single model with bifurcated inputs and outputs (coming down to the same latent layer and originating from that same layer) or are you doing something else?

Thank you for taking the time to assess my issue. I updated my question for more details on my goal.

@DavidHoelzer I'm using separate networks

@nbro let me know if you need more details

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why doesn't this Monte Carlo Tree Search algorithm work properly?
PROBLEM
I'm writing a Monte-Carlo tree search algorithm to play chess in Python. I replaced the simulation stage with a custom evaluation function. My code looks perfect but for some reason acts strange. It recognizes instant wins easily enough but cannot recognize checkmate-in-2 moves and checkmate-in-3 moves positions. Any ideas?
WHAT I'VE TRIED
I tried giving it more time to search but it still cannot find the best move even when it leads to a guaranteed win in two moves. However, I noticed that results improve when I turn off the custom evaluation and use classic Monte Carlo Tree Search simulation. (To turn off custom evaluation, just don't pass any arguments into the Agent constructor.) But I really need it to work with custom evaluation because I am working on a machine learning technique for board evaluation.
I tried printing out the results of the searches to see which moves the algorithm thinks are good. It consistently ranks the best move in mate-in-2 and mate-in-3 situations among the worst. The rankings are based on the number of times the move was explored (which is how MCTS picks the best moves).
MY CODE
I've included the whole code because everything is relevant to the problem. To run this code, you may need to install python-chess (pip install python-chess).
I've struggled with this for more than a week and it's getting frustrating. Any ideas?
import math
import random
import time

import chess
import chess.engine

class Node:

    def __init__(self, state, parent, action):
        """Initializes a node structure for a Monte-Carlo search tree."""
        self.state = state
        self.parent = parent
        self.action = action

        self.unexplored_actions = list(self.state.legal_moves)
        random.shuffle(self.unexplored_actions)
        self.colour = self.state.turn
        self.children = []

        self.w = 0 # number of wins
        self.n = 0 # number of simulations

class Agent:

    def __init__(self, custom_evaluation=None):
        """Initializes a Monte-Carlo tree search agent."""

        if custom_evaluation:
            self._evaluate = custom_evaluation

    def mcts(self, state, time_limit=float('inf'), node_limit=float('inf')):
        """Runs Monte-Carlo tree search and returns an evaluation."""

        nodes_searched = 0
        start_time = time.time()

        # Initialize the root node.
        root = Node(state, None, None)

        while (time.time() - start_time) < time_limit and nodes_searched < node_limit:

            # Select a leaf node.
            leaf = self._select(root)

            # Add a new child node to the tree.
            if leaf.unexplored_actions:
                child = self._expand(leaf)
            else:
                child = leaf

            # Evaluate the node.
            result = self._evaluate(child)

            # Backpropagate the results.
            self._backpropagate(child, result)

            nodes_searched += 1

        result = max(root.children, key=lambda node: node.n)

        return result

    def _uct(self, node):
        """Returns the Upper Confidence Bound 1 of a node."""
        c = math.sqrt(2)

        # We want every WHITE node to choose the worst BLACK node and vice versa.
        # Scores for each node are relative to that colour.
        w = node.n - node.w

        n = node.n
        N = node.parent.n

        try:
            ucb = (w / n) + (c * math.sqrt(math.log(N) / n))
        except ZeroDivisionError:
            ucb = float('inf')

        return ucb

    def _select(self, node):
        """Returns a leaf node that either has unexplored actions or is a terminal node."""
        while (not node.unexplored_actions) and node.children:
            # Pick the child node with highest UCB.
            selection = max(node.children, key=self._uct)
            # Move to the next node.
            node = selection
        return node

    def _expand(self, node):
        """Adds one child node to the tree."""
        # Pick an unexplored action.
        action = node.unexplored_actions.pop()
        # Create a copy of the node state.
        state_copy = node.state.copy()
        # Carry out the action on the copy.
        state_copy.push(action)
        # Create a child node.
        child = Node(state_copy, node, action)
        # Add the child node to the list of children.
        node.children.append(child)
        # Return the child node.
        return child

    def _evaluate(self, node):
        """Returns an evaluation of a given node."""
        # If no custom evaluation function was passed into the object constructor,
        # use classic simulation.
        return self._simulate(node)

    def _simulate(self, node):
        """Randomly plays out to the end and returns a static evaluation of the terminal state."""
        board = node.state.copy()
        while not board.is_game_over():
            # Pick a random action.
            move = random.choice(list(board.legal_moves))
            # Perform the action.
            board.push(move)
        return self._calculate_static_evaluation(board)

    def _backpropagate(self, node, result):
        """Updates a node's values and subsequent parent values."""
        # Update the node's values.
        node.w += result.pov(node.colour).expectation()
        node.n += 1
        # Back up values to parent nodes.
        while node.parent is not None:
            node.parent.w += result.pov(node.parent.colour).expectation()
            node.parent.n += 1
            node = node.parent

    def _calculate_static_evaluation(self, board):
        """Returns a static evaluation of a *terminal* board state."""
        result = board.result(claim_draw=True)

        if result == '1-0':
            wdl = chess.engine.Wdl(wins=1000, draws=0, losses=0)
        elif result == '0-1':
            wdl = chess.engine.Wdl(wins=0, draws=0, losses=1000)
        else:
            wdl = chess.engine.Wdl(wins=0, draws=1000, losses=0)

        return chess.engine.PovWdl(wdl, chess.WHITE)

def custom_evaluation(node):
    """Returns a static evaluation of a board state."""

    board = node.state

    # Evaluate terminal states.
    if board.is_game_over(claim_draw=True):
        result = board.result(claim_draw=True)
        if result == '1-0':
            wdl = chess.engine.Wdl(wins=1000, draws=0, losses=0)
        elif result == '0-1':
            wdl = chess.engine.Wdl(wins=0, draws=0, losses=1000)
        else:
            wdl = chess.engine.Wdl(wins=0, draws=1000, losses=0)

        return chess.engine.PovWdl(wdl, chess.WHITE)

    # Evaluate material.
    material_balance = 0
    material_balance += len(board.pieces(chess.PAWN, chess.WHITE)) * +100
    material_balance += len(board.pieces(chess.PAWN, chess.BLACK)) * -100
    material_balance += len(board.pieces(chess.ROOK, chess.WHITE)) * +500
    material_balance += len(board.pieces(chess.ROOK, chess.BLACK)) * -500
    material_balance += len(board.pieces(chess.KNIGHT, chess.WHITE)) * +300
    material_balance += len(board.pieces(chess.KNIGHT, chess.BLACK)) * -300
    material_balance += len(board.pieces(chess.BISHOP, chess.WHITE)) * +300
    material_balance += len(board.pieces(chess.BISHOP, chess.BLACK)) * -300
    material_balance += len(board.pieces(chess.QUEEN, chess.WHITE)) * +900
    material_balance += len(board.pieces(chess.QUEEN, chess.BLACK)) * -900

    # TODO: Evaluate mobility.
    mobility = 0

    # Aggregate values.
    centipawn_evaluation = material_balance + mobility

    # Convert evaluation from centipawns to wdl.
    wdl = chess.engine.Cp(centipawn_evaluation).wdl(model='lichess')
    static_evaluation = chess.engine.PovWdl(wdl, chess.WHITE)

    return static_evaluation

m1 = chess.Board('8/8/7k/8/8/8/5R2/6R1 w - - 0 1') # f2h2
# WHITE can win in one move. Best move is f2-h2.

m2 = chess.Board('8/6k1/8/8/8/8/1K2R3/5R2 w - - 0 1')
# WHITE can win in two moves. Best move is e2-g2.

m3 = chess.Board('8/8/5k2/8/8/8/3R4/4R3 w - - 0 1')
# WHITE can win in three moves. Best move is d2-f2.

agent = Agent(custom_evaluation)

result = agent.mcts(m2, time_limit=30)
print(result)
````


text:
How does GPT like Decoder only conversational models distunguish the source of text?
In a conversational setting where two sources of text (user and the model) follow each other like below
User: some text bla bla
Model: another text bah bah
User: bla bla bla
Model: bah bah
and so on, how does the model differentiate the texts written by the user and the model?
I assume in an encoder-decoder setting (like T5 or BART), texts from two sources can be differentiated by giving user texts as encoder input and model's earlier responses as decoder input.
How about GPT-like Decoder only models?
Relating to that, what is the common method to train models for long conversational setting like above?

you can definitely encode it with some special token

A decoder-only conversational model, like GPT-3, generates text based on the context provided to it. It doesn't inherently "distinguish" the source of the text in the way humans might identify different speakers in a conversation. Instead, it generates text continuation based on patterns and information it has learned during training.
Here's how it generally works:

Training Data: The model is trained on a vast amount of text data from the internet. This data includes various sources of information, conversations, articles, books, and more. It learns patterns, language structures, and associations from this diverse dataset.

Contextual Understanding: When given a prompt or context, the model uses its training to understand the context and generate text that follows coherently from that context. It doesn't "remember" specific sources or authors; it generates text based on patterns it has learned.

Pattern Recognition: The model is adept at recognizing patterns in language, such as syntax, semantics, and even subtle context cues. It uses these patterns to generate text that appears to be a continuation of the given context.

No Source Attribution: The model doesn't inherently attribute sources to text, unless specific source information is provided in the prompt. It doesn't have a memory of past interactions or contexts unless provided in the current conversation.

Generating Responses: When generating responses in a conversation, the model generates text that fits the established conversational context. It's not aware of who said what in previous turns of the conversation; it simply generates text that continues the conversation based on patterns it has learned.

Context Window: One limitation of the decoder-only architecture is that it has a limited "context window" or memory of previous input. Earlier parts of the conversation might not have as much influence on the current response as the most recent context.

In summary, the decoder-only conversational model generates text based on patterns it has learned during training and the context provided to it. It doesn't inherently differentiate between different sources of text, and it doesn't have an inherent awareness of the sources of its training data or specific conversations unless explicitly provided in the current context.
Thanks!!!


text:
Is LSTM a subcategory of RNN?
Is the LSTM-Architecture a subcategory of RNNs? Or are they totally different?
Literature doesn't seem to be unitary on this.
This figure appears to explain the models to be alternatives, but I thought of them otherwise (LSTM to be a subcategory of RNN)
LSTM as a subcategory of RNN is mentioned in the Wikipedia article on LSTMs:

Long short-term memory (LSTM) is an artificial recurrent neural
network (RNN) architecture...

The Wikipedia article is more technically correct, in that the term RNN is formally taken to mean "a neural network with recurrent connections", and that includes many architectures that match this description, including LSTMs.
However, it is also common to see "RNN" used as a short-hand for a kind of "Vanilla RNN" or "basic RNN", where one or more layers have weights connecting the layer to itself (its own activations from $t-1$ are concatenated to the external inputs at $t$), and there are no other gates or special combinations, just those recurrent connections.
Oddly, this basic layer-based RNN archtecture is not listed in all the options on the Wikipedia page on RNNs - probably the closest are Elman networks and Jordan networks which are ways to implement the recurrent connection. It is a valid architecture choice, and can be effective. The LSTM and GRU architectures improve on it in terms of handling longer sequences and preserving important signals over them when training (e.g. matching a starting and ending quote in text processing).

So, I would clarify that this is a matter of terminology. If you define RNN (a short for recurrent neural networks) as the set of all neural networks with recurrent connections, then LSTMs could be considered a subset of RNN. However, if you define RNN as the vanilla RNN, then LSTMs are not a subset of RNNs (because vanilla RNNs do not have gates): it probably would be more the other way around (it may be possible to represent vanilla RNNs as LSTMs, though I don't remember the details now, so I am not really sure if this is possible).

@nbro: Yes. I think the looser terminology is very common, because for many problems "basic" RNNs and LSTMs/GRUs are interchangeable - in the sense that they can have same inputs and outputs, so using LSTM or GRU is a hyperparameter choice. For some reason that is phrased as using one of (RNN, GRU, LSTM) to solve a problem as opposed to one of (Elman, GRU, LSTM). I am not sure why


text:
Is the multi-head attention in the transformer a weighted adjacency matrix?
Are multi-head attention matrices weighted adjacency matrices?
The job of the multi-head-attention mechanism in transformer models is to determine how likely a word is to appear after another word. In a sense this makes the resulting matrix a big graph with nodes and edges, where a node represents a word and an edge the likelihood to appear after that. So basically it is an adjacency matrix that is created.

Hello. Please, edit your post to provide more context (this post has been flagged as "needs more details") and explain why you're asking this question (what confuses you?).

No, that isn't the job of the multi-head-attention mechanism.

Short answer, yes I believe we can! One way feels more meaningful that the other. First, let's look at some nuance in the definition of attention. If $\text{score}(x_i, x_j) = \text{score}(x_j, x_i)$, then the attention matrix is symmetric and naturally has the form of a weighted adjacency matrix. For example, this happens when attention is given by a simple dot product $\text{score}(x_i, x_j) = \langle x_i, x_j \rangle = x_i^Tx_j$. This also happens if we have a learnable matrix $A$ and $\text{score}(x_i, x_j) = x_i^TAx_j$ if $A$ is symmetric. We can then view the attention matrix $\alpha_{i,j} = \text{Attn}_{i,j}(X)$ as a weighted adjacency matrix where the nodes represent input tokens, and edge weights correspond to similarity scores (as defined by the inner product, scaled inner product, or the symmetric matrix $A$). Now, for the following definition of attention, this makes a little less sense, as the edge connecting token nodes in graph corresponding to tokens $x_i$ and $x_j$ is not the same as the weight on the edge connecting $x_j$ to $x_i$, in general. Suppose $X \in \mathbb{R}^{d \times n}$ has as columns $X_i$ the $d$-dimensional embeddings of the $n$-tokens $x_1, x_2, ..., x_n$ from your input. Now, let
$$W_QX = Q$$
$$W_KX = K$$
$$W_VX = V$$
be the learned weight matrices giving the "query" $q_i = W_QX_i$, "key" $k_i = W_KX_i$, and "value" $v_i = W_VX_i$ vectors. Then we can define attention as
\begin{align}
\text{Attn}(X) &= \text{softmax}\left( \frac{Q^TK}{\sqrt{d}} \right)V \\
               &= \text{softmax}_j\left(\text{score}(x_i, x_j)\right)
\end{align}
From this we can derive,
$$ \text{Attn}_{i,j}(X) = \frac{\exp \left( \frac{\langle q_i, k_j \rangle}{\sqrt{d}}\right)}{\sum_k \exp\left( \frac{\langle q_i, k_k \rangle}{\sqrt{d}} \right)}.$$
Now, note, we have turned each column into a probability distribution by applying the softmax so we have
$$
P(X_i) = \text{softmax}\begin{pmatrix}
\frac{\langle q_i, k_1 \rangle}{\sqrt{d}}\\
\frac{\langle q_i, k_2 \rangle}{\sqrt{d}}\\
\vdots \\
\frac{\langle q_i, k_n \rangle}{\sqrt{d}}
\end{pmatrix}.
$$
Now, if we adjust for masked self attention, we can represent the attention mechanism as a directed graph (with weighted self-loops) as the entries in the attention matrix above the diagonal are zero, and so all edges are directed at token nodes that come "after" them.
There is another way to understand attention using graphs when we view attention through the lens of (complete) graph attention networks as explained here.
There is also a visualization you can find here that shows the attention mechanism as a graph. It is equivalent to the graph attention formulation, but is more interactive and pretty.


text:
Defining states and possible actions in Q learning
I am trying to define the number of states and possible actions for a reinforcement learning problem that I want to solve with Q-learning, but I am a bit confused, as I'm totally new to reinforcement learning.
The problem I'm trying to solve is to assign different groups with people in the sample group having sequential numbers. Let's say there are three in each group.
Group 1, Group2, Group3.
1:{"group: Group1, "number": 1},
2:{"group: Group2, "number": 2},
3:{"group: Group3, "number": 3},
4:{"group: Group2, "number": 4},
5:{"group: Group1, "number": 5},
6:{"group: Group3, "number": 6},
7:{"group: Group3, "number": 7},
8:{"group: Group2, "number": 8},
9:{"group: Group1, "number": 9},

An optimal output will be a case where the numbers are sequential with respect to the group. For example, all in group1 should have number 1, 2, 3, or 4,5,6 or 7, 8, 9. and not 1, 5, 9 as in the dictionary above.
In other words,  group1, group2, group3 represent the group ids, which means I have 3 groups. The number represents seat numbers. All in each of the groups need to sit close to each other e.g seat numbers 1,2,3 or 4,5,6, or 7,8,9.
I am wondering if the possible state will be all possible combinations of the group and numbers, in which case it will be 1680 and the possible action is the number of numbers to swap to get the desired output which is 9.
Any useful information will be very much appreciated.

Hello. Welcome to AI SE. I don't fully understand your problem even with your example. Particularly, I don't understand this description "assign different groups with people in the sample group having sequential numbers.". What is a "sample group" here? It seems that you have 3 groups. How are these groups different from the "sample group"? What do these sequences of numbers 1, 2, 3 or 7, 8, 9 represent in your problem? Please, edit your post to clarify all these points.

@nbro thank you for your comment. I have added more information. Hope it is clearer now. The same group is Group1 or Group2, or Group3. The numbers can be 1,2,3 or 4,5,6 or 5,7,6 for any of the groups.

Thank you for the edit. It is easier to understand your goal state here, but not easy to understand the rules of your environment. It seems trivial to sort by group id and then assign "seat numbers" sequentially, and that appears to be a perfect solution. As you are only asking about the state space, that may not be a problem, but I think you need to give a little bit more background so that readers understand why you want a state representation at all

Also I think you need to be clearer on whether the goal is simply that a group must consist of sequential number assignments, or whether that sequence relates to the row (person?) id. E.g. if it is the former, then your starting state could be 123213321 and an acceptable end state might be 333111222 (there are 6 such end states, and yes I count 1280 states overall using this representation)


text:
How to approach a two-agent two-step action game?
A simple two-player sniper game:

Each player has 9 houses that he can reside in. So 18 houses in total. The houses can be considered in a row: e.g. 1-9 for player A, and 10-18 for player B.

Each step, the player should make two actions! First, he can use his gun's limited view to check out three consecutive houses of the enemy to see if he is there (for example, he choose 3,4,5.). Then, based on that result, he can choose one house to shoot. That means if he guessed correctly, he will know the other player is in one of those three houses. Otherwise, he can shoot one of the remaining six houses.

The killer wins!

Please note that in each step, the player has to perform two actions without interruption from the other player. Based on the result of the first action (limited view), he will have more information to select his second action (shooting). Thus, the first action is informative to reduce action space.

I have decided to use stable-baselines3. I have to create an environment. I am not sure about the policy network.
How should I approach this game for training an AI agent? I would really appreciate it if you can guide me on env creation, policy selection, or any general tips.

I am not sure how in-depth the information you want need to be, but maybe I can share some thoughts that may help you!
Environment
I would highly recommend using OpenAI Gym with your environment, since most of the already implemented RL-algorithms are designed to use gym environments.
You can design your environment any way you want to and then use gym to exchange action, observation and reward handling.
If you want a 3D environment you could look into Unity3D which has a machine learning toolbox, but as far as I understood you can just model the game in python.
Algorithm
Now the tricky part. It really depends on how you design your game and how you want to handle the agents.
A simple scenario would be only using one agent (Single-Agent RL). In this case, you would have to code a simple opponent by yourself and the agent could only be as good as the opponent you coded.
If you want them both to be as good as possible, you would have to use a Multi-Agent RL (MARL) approach.
MARL comes with its own problems and difficulties. The main problem is that the learning environment is unstable, because the resulting states and rewards do not only depend on the action of one agent, but of the actions of multiple agents.
In a competetive game, where the agents do not communicate, Independent Reinforcement Learning (IRL) could work. In this case you would use two completely seperated agents with a single-agent RL algorithm for each agent.
However other MARL algorithms make use of centralised training, which guarantees a stable learning environment and decentralised execution.
MADDPG is a wildley used algorithm for MARL problems, however there are a lot of different and newer algorithms.
I would recommend checking recent review papers like this one.
How I would approach the problem
Now I don't know how helpful this was. If I had to train agents for your scenario, I would follow these steps:

Create the environment and test it
Design an observation space - what does the agent see in each time step?
Decide on action spaces (discrete in your case), which action can he choose?
Design a reward function / reward to tell the agent when the action was successful
Decide on an algorithm and train the agents with scenarios as simple as possible, then make it more difficult if the training is working.

Hope this helps!

Thank you very much. Indeed this is helpful, both the proposed approach and explanations of the first parts.


text:
Is using separate channels of a RBG image a valid data augmentation technique?
Suppose there is a ML network that takes grayscale images as the input. The images that I have are RGB images. So, instead of converting these RGB images to grayscale, I treat each individual colour bands as distinct inputs to the network. that is, instead of feeding RGB image A to the network, I feed the R matrix of A as the first input, followed by the G matrix and then the B matrix. This leads to 3 times more data for the network. Can this be considered as data augmentation (since we are increasing the amount of data)? If yes, what is the name of this data augmentation technique?

I doubt it has a name. In your example, how close are the separate channel images to the target population where the network will be used to make predictions?

Neil is right, data augmentation is suppose to add data that respect the same distribution as the original data.
Grayscale is a linear combination of individual channel, therefore the single channels have different distributions (histograms) compared to their combination. And that is always gonna be the case, and unfortunately there's no work around to it, the only way would be to match the single channel histograms to the grayscale one, but then you'll end up with 3 copies of the same image.
You can easily check the differences with this code snippet.

import numpy as np
import skimage as sk
import matplotlib.pyplot as plt

def plot_hist():
    img = sk.data.chelsea()

    red = img[:, :, 0]
    green = img[:, :, 1]
    blue = img[:, :, 2]

    gray_crt = 0.2125 * red + 0.7154 * green + 0.0721 * blue
    gray_avg = img.mean(axis=2)

    fig, ax = plt.subplots(2, 3)

    bins = np.arange(-0.5, 255 + 1, 1)
    ax[0, 0].imshow(img)
    ax[0, 1].hist(gray_crt.flatten(), bins=bins, color="k")
    ax[0, 2].hist(gray_avg.flatten(), bins=bins, color="k")

    colors = ["r", "g", "b"]
    for c in range(img.shape[2]):
        ax[1, c].hist(img[:, :, c].flatten(), bins=bins, color=colors[c])
    plt.show()

if __name__ == "__main__":
    plot_hist()


text:
How do I show the relationship between theories and models using Conceptual Graphs?
The Mereology Theory below contains three first-order axioms that represent a part of a mereology theory. For this posting, it is important that the set of axioms should be considered as a theory.
Mereology Theory
Reflexivity $\forall x : part(x,x)$
Antisymmetry $\forall x  \forall y : ((part(x,y) \land part(y,x)) \implies (x = y))$
Transitivity $\forall x  \forall y \forall z :((part(x,y) \land part(y,z))  \implies part(x,z))$
Here is my naïve attempt to present the theory as a set of conceptual graphs (CG).

My understanding of the above CGs is as follows:

The variables are universally quantified, not default for CGs, but allowed in extended CG (ECG).
The inner graphs are all related by conjunction, which is default for GCs and I assume for ECGs.
The arrow on graph representing reflexivity is bi-directional.
Both antisymmetry and transitivity are represented by an IF-THEN contexts.
Dotted lines are co-references.
Equality (=) is actually commutative, but is represented as a directed relation .
Each inner graph asserts a single proposition, labelled Proposition.
The outer graph is labeled MereologyTheory, I am not sure that this is correct ECG syntax.

Below is a possible model of the above theory:
Mereology Model Mathematical notation
$Entities = \{ a,b  \}$
$Relations = \{part(a,a),part(a,b),part(b.b)\}$
Obviously there are many other possible models. MereologyModel below is my attempt to visualize this model as a CG. I am not sure that putting the label MereologyModel is correct CG syntax or denotes it as a model of  MereologyTheory .

Question:
In CG visual notation, how are theories and models related?
It seems to me that basic CGs can represent FOL sentences and the relation between such sentences. According to Chein and Mugnier the subsumption relation is defined by graph homomorphisms between CGs. Is  the model/theory relation for CGs also defined in terms of graph homomorphisms? I am aware that in general a model satisfies a theory i.e. $M \vDash T$. Does the graph homomorphism provide the necessary syntactic mapping to enable model and theory to be related?
Note  CGs can be represented in Common Logic (ISO zipped PDF), which would permit a formal proof that all the axioms of the theory are satisfied in the model.

For the approach of formal ontology, it is usual to keep theory and its model together, and name this "ontology" subsuming "formal". So, just put theory and model on one CG.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

@Alex In general theories and models are distinct entities that are related by the satisfaction relation, M⊨T. So embedding the model and the theory in one graph does not seem to me to be appropriate. Further, the model and theory may be presented in different logical systems (see Goguen and Burstall's Theory of Institutions)


text:
How are theories represented using Conceptual Graphs?
Theory 1  shows three axioms and two definitions, written in First Order Logic (FOL), that represents a fragment of a mereology theory. For this posting, it is important that the set of axioms is considered as a theory (i.e. a set of axioms together with theorems as a logical consequence of those axioms). In the context of this question, the particular axioms are not significant. Any other set of axioms forming a consistent theory would be equally acceptable.
Theory 1
Axioms
Reflexivity $\forall x : part(x,x)$
Antisymmetry $\forall x  \forall y : ((part(x,y) \land part(y,x)) \implies (x = y))$
Transitivity $\forall x  \forall y \forall z :((part(x,y) \land part(y,z))  \implies part(x,z))$
Definitions
Overlap : $\forall x,y \colon (overlap(x,y) \iff(\exists z \colon  (part(z,x) \land part(z,y)))$
Proper Part : $\forall x,y \colon (proprPart(x,y) \iff (part(x,y) \land \neg part(y,x)))$
I am using CafeOBJ to represent the above logical axioms and definitions, shown in Listing 1:
Listing 1
   mod M{
    [E]
    preds overlap part properPart : E E
 -- axioms
 ax [M1] : \A[x:E] part(x,x) .
 ax [M2] : \A[x:E]\A[y:E] ((part(x,y) & part(y,x)) -> (x = y))  .
 ax [M3] : \A[x:E]\A[y:E]\A[z:E]((part(x,y) & part(y,z))  -> part(x,z)) .
 -- definitions
 ax [DM1] : \A[x:E]\A[y:E] (properPart(x,y) <-> (part(x,y) & ~(part(y,x)))) .
 ax [DM2] : \A[x:E]\A[y:E] (overlap(x,y) <->  (\E[z:E] (part(z,x) & part(z,y)))) .
    }

Note that the logical theory is contained in a named module called M. The variables are over a domain of generic entities E, universal and existential quantification are denoted  by  \A[x:E] and  \A[x:E] respectively. In CafeOBJ, named modules allow one to structure signatures, theories, sub/super theories, and models using the Theory of Institutions (TOI).
Below is my naïve attempt to present the axioms as a set of conceptual graphs (CG).  My motivation for using CGs is that they provide an intuitive visualization of logic and have a direct relation to Common Logic (ISO zipped PDF).

The above CG was produced using CharGer software as Java zip file (manual).
My understanding of the above CGs is as follows:

The variables are universally quantified, not default for CGs, but allowed in extended CG (ECG).
The three graphs are all related by conjunction, which is default for GC.
The arrow on graph representing reflexivity is bi-directional.
Both antisymmetry and transitivity are represented by an IF-THEN contexts.
Dotted lines are co-references.
Equality (=) is actually commutative, but is represented as a directed relation .
Each CG asserts a single proposition, labelled Proposition.

Question:
How do I present Theory 1 using CGs?  Do I need some labeling that indicates that a set of concepts represent a theory. Or are theories represented by some enclosing special type of concept?

DOL project may be helpful https://www.omg.org/spec/DOL. see for ex.

@Alex Thanks the link is useful, but I am more interested in FOL <-> CG mappings and model/theory representation using CGs.

It seems that this post starts to become more complex. If you have multiple questions (even if they are related), I would recommend that you split them into multiple posts.

@nbro I have tried to simplify the question.


text:
Train Validation Test Splitting After or Before Data Augmentation?
I have seen tutorials online saying that you should do data augmentation AFTER doing the train/val/test split. However, when I go online to read some research papers, I see numerous instances of authors saying that they first do data augmentation on the dataset and then split it because they don't have enough data. Is it just that these are silly mistakes, even for papers with many citations, or is this acceptable?
Example: Research paper.
they say:
"Among these selected 480 images, 94 images were col-lected while changing the viewing angle, including images of 30 youngapples, 32 expanding apples, and 32 ripe apples.These 480 images were then expanded to 4800 images using dataaugmentation methods, yielding the training dataset. The training da-taset is used to train the detection model. The remaining 480 images areused as the test dataset to verify the detection performance of theYOLOV3-dense model".

I personally can’t think of any good reason to apply data augmentation before splitting the dataset, though one may exist. The issue is that if you augment first and split later, you risk introducing unwanted correlations between your training and test datasets.
In the paper you linked it sounds as if the training data is all procedurally derived from the test set, which makes it hard to trust the test accuracy. How do you know the network is doing any more than picking up some basic (non-generalizable) similarity between the various augmentations and their source input?

In my personal experience, that depends. Augmenting data for training purposes is valid, and can even improve performance, as you may be aware. For testing purposes, it may be valid. Let me give you two examples when that may be the case:
Facial Recognition. Imagine that you have an augmentation function that can change the face pose (left/right pose, for simplicity). You may want to include augmented pose images for testing your models robustness.
The paper you mentioned. In this case, you have apple detection. As the authors in [1] say:

Apples in orchards were detected and the growth stages of apples were judged. Since the angle and intensity of sunlight illumination varies greatly during the day, whether the neural network can process the images collected at different time of the day depends on the integrity of the training dataset. In order to enhance the richness of the experimental dataset, the collected images were pre-processed in terms of colour, brightness, rotation, and image definition.

After this brief introduction, the authors proceed into discussing the augmentation types they used for enhancing the richness of the dataset. As for the case of Facial Recognition, augmenting the test data follows the same idea of having a diverse testing data.
References
[1] Tian, Y., Yang, G., Wang, Z., Wang, H., Li, E., & Liang, Z. (2019). Apple detection during different growth stages in orchards using the improved YOLO-V3 model. Computers and electronics in agriculture, 157, 417-426.


text:
Understanding the features given in Example 13.1 of Sutton and Barto
I'm struggling to understand the notation used to represent the features within Example 13.1 (Short corridor with switched actions" in the Sutton and Barto RL book.
I assume as it is a free pdf it is ok to quote this content here, but if I am wrong, please let me know and I can remove it.

Consider the small corridor gridworld shown inset in the graph below. The reward
is -1 per step, as usual. In each of the three nonterminal states there are only
two actions, right and left. These actions have their usual consequences in the
first and third states (left causes no movement in the first state), but in the
second state they are reversed, so that right moves to the left and left moves to
the right. The problem is difficult because all the states appear identical under
the function approximation. In particular, we define $\mathbf{x}(s,right)=[1,0]^T$ and
$\mathbf{x}(s,left) = [0,1]^T$ , for all s. An action-value method with $\epsilon$ -greedy action selection
is forced to choose between just two policies: choosing right with high probability
$1-\epsilon / 2$ on all steps or choosing left with the same high probability on all time
steps. If $\epsilon = 0.1$, then these two policies achieve a value (at the start state)
of less than -44 and -82, respectively, as shown in the graph. A method can
do significantly better if it can learn a specific probability with which to select
right. The best probability is about 0.59, which achieves a value of about -11.6.

Screenshot of plot below:

In the example it states $\mathbf{x}(s,right)=[1,0]^T$ and $\mathbf{x}(s,left) = [0,1]^T$ for all s. Seeing feature vectors with relation to states (and not actions) seemed to make sense to me, e.g. for the polynomial construction:
$\mathbf{x}(s)=\left(1, s_1, s_2, s_1 s_2\right)^{\top}$
But I don't recall ever seeing it spelled out like this for features incorporating actions as well. In the polynomial case given above, we have a 1x4 vector. What is the dimension of the feature vector given in Example 13.1? Is it 1x2x2? How can we write it out in terms of $x(s,a)$ rather than two explicit definitions for each action? $x(s,a) = [[1,0],[0,1]]$?
Also, I'm confused on why this example makes a positive case for using policy gradient over value-based methods. If we just used the action values to determine a policy, wouldn't the agent be able to learn the choosing left from the 2nd state result in a higher action value? In such a case I would expect the value under such a policy would be [-3, -2, -1, 0] for each state.
Finally, I don't understand the y-axis of the plot. This is the value under policy pi (defined by theta) for some state S. What is S here? A single state? The average across all states?

Please don't upload text as images, it isn't searchable. There are tools available to convert PDF to text and MathJax, please edit to limit the image to the graph and its labels only; the remainder should be quoted text ( > "text". ) and MathJax. Thanks.

Thanks, didn't know about MathJax. Updates made.

Thanks, I searched for the text and found (an answer?): this: https://towardsdatascience.com/reinforcement-learning-policy-approximation-f5606ad3d909 and http://download.cmutschler.de/lectures/FAU_RL_2021/slides/6-02_Policy_Gradients.pdf - including a link to the canonical paper or lecture avoids another resource (book) which copied the content. --- A link to the full paper is useful, and on stickler sites any copied text must appear within quotes, or you get a last warning; not sure about here, but it's fairly standard. Thanks for making the previous changes.

Looks like the towardsdatascience link contains some code that would affirm my assumption about the shape of the feature vector, though I'm not seeing anything new to answer my question about why policy gradient would be useful here over an action-value based method. Finally, my last question about what S is in the plot might be answered by Equation 13.4 in the book where it defines $J(\theta)$ to be relative only to the initial state.


text:
Is it possible to successively train an RL agent on the same environment with different data
I have a scheduling problem that I am trying to solve with RL (if you are interested in more details you can read about it here Reinforcement learning applicable to a scheduling problem?).
I have created an own environment (OpenAI-Gym) and I have trained the model for one specific day of the simulation. So I have 288 timesteps for one day (1 for each 5 minutes) and the simulation last until the end of the day. So the agent needs to make 288 decisions when having 1 control variable.
Now my question is whether it is possible to successively train an RL agent on the same environment for different days? The environment and reward function will stay the same but the input data will change as every day has different input data (temperature, heat demand, electricity price etc.). So I would like to train an agent for one day and then tell the agent to train on another day but not forget everything it has learned during the training of the first day. Thus I can make sure that the agent is not overfitting to one special input data but also has the ability to generalize and thus be applicable for different days.
Do you know if and how I can do this?
Reminder: Can anybody tell me more about this by now. I'll highly appreciate every further comment as I still don't know how to do this.

You can mitigate catastrophic forgetting by storing the trajectories generated by the actors during training in a replay buffer. Then, you sample trajectories from that replay buffer. This way, each mini-batch of experience will contain data from multiple days.
There are many strategies to do this sampling, but you can start with uniform sampling. From what you're describing, it doesn't seem that storage is going to be an issue (288 data points per day is small), so you can keep all trajectories. If you can't afford to store all trajectories, then you should also design a strategy to remove them from the replay buffer.
You can refer to this handy guide describing how to implement a replay buffer in TensorFlow.

Thanks Raphael for your answer. I have to admit that I have problems understanding it. You mentioned the catastrophic forgetting. Actually, I don't want to forget the information from the previous days. And what do you mean by sampling trajectories from a replay buffer? And just for the record: I am not using TensorFlow for reinforcement learning so the posted link is not suitable for me (altough I appreciate that you have posted it).

Thanks Rapahel for your answer. Any comments to my last comment? I'll highly appreciate every further comment from you.

Any further comments? I have problems understanding your answer. Would you mind elaborating a little bit more on it (see my first comment).

Apologies I was on holidays :) The method I'm describing is supposed to prevent catastrophic forgetting. The idea is to store the 288 decisions the agent has made on day 1 (with associated state and reward). That's a trajectory. Then the next day, you train on a mixture of the decisions made on this day and those of day 1. On day 3 you train on a mixture of decisions made on this day and those of day 1 and 2, and so on. Is that clearer?

Thanks for your comment Raphael. Actually I still don't understand it. Shall I train an agent and then test it on day 1 and store the values for each timeslot? But how shall this work. After training the model, I can just apply it on other days. How can I tell the model to futher learn another way and still somehow consider the values from the training of day 1?

I don't know what ML framework you use, but it's fairly standard in most to continue training the same model architecture by restoring the weights from a previous training run as the starting point for a new training run. So, on day two, reload the weights from the training done on day one, and train the model on a mixture of day one and day two data. On day three, reload the weights from the training done on day two and train the model on a mixture of day one, two and three data, and so on. Does it make sense?

Thanks for your answer and effort. I really appreciate it. I use the Framework Stable-Baselines 3 (https://stable-baselines3.readthedocs.io/en/master/) and currently the algorithm A2C. So I have to find a way to continue training the model after having loaded weights from a previous run. What I don't understand in your comment is how to "train the model on a mixture of day one and day two data". My environment is a simulation of different days. One run is for one day. So it does not make sense to "mix" the days as the data would be inconsistant and unrealistic if you create a new day

A2C is not a good fit for experience replay because it's an on policy algorithm (there are off policy variants though). You could switch to something like DQN or offline RL which are better suited to replaying experience.
In DQN, it's fine to mix data from several days because you learn on transitions $(s_t, a_t, r_{t+1}, s_{t+1})$ rather than whole episodes.
PS: don't forget to upvote my comments if you find them useful!

Thanks for your comment. The reason why I don't use DQN is that I have a continious 3-dimensional action space (meaning I have 3 continuous variables). To be totally honest, I still have fundamental problems with your suggested approach. Mixing the days is not reasonable as I also get a cumulative reward at the end of the episode (next to the reward after every action).

Further, I don't understand how your trajectory approach should work. You said I should "store the 288 decisions the agent has made on day 1". Are you talking about the weights of the model or why else should I store the decisions of the agent after training? Your described pipeline is not clear to me. When I train the model on day 1, then I could store the weights of the model. Shall I then run the agent again on day 1 with the stored weights of the model and test the agent on the same day and store which decisions it is making? If so, what shall I do with the stored actions?

The terminal reward is not a problem, it's part of the final transition. You train your agent on batches of transitions, not on whole trajectories. That's what you're mixing: in your batches, you'll have transitions from day 1, day 2, etc. When I say "store the 288 decisions the agent has made on day 1", I mean the 288 transitions $(s_t, a_t, r_{t+1},  s_{t+1})$. At the end of training on these 288 transitions, you should separately store the weights of the model.

Thanks Raphael for your comment and effort. I really appreciate it. Unfortunately I have to admit that I don't understand your answer and still  don't know what to do. In my current setting, I train the agent only for one day and afterwards I can test it. Now I don't understand what to do. One training episode always ends after one day. I have to find a way to tell the training algorithm to switch the days during the training after one episode. How can I do that? I think this should be independant from your trajectory and replay buffer approach

Thanks Rapahel for your answer. Any comments to my last comment? I'll highly appreciate every further comment from you.

I'm not sure how I can be of more help unfortunately. Hopefully someone else understands your problem better than I do

@Rapahel: Thanks for your comment. Would you mind giving a little bit more information about how to do the things you suggest? How can I generate the trajectories and how can I train on a mixture of day 1 and 2 as you describe? For example you wrote "So, on day two, reload the weights from the training done on day one, and train the model on a mixture of day one and day two data. " --> How can I mix those 2 days. Maybe you can describe how to successively train an RL agent on the same environment with different data on your own or maybe you know a link where this is described

Thanks for your comments. Any comment to my last comment? I'll highly appreciate every further comment from you.

@ Raphael Lopez Kaufman: Thanks Raphael for your comments so far. Would you mind elaborating a little bit more on my previous comment. There I describe what I don't understand about your given answers so far.


text:
Reinforcement learning applicable to a scheduling problem?
I have a certain scheduling problem and I would like to know in general whether I can use Reinforcement learning (and if so what kind of RL) to solve it. Basically my problem is a mixed-integer linear optimization problem. I have a building with an electric heating device that converts electricity into heat. So the action vector (decision variable) is $x(t)$ which quantifies the electrical power of the heating device. The device has to take one decision for every minute of the day (so in total there are $24$ hours $\times 60$ minutes $= 1440$ variables). Each of those variables is a continuous variable and can have any value between $0$ and $2000 W$.
The state space contains several continuous variables:

External varying electricity price per minute: Between $0$ Cents and $100$ Cents per kWh (amount of energy)
Internal temperature of the building: Basically between every possible value but there is a constraint to have the temperature between $20 °C$ and $22 °C$
Heat demand of the building: Any value between $0 W$ and $10.000 W$
Varying "efficiency" of the electrical heating device between $1$ and $4$ (depending on the external outside temperature)

The goal is to minimize the electricity costs (under a flexible electricity tariff) and to not violate the temperature constraint of the building. As stated before, this problem can be solved by mathematical optimization (mixed-integer linear program). But I would like to know if you can solve this also with reinforcement learning? As I am new to reinforcement learning I would not know how to do this. And I have some concerns about this.
Here I have a very large state space with continuous values. So I can't build a comprehensive $Q-$table as there are to many values. Further, I am not sure whether the problem is a dynamic programming problem (as most/all?) of the reinforcement problems. From an optimization point of view it is a mixed-integer linear problem.
Can anyone tell me if and how I could solve this by using RL? If it is possible I would like to know which type of RL method is suitable for this. Maybe Deep-Q-Learning but also some Monte-Carlo policy iteration or SARSA? Shall I use model-free or model-based RL for this?
Reminder: Does nobody know whether and how I can use reinforcement learning for this problem? I'd highly appreciate every comment.
Can nobody give me some more information on my issue? I'll highly appreciate every comment and would be quite thankful for more insights and your help.

Is the output you need an advanced and fixed plan of all decisions for the day, or can decisions be made at each minute dynamically in response to state values at each time $t$?

Hi Neil, thanks for your comment. Basically both options would be possible. The output could be a fixed plan (schedule) of all decisions using some predictions of external variables (like the outside temperature). But the output can also be made dynamically at each timeslot t based on the current status.

@NeilSlater: Hi Neil, it is possible with RL to only generate the actions for 1 timeslot or is it also possible to generate a schedule (=action for multiple time slots in the future)?

It is possible for an action to be a schedule for multiple time steps, but it is more natural in RL to consider one time step at a time.

Details matter, and it is possible that your problem is best solved using classic control (solving the state equations) or operations research style optimisation. However, RL is also a good choice because it can be made to learn a controller that is not brittle when things go wrong.
One thing you will have to accept with RL is that the constraints will be soft constraints, even if you penalise them heavily. That is, you can expect that the internal temperature could drift outside of bounds. It definitely will during learning. A major design concern when framing the problem for reinforcement learning is how to weight the different rewards that represent your goals. You can weight your strict constraints higher, but at least initially they need to be low enough that the cost saving is not completely swamped.
I would suggest that your worst constraint failure penalty is slightly larger than the highest possible electricity cost for a single time step. That would mean the agent is always incentivised to spend money if it has to, as opposed to break the constraints, whilst still being able to explore what happens when it does break the constraints without having to cope with predicting large numbers.
There are lots of types of RL. Some are better at different kinds of problems. I would characterise your problem as you have described it as:

Episodic - but only for convenience of describing the problem. In fact your agent with a 24 hour episode will be incentivised to allow internal temperature to drop at the end of the 24 hours to save money, when it does not care what might happen immediately afterwards. Depending on price of electricity at that point, it could well be more optimal to spend more. This may only be a small difference from truly optimal behaviour, but you might play to strong point of RL by re-framing the problem as a continuing one (where mixed-integer linear optimisation may be harder to frame).

Continuous state space, with low dimensionality.

If prices are known in advance, you may want to augment the state space so that the agent knows how long it has at current price and whether the next price will be higher or lower. Alternatively, if they always follow the same time schedule, you could add the current time as a state variable. Either way, that allows the agent to take advantage of the temperature bounds. For instance, it could load up on cheap heating before a price hike, or allow the temperature to drop to minimum acceptable if cheaper electricity is about to become available.

Large, possibly continuous action space. You might want to consider approximating this to e.g. 21 actions (0 W, 100 W . . . 2000 W) as optimising this simpler variant will be easier to code (a DQN could do it), whilst it may not significantly affect optimality of any solution.

I don't think you could simplify your state space in order to use Q tables. So the DQN agent is probably the simplest that you could use here, provided you are willing to discretise the action space.
If you don't want to discretise the action space, then you will want to use some form of policy gradient approach. This will include a policy network that takes current state as input and then output a distribution of power level choices - e.g. a mean and standard deviation for the power choice. In production use you may be able to set the standard deviation to zero and use the mean as the action choice. A method like A3C can be used to do train such an agent.
I suggest that you discretise the actions and use a DQN-based agent to learn an approximate optimal policy for your environment. If that returns a promising result, you could either stop there or try to refine it further using continuous action space and A3C.
Also, you will want to practice using DQN on a simpler problem before diving in to your main project. Two reasonable learning problems here might be Cartpole-v1 and LunarLander-v2 which also has a continuous actions variant. Learning enough about setting up relevant RL methods to solve these toy problems should put you on a good footing to handle your more complex problem.
Keras documentation includes an example DQN for Atari Breakout, that you may be able to use as the basis for building your own code.

Thanks Neil Slater for your answer. Regarding the "Episodic ": Basically I also have a constraint at the very end of a day that the temperature should be (almost) the same as the initial temperature (which is set to 21 °C). Regarding the action space: I will discretize the action space such that I can use DQN. About the constraint violation: Basically the RL-controller should be trained and later it will used with a secondary controller that can overrule the RL-controller if a constraint violation is about to happen.

What framework or library would you advice me to use for DQN? Maybe Keras? Further, do you know a good tutorial for a framework that I can use and then adjust to my problem? I am honestly not such a big fan of these toy examples as they are quite different from what I inted to do and I also have to spent quite much time to learn other frameworks (like the gym) that I will not need at all. This is why I would like to directly start with my problem.

Keras is fine to build a DQN with. I would still advise you to study the toy problems, as per the answer, and I have picked three that are quite similar to your problem - continuous state space and simulations of physical environments.

Thanks a lot Neil for your great answer. I really appreciate it. I upvoted and accepted it. So I will then try to understand the Atari example and later try to implement my problem.

When I have learned the Q-function would you advice me to also use policy iteration or value iteration after the Q-learning (or in combination)? Or shall I use the SARSA-algorithm?

@PeterBe: DQN also learns an implied policy $\pi(s) = \text{argmax}_a \hat{q}(s,a, \theta)$, so when you have learned an (approximate) optimal action value function, you also have your (approximate) optimal policy. The other algorithms you suggest won't help you. SARSA is an alternative to Q learning that may work, but I suggest you stick to DQN for discretised actions. There are extensions to DQN that may help, although difficult to say much in advance. Use A3C, PPO or DDPG if you want to explore continuous action space.

Thanks for the answer Neil and your effort. I really appreciate it. What I am wondering is why DQN needs a discrete action space? As far as I understood, DQN just uses an articial neural network (e.g. a multi-layer perceptron or a LSTM) to learn the Q-function. Normally these ANNs can also cope with continuous input and outputs.

@PeterBe: It is because to derive the policy from the neural network you have to perform $\text{argmax}_a \hat{q}(s,a,\theta)$ (where $\hat{q}$ is the neural network function and $\theta$ is its learned parameters). The more actions there are, the longer this will take to do precisely. In a continuous action space it is intractable without some kind of approximation of the space or the $\text{argmax}$.

Thanks Neil for your answer. Basically I do not really understand your last comment. As far as I know ANN are also often used to map continouous inputs to continouous outputs. Or am I wrong on this one? Of course discrete inputs and outputs are easier to handle, but still I would say that it should also be possible to continous variables.

@PeterBe I suggest you ask a separate quiestion about your last issue because it is quite different to your original problem and this comment thread has gone on far too long.

Hi, it's me again with 2 follow up questions to your answers. Meanwhile I learned about one of the examples you suggested with gym (I do not want to learn the atari example as it is quite far away from my use case). 1) I have to implement my own environment. Can I use gym for that or can I just set up some simulation of a building in Python without any reference to gym (I would prefer not using gym) 2) Eventually my agent has to make 2 non-independant decisions at every time slot (I did not mentione this in my initial question). Is it possible to do this with RL learning?

@PeterBe 1)You can do either, you will find that you end up writing an environment very similar to Gym ones if you write your own, but it is not a problem. 2) Yes, you can have compound actions. For more/better details please ask a new question on the site. Comments are for clarifying details on the answer, and I cannot give proper details here

Hi Neil, I have a question to the constraint part of your answer "I would suggest that your worst constraint failure penalty is slightly larger than the highest possible electricity cost for a single time step. That would mean the agent is always incentivised to spend money if it has to, as opposed to break the constraints, whilst still being able to explore what happens when it does break the constraints without having to cope with predicting large numbers."

Basically I have 3 goals with normalized rewards between 0 and 1 for every timeslot and I have 10 constraints. Should the constraints' reward also be normalized for all 10 constraints and then I should choose a higher weight for the most important constraint than for all 3 goals combined? Is it also possible to tell the agent some rules directly without any constraints? E.g. I have 2 storage systems and the agent is only allowed to heat up 1 for every timeslot.  Further the agent should not start and stop heating up quite frequently but just have e.g. 4 starts of the device a day

@PeterBe Sorry this comment thread has gone on far too long. Please ask follow-ups as a new question. You may also benefit from another point of view than mine. You could link this question and answer from your new question, but do please make it as self-contained as you can.

Hi Neil, it's me again coming back to my follow-up question about your suggested approach regarding the weights of the constraints and the rewards. I ask a separate question as you advised (https://ai.stackexchange.com/questions/30469/how-to-weigt-constraints-in-a-control-problem-with-reinforcement-learning) but unfortunately nobody has answered during more than 3 weeks altough having set reminders mutiple times. This is why I'll highly appreciate any further comment on your suggested approach.


text:
Is it possible to tell the Reinforcement Learning agent some rules directly without any constraints?
I try to apply RL for a control problem, and I intend to either use Deep Q-Learning or SARSA.
I have two heating storage systems with one heating device, and the RL agent is only allowed to heat up 1 for every time slot. How can I do that?
I have two continuous variables $x(t)$ and $y(t)$, where $x(t)$ quantifies the degree of maximum power for heating up storage 1 and $y(t)$ quantifies the degree of maximum power for heating up storage 2.
Now, IF $x(t) > 0$, THEN $y(t)$ has to be $0$, and vice versa with $x(t)$ and $y(t)$ element $0$ or $[0.25, 1]$. How can I tell this to the agent?
One way would be to adjust the actions after the RL agent has decided about that with a separate control algorithm that overrules the actions of the RL agent. I am wondering if and how this can be also done directly? I'll appreciate every comment.
Update: Of course I could do this with a reward function. But is there not a direct way of doing this? Because this is actually a so called hard constraint. The agent is not allowed to violate this at all as this is technically not feasible. So it will be better to tell the agent directly not to do this (if that is possible).
Reminder: Can anyone tell me more about this issue? I'll highly appreciate any further comment and will be quite thankful for your help. I will also award a bounty for a good answer.

Hello @PeterBe, if I understand correctly, yes, an off-policy algorithm fit your problem. E.g. Q-Learning (but maybe a DDPG for the continuous case), SARSA is an off policy so you couldn't do that. Could you please tell me how you define your action space and state space?

@Pulse9: Thanks for your comment. A simplified basic version of my problem is described here https://ai.stackexchange.com/questions/28888/reinforcement-learning-applicable-to-a-scheduling-problem . You wrote "SARSA is an off policy so you couldn't do that"? Why can I not use an off policy learning algorithm for my problem? Or do you just want to say I can't use an off-policy algorithm to tell the RL agent some rules directly without any reward function (but I still can use an off-policy algorithm for my general problem)?

I'm not an expert, but as far as I understand, you should use an off-policy algorithm, the difference between is:
On-Policy: The agent learns the value function according to the current action derived from current the policy being used.
Off-Policy: The agent learns the value function according to the action derived from another policy.
This means that you can use another policy to explore. For example, if you use a Q-Learning (not your case because of the continuos values of your problem) that is an off-policy approach, you can explore with a particular policy to get the actions (you can only select valid actions) then you can update your q-table with the Q-Learning equation.
In your case you can use an off-policy deep approach. I suggest DDPG/TD3, you can look about some of them briefly here.
The idea is to use an exploration policy, the one you restrict to only select valid values (hard-constraint), and integrate the State, Action, Reward, State' in the replay buffer. The Stable_Baseline library doesnt allow that, but you could check the original source code of TD3.
Edit1:
If you see in the Q learning algorithm, the e-greedy consist on selecting with a probability of $\epsilon$ $a \gets \text{any action}$, and with $1-\epsilon$ the $a \gets max_{a}Q(s,a)$. This $\text{any action}$ is the part of the code that you use this "controller" to only select random (but valid) actions. This is because you want to explore but only explore with valid actions. Then the Q learning can "exploit" picking the best action from the exploration you did before. Now, for your case with continuos actions, you can use DDPG/TD3 to do something similar but you store these valid actions in a replay buffer, so your Neural Network can learn for this "data" of only valid actions.

Edit 2:
In your custom environment you can define your action space like:
self.action_space = gym.spaces.Box(low=-1, high=1, shape=(1,))

Now, as you said, in the step function of your environment you can establish the x(t) and y(t)
maxX=10 #Depends on the maximum value of your x(t), I assigned a 10
maxY=10 #Depends on the maximum value of your y(t), I assigned a 10
x=0
y=0
if action>0:
    y=0
    x=action*maxX
elif action<0:
    x = 0
    # you need to multiply by -1 because your action is negative
    y = -1*action * maxY
# do the rest of the code of your controler with x and y

In this way, your RL agent will learn which action (between -1 and 1) will get the best reward, but in the step function, you map the action [-1 +1] to your true values.

Thanks Pulse9 for your answer. I don't understand why I can't use Q-learning for that? I intend to use deep-Q-learning to map the inputs (states) to the outputs (actions). Further your wrote "The idea is to use an exploration policy, the one you restrict to only select valid values ". This is exactly my question. How can I restrict the agent to only select valid values? Of course I can just add another superior controller that can overrule the actions of the RL agent. But I would like to know if and how I can directly tell this to the RL agent while learning

Moreover I have some problems understanding your explanations about on and off policy learning. You wrote "Off-Policy: The agent learns the value function according to the action derived from another policy." What do you mean by another policy in this context? How can I change the policy?

I just improved my answer

Thanks for your answer Pulse9. If I understood correctly your advice is to implement the Q-learning agent on my own and not use e.g. the DQNAgent from Keras? I would have to admit that I would not know how to implement this agent on my own and I assume that the effort would be way to high for that. Is it not possible to tell a predefined agent about some rules directly without implementing the whole agent from scratch?

I forgot to specify the algorithms that I recommend for your case (DDPG/TD3) with continuos actions, the current libraries (Keras/Stable baseline...) do not allow you to define these exploration policies, but you can use others code from the authors that is ready to use, for example here: https://github.com/sfujim/TD3

Thanks Pulse9 for your answers and effort. I really appreciate it. I have to admit that I am kind of confused at the moment. So I gernerally have the following questions 1) Why can I not use deep-Q-learning for my problem? As always I could discretize the continuous action space. Why is Q-learning still not applicable to my problem

Is there a direct way to tell the RL agent some rules e.g. in the step function of Keras RL? If not I'd use another super-ordinate controller that would overrule the actions of the RL agent. Implementing my own RL agent is not in the scope of my work due to the tremendous effort for that. 3) I do not understand your explanations about on and off policy learning at all. You wrote "Off-Policy: The agent learns the value function according to the action derived from another policy." What do you mean by another policy in this context? How can I change the policy? Why do I have to distinguish

Any comment to my last 2 comments? I'll highly appreciate every further comment from you.

Thanks Pulse9 for your answers. Any comments to my last comment?

Hello PeterBe, I just had a new idea. I think you can use any RL library but using only 1 action, for example from -1 to +1, if the value is 0, then x(t) and y(t) is 0, if the value is 1, you can map it to be x(t) = 1*(yourMaxValue) and y(t)=0, and backwards, if your action is -1 then y(t) = 1* (-yourMaxValue) and x(t)=0. And in this way the agent can learn and you can map the actions to your need. What you think =)?

You can use PPO for example and its very easy to implement, I suggest to use the Stable Baseline 3 Library

Thanks for your answer. Honestly I do not understand how your suggested approach can ensure for example that x and y are never greater 0 at the same time. Further, you suggest to adjust the actions, right? Where shall I do this (maybe in the step function of OpenAI Gym) and how it this different from a super-ordinate additional controller that can overrule the actions of the RL agent?

I imagine that you are doing a custom environment right? I will edit (edit 2) my answer

Thanks Pulse9 for your answer and effort. I really appreciate it. I don't understand why you are using negative values? Neither x nor y can be negative. They are either 0 or between [0.25, 1] as written in my question. The second rule is that if x(t) is greater 0 then y(t) has to be 0 and vice versa. I don't see how you make this in your custom environament. Further, the action_space of my gym environment contains multiple variables that have different boundaries. So I can't just use one box variable for the actions. I need multiple variables with different limits for my actions.

And where do you use the PPO method that you suggested? As stated before I would like to use Deep-Q-Learning first and then also maybe SARSA. Can I combine the PPO method with them?

As I said, you map the RL action to your actions, you can see an example here: https://github.com/openai/gym/blob/master/gym/envs/classic_control/acrobot.py they map the action "a" in the step function using the variable AVAIL_TORQUE = [-1.0, 0.0, +1], you can see that they define the action space to a discrete (3) value: self.action_space = spaces.Discrete(3). The agent learns the best action 0,1,2, but in the step function, it is mapped to -1, 0, and +1.

Thanks Pulse9 for your comment. Maybe some last follow up questions: 1) Is your suggested approach (mapping discrete actions to other actions in the step-function) the PPO method? 2) Can I use this method in combination with Deep-Q-Learning and SARSA (because I want to use those 2 approaches)?

can you tell from your personal experience whether it is really worth doing such a mapping in order to tell the agent some rules directly. Or is it just better to use a superordinate controller that can overrule the actions of the RL agent? I am asking because in my case there are 2 discrete variabels x and y that both have 10 discrete values. So when I combine them into 1 variable that will have 10 x 10 = 100 discrete values (each of them represents 1 tuple with two componentes (x,y) and will be mapped to it). I think this will make the learning way more difficult.

I upvoted and accepted your answer. Further, I awarded the bounty to you because of your tremendous help. I still hope that you will answer my 3 follow up questions.

Yes, I think it will be much easier for you controlling the mapping but letting the agent learn their own actions. Similar thinking that a videogame, you dont focus on moving your fingers, you focus on the game but your brain do the mapping without you realizing it. You can do it with PPO using stable baseline library, you can see here that it accept discrete and continuos (box) https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html

DQN only accept discrete actions, you can see here: https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html and SARSA is also a discrete action, so I dont think you can map it to this range [0.25,1]

I think is good, normal Deep RL works with continuos actions, is way more than 100 actions =), think about how many different values you can have between 2 real values ;). Try it, I think it will work great. Good Luck!

Thanks for your comments Pulse9. I still have several questions because for me it is still quite unclear what to do. 1) Why can I not use DQL or Sarsa? Actually you  wrote  in 2) "DQN only accept discrete actions" and then in 3 "normal Deep RL works with continuos actions," Actually I can just discretize my continuous action space so I should be able to use DQN. 2) Why shall I not be able to map [0.25,1]? I can just do any mapping with the method you described. E.g. if the discrete action is 0 then the value is 0. If the discrete action is 1 then the value is 0.25. If action 2, value =0.3 e.g.

If you can discretize your space, then is fine, you can use DQL or SARSA.
I was saying about the mapping so your agent can learn that positive values (0,1] represent an x>0 and y=0 and negative values [-1,0) represent a y>0 and x=0, but these actions are coded inside your environment, the agent will learn only de action a=[-1,1].

Thanks once again for your tremendous effort. I really appreciate it.

You could just tweak your reward function to include this restrictions.
In the most simple case, you could reward your agent -1 if $x(t) > 0$ and $y(t) \neq 0$.
The scale of your negative reward depends on your general reward scaling of course.

Thanks tnfru for your answer. Of course I could do this with a reward function. But is there not direct way of doing this? Because this is actually a so called hard constraint. The agent is not allowed to violate this at all as this is technically not feasible. So it will be better to tell the agent directly not to do this (if that is possible)

Your question might be misleading then. without any constraints implies you don't want to do this as a hard constraint.
I'd suggest then, that you overrule the agent and give negative reward so the agent stops selecting the action and it is never exectued.

Thanks tnfru for your comment and effort. I really appreciate it. My question is if there is a direct way of telling this to the agent without any reward function. As far as I understand your answer you tend to say that there is not direct way of doing this?

To my knowledge none other than the overruling suggested above.

When you take a step in the DQL process, you sample a move based on the estimated qualities of each possible action. During that step, you can restrict your sampling method to have probability 0 of choosing the forbidden action.

Thanks nnolte for your answer. I have to admit that I do not really understand how I can do what you suggest. Would you mind elaborating a little bit more on that? Here are some questions that I have: 1) How can I estimate the quality of each possible action and where shall I do this? In the step function (of Open AI Gym)? 2) How can I restrict the undesired actions to probability 0? 3) Is this approach not the same as having a super-ordinate additinal controller that can overrule the actions of the RL agent?


text:
Batch normalization for multiple datasets?
I am working on a task of generating synthetic data to help the training of my model. This means that the training is performed on synthetic + real data, and tested on real data.
I was told that batch normalization layers might be trying to find weights that are good for all while training, which is a problem since the distribution of my synthetic data is not exactly equal to the distribution of the real data. So, the idea would be to have different 'copies' of the weights of batch normalization layers. So that the neural network estimates different weights for synthetic and real data, and uses just the weights of real data for evaluation.
My question is, how to perform batch normalization in the aforementioned case? Is there already an implementation of batch norm layers in PyTorch that solves the problem?

Does the second part of the question falls in to the category of asking for an API, library, or dataset (to solve a specific problem). which is off-topic? @nbro

I am interested in first part of the question.

Oh, in this case? Where should i post this question?

The question is nice. But i am not sure about the part  Is there already an implementation of batch norm layers in PyTorch that solves the problem? only

The question what is the best and easiest way to implement that? is on-topic since it is asking for reconciling batch norm over multiple distributions.

Oh I see. I would like to implement it, but I dont know enough pytorch to change it alone. That is why I would like some hints

You can remove the words best and easiest since they are superlative. Instead you can ask, how to perform batch normalization in aforementioned case?

@hanugm If "Is there already an implementation of batch norm layers in PyTorch that solves the problem?" is just a secondary/additional question, i.e. it's not the main question, and by "how to perform batch normalization in the aforementioned case?" the OP is asking, for example, for the pseudocode that would allow them to implement the approach to solve their problem, the question is fine (I think).