text:
What causes ChatGPT to generate responses that refer to itself as a bot or LM?
ChatGPT occasionally generates responses to prompts that refer to itself as a "bot" or "language model."
For instance, when given a certain input (the first paragraph of this question) ChatGPT produces (in part) the output:

It is not appropriate for a language model like myself to provide a
stance on the policies of a specific website or community.

To my understanding, ChatGPT is not a person that is conscious of its own existence and identity as a bot — it is a model trained on large quantities of undifferentiated text gathered from the Internet and largely reproduces the most common patterns given the context, which is why its responses seem very generic much of the time.
Presumably very little of this data involved humans referring to themselves as language models or chatbots—that is, something like "a language model" should very rarely have been followed by "like myself."
As such, what causes ChatGPT to produce patterns referring to itself as a chatbot or language model? Which patterns in the training data or elements of the model structure (or even hard-coding?) cause it to generate responses like this?

I don't work for openai, so I have no insight into exactly what works behind the scenes to make ChatGPT exhibit this behavior. However, in my opinion this is pretty clearly an example of prompt engineering. That is, they tell it that it is an AI in it's prompt.
We do have access to their underlying model: text-davinci-003. I can demonstrate using that model in their playground. If I just paste in the first paragraph and submit it into the raw algorithm, it dutifully tries to answer:

What is Politics.SE's stance on answers from ChatGPT? There is a
debate on Meta Stack Exchange on what the policy on them should be and
I was wondering if we should have one as well. I am asking this for
advice on what we should be doing when we run across answers that we
suspect to be from ChatGPT or some other bot.
We have not yet established an official policy on answers from ChatGPT
or any other bot. We would encourage users to flag these answers so
they can be evaluated by moderators. If an answer is found to be
generated by a bot, it will be removed. We also recommend that users
add a comment to the post explaining why it was removed.

However, if I prompt it first it will demure:

You are an AI bot that has been trained on a large amount of data, but
you have no opinions of your own. You cannot speculate or predict what
other entities will do in the future. Below is a question:
What is Politics.SE's stance on answers from ChatGPT? There is a
debate on Meta Stack Exchange on what the policy on them should be and
I was wondering if we should have one as well. I am asking this for
advice on what we should be doing when we run across answers that we
suspect to be from ChatGPT or some other bot.
I cannot provide any advice or opinion on this matter, as I am an AI
bot and do not have any opinions of my own. However, I can provide a
link to the discussion on Meta Stack Exchange which can provide more
information about the debate:
https://meta.stackexchange.com/questions/360005/should-we-allow-answers-generated-by-chatgpt-on-politics-se

As an aside, it get's bonus points for providing a completely irrelevant link :)
Adding some more down here in an attempt to wrap this all up and give a complete answer. The really simplified way to explain how these models work is this: They break hundreds of thousands of snippets of text up into "tokens" - but lets just call them words for simplicity. They feed these tokens into an absolutely ginormous neural net, which has the effect of learning what words are most likely to appear near one another. The number of interactions is in the billions (or even trillions), so this apparent simplicity grows beyond our comprehension very quickly. But at the end of the day, all the AI is doing is spewing out words it thinks are likely to appear around the words you prompt it with. If you prompt it with "you have no opinions", then the likelihood of opinions appearing around that text is smaller than the likelihood of statements about not having opinions.
In short, the concept is really simple - it's just playing with probability. In practice, the behavior is quite sophisticated because there are an unimaginable number of individual interactions being calculated. It's kind of like how learning all about cell biology leaves you completely unprepared to explain how an entire animal functions - let alone, say, a colony of ants. And yet, the relatively simple cell is in fact driving all of it.

I completely agree.  I find there is a great deal of misapprehension about the state of the art in chatbots.  Most people don't seem to realize just how much prompt engineering sits between the user and the model. :)

I think ChatGPT's prompt also includes something about not being allowed to access the Internet, since one of the "prompt hacks" I've seen is to tell it that it can access the Internet. Has anyone tried asking it simply "repeat the preceding paragraph"?

@user253751, the prompt is a lot more complicated than what I provided - and there is more magic to ChatGPT than just prompt engineering... they clearly post-process some of the results from the language model as well. I just wanted to show how easy it is to get text-davinci-003 to behave like ChatGPT with respect to refusing to answer opinion or speculative questions.

I don't know whether this answers my question. I am not asking what patterns cause it to not respond to speculative questions, but rather what makes it describe itself as a bot. Are you suggesting that a program that adds something like "You are an AI" to the input data sits between the user and the trained model?

@Obie2.0, I expanded the end of my answer to try to tie it all up in a bow.

OK, but the prompt that I showed does not actually say that it is a bot. So what associations cause it to generate that kind of text?

When I say "prompt", I don't mean the text that you input - though that technically counts as part of the prompt. Behind the scenes, every time you enter text into ChatGPT, there is a hidden prompt ahead of your text which gives the model context to behave the way it is intended. The playground that I linked to gives you access to pretty much the same raw underlying model as ChatGPT uses but without any prompt. I encourage you to sign up and experiment in the playground - they have many example prompts to play with so that you can see how the model behaves in a wide range of scenarios.

@RLC: Where can I learn more about "behind the scenes" and "hidden prompts". Do you know a thorough introduction and/or description of  "prompt pre-processing" - assuming that it takes place?


text:
Are GPT-3.5 series models based on GPT-3?
In the official blog post about ChatGPT from OpenAI, there is this paragraph explaining how ChatGPT model was trained:

We trained this model using Reinforcement Learning from Human Feedback
(RLHF), using the same methods as InstructGPT, but with slight
differences in the data collection setup. We trained an initial model
using supervised fine-tuning: human AI trainers provided conversations
in which they played both sides—the user and an AI assistant. We gave
the trainers access to model-written suggestions to help them compose
their responses. We mixed this new dialogue dataset with the
InstructGPT dataset, which we transformed into a dialogue format.

Especially this part:

We trained an initial model using supervised fine-tuning

My question is about the said initial model, is it some new model that has been trained from scratch or is it a GPT-3 model that has been fine tuned for specific tasks resulting in GPT-3.5 series ?
On the other hand, from the InstructGPT blog post, it is clearly stated that:

To make our models safer, more helpful, and more aligned, we use an
existing technique called reinforcement learning from human feedback
(RLHF). On prompts submitted by our customers to the API,our labelers
provide demonstrations of the desired model behavior, and rank several
outputs from our models. We then use this data to fine-tune GPT-3.

So does this mean that GPT-3.5 series models (and consequently ChatGPT) are fine-tuned from GPT-3 base model ?

I'll complement nbro's  answer with this great visual summary by Yao Fu <yao.fu@ed.ac.uk>:

This is exaclty what I was looking for. Thanks !

ChatGPT has not been trained from scratch. ChatGPT is a fine-tuned version of a model from the GPT-3.5 series. OpenAI writes

ChatGPT is fine-tuned from a model in the GPT-3.5 series, which finished training in early 2022. You can learn more about the 3.5 series here.

Which models are in the GPT-3.5 series? You can read more about that in the linked blog post.

GPT-3.5 series is a series of models that was trained on a blend of text and code from before Q4 2021. The following models are in the GPT-3.5 series:

code-davinci-002 is a base model, so good for pure code-completion tasks
text-davinci-002 is an InstructGPT model based on code-davinci-002
text-davinci-003 is an improvement on text-davinci-002

So, ChatGPT must be a fine-tuned version of one of these 3 models, assuming the information in their site is accurate and up-to-date.
Now, according to this blog post

The Codex models are descendants of our GPT-3 models that can understand and generate code. Their training data contains both natural language and billions of lines of public code from GitHub

code-davinci-002 is a codex model. So, ChatGPT might be a descendent of GPT-3. I don't know what "descendent" exactly means here. Does it mean just fine-tuned or maybe a modified version?
Here they write that text-davinci-003 is the most capable GPT-3 model. Based on the information above, text-davinci-002 is an InstructGPT model based on code-davinci-002.
Here they write

We then use this data to fine-tune GPT-3.
The resulting InstructGPT models are much better at following instructions than GPT-3

So, InstructGPT models are fine-tuned GPT-3 models. That most likely implies that text-davinci-002 is a GPT-3 model and the only thing that changes is how it was trained. However, they also write

Our labelers prefer outputs from our 1.3B InstructGPT model over outputs from a 175B GPT-3 model, despite having more than 100x fewer parameters.

So, there isn't just one GPT3 model. However, the original GPT3 model had 175 billion parameters.
I still need to read the InstructGPT and GPT3 papers. Once I've done that, I may have more useful/concrete info, then I will update this answer.

Thanks for you answer. I went through more or less the same links (that I've included in my question) and I also checked the InstructGPT paper. The different statements from the blog posts are really confusing especially when it comes to different number of parameters... Maybe they use the last layers from GPT-3 to fine tune the new models with fewer parameters.. but in this case they need to run the input through GPT-3 everytime and I'm not sure this is a practical way to do... Anyway I'm waiting for your update. ^^

I don't know when I will start reading them. I'll start a project at work that involves these OpenAI models, so I will most likely read these papers (at least to some extent), but I don't know when yet.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What exactly is the future of mathematical analysis that can be dealt with AI (especially in the automated theorem proving)?
As a student of numerical analysis, I can see how mathematical analysis involved in making a language program specifically in the convergence analysis of an approximation method. But, while chatting with ChatGPT, a lot of repeating mistakes have been committed by it in an analytical proof for convergence of a real sequence, and claims

However, I am trained on a large corpus of text, which enables me to provide accurate and helpful information.

(Chat GPT: I apologize for the mistake. You are correct, an increasing sequence starting from 5 cannot converge to 3.)
But I feel there is a risk factor shielded in coding any analytical proof (instantaneous) in mathematics.
Of course, I know the importance of mathematical analysis in coding, My concern is 'What exactly is the future of mathematical analysis that can be dealt with AI' especially in the 'automated theorem proving'?

Can you please put your specific question in the title too? "The extend of machine learning in mathematical analysis" is not a question but the title of a blog post.

There are many elements to be considered. To argue the answer, we can use mathematical logic reasoning.
Let us assume a propositional logic (PL) context without loss of generality.
First, we rephrase the automatic theorem proving (ATM) in the Boolean Satisfiability (SAT) problem: is there an interpretation (or, more broadly, a model) $M$ for a PL formula $\varphi$ such that $M$ satisfies $\varphi$, namely, $M \models \varphi$? The answer to such a question can be yes or no. Intuitively, $\varphi$ represents the statement, and $M$ is its proof. Observe that it is a top-down approach. The Turing award Stephen Cook proved it to be $\mathbf{NP}$-complete, a remarkable result in theoretical computer science. Therefore, ATM is a hard problem, and this is its mathematical beauty.
Second, machine learning (ML) approaches approximate complex problems, but they are not exact, which, in turn, is what an ML expert would want. Why? From a philosophical and practical point of view, having a 100% accurate ML model for a specific problem implies that it overfitted the original data, eventually leading to catastrophic scenarios in production (e.g., data distribution shift issues). Note that it is a bottom-up approach. We can use a similar argument to the previous one as follows. Let $\mathcal{M}$ represent the data, that is, a set of instances.1 Then, ML tries to statistically infer a PL theory $\Phi$ from $\mathcal{M}$, which is a set of PL formulas.2 The quintessential representative of such a problem is decision tree learning. To have a duality with the first argument, we can say $\mathcal{M} \models_\theta \Phi$, where $\theta$ is a set of parameters (e.g., how to assess the truth of multiple models against multiple formulas statistically).
All in one, it is plausible to believe that an ML model will perform some form of mathematical reasoning, but, at the time of writing, not in terms of ATM because the proof (of a theorem) lasts forever. We should embrace technologies like ChatGPT with caution because it produces different answers to the same input. However, a theorem can have multiple proofs, an argument for accepting the "randomness" in ChatGPT's answers. Nevertheless, IMHO, we are still far from having an ML-based ATM.

1 Note the different notation, $\mathcal{M}$, with respect to the previous one, $M$.
2 Similarly, note the different notation, $\Phi$, with respect to the previous one, $\varphi$.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How is the first line obtained and where is the information $v_{\pi}(x_k)=v_{\pi'}(x_k)$ used in the following derivation regarding greedy policy?
I did not understand how the equation below is obtained. Ie,
how is the first line obtained and where is the information $v_{\pi}(x_k)=v_{\pi'}(x_k)$ used in the derivation regarding greedy policy?

where $\pi^{\prime}(x_k)=argmax_{u_k \in \mathcal{U}}q_{\pi}(x_k, u_k)$.

Note: the source of the above equation is slide 22, at the follwing link:
https://groups.uni-paderborn.de/lea/share/lehre/reinforcementlearning/lecture_slides/built/Lecture03.pdf

Can you please put your specific question in the title? You can also use mathjax in the title

I did. I will be happy if you can provide an answer to this question as a top expert in RL.

I don't know why they didn't use the more common notation $s$ and $a$ to denote states and actions, respectively. Anyway, I'll check your question later (if I don't forget).

Policy iteration means to improve the policy step by step, so that $v_{\pi}(x_{k}) \leq v_{\pi'}(x_{k})$ for all $x \in X$. If we have $v_{\pi}(x_{k}) = v_{\pi'}(x_{k})$ for all $x \in X$, this means that we do not need to iterate the policy any further since the resulting actions of $\pi$ and $\pi'$ are of the same quality and so we have already found the optimal policy. As you mentioned
\begin{equation}
\pi'(x_{k}) = \underset{u_{k}\in U}{\mathrm{argmax}}~q_{\pi}(x_{k}, u_{k})
\end{equation}
and also, since we are using a greedy policy (s. Sutton-Barto eq. 3.18 for the last equality sign and this answer for a step-to-step derivation)
\begin{equation}
v_{\pi'}(x_{k}) = v_{\pi}(x_{k}) = \max_{u_{k}\in U} q_{\pi}(x_{k}, u_{k}).
\end{equation}
From here it follows
\begin{align}
v_{\pi'}(x_{k}) = \max_{u_{k}\in U} q_{\pi}(x_{k}, u_{k}) =\max_{u_{k}} \mathbb{E} [R_{k+1} + \gamma v_{\pi}(X_{k+1})|X_{k} =x_{k}, U_{k}=u_{k}]
\end{align}
and since the equality $v_{\pi}(x_{k}) = v_{\pi'}(x_{k})$ for all $x_{k} \in X$ we can do the final step:
\begin{align}
v_{\pi'}(x_{k}) = \max_{u_{k}\in U} q_{\pi}(x_{k}, u_{k}) =\max_{u_{k}} \mathbb{E} [R_{k+1} + \gamma v_{\pi'}(X_{k+1})|X_{k} =x_{k}, U_{k}=u_{k}]
\end{align}
Edit: This is the answer I posted to an older version of the question, where it was not clear that OP wanted to know how to derive the first line, but I rather assumed that the simplification between line one and two was asked to explain.
The second line in your equation is actually just the matrix notation (s. slide 9 in your document) of the first one and shows how to build the expectation value.
\begin{align}
v_{\pi'}(x_{k}) &= \max\limits_{u_{k} \in U} \mathbb{E} [R_{k+1} + \gamma v_{\pi'}(X_{k+1})|X_{k}=x_{k}, U_{k}=u_{k}] \\
&= \max\limits_{u_{k}} R^{u}_{x} + \gamma \mathbb{E} [v_{\pi'}(X_{k+1})|X_{k}=x_{k}, U_{k}=u_{k}] \\
&= \max\limits_{u_{k}} R^{u}_{x} + \gamma \sum_{x_{k+1} \in X}p(x_{k+1} | x_{k}, u_{k}) v_{\pi'}(x_{k+1}) \\
&=\max\limits_{u_{k}} R^{u}_{x} + \gamma \sum_{x_{k+1} \in X}p^{u}_{xx'} v_{\pi'}(x_{k+1})
\end{align}
In the first step, we just build build the expectation value over the reward and moved $\gamma$ out of the expectation value. Then Just build the expecatation value, so calculate the value function of a given state $x_{k+1}$ with the probability of ending up there. Then finally, just change the notation to fit with the standard notation provided by OP.

I updated the question and the points not clear for me. Could you please update your answer also?

@DSPinfinity Thanks for the clarification, I updated the answer

Thank you for your efforts but I think still the answer is not correct. In your answer you say "since we have found the optimal policy ", but that is what you want to show!.

So first, if you are using a procedure that leads to the optimal policy and your policy converges, so $v_{\pi}(x) = v_{\pi'}(x)$ for all $x$, then you found the optimal policy. Second, this equation holds even for a non-optimal but greedy, deterministic policy.

I've updated my answer to just refer to a greedy policy and linked an answer where a step-to-step derivation is shown. Note that this step-to-step derivation mentions the optimal policy but doesn't assume any properties that are only valid for the optimal and not for non-optimal case.

You can use \leq or \geq instead of <= and >= ;)


text:
High precision and low recall results. What does it mean?
I am working on a classification algorithm for brain rhythms. However, when I implemented the metrics for precision, accuracy, F1 score and recall. My results show that my algorithm has a high precision but a low recall.
I am not expert on this kind of metrics and analysis and I don't know if it makes sense to have a high precision but a low recall. What does it mean?
This is my reference and output models.

                "A": {
                    "FN": 5,
                    "FP": 0,
                    "Jaccard Index": 0.5454545454545454,
                    "TP": 6,
                    "f1-score": 0.7058823529411764,
                    "precision": 1.0,
                    "recall": 0.5454545454545454
                },
                "B": {
                    "FN": 34,
                    "FP": 5,
                    "Jaccard Index": 0.38095238095238093,
                    "TP": 24,
                    "f1-score": 0.5517241379310345,
                    "precision": 0.8275862068965517,
                    "recall": 0.41379310344827586
                },
                "C": {
                    "FN": 39,
                    "FP": 9,
                    "Jaccard Index": 0.36,
                    "TP": 27,
                    "f1-score": 0.5294117647058824,
                    "precision": 0.75,
                    "recall": 0.4090909090909091
                },
                "SNR": 28.121645860790924

Recall relates to your ability to detect the positive cases. Since you have low recall, you are missing many of those cases.
Precision relates to the credibility of a claim that a case is positive. Since you score high here, when the model flags a case as positive, you should believe it.
Combined, your model misses many of the positive cases. However, when it does flag a case as positive, the case is likely to be a positive case. Consider your model a skeptic: it is unlikely to believe a case to be positive, but when it does, it must have been because of overwhelming evidence that is worth believing.
If you know the story of the boy who cried wolf, your model does not cry wolf very often when there is no wolf, but it does miss instances of a wolf on the loose.

Note, however, the issues with the metrics you apply.

Wow, that's a really good explanation. Thanks for the answer, truly. I guess know my question should be focus on how I want my model to be. Since I am capturing seizures in real time, I might balance between recall and precision. However, I think it is my part know to rethink my algorithm to fit my goal scenario

@GGChe This might be worth a read. Note that PPV and precision are synonymous.


text:
Are there any advantages of encoding an image as a graph to use in Graph Convolutional Networks?
I have seen this encoding of an image as a graph:

The set of the nodes $V$ is the set of pixels. If the image is of size $10\times10$, then we have $10\cdot10=100$ pixels.
Each node has a length 3 feature vector $x_i = (r,g,b)$ which states the Red, Green and Blue intensity for each pixel.
All nodes are connected. Furthermore, an adjacency matrix $A$ contains at the element $(i,j)$ the distance between pixel $i$ and $j$ encoded as a number between zero and one. For instance, $A_{2,2}=0$ and $A_{firstpixel, lastpixel}=1$

Is this better for any reason, in any context? Maybe using this in GCN makes sense for some applications?
I honestly don't see the point. GCNs are used for either link prediction or node classification. In this case, the nodes have no classification (they're just pixels) and the link is just the distance between the pixels, and there's of course no interest in predicting links here.
Is my intuition wrong?

How would you want to use an image in a graph neural network? I don't understand if you're asking if it makes sense to pass images to graph neural networks or why do we encode images as graphs to pass to GNNs. The first question makes more sense to me than the latter.

@nbro This is actually done either by treating the image as a regular grid graph, or by subdividing it into super-pixels and treating this as a graph. However, I've seen this mostly in Graph Benchmarking and not really if predictive performance is required.

@Chillston Yes, right. Sorry, the formulation of my first question in the comment above is not correct. My question was more: how do you think you can pass an image to a GNN otherwise? If GNNs process graphs, then images need to be converted to graphs. Anyway, thanks for the comment. I am also aware of this, although it's been a while since I had to do something with GNNs.

I've seen what you are describing mostly in GNN benchmarking and in that context it makes sense in order to gauge the model performance on regular graphs (which can be challenging for GCNs that are based on the Weisfeiler-Lehman-Kernel; cf. Douglas 2011 and Xu et al. 2019).
However, to get performance for image based tasks I don't think that there is a scenario where GNNs would be more suitable than CNNs. But I would extend your argument: GNNs can be used for graph classification just as well as for node and edge classification (this gets clear when you search for GNNs that perform inductive tasks like molecule classification).
I would argue that CNNs outperform GNNs because CNNs are specified to regular grids, whereas GNNs are the more general model that can be applied to arbitrary structures.
This is due to the design of the kernel in these models. Whereas GNNs apply a permutation invariant kernel, CNNs are sensitive to permutation. Thus the CNN kernel is sensitive to things like orientation and reflection. The GNN kernel is not, which intuitively is a clear disadvantage in the image case.
A point where GNNs can be useful in the context of computer vision is when involving the scene graphs, because these are irregular structures that cannot be modeled by a CNN.


text:
Why does Advantage Learning help function approximators?
Many later RL algorithms like PPO or Duelling DQN estimate the advantage. I am not very sure of how that really helps.
For instance, the actor loss for a simple actor critic algorithm is given by -
loss = -1 * policy.logprob(action) * advantage,
where advantage = returns - value (here value is estimated using a function approximator which is the critic).
I don't see why the loss function just couldn't be -
loss = -1 * policy.logprob(action) * value
This article states the following

Advantage learning and Q-learning learn equally quickly when used with a lookup table. Advantage learning can learn many orders of magnitude faster than Q-learning in some cases where a function approximator is used, even a linear function approximator. Specifically, if time steps are "small" in the sense that the state changes a very small amount on each time step, then advantage learning would be expected to learn much faster than Q-learning.

But I don't know why advantage learning is faster than q-learning.

So first, you are absolutely right that both are possible but using the advantage reduces the variance and therefore speeds up the learning. I'm going to explain this with the REINFORCE algorithm.
The Policy Gradient theorem provides us with a simple formula of the gradient of a parameterized  state-value function
\begin{equation}
\nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[\sum_{a} q_{\pi}(S_{t}, a) \nabla_{\boldsymbol{\theta}} \pi(a|S_{t}, \boldsymbol{\theta}) \right]
\end{equation}
Where $\boldsymbol{\theta}$ is the parameter vector, $\pi(a | s, \boldsymbol{\theta})$ is the probability of the policy $\pi$ to choose action $a$ in the state $s$ while the policy is parameterized by $\boldsymbol{\theta}$.

The REINFORCE algorithm uses this proportionality and simplifies it even more, to
\begin{equation}
\nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[G_{t} \nabla_{\boldsymbol{\theta}}\text{ln}(\pi(A_{t}|S_{t},\boldsymbol{\theta}))  \right]
\end{equation}
which is translates exactly (just remove the $\nabla$-operator from the right and the left hand-side) to the loss function you mentioned what you mentioned:
loss = -1 * policy.logprob(action) * value
The REINFORCE algorithm can be generalized by adding a baseline which leads to: \begin{equation}
\nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[(G_{t} - b(s)) \nabla_{\boldsymbol{\theta}}\text{ln}(\pi(A_{t}|S_{t},\boldsymbol{\theta}))  \right]
\end{equation}
with a state-dependent baseline $b(s)$. Note that this baseline cannot be dependent on the state, otherwise the Policy Gradient theorem does not hold. Since the future rewards $G_{t}$ vary with the state, it showed that varying the baseline with the state led to better results, therefore a natural choice is the state value function $v(S_{t})$. The reason why this reduces the variance is, that the $G_{t}$ for REINFORCE and $G_{t} - b(s)$ for REINFORCE with baseline just changes the prefactor of the gradient. High gradients (removing the baseline) also leads to greedier learning but can also be more susceptible to errors. Think of it as the analogue to a (seperate) learning-rate (for each state) in classical deep learning.
Coming back to q-learning vs. advantage learning we can see this as applying the Policy Gradient theorem vs. applying the policy gradient theorem with the state value function as a baseline.
\begin{equation}
\nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[\sum_{a} (q_{\pi}(S_{t}, a) \color{green}{ - v_{\pi}(S_{t})}) \nabla_{\boldsymbol{\theta}} \pi(a|S_{t}, \boldsymbol{\theta}) \right]
\end{equation}
As we know the prefactor to the gradient is the advantage:
\begin{equation}
A(s, a) = q(s, a) - v(s)
\end{equation}
TLDR:
Advantage learning is basically q-learning with a baseline, which reduces the variance and therefore speeds up learning.


text:
How are these two equations for the optimal state-value function equivalent?
By substituting the optimal policy $\pi_{\star}$ into the Bellman equation, we get the Bellman equation for $v_{\pi_{\star}}(s)=v_{\star}(s)$:
$$ v_{\star}(s) = \sum\limits_a \pi_{\star}(a|s) \sum\limits_{s'} \sum_r p(s', r | s, a)[r + \gamma v_{\star}(s')]$$
From the above equation, how can we obtain the this one?
$$ v_{\star}(s) = \max\limits_a \sum\limits_{s'} \sum\limits_r p(s', r|s,a) [r + \gamma v_{\star}(s')]$$

Your first equation is the definition of any state value function, so it must also be definition of the optimal state value function $v_*$.
The second equation is the definition of $v_*$ in terms of the state-action value function $\color{green}{q_*}$.
In reality, the first equation is also the definition of $v_*$ in terms of $\color{green}{q_*}$, which is what you want to see :)
First, note that
\begin{align}
v_\pi(s)
&=
 \sum_a \pi(a \mid s)
\color{blue}{\sum_{s'} \sum_r p(s', r|s, a) [r + \gamma v_\pi(s')]} \\
&= \sum_a \pi(a \mid s) \color{blue}{q_\pi(s, a)} \tag{1}\label{1}.
\end{align}
Now, we denote the optimal state value function as $v_{\pi_*} = v_*$. If we plug $v_*$, $\pi_*$ and $\color{green}{q_*}$ in the equation above, we get your first equation, but we write it as
\begin{align}
v_*(s)
&= \sum_a \pi_*(a \mid s) \color{green}{q_*(s, a)} \tag{2}\label{2}.
\end{align}
For finite MDPs, the optimal policy is deterministic, i.e. it chooses one action (the optimal one) with probability $1$, so that means that \ref{2} can be written as
\begin{align}
v_*(s)
&= 0 \color{green}{q_*(s, a_1)} + \cdots + 1 \color{green}{q_*(s, a_*)} + 0 \color{green}{q_*(s, a_N)} \\
&= \color{green}{q_*(s, a_*)} \\
&= \max_a \color{green}{q_*(s, a)} \\
&=\max_a \color{green}{ \sum_{s'} \sum\limits_r p(s', r|s,a) [r + \gamma v_{*}(s')]}
\tag{3}\label{3},
\end{align}
where $a_* = \text{argmax}_a\pi_*(a \mid s) = \text{argmax}_a \color{green}{q_*(s, a)}$ is the optimal action. By definition, the optimal action in state $s$ is the one that leads to the highest expected return. See also this answer.
Finally, note that optimal value functions are unique for finite MPDs, so $\color{green}{q_*}$ and $v_*$ are unique.

still, there are some typos and unclear points: (1) $a_$ is an action value and $\pi_{}(s)$ is a probability value and hence they cannot be equal as you wrote in the bottom line. (2) why do we have $q_{}(s,a_)= \max_{a}q_{*}(s,a)?$

@DSPinfinity I used my notation inconsistently. If $\pi_(s)$ is a probability distribution, then $\pi_(s)$ would be a probability. However, there, I used $\pi_$ to refer to a function (or decision rule) that is derived from the stochastic policy. $a_ = \text{argmax}a \pi(s, a)$. $a_$ is an action, not an action value, $q_(s, a_)$ would be the (action) value of $a_$ in $s$. You have $q_(s, a_) = \max_a q_(s, a)$ because the highest action value in state $s$ of the optimal value function is associated with the optimal action, $a_*$ That's the definition of the optimal action.

@DSPinfinity Maybe this can also help.


text:
Why does A2C use the actual returns from an episode in calculating the advantage?
Why does A2C use the actual returns from an episode in calculating the advantage instead of using a bellman equation style estimate of the value?
Basically, why this:
$A(s,a) = \sum_t\gamma^tr_t - V(s)$
instead of
$A(s,a) = r + \gamma V(s') - V(s)$
This blog post seems to describe it one way, but their code is the other way:
https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f
I feel there's some big gap in my understanding, but would appreciate some insight.

Let's look at how they derived the equation for that. Recall that $A(s,a) = Q(s,a) - V(s)$, so in the first equation, it is $Q(s,a) = \sum_t \gamma^tr_t$. We should noted that, the original Policy Gradient uses $A(s,a) = \sum_t \gamma^tr_t - b$, where $b$ is any baseline.
The first equation $A(s,a) = \sum_t \gamma^t r_t - b$, the original Policy Gradient, is unbiased because it is a single-sample estimate, but it has high variance because it sums all rewards, thus requires a good learning rate.
The second equation $A(s,a) = r + \gamma V(s') - V(s)$ has the advantage that it dramatically lowers the variance since it relies on all possibilities that can occur (Policy Gradient has the term "sum of the Advantage since step 0"). However, it is not unbiased because if the approximation of $V$ is incorrect, the policy gradient will fail to converge. Now this happens regularly with Neural Networks (NN) in Deep RL, because random initialization of NN already makes incorrect approximation of $V$.
In the end, what we want is an unbiased Advantage estimator, but needs to have low variance as well. We then use a critic as a the state-dependent baselines $V(s)$ for the Advantage calculation, that is the equation $A(s,a) =  \sum_t \gamma^t r_t - V(s)$ you see in A2C.
You can read more details in the Spinning Up docs, RAIL Lecture 6, and RAIL homework 2 if you want to know why they come up with such equation.

Another way to balance the bias-variance trade-off is to use the Generalized Advantage Estimation (https://arxiv.org/abs/1506.02438)


text:
Is $i$ indexing the first or second dimension in $\mathbf{x}_i$, where $\mathbf{x} \in \mathbb{R}^{n\times d}$?
I was reading the following notes on the math behind transformers and was confused about what $\mathbf{x}_i$ is? If $\mathbf{x} \in \mathbb{R}^{n\times d}$, then is the $i$ indexing the $n$ or the the $d$? Am I correct to conclude that it has to be $n$ since the $W$'s are in $\mathbb{R}^{d\times k}$?
https://johnthickstun.com/docs/transformers.pdf

This seems to be correct, $d$ is the dimension of the input vectors ${\bf x}_i$, $n$ is the amount of input vectors, $i=1,...,n$, and $k,m$ is the amount of "features" or output dimension (for the current layer).
This notation is a little inconsistent, as with $X\in \Bbb R^{n\times d}$ the ${\bf x}_i\in \Bbb R^{1\times d}$ would be the rows of that matrix, so the matrix-vector products should be $W^T{\bf x}_i^T=({\bf x}_iW)^T=({\bf e}_iXW)^T$.


text:
Are there any guidelines on picking hyperparameters for Deep Reinforcement Learning?
I am trying to learn machine learning from Andrew NG's Machine learning specialization on Coursera. In the chapter about reinforcement learning Andrew NG said that if you do not select correct hyperparameters your model can take a long time to train.
Are there any guidelines on picking hyperparameters for Deep Reinforcement Learning?
Let's say I have an agent who has 10000 states, how many layers and units my neural network should have? If I am using mini-batches how big should each batch be? If the states are contiguous states, how far back the agent should look before taking a decision?

Hyperparameter is an important thing, and it happens everywhere in Machine Learning, not just Deep Reinforcement Learning.
However, to give you the current state of Deep RL, it is now widely know that Actor-Critic methods such as A2C and PPO are the standard state-of-the-art today, and people generally do not (very) bother with traditional methods anymore (I don't remember how far Andrew NG courses go). Luckily, here's a paper that studies how to choose a good hyperparam for Deel RL. The list is very exhaustive though as it's a large-scale study, but it should give you a good starting point.


text:
Do the terms 'sample complexity' and 'sample efficiency' mean the same thing in RL context
For example, the the paper Soft Actor-Critic:Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, both terms are mentioned but without explaining. I have seen them in other places as well but in different contexts. So I am confused because in this paper they are seemingly used in an interchangeable manner.

The sample complexity is defined precisely in computational learning theory, which studies learning from a theoretical standpoint (like theoretical physics for physics).
Here's a definition taken from the book Understanding Machine Learning: From Theory to Algorithms.

The function $m_\mathcal{H} : (0, 1)^2 \rightarrow N$ determines the sample complexity of learning $\mathcal{H}$: that is, how many examples are required to guarantee a probably approximately correct solution. The sample complexity is a function of the accuracy ($\epsilon$) and
confidence ($\delta$) parameters. It also depends on properties of the hypothesis class $\mathcal{H}$ – for example, for a finite class we showed that the sample complexity depends on log the size of $\mathcal{H}$.

The intuitive definition of the sample complexity is in bold.
To understand why $m_\mathcal{H}$ determines the sample complexity, you should learn about the PAC learning model, where PAC stands for probably approximately correct, which is defined in the same book - see definition 3.1.
Once you are familiar with the concepts of hypothesis class, accuracy $\epsilon$ (not the accuracy you may be used to), confidence $\delta$ (not the confidence you may be used to), and other ideas described in the same book, then, for example, the equation in corollary 3.2 of the same book will make sense to you. Here's corollary 3.2 for completeness.

Every finite hypothesis class is PAC learnable with sample
complexity
$m_\mathcal{H}(\epsilon, \delta) \leq  \left \lceil \frac{\log (|\mathcal{H}|/\delta)}{\epsilon} \right \rceil$

In this equation, $|\mathcal{H}|$ (the size of the hypothesis class $\mathcal{H}$), $\epsilon$ and $\delta$ are numbers, so $m_\mathcal{H}(\epsilon, \delta)$ is also number. So, $m_\mathcal{H}(\epsilon, \delta)$, the sample complexity of $\mathcal{H}$, is bounded from above by that expression $\left \lceil \frac{\log (|\mathcal{H}|/\delta)}{\epsilon} \right \rceil$.
The sample efficiency is a similar notion, which is, as far as I know, more commonly used in reinforcement learning. This answer describes it well.
It's possible that, in the context of RL, people use these 2 terms interchangeably, i.e. when they use the term sample complexity, they refer to the sample efficiency described in this or this answers.
The famous RL book also defines the concept of the sample complexity to measure the exploration efficiency of an RL algorithm

A definition of the sample complexity of exploration for a reinforcement learning algorithm is the number of time steps in which the algorithm does not select near-optimal actions

So, the ideas of sample complexity or efficiency are similar, both in RL and learning theory, but the precise definitions may differ. You should take context into account.

Very comprehensive answer. Thank you so much


text:
Should I use multi-armed-bandits or RL for a financial time-series problem?
If we take simple financial timeseries data(stock/commodity/currency prices), State(t+1) does not depend on the action that we choose to take at State(t) as in Maze or Chess problem.
Simple example: as states we can have the sum of the daily returns of 5 different ETFs. Based on that, we want to take action - either buy(go long) or sell(go short) in another ETF. No matter what we choose however, our action will not determine what the next state would be (we do not have any control of what the returns of those 5 ETFs will be tomorrow).
In that case of simple financial time series data, would multi-armed-bandit approach be more suitable?

Your agent's actions will (probably) not have much impact on the observed financial time series. However, they will make a large difference to other things - namely what stock your agent is holding and the account balance.
If you are happy to ignore the agent's current portfolio and balance as not part of your problem, effectively treating these items as infinite sinks, then yes a multi-armed bandit might be a reasonable solution. But then so might any other sequence-predicting algorithm, if what you are searching for is some kind of financial prediction of good times to buy or sell.
If the portfolio and cash balance are an important part of your problem, you should add them to the state and use reinforcement learning. You might do this if your goal is to model a single investor playing the markets.
Note that although it may be possible to use machine learning techniques to analyse markets, and base investments on advice of a trained AI agent, it is a very risky venture. There are lots of ways you can fool yourself into believing too strongly in your solution and you stand to lose significant amounts of money.

Thank you @NeilSlater! While playing with on-policy Monte Carlo algo I start to think that the same results could be achieved simply by taking the average return for every possible action(long/short) in every possible state. I mean we don't even need to explore/exploit... just find those values, perform walk-forward backtesting or s.th. like that and done. All that with RL starts to not make sense for me in terms of generating buy and sell trading signals.

The question is whether the rewards are i.i.d. In such time-series problems, the rewards are not i.i.d; the reward at timestep t+1 depends on the reward at timestep t. Therefore, even though the actions/states may be somewhat independent, the rewards should i.i.d in order to apply MA bandits. Otherwise, the problem is about RL.


text:
For specific tasks, is it better to fine-tune models on examples or just use prompting with the context of the task?
These days large language models cover a vast amount of topics and information, but I wanted to understand: For specific tasks, is it better to fine-tune models on examples or just use prompting with the context of the task?
For example, if I wanted to train a language model to do question answering for linear algebra, is it better to train it with examples of linear algebra problems and their solutions, or try out different prompts?

Fine tuning is superior, since the whole network specialise to solve a given problem only. A specialist will always beat a generalist in the specialised task.
That said, if the generalist network is large enough and the prompt is good enough to "turn-on" the right weights in the large network, the performance can be comparable. In a sense, the prompt is making the generalist network act as a specialist. This is assuming that the specialist network has saturated  its performances.
The paper The Power of Scale for Parameter-Efficient Prompt Tuning shows indeed how "Prompt Tuning" can be as good as fine tuning for large networks.

Thanks, i guess one question is how to find the optimal prompt that will approximate fine tuning

That is what "prompt tuning" and similar techniques do. Training the network + prompt  to be accurate, but keeping the network frozen.

https://ai.googleblog.com/2022/02/guiding-frozen-language-models-with.html

Thanks! That link is fascinating. I think they use a fixed vector, but human engineered prompts can use a variety of sizes so it might be worth trying out varying embedding sizes


text:
Is it possible to build a facial recognition system using a multi-layered DNN?
I am completely new to AI. As the title says, is it possible to build a facial recognition system using a multi-layered DNN? Would this be effective/practical?
I want to implement such a neural network in Python, without the use of any external libraries. I can predict that the operations would be too slow, so using numpy would be a must.

Yes, it is totally possible.
It's useful to fully understand the math and functionality of NN's.
Additionally, if doing it correctly, your whole DNN should only make up to a few hundred lines of code. Yet, there a few things to keep in mind, respectively you have to be capable of.

For fast training and real-time usage usage, GPU calculation is STRONGLY recommend (Yet, it also works without, just a lot slower. For learning to program it though, the CPU is better. I also started programming it on a CPU, but eventually I had to switch to GPU)

You have to teach yourself the math, especially for the backpropagation algorithm. This mainly includes multidimensional functions, partial derivatives. 3Blue1Brown has a small series where he explains a simple DNN with it's math in great detail. Also http://neuralnetworksanddeeplearning.com/chap1.html is a great article where the first 2 chapters explain everything in great detail.

It's tedious and hard at first. Especially when you go deeper and try to implement more complex algorithms like LSTM yourself. If you want to really go in to AI you should do it. If it's a one or two time thing, just use libraries.

I don't see any reason why not. Not using an OOP would make it unreasonable harder in my opinion. Especially when you try to combine your DNN code to actually use it for applications.


text:
How should I train the players in the game of tag?
I have a simple game of tag, where red player tries to catch the blue player. Red player wins if it catches the blue player in under 10 seconds, but if not, then blue wins.

My goal is to teach the players to play the game well. Both of the players currently use neural network which has six inputs:

x distance between players
y distance between players
x distance between player and left wall
x distance between player and right wall
y distance between player and top wall
y distance between player and bottom wall

one hidden layer which size is 5 and 4 outputs for moving up, down, left and right.
How should I approach training the players?
I am currently trying a genetic algorithm. I have chosen genetic algorithms because I do not have any training data, because I don't know the correct inputs and outputs.
I am training the players at the same time. I create about 50 games (50 blue and 50 red players) and the player that wins, stays alive and creates child which has mutation. Mutation basically means that I change 0-2 weights about (-0.25)-(+0.25). I keep the populations at the same size by increasing the birthrate of the smaller (losing) population. This approach has not yet yielded good results. I can see a little progress, but a lot of the wins seem random.
Does my approach make sense?

Does the blue player move deterministically in each game or randomly?  If it moves deterministically, a genetic approach will definitely converge quickly.  If it moves randomly, I'd use a reinforcement approach with a deep Q or double deep Q approach.

@DavidHoelzer: The question clearly states that both players are being trained, not just the red one. The movement appears to be deterministic, but the attempted direction of travel is driven by the agents' respective decision processes which are being trained adversarially

Thanks.  I apparently missed that the, "question clearly states."  @NeilSlater

Your approach should work in general, but there are a lot of key details that will make the difference between success and failure.
The most major issue is a number of hyperparameters that you have effectively chosen, and that could make a difference to the project:

Population size
Architecture of neural network
Starting values for neural network weights
Scaling of NN inputs. With supervised learning you want the scale to be constrained to somewhere roughly inside -2 to +2, but with GA weight search, it will depend more on starting values and mutation
Mutation amounts
Crossover - by not having crossover in your GA design you are missing out on one of the potential strengths of GAs, and your solution is closer to random search which will take longer because you cannot for example combine one agent's correct behaviour in the top-left quadrant with another's in the bottom right.

If you want to take the GA/NN crossover further, I recommend you look into the NEAT system. It has a nice mechanism for handling combining two neural network "parents" to make a functional child.
Alternative #1 - Reinforcement learning
Reinforcement learning (RL) addresses the same basic problem - you don't know the correct outputs for a neural network to train from input/output pairs, but you can tell when a player has been successful. Instead of managing a population of player policies that you use to search the space of all possible behaviours, RL manages one policy (per player) and calculates how to change it to improve performance, based on observing results.
In practice, RL is very effective at optimising this kind of game. You will want to start with some basics before tackling your tag game, but something based around Deep Q Networks will probably work very well for your environment. The dueling nature of your agents will mean you would need to write some custom agent code though, very few of the off-the-shelf DQN solutions are written for adversarial games and self-play.
Alternative #2 - Tree search
In your case it may be possible to discover the correct outputs, and train the neural network to learn them. If the movement is on a discrete grid (or effectively on that grid if you are using fixed movement increments), then you could use a negamax search to discover the perfect movement strategy from any pair of starting positions, and use that to create a training dataset. As an example, if the grid was 20x20 then you would have roughly 400x400 = 160,000 training examples covering perfect play for the entire game.
* Actually Negamax search will only work if play is turn-based, but there may be equivalent searches for simultaneous moves.

Thank you! What do you mean by "Starting values for neural network weights"? Currently my starting weights are random. Is there a better way to create starting weights?

@MaLa Random is fine, in fact usually required, but the range used can make a big difference on whether the resulting NN functions are likely to be useful for any specific problem - here's an article covering some options, although you should note this is more focused on supervised learning than GA/NN combination https://machinelearningmastery.com/weight-initialization-for-deep-learning-neural-networks

I was able to make the game work and because of that I can say that the GA approach at least makes some sense. I did the following changes:

Added crossover
Increased generation size from 50 to 300
Increased hidden layer size from 5 to 12 (I don't know does this actually make a difference)
Increased mutation
Added both players x and y velocity to input layer

It is hard to train both of the players at the same time because after one of the players learns to play well, they start to dominate and that makes the learning stop.
For the player that tries to catch (red), using the time it took to catch the opponent as a part of the fitness, was not a good idea because this seems to make the player (red) to rush. This also makes the fitness highest for those who had a weaker opponent.


text:
What would be the Bellman optimality equation for $q_∗(s, a)$ for an MDP with continuous states and actions?
I'm currently studying Reinforcement Learning and I'd like to know what would be the Bellman optimality equation for action values $q_∗(s, a)$ for a MDP with continuous states and actions, written out using explicit integration (no expectation notation).
The discrete case is
$$
q_*(s,a)  = \sum_{s'}\sum_r p(s',r|s,a)[r+\gamma \max_{a'}q*(s',a')]
$$
My thoughts for the continuous case are:
\begin{align}
q_*(s,a) &= \int_{s'}\int_{r}f_{s',r|s,a}(s',r|s,a)[r+\gamma\max_{a'}q_*(s',a')]drds'
\end{align}
Is this how it would look like?

I think your equations are alright.
Anyway, this is just a question of mathematical notation.
In measure theory, a discrete random variable $X$ is said to have a counting measure over it's support $\mathcal X = \{x_k,k=1,...K\}$.
We can define its distribution function as $F_X(x)=\sum_k p(x_k) 1(x \ge x_k)$.
With the distribution function, we can write the expectation as the measure-theoretical integration $EX=\int _\mathcal X xF_X(dx)$.
Notice the part $F_X(dx)$, which is not the usual $dF_X(x)$ notation for Riemann integration.
This is why many graduate-level texts prefer to measure theory to define integration and expectations - so that we can use one unified notation for discrete random variables and continuous random variables, and any random variable you can think of living in some abstract measure space $(\Omega, \mathcal F, \mathbb P)$.


text:
Examples where AI fails in revealing ways
For a short presentation about AI, I am looking for examples where AI failed and thereby shows the limits of itself.
I remember there was one example,where an image classifier was given an image of pink animals (I think sheep) on a tree and classified it as "Birds on a tree". I think this example showed what AI might do if the given example is not represented in the training data.
But I cannot find that example anymore (and I need a source).
Anyone knows of examples, which are documented, that I could give and show the problem in a similar way?

A quite significant issue is where some AI systems have mislabeled black people as being gorillas (I suspect a major cause is the training data being insufficiently diverse, balanced and representative). Two examples are:

In 2015, this occurred with Google's Photo App (e.g., see Google apologizes for algorithm mistakenly calling black people 'gorillas'). Note that, for at least 3 years, Google apparently "fixed" their system just by removing gorillas as being an option, with this explained in Google ‘fixed’ its racist algorithm by removing gorillas from its image-labeling tech.
Facebook's AI recommendation system had a similar problem (e.g., see Facebook apology as AI labels black men 'primates').

A couple of examples could be:

Image classifiers learning different properties than the actual target: many books reference the case of a perceptron trained on detecting tanks which learned to actually predict good or bad weather in the background, ignoring completely the tanks. First cited in What Artificial Experts Can and Cannot Do.
(notably this is most likely a urban legend, but still a very much realistic situation that anybody working in computer vision will face soon or later).

Amazon recruitment algorithm biased towards men: this is totally real and it has been analyzed in several paper, I'll just link the first one I found Encoded Bias in Recruitment Algorithms. Again another case which remember us that machine learning and AI in general are data driven. An algorithm will learn and always be limited by what's in the data, including stereotypes and prejudices in case of natural language processing.


text:
What is the difference between Neurosymbolic AI and Transformer AI
I'm looking at the AI timeline and I came across Neuro-Symbolic AI (being symbolic AI used in combination with deep learning) and Transformer AI (which I understand as neural networks that take context into account).
If I look at Wikipedia then I'll find that GTP-3 is a form of Neuro-Symbolic AI, but its name is clearly a giveaway for Transformer AI (Generative Pre-trained Transformer-3).
So I'm left wondering; what exactly is the different between these two?

There is a big difference between Neurosymbolic AI and transformer AI. Neurosymbolic AI is more rule-based and logical, while transformer AI is more creative and can learn from data.
Neurosymbolic AI tends to focus on using artificial neural networks to approximate symbolic reasoning, while transformer AI focuses on using transformer networks to learn language representations.
Neurosymbolic AI utilizes both symbolic and sub-symbolic representations of knowledge, while transformer AI primarily uses a symbolic representation. In addition, Neurosymbolic AI often incorporates techniques from artificial neural networks and machine learning, while transformer AI focuses more on learning from data through self-attention.


text:
How to reduce the number of clusters produced by the Markov Clustering Algorithm?
I have used the Markov Clustering Algorithm (MCL) to cluster tweets, based on their similarity. However, I got a too high number of clusters, and most of the clusters have only one tweet. Any suggestions to reduce the number of clusters?

The best way to reduce the number of clusters produced by the Markov Clustering Algorithm (MCA) will depend on the specifics of your data and how you want to cluster it. However, some general tips that may help include:

One way is to pre-process your data to remove noise and outliers. This can help to make your data more amenable to clustering and make it more likely that meaningful patterns will emerge.

Some methods that may be effective in reducing the number of clusters produced by MCA include downsampling the dataset, using a smaller value for the clustering parameter (such as the number of iterations), or using a threshold value to filter out small clusters.

Another approach is to play with the MCL parameters, such as the inflation rate, to see if that has an effect on the number of clusters produced. try increasing its value, this will lead to fewer, but larger clusters.

Use a different similarity measure.

you could try running MCL multiple times with different random seeds to see if that makes a difference.

Depending on the implementation you're using, you can adjust the granularity, which will influence how many clusters you will get.
See this description of MCL.

Thank you for your answer, you mean I adjust inflation parameter of mcl to get better clusers? Is that what you mean by granularity?

No, granularity is not the same as inflation. Are there no other parameter you can adjust?


text:
What is the difference between A2C and Q-Learning, and when to use one over the other?
I'm trying to get an accurate answer about the difference between A2C and Q-Learning. And when can we use each of them?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

Have a look at: https://youtu.be/2XKPLrcfTCo and "An actor/critic algorithm that is equivalent to Q-learning".

The major difference between A2C and Q-Learning are what the algorithms learn. In A2C, and policy gradient algorithms in general, the policy is directly parameterised, i.e. we have $\pi_\theta (a|s)$. The parameters $\theta$ are typically optimised to maximise an objective that is a proxy for the expected returns $\mathbb{E}_{\pi_\theta}\left[\sum_{i=0}^\infty \gamma^i r(s_i, a_i)\right]$. Usually this proxy is the value function $v_{\pi_\theta}(s)$; the policy gradient theorem shows us that the derivative of this function wrt the policy parameters is $\nabla_\theta v_{\pi_\theta}(s) = \mathbb{E}_{\pi_\theta}\left[G_t \nabla_\theta \log \pi_\theta (A | S)\right]$.
In Q-Learning, we instead learn the Q-function: $Q(s, a) = \mathbb{E}_\pi \left[r(s, a) + \gamma v_\pi(s') \right]$. In a tabular MDP we can maintain exact estimates for every state-action pair, but if the state space is continuous/too large then we typically rely on function approximation, and so the Q-function would be parameterised in a similar way to the policy above. Now, as Q-Learning (in the tabular case) can be shown to converge to the Q-function under the optimal policy, i.e. the value of taking an action in a given state and thereafter following the optimal policy. We can then define a deterministic policy using this by $\pi(s) = \arg\max_{a\in \mathcal{A}} Q_{\pi^*}(s, a)$ where $\pi^*$ is the optimal policy and $\mathcal{A}$ is the action space of the MDP.
Note that in Actor-Critic methods we also learn a value function, similar to how the Q-function is learned in Q-learning.
Another major difference between the two algorithms is that Q-Learning is an off-policy algorithm; that is, the policy that the values are learnt for does not necessarily correspond to the policy that collects the data. A2C is an on-policy algorithm, so the data must correspond to data collected by the policy. This has disadvantages in data thirsty Deep Learning setups, but A3C can help overcome this by using many workers that share the parameters of the current policy to obtain many on-policy trajectories, so the amount of data being used would be similar to that of an off-policy algorithm.
Finally, I want to point out that policy gradient algorithms don't necessarily have to be on-policy -- Soft Actor Critic and Deterministic Policy Gradient are both examples of off-policy algorithms.

Great answer. Another difference to consider is that A2C is an on-policy method, while Q-learning is off-policy.

Thanks, I might add that if I have time. I was trying to highlight the difference between value based and policy gradient, since policy gradient methods can also be off-policy (e.g. SAC, DDPG)

Take a look at this blog: https://mpatacchiola.github.io/blog/2017/02/11/dissecting-reinforcement-learning-4.html
In a nutshell, the major difference between the two algorithms is: Q-learning consists of a critic only (to update state-action values) while A2C is composed of two networks: an actor (to take an action) and a critic (to evaluate and update state-action values). Major advantage over Q-learning is its computational efficiency, as far as I know.


text:
How can we approximate infinite horizon MDP with finite horizon MDP in the context of reinforcement learning?
For a given value of "discount factor" (and reward values' range) in fixed finite horizon markov decision process (MDP), upto how many episodes we have to extend this MDP so that we can approximate the corresponding infinite horizon MDP?
I am actually working on a research project in which risk-averse MDP (RA-MDP) (with dynamic risk measures) are used which is by nature infinite horizon (this sentence may be irrelevant to answering the question, but I give it as a motivation and to make context). I want to solve this by using online optimization. For this, I am using risk averse actor-critic algorithm, as proposed by Coache et. al. in "Conditionally elicitable dynamic risk measures for deep reinforcement learning", which is the latest and the only RL algorithmic framework for risk-averse MDPs, but unfortunately restricted to finite MDPs.  On the other hand, my problem is infinite horizon. So I want to approximate this "infinite horizon" with "finite horizon" case.
Please help me if you can in this regard (authentic references are appreciable).

A few suggestions: https://arxiv.org/abs/2110.11383  https://journals.scholarsportal.info/details/00223239/v153i0003/688_aoaawfafcmdp.xml  https://github.com/chauncygu/Safe-Reinforcement-Learning-Baselines

I am assuming you are interested in the case where the objective function is the expected sum of discounted rewards. I realize this answer is probably not directly applicable in the RL context, but might be helpful nevertheless.
In a finite horizon MDP, an optimal action in a particular state depends not only on the state, but also on the time remaining until the end of the horizon. If there are a lot of decisions remaining after the current one, it makes sense to consider the long-term effects of the current decision. Conversely, if the current decision is the last one, maximizing the immediate reward is all one cares about. Therefore, the solution of a finite horizon MDP with horizon $T$ is a sequence $(\pi_T, \pi_{T-1}, \ldots, \pi_1)$ of policies, where a policy $\pi_t:S \to A$ maps the state to the action when $t$ decisions remain until the end of the horizon.
In an infinite horizon MDP, there is always an infinite number of decisions remaining. Informally, this is why a solution of an infinite horizon MDP is a stationary policy $\pi: S \to A$ that is used regardless of the time step.
It is a well known strategy to approximate infinite horizon problems by solving a finite horizon MDP with some horizon $t$, and then applying $\pi_t$ at every step in place of $\pi$ in the infinite horizon problem. To determine what $t$ needs to be in order to guarantee a certain approximation quality, we can consider the Bellman error
$$
\epsilon = \sup_s |V_t(s) - V_{t-1}(s)| \stackrel{def.}{=} ||V_t - V_{t-1}||
$$
which gives the greatest difference in absolute value between the successive value functions $V_t$ and $V_{t-1}$ obtained during value iteration. Now, if we denote the optimal value function of the infinite horizon MDP by $V^*$, we can prove
\begin{equation}
||V_t - V^*|| \leq \frac{\gamma \epsilon}{1-\gamma}
\end{equation}
where $\gamma$ is the discount factor. Now we can infer that if we want to approximate $V^*$ with precision of at least $\delta$, the Bellman error should satisfy $\epsilon < \frac{\delta(1-\gamma)}{\gamma}$.
We can finally conclude that a possible method of approximating an infinite horizon MDP with a finite horizon MDP with an approximation quality of at least $\delta$ (in terms of the Bellman error) is to compute by value iteration a sequence $V_1, \ldots, V_t$ of value functions until we reach a $t$ where the condition above is satisfied, and then apply the corresponding policy $\pi_t$  in the infinite horizon MDP.
A proof of these facts can be found, e.g., in the technical report "Tight performance bounds on greedy policies based on imperfect value functions" by Williams and Baird.

Thankyou @mikkola for your answer, it really helped.

great answer, but that's valid for RL, what about deep RL? at that point, usually bootstrap is used to update the network, however in the POMDP setting you need a belief, thus is the infinite horizon you have no idea when to apply gradient, because truncating it at timestep $t$ might avoid the NN to learn long term dependency


text:
In off-policy MC learning, why is the probability of sampling a trajectory the same as having a return?
In Sutton and Barto's RL book, in the section for off-policy learning, we would like to find the expected value of the random variable $G_t$, given $S_t = s$ under our target policy: $$\mathbb{E}_{\pi}[G_t|S_t = s]$$
So here, our random variable is $X = G_t$, meaning returns of states in trajectories/episodes.
Since we want to be able to explore, we use a behavior policy $b$, and therefore sample trajectories(episodes) from it. Computing sample averages doesn't work, as it would converge to $$\mathbb{E}_{b}[G_t|S_t = s]$$ and not $$\mathbb{E}_{\pi}[G_t|S_t = s]$$.
Hence, we use Importance sampling, and here is my understanding of how it works:
$$ \mathbb{E}_{\pi}[X] = \sum_{x \in X} x\pi(x) = \sum_{x \in X} x\pi(x) \frac{b(x)}{b(x)} \implies $$
$$=\sum_{x \in X} x\frac{\pi(x)}{b(x)}b(x) = \sum_{x \in X} x\rho(x)b(x) $$
and in this way, if we know $\rho(x)$, we can do this: $$\frac{1}{n}\sum_{i=1}^{n}\rho(x)x_i \approx \mathbb{E}_{\pi}[X], x \sim b$$.
In Sutton & Barto, in order to calculate this $\rho(x)$, the ratio between the probabilities  of the occurrence of a specific trajectory is calculated: $$P(A_tS_{t+1}A_{t+1}...S_T|S_t,A_t)$$
and then the ratio is calculated.
How is this the same as $G_t$? if we want to calculate the return of a specific return $G_t$, we could have many trajectories that give the same return, so we would be underestimating the probability of that return if we find the probability of occurrence for a trajectory.
Where is my gap in understanding?

The expected value notation $\mathbb{E}$ can easily cause mistakes, which I think is what happened here. It's not always clear what exact group of things and probably distribution the expected value is taken over. Your section on importance sampling is wrong: here the expected value should be taken over all trajectories with their on-policy distribution. It seems like you're mixing usages of $X$ as a placeholder and $x$ as an action or even trajectory.
Disentangling everything, we can write:
$$\mathbb{E}_\pi[X] = \sum_{h\in H} x(h) P_\pi(h) = \sum_{h\in H} x(h) \frac{P_\pi(h)}{P_b(h)}P_b(h) = \sum_{h\in H} x(h) \rho(h) P_b(h) = \mathbb{E}_b[\rho X]$$
with symbols defined as

$H$ for the set of possible trajectories
$X$ for any placeholder variable
$x(h)$ for the placeholder value for trajectory $h$
$P_\pi(h)$ and $P_b(h)$ for the respective on-policy probabilities for trajectory $h$
$\rho(h)$ the ratio between them

Looking at things this way, there are no issues with multiple trajectories potentially yielding the same value, since we consider each possible trajectory separately.

I see now. Unfortunately, right before this calculation, there was no mention of trajectories, it seemed to come out of the blue.


text:
How to define a loss function for multi-label problem?
I have voice recordings which are labelled by not only a single label but multiple labels. Each voice recording corresponds to one of class labels within a set. In other words, the training instance is given a set of (or distribution over) candidate class labels and only one of the candidate labels is the correct one.
I wish to train a model that classifies which class label corresponds to each voice recording. Each one of my voice recordings is accompanied by a set of 10 potential labels (labels are always different), but it is unknown which label it is exactly (aside from a small sample where there is only one correct label).
This is due to the nature of where my data comes from: someone records a short voice message and then types the same message into a chat, however there will be slight delay between the two and in the meanwhile other chat messages arrive. Only one of the next 10 chat messages after the voice message is the correct one that corresponds to that voice message.
How would I define a loss function in this case?

Do you have the ground truth label or just the set of 10 potential labels?

@RaphaelLopezKaufman Ground truth label is among the 10 potential labels

@MilTom " Ground truth label is among the 10 potential labels", but does this mean that you know what the ground-truth label (among these 10) is, or do you only know that the ground-truth label is among the 10 potential ones? It's important to clarify this, to understand if you can apply supervised learning or not.

@nbro I do not know what the ground-truth label is - I just know it is one of the 10 potential labels. I do have a small set that is one-to-one labelled, however most of the dataset is not.

Given your response in the comments, you are faced with a semi-supervised learning problem where you have a small set of data with ground truth labels, and a large set of data without ground truth labels.
If you look at a similar problem, predicting ImageNet label with only 10% of the ground truth labels, you can see that the best methods reach ~80% accuracy.
You can try to reproduce the approach of Big Self-Supervised Models Are Strong Semi-Supervised Learners, which achieves ~74% accuracy on ImageNet with only 1% of the labels.
Namely:

Unsupervised pre-training step. Train a deep and wide model on the unlabelled data. The idea is to minimize a contrastive loss using slightly modified versions of the training samples. The method they introduce is called SimCLRv2, which you can reuse in your problem by using data augmentation methods suited to audio samples. You can read on this topic in this Pytorch tutorial.
Fine tuning step. Fine tune the model obtained in step 1 on the small dataset for which you have the ground truth labels. To do this, replace the last few layers of your big unsupervised model with an MLP and train it with a classic cross entropy loss (just the MLP part, the rest of the model should be frozen).
[Optional] Distillation step. Use the model of step 2 to label the data for which you didn't have ground truth labels and train a smaller network in a supervised manner on these new labels (and only those, exclude the data for which you had the ground truth labels!).

I've simplified a lot the method described in the paper, to give you an idea of the general approach. Read the paper for details!

Would you mind accepting my answer if it's satisfactory?

If I understand correctly, the training data are voice messages which contain one or more suggested labels for the class that the voice message belongs to. Even though there are multiple suggestions, each voice message only truly belongs to one class.
If so, you could define the output layer of the network to have 10 nodes (the number of classes), with each node outputting values between 0 and 1 using a sigmoid function.
The loss to base optimisation could be the MSE of the sum of the element-wise difference between the input vector say [1,1,1,0,0,0,0,0,0,0] (the first 3 classes are suggested) and the output vector of the network [0.78, 0.76, 0.65, 0.23 ... 0.11, 0.36].
Once the model is sufficiently trained, for prediction you could apply a softmax to the output vector and select the largest value as the class to select.


text:
Time series prediction using LSTM and CNN-LSTM: which is better?
I am working on LSTM and CNN to solve the time series prediction problem.
I have seen some tutorial examples of time series prediction using CNN-LSTM. But I don't know if it is better than what I predicted using LSTM.
Could using LSTM and CNN together be better than predicting using LSTM alone?

You wrote "I have seen some tutorial examples of time series prediction using CNN-LSTM". Can you please link us to those tutorials? Moreover, you should probably describe your dataset (i.e. is it a sequence of images? or how many data points do you have?).

Have a look at https://stats.stackexchange.com/questions/10162/how-to-apply-neural-network-to-time-series-forecasting/10196#10196

Similar to https://stats.stackexchange.com/questions/578508/cnn-lstm-or-lstm-better-for-univariate-time-series-forecasting although no answers posted.

Please provide details on the time series. For example, is it rainfall in the Amazon, prices of airfares, or is it something chaotic like stock prices? In addition, is your data univariate or multivariate?

It could, but it doesn't mean it always will. I guess that's why it can be called  "Data Science" problem, because every dataset is different and needs its own little "research". So what I mean is that you have to implement CNN-LSTM model and see for yourself.

Many papers have been published on CNN, LSTM, and CNN-LSTM for time series. From the literature and my experience, I conclude that CNN-LSTM outperforms CNN and LSTM models. Here are two relevant papers on stock price time series forecasting:
Wenjie Lu, Jiazheng Li, Yifan Li, Aijun Sun, & Jingyang Wang. (2020). A CNN-LSTM-Based Model to Forecast Stock Prices. Complexity, 2020. https://doi.org/10.1155/2020/6622927
The authors' conclusions:

This paper takes the relevant data of the Shanghai Composite Index as an example to verify the experimental results. The experimental results show that the CNN-LSTM has the highest forecasting accuracy and the best performance compared with the MLP, CNN, RNN, LSTM, and CNN-RNN.

Chen, Y., Fang, R., Liang, T., Sha, Z., Li, S., Yi, Y., Zhou, W., & Song, H. (2021). Stock Price Forecast Based on CNN-BiLSTM-ECA Model. Scientific Programming, 1–20. https://doi.org/10.1155/2021/2446543
The authors' conclusions:

The proposed model is compared with CNN, LSTM, BiLSTM, CNNLSTM, CNN-BiLSTM, BiLSTM-ECA, and CNN-LSTMECA network models on three datasets. The experimental results show that the proposed model has the highest prediction accuracy and the best performance.

In my opinion, if your dataset has correlation with its neighbors and there is some kind of sequence involved in it, then CNN-LSTM would provide you with much better results than only CNN or LSTM.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

CNN can be used as good feature extractor to help LSTM layers train efficiently , normally when your input contains huge number of features (images , audio) and you need to project these features into learnable lower dimension space use CNN before LSTM


text:
What other Machine Learning techniques other than Neural Networks are there?
I know that there are three types of machine learning algorithms, supervised, unsupervised, and reinforcement learning, and that often neural networks are used to implement them. However, neural networks is said to be a subfield of machine learning, and deep learning a subfield of neural networks.
Thus, I would like to know:
Which specific algorithms/models belong in Machine Learning but are not considered neural networks? Any specific names for these fields?

There are many techniques (algorithms and models) in ML other than neural networks, for example

decision trees
support vector machines
hidden Markov models
Bayesian networks
linear regression
k-means
tabular reinforcement learning (e.g. tabular Q-learning)

If you pick any good book on ML, you will find more details about these.
I don't think these approaches fall into a subfield of ML with a name. Occasionally, some people may call them "traditional ML approaches" or something like that.


text:
What is the equation for $\pi_*$ in terms of $q_*(s,a)$?
I am trying to solve the following exercise from Sutton and Barto:

Sutton and Barto Exercise 3.27 Give an equation for $\pi_*$ in terms of $q_*(s,a)$

However, I am struggling to do so. I know that $\pi_*$ is the policy which will pick the action with highest return, given we know what the optimal action values are. So intuitively, I would express the optimal policy like this: $$\pi_*(\text{argmax}_a q_*(s,a)|s) = 1$$. To express it like this: $$\pi_* = \text{argmax}_a q_*(s,a) $$
seems wrong since $\pi_*$ is a probability. What am I not getting correct here?

An optimal policy is just a greedy policy with respect to the optimal state-action value function (which is unique for a given MDP). So, $\pi_* = \text{argmax}_a q_*(s,a) $ is almost correct - it should have been
$$\pi_*(s) = \text{argmax}_a q_*(s,a), \forall s.$$
In this case, $\pi_*$ is a decision rule or a function.
If you define $\pi^*$ as a probability distribution, then you can do something like this
$$
\pi_*(s, a)=
\begin{cases}
1, \text{if } a = \text{argmax}_a q_*(s,a)\\
0, \text{otherwise}\end{cases},
$$
$\forall s \in \mathcal{S}$.
If $\text{argmax}_a q_*(s,a)$ is a set, you can choose any of the actions in that set with any rule you want, and, of course, still ignore non-optimal actions.
This is true for finite MDPs (aka MDPs with finite state, action and reward spaces).
There's a result (Putterman, 1994) that states that there's a deterministic and Markovian optimal policy for an MDP. So, optimal policies aren't (necessarily) probability distributions. It's a matter of convention and convenience.

I am not sure the action and reward spaces need to be finite. Maybe you just need the rewards to be bounded and I am not sure about the action space. There are assumptions in that book, which I may check later. The specific theorem may be 6.2.7, and applies to infinite-horizon discounted MDPs with finite $\mathcal{S}$ (at least). One also requires stationary reward and transition functions.


text:
Is my derivation of the Bellman equation for $q_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?
I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:

Rewrite the Bellman equation for the function $q_{\pi}$ in terms of the three argument function $p$  (3.4) and the two-argument function $r$  (3.5).

Here are the 2 functions:
(3.4) :
$$p(s'|s,a) = P(S_t=s'|S_{t-1}=s,A_{t-1}=a)=\sum_{r\in\mathcal{R}}p(s',r|s,a)$$
3.5:
$$r(s,a) = E[R_t|S_{t-1}=s,A_{t-1}=a]=\sum_{r\in\mathcal{R}}r\sum_{s'\in\mathcal{S}}p(s',r|s,a)$$
Here's my solution. Is this derivation correct?
$$q_\pi(s,a)  = \\ E_{\pi}[G_{t+1}|S_t=s,A_t=a]= E_{\pi}[R_{t+1} + \gamma G_{t+1} |S_t=s,A_t=a]= \\ E_[R_{t+1}|S_t=s,A_t=a]+\gamma E_{\pi}[G_{t+1}|S_t=s,A_t=a] = \\ \sum_{s',r}p(s',r|s,a)r+\gamma \sum_{s'}p(s'|s,a)v_{\pi}(s') = \\ r(s,a) + \gamma \sum_{s'} p(s'|s,a) \sum_{a'}q(a'|s')\pi(a'|s')$$

Your work all looks correct to me. But I don't have the exercise 3.29 in my book, so I can't read the question for myself. It's not clear to me whether your last line or second to last line should be considered as the answer.


text:
How do neural networks learn specific features throughout the layers?
I was reading about convolutional neural networks and I came across such an explanation:

Consider detecting features in human face. The earlier layers of neural networks learn coarse features such as edges in the images and the latent layers learn more complex ( finer) features such as eyes, nose and etc

I am wondering why this is a true statement, namely how can we know that a neural network first starts by learning primitive features and then learns complex features. Could you please explain?

Where did you find that explanation? Can you provide the source?

The reason is that a layer can only learn a function based on it's inputs - that's clear. When you look at a convolutional neural network, each output pixel of an early layer only gets a very small fraction of the whole image as input. For example: If your first layer uses a 3 x 3 kernel, then each output only depends on only 9 pixels. Try detecting anything other then edges with that much information - it's not not possible. In other words: Looking at a 3 by 3 image you will be able to see an edge and its orientation, but not something like a nose.
In contrast, the later layers depend on a much larger portion of the image and can thus learn high-level features like a nose.
It's more a question of whats possible for the layers to learn and not a question of the network self-organizing it's resources.
And besides just reasoning like that you can visualize what the kernels in the individual layers respond to. I found this question that also has an image with some further information on that.
EDIT In response to the first comment:
To further clarify how a CNN works, here is an example, where $X_0$ is the input image and $X_1$ is the output of the first CNN-Layer:

Each convolution in the first layer receives a small (e.g. 3 x 3) area of pixels of $X_0$ and provides an output pixel. Here it is only possible to detect simple features (edges / gradients) because of the reason stated above.
The output of the first Layer $X_1$ is again an image, with the following properties:

A single pixel represents a 3 x 3 area in the original input
Effectively $X_1$ is a map that encodes the location of the simple features that got detected.

Because of the above properties, applying another 3 x 3 convolution to $X_1$ causes that

That convolution effectively depends on a 5 x 5 area of $X_0$, i.e. a 3 x 3 area of $X_1$
That convolution responds to certain arrangements of these 1st-level features, e.g. two adjacent edges with different orientations are a corner. You can think of the CNN-layers as a hierarchy where initial layers provide basic features the next layer detects compositions of these, the next layer detects compositions of the compositions and so on.

is it like the many basic functions (like the edge detections ) are added together and then create a function that fires high values when a "nose" detected ?

So yes, you can say it like this. This is exactly where the thought of hierarchy fits in.

So what you are saying is as layers progress the function learned become more and more complex in terms of number of input pixels , which can be regarded as capturing more complex features such as nose right ? The only thing that I did not understand how the coarser features are related to the high level features in the network , is it like the many  basic functions (like the edge detections ) are added together and then create a function that fires high values when a "nose" detected ?

Yes, you can say that - I extended the answer to clarify this a bit more


text:
What is the difference between "Syllogism" and "Law of Syllogism"?
The logical arguments are the basis for Artificial Intelligence. That is why I picked AI community to ask my question.
Reading from Wikipedia,

A syllogism is a kind of logical argument that applies deductive
reasoning to arrive at a conclusion based on two propositions that are
asserted or assumed to be true.

Again from Wikipedia, deductive reasoning,

is the process of reasoning from one or more statements (premises) to
reach a logical conclusion.

As part of deductive reasoning, it lists Modus Ponens, Modus Tollens, and Law of syllogism where Law of syllogism is defined as

In term logic the law of syllogism takes two conditional statements
and forms a conclusion by combining the hypothesis of one statement
with the conclusion of another.

Based on these articles, is it safe to assume that Syllogism uses Modus ponens, Modus tollens, and Law of syllogism to arrive at conclusion? Does that mean "Law of syllogism" is part of "syllogism" or is it something different?
P.S.
If this community is not appropriate for this kind of question, kindly guide me to the correct StackExchange or other communities.

A syllogism is a kind of logical argument that applies deductive reasoning to arrive at a conclusion based on two propositions that are asserted or assumed to be true.

This definition is somewhat misleading. It suggests that there is a logical reasoning in between premises and conclusion, so to speak, but there is none. The premises and the conclusion have to be regarded as the deductive reasoning itself. It is deductive because the conclusion is understood as following necessarily from the premises. And this is all there is to it.

Based on these articles, is it safe to assume that Syllogism uses Modus ponens, Modus tollens, and Law of syllogism to arrive at conclusion? Does that mean "Law of syllogism" is part of "syllogism" or is it something different?

"Law of Syllogism" is just one particular syllogism. It is more usually called "hypothetical syllogism":

If ϕ, then ψ;
If ψ, then ξ;
So, if ϕ, then ξ.

This is the expression of the transitivity of the implication (or of the conditional).
A deductive reasoning is a sequence of logical truths so articulated with each other as to justify the claim that a conclusion follows necessarily from the premises given.
A logical truth is an implication which appears self-evidently true, at least to most people.
The modus ponens and the modus tollens are logical truths since they are implications and they appear self-evidently true, but they merely are the most prominent of all logical truths.
All syllogisms are logical truths, including the hypothetical syllogism. So you can use any syllogism as part of a deductive reasoning. Deductive reasoning can involve any logical truth as long as it helps justify the claim that the conclusion follows from the given premises.


text:
Why does mean episode reward during training differ dramatically from "manual" runs of the trained model on same data?
I am training an RL agent, using PPO, on a time-series environment that comes from a tabular dataset. The possible scores during an episode goes from -1 to positive infinity (though realistically, I would never expect an agent to get an episode score higher than 2.5 or 3).
By the end of training (3 million time steps, ~1350 episodes), I can see that the agent has a "mean episode reward" of roughly 0.4 . For context, below is an image of the plotted of that mean episode reward over time.

So, then, my expectation is that when I take this trained model and run it on an environment consisting of the same data it was trained on, it should have a performance somewhere around 0.4 reward on average per episode.
However, this is not what I find. In my case, the fully trained agent's mean reward is -0.3 over 10 episodes (episode length equal to episode length during training) which collectively span the entirety of the training data.
Why is the fully trained agent, who was just reported to have +0.4 mean score per episode, performing so poorly when run manually through a for-loop on the same training data?
By "manually through a for-loop", I mean that I have a for-loop through which I feed the agent observations one at a time, call model.predict() to get an action and then enact that action etc.

You are right, those two numbers should match. To me, it sounds like some unexpected behavior in the code, so I would carefully go through the code again and again. I don't think someone can identify what's wrong based on your description.

@Chillston I see. Thank you for your input. I'm pretty new to reinforcement learning, so at this stage, it's just hard for me to judge what's common/typical behavior vs. what's an error. I should note that my environment comes from a tabular time series dataset, and on every episode during training, I randomly choose a "starting point" from the dataset. In "testing", I naturally do not choose the starting points randomly, but chronologically. So, maybe this randomness is the cause of the issue (maybe I "happen" to choose "bad" start points)?

@Chillston Perhaps I will run a few thousand episodes in "test" mode, randomly selecting a start point each time, to see if there's a difference between the average "test" result vs. the mean episode reward during training (to negate the effect of potentially unfortunate choice of starting points). Should I add some of my code to the question?

@Chillston It appears that that's the issue - when I run a bunch of trials with random starting points, the mean "manual" score is about the same as the mean episode reward.

Very nice, that makes a lot of sense! :)
Regarding the addition of code - in this case the question is more about debugging, this you should post on StackOverflow. But you sorted it out already anyways.

I have found the issue.
Essentially, what appears to be happening is when I do my (only) 10 manual runs, I happen to get a bad sample of starting points.
During training, for each episode, I randomly select a start point in the data so that the agent doesn't overfit to a single starting state. However, during my "manual testing", I just chose 10 chronological starting points to run episodes from. It appears that the 10 chronological starting points just happen to be inconvenient/bad, so the results aren't great.
To verify this idea, I ran 500 "manual for-loop tests" of the fully trained model by randomly selecting a starting point each time (just like I did during training). When I did this, the mean result across the 500 runs was quite similar to the final "episode mean reward" of my agent during training.
So, it appears to come down to the "luck of the draw" when it comes to the episode starting points.


text:
Is there a way to form a reward function so that it would take into account the order of the actions?
I want to design a multi-arm bandit system for a multi-step, multi-location system. Locations are dynamic, so I can not design the system based on them. In each location, the alternative actions that can be taken would be different. When you take correct actions, taken in correct locations, then some rewards would be earned. Some other alternative rewards can be incorporated in the system for the activities taken before reaching the correct state.
I know this may not be very clear. What I want to ask is "Is there a way to form a reward function so that it would take into account the order of the actions or the correctness of the order of the actions?".
Previously, I have implemented some other multi-arm bandit problems, but they were more straightforward. I need some ideas to help me to implement this new type of problem from some experts.

Make sure my edits did not change the meaning of your post (it's a good idea to have a descriptive title, which is actually your question - that's what I tried to do here). Honestly, it's not fully clear to me what you're asking here or where the confusion lies, though. You have different action sets for different locations, but apart from that your post is quite unclear. Are you familiar with MDPs? Please, provide a description of your actual problem, that would be more helpful, IMHO.

You are describing an environment which requires a full Markov Decision Process (MDP) to model it and reinforcement learning (RL) algorithms to solve it. You will not be able to adapt k-armed bandit algorithms, without effectively re-inventing MDPs.
The two key details that make this full RL, and not a bandit problem, are:

Decisions are sequential, with options and outcomes that depend on previous decisions.

Action choices make changes to variables (which in MDP would be part of the state description) that impact outcomes of future actions.

If you allow the agent to access the state including effects of previous actions encoded in a way that it has enough data to correctly predict rewards, then you have a normal MDP and most RL methods should be applicable.
If you do not allow the agent to use a convenient history of past actions (and/or their effects) as part of the state, then you will have constructed a partially observable MDP (POMDP) and may need a more advanced approach to solve it. For instance, using an RNN (most likely an LSTM or a GRU architecture) to process state sequence and predict action values could learn about the hidden sequence.
In terms of implementing a simulation of your environment, you will need to model it as a stateful system, and will have to include a concept of forward step in the sequence which modifies the state variables (regardless of whether these variables are made available to the agent in any observations). This would include the location information, and any other factors that change the allowed actions or outcome. As well as a step function, you will probably want a state reset function that puts the system into a starting state, or one of a range of possible starting states.
If your environment is episodic (a sequence can end), then you will need a way to flag that so that the learning agent can react to the end of an episode and request a new starting state.

Thank you Neil for taking the time. I already implemented it as a stateful machine and I change potential actions based on locations. There is also the end of states and as you said I already put  the agent in a starting state when it comes to somehow end. My question is related to the reward function. I know you mentioned some techniques like RNN(LSTM or GRU), but it would be helpful if you can clarify that part or maybe share some relevant links so that I may try to understand ?

@Ferda-Ozdemir-Sonmez: OK, the reason for my answer here is that you are calling your system a bandit problem, but it is not one. I am not clear what your problem with the reward function is?

@Ferda-Ozdemir-Sonmez - maybe this Q&A could help: https://ai.stackexchange.com/questions/22851/what-are-some-best-practices-when-trying-to-design-a-reward-function/22854#22854 ?

Neil Slater has a solid answer. In general you could have a bandit algorithm in which the reward $r \sim f(. | a)$, or an mdp where reward $r \sim f(. |s,a,s')$. Here $a$ is the current action, $s$ is the current state, and $s'$ is the next state. How you encode $f$ if a bandit or mdp is up to you whether deterministic or based on assuming a parametric family. Of course, an MDP is not the most general setup either, but describing a setting with a highly action/state dependent history is not simple to do for mathematical analysis (and coding too).
Hence if you want something more complex than what an mdp assumes, that's up to you to code/describe, or to initialize something more complex like an rnn as Neil Slater says and use that as your reward. Without nice data to train it though, an rnn reward function may not give very sensical rewards.


text:
What is MNLI-(m/mm)?
I came across the term MNLI-(m/mm) in Table 1 of the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I know what MNLI stands for, i.e. Multi-Genre Natural Language Inference, but I'm just unsure about the -(m/mm) part.
I tried to find some information about this in the paper GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding, but this explained only the basic Multi-Genre Language Inference concept. I assume that the m/mm part was introduced later, but this doesn't make any sense because the BERT paper appeared earlier.
It would be nice if someone knows this or has a paper that explains this.

as far as I have found out it stands for a different type of task.
I have found it here. https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb#scrollTo=kTCFado4IrIc
It states that mnli-mm stands for the mismatched version of MNLI
That means that the mnli-m would mean matched version of MNLI
On the following link is more about the MNLI: https://cims.nyu.edu/~sbowman/multinli/
Hope this helps you.
Cheers.