text:
How to derive "value iteration" from "policy iteration"?
This is the equation for "value iteration" from Sutton-Barto:
\begin{align}
v_{k+1}(s)=& \max_{a \in \mathcal{A}}\mathbb{E} \Big(R_{t+1}+\gamma v_k(S_{t+1}) \big|S_t=s, A_t=a\Big) \\
=& \max_{a \in \mathcal{A}}\bigg\{
\mathcal{R}_s^a+\gamma\sum_{s'\in \mathcal{S}}\mathcal{P}_{ss'}^av_{k}(s')\bigg\}
\end{align}
It is said that this is a special case of "policy iteration" when policy evaluation is stopped after just one sweep (one update of each state).  I do not understand how this gives rise to the value iteration equation above. Can someone provide a step-by-step mathematical derivation of value iteration from policy iteration with one sweep?

From S&L's value iteration section:

For arbitrary $v_0$, the sequence $\{v_k\}$ can be shown to converge to $v_*$ under the same conditions that guarantee the existence of $v_*$... value iteration is by reference to the Bellman optimality equation (4.1). Note that value iteration is obtained simply by turning the Bellman optimality equation into an update rule.

Therefore to derive value iteration from policy iteration as its extremely simplified case in a step-by-step manner as requested in your updated question is to formally express the above quoted "turning the Bellman optimality equation into an update rule". And indeed, as you commented, value iteration first does policy improvement and then policy evaluation (normally we have first policy evaluation and then policy improvement in GPI framework of RL). The key here is value iteration can only be rigorously derived from Bellman optimality equation (BOE), not from policy improvement theorem. And once you do the derivation you'll understand there's no conflict as complained in your comment.
So first see BOE (4.1) as follows which is already very close to your own quoted first equation:
$$v_*(s)=\max_{a \in \mathcal{A}}\mathbb{E} \Big(R_{t+1}+\gamma v_*(S_{t+1}) \big|S_t=s, A_t=a\Big)$$
To derive a practical algo to solve for optimal state value $v_*$ in BOE is simply to turn it into an iterative update rule as hinted above in S&B's book. So formally you just simply need to replace LHS's $v_*$ with $v_{k+1}$ and RHS's $v_*$ with $v_k$ which is exactly your requested equation! Your second equation is just further calculating the expectation  statistics with model-based RL.
Finally note the $v_k, v_{k+1}$ in your value iteration are not state values themselves because there's no policy evaluation based on state value here at all, thus you cannot rely on policy improvement theorem to prove convergence. To prove convergence of value iteration as hinted above you need BOE and its contraction mapping fixed point theorem to step-by-step prove the uniqueness and convergence of its iterative solution which is entirely another question. There's a diagram from Zhao's Mathematical Foundation of Reinforcement Learning to illustrate the non-conflict relation between value iteration and policy iteration as follows:

And the same reference specifically warned:

One problem of value iteration that may confuse beginners is whether $v_k$ is a state value. The answer is no although $v_k$  converges to the optimal state value $v_*$ eventually. That is because $v_k$ is not guaranteed to satisfy any Bellman equation... This might be one of the most confusing problems for beginners studying this algorithm. It is easier if we simply treat them as intermediate variables that emerge when solving the BOE.

That's why strictly speaking one shouldn't say policy improve (PI) in value iteration since there's no policy improvement theorem without state values evaluation (note all $v_k$'s in value iteration has no subscript $\pi$ compared to policy iteration), one should only say policy update (PU) in value iteration though it's also consistent with GPI (generalized policy iteration) as shown in above diagram.

unfortunately your explanations are  not clear for me. I want to see the equivalence through a step by step mathematical derivation.

What do you mean by $v_k$ and $v_{k+1}$ are not state values? Are you using ChatGPT or similar to generate your answers? They are what then? Unicorns?

@Neil Slater,  could you please help answer this question?

A good explanation is in this video but here it seems that now first policy improvement and then policy evaluation (normally, we have first policy evaluation and then policy improvement):  https://www.youtube.com/watch?v=64RljWXMpko

@DSPinfinity Your last comment makes sense and I edited to simplify my answer and tried to meet your added step-by-step derivation request.

@cinch, your answer does not provide a proof: you cannot use what is to be proved in mathematical proofs.


text:
Confusing point in Sutton-Barto: replacing $a$ in $q(s,a)$ with a stochastic policy $\pi^\prime$
Sutton-Barto, page 101, Eq (5.2):
Assume that $\pi^\prime$ is the $\epsilon$-greedy policy. Then,
\begin{align}
 q_{\pi}\big(s,\pi'(s)\big)&= \sum_{a}\pi'(a|s)q_{\pi}(s,a) \\
                           &= \frac{\varepsilon}{|\mathcal{A}(s)|}\sum_{a}q_{\pi}(s,a)+(1-\varepsilon)\max_{a}q_{\pi}(s,a)\\
& \cdots
\end{align}
I am confused by the first line in first equation. My confusion points are as follows.
(1) On the left we have $\pi'(s)$ but on the right $\pi'(a|s)$. Since $\pi^\prime$ is stochastic, how can one write $\pi'(s)$?
(2) Why does the first line in equation follow?

Sutton and Barto are being a bit loose with notation here.
If you had

a random variable  $\mathbf{X}$, with individual observable values $x$ drawn from set $\mathcal{X}$
a discrete probability function for $\mathbf{X}$: $p(x)$ where $\sum_{x \in \mathcal{X}} p(x) = 1$
and a function $f(x): \mathcal{X} \rightarrow \mathcal{Y}$ i.e that maps any $x$ to another type

Then you can process $f(\mathbf{X})$, the same function applied to all possible $x$. This returns the distribution for a new random variable $\mathbf{Y}$ - all the output values of $f(x)$ - with the probability of each value depending on $p(x)$
In Sutton & Barto, they are treating $\pi'(s)$ as a random variable for the action choice, like $\mathbf{X}$ (this is where they are being loose, it's not the deterministic policy). Whilst $\pi'(a|s)$ is equivalent to $p(x)$ in the above description.

In your first item, you wrote three different versions of the letter X. How do you pronounce them to differentiate them from each other?

@evaristegd There is an observable value, a set, and a distribution for the same variable. So you would say "the observed value x", or "the set of all possible x", or "the probability distribution for observing specific x" etc. For concise notation, it makes sense to use some variants of the same letter to show that they are related. I used the font variants that are typically used in S&B.

i would also say that makes no sense to define Q using a distribution as action, because that’s literally the definition of $V^\pi(s)$…

@Alberto except that is the point of the proof for Policy Improvement Theorem over epsilon-greedy policies, that this is from. It's showing it in a similar way to the deterministic policy version.

@Alberto - also, the value that results is neither $V^{\pi}(s)$ nor $V^{\pi'}(s)$, but something in-between. The proof eventually rolls out the change of policy over the whole trajectory, transforming $V^{\pi}(s)$ to $V^{\pi'}(s)$ and demonstrating $V^{\pi}(s) \leq V^{\pi'}(s)$

@NeilSlater sorry, it didn't wanted to sound like a critique of your answer, but just of the notation used in general

@NeilSlater, what do you mean by "distribution "?  What is the difference between distribution  and p(x)? I think by "distribution "  you mean a random variable "", right?

@DSPinfinity Yes, random variable is what Sutton&Barto call it, and probably the better term. However, we do want to make the distribution of it explictly known.

@NeilSlater  Then, could you please update your answer by using the term random variable  for "distribution "?

@DSPinfinity done

@NeilSlater, I think still your explanation  needs an update to make the connection between  $X$, $f(X)$ and $Y$ and the  stated problem variables and/or expressions. I.e., what are  $X$, $f(X)$,  $Y$  in the stated problem? It will be perfect if in your answer you make this correspondence clearer.

LHS $\pi’(s)$ is just $\pi’(.|s)$ which denotes the whole probability distribution of the said policy given state $s$, while your RHS $\pi’(a|s)$ is just a specific conditional probability when the random variable A=a, given the same state $s$. Once you understand this, you can also immediately see the reason for the first line of your equation.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Violation of Markov property
Consider the following cases:
a-) In solving an episodic problem we observe that all trajectories from the start state to the goal state pass through a particular state exactly twice.
b-) In solving an episodic problem we observe that some trajectories from the start state to the goal state pass through a particular state exactly twice.
Which case(s) violates Markov property and why?

Where did you take this exercise from? Please, edit your post to provide it and if this a homework, tag it with [tag:homework]

Markovian environments contain everything the agent needs to make an optimal decision in the present state.  There cannot be a state where an agent passes through it once and the optimal action is one action and when it passes through it again, the optimal action is a different one.
"b-)" is likely Markovian.  The simplest example is a triangle of points.  To one of the points of the triangle let us make a path from the starting state.  From another point let’s make a path to the terminal state with some fixed probability.

Trajectories through this arrangement pass through any given point not only twice but sometimes none (A, B, end) or many times (A, B, C, A, B, C, A, B, end).
"a-)" on the other hand is non-Markovian because the trajectory passes through a point twice and at some other point exits deterministically.  If exiting was always optimal we’d see some paths with one or zero traversals of a given state.  So the optimal policy must be determined based on something not in the state, violating the markov property.

(x,y) here marks the probability of transition on the first pass (x) vs second pass (y).
Extra:
To be exact, there is a markovian representation of this graph but states where we pass through twice are two distinct states which doesn't fit the wording of the question.

Could you re-express the last paragraph because it was not clear to me?

I gave it a shot and added some diagrams.

Really, very hard to understand your explanations and diagrams.

I've tried to be more exact in my wording once more and removed the errant A in the graph.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy?
This is question 6.13 in Sutton-Barto,page 136.

What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy?

The answer is given as follows:

Let $Q_1$ and $Q_2$ be the two action-value functions and let $\pi_1$
and $\pi_2$ be their $\epsilon$-greedy policies, respectively. Then, the updates would be:

$Q_1(S_t, A_t) \leftarrow Q_1(S_t, A_t)+\alpha\Big[R_{t+1}+\gamma \displaystyle \sum_{a}\pi_2(a|S_{t+1})Q_1(S_{t+1},a)-Q_1(S_t, A_t)\Big] $

$Q_2(S_t, A_t) \leftarrow Q_2(S_t, A_t)+\alpha\Big[R_{t+1}+\gamma \displaystyle \sum_{a}\pi_1(a|S_{t+1})Q_2(S_{t+1},a)-Q_2(S_t, A_t)\Big] $

Could someone explain how they were obtained similar to a reasoning corresponding to double Q-learning?

I thought you had the official answers already? You do know you can get them from the author?

Yes, I have but they just put the equations without any explanation.

Now, I also put the equations  here. What I need is a good explanation on how they were obtained similar to a reasoning for double Q-learning.

That's not how they were obtained though. The two component ideas - double-learning, and expected SARSA - make entirely different changes to the formulation of the update rules, and are compatible. So you simply make both changes: Sum over expected results from policy, and alternate action choice table vs bootstrap estimate table. There's no need to derive these from scratch, and no benefit to doing so at all. In fact I think that defeats the purpose of understanding what is going on - having a feel for these changes as "operators" that modify the equations is more useful and intuitive

What are "action choice table" and "bootstrap estimation table"? Where is the "alternate action choice table vs bootstrap estimate table" in the given updates?

This is a tabular method, so Q(s, a) is a table


text:
Can GPT-3 write an entire program if provided with detailed UML and ER Diagrams?
We already have tools like Denigma.app & Figstack.com (AI-powered code explanation in English), Codesee.io (Codebase and dependency mapping).
There's also a tool called Enzyme.so which converts English instructions to code.
All of us know about the capabilities of Github Copilot, OpenAI's Codex and recently Deepmind's AlphaCode.
Taking all these advances into consideration, is there a possibility that GPT-3 or any sufficiently advanced engine can generate an entire codebase or program out of thoroughly detailed UML and ERD documents?
Have there been any advances in this direction already?

UML created strictly enough can drive non-AI code generators such as Rational Rose - https://whatis.techtarget.com/definition/Rational-Rose - that's not really an AI system though. I doubt GPT-3 or derived tools could do it, as they are typically not coherent enough to tackle large systems. I don't know enough about the topic to answer solidy, but thought the RR link might be helpful to you or anyone thinking of answering


text:
Mask R-CNN: how is the inference done?
According to the Mask R-CNN paper and the picture below (taken from the paper), the mask branch is computed in parallel with the bbox classification and regression branches.

However in the paper they write that inference is done differently from training, not in parallel:

Inference: At test time, the proposal number is 300 for the C4
backbone (as in [36]) and 1000 for FPN (as in [27]). We run the box
prediction branch on these proposals, followed by non-maximum
suppression [14]. The mask branch is then applied to the highest
scoring 100 detection boxes. Although this differs from the parallel
computation used in training, it speeds up inference and improves
accuracy (due to the use of fewer, more accurate RoIs).

How is this actually done? By construction, the masks are outputted in parallel with the bounding boxes. So how can they run the mask branch after the bounding box prediction? Do they run it twice?

Can you please put your specific question in the title? "Mask R-CNN inference vs training" is not a question and it's also not specific.


text:
Mask R-CNN: How are the computed masks projected back to the input image?
The computed masks by Mask R-CNN are of fixed size $m \times m$ each. How are they projected back to the image?

Is this the same question as your previous one https://ai.stackexchange.com/q/34551/2444? If not, can you clarify why. If yes, I recommend that you delete one of the two.

No it's a different question, about how the masks are back-projected to the input image.

Each computed mask is simply resized to the corresponding computed bounding-box. For example, using OpenCV:
mask = cv2.resize(mask, (bboxW, bboxH), interpolation=cv2.INTER_NEAREST)

Then, after converting it to a binary mask by thresholding it, you can overlay this mask on the input image using the coordinates of the corresponding computed bounding-box.
See here for more details.


text:
Why Soft Actor-Critic (SAC) uses a double Q trick?
Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, even so, it still uses the double Q trick. So it confuses me that if it doesn't serve the purpose of mitigating the maximum overestimation bias, what is the real intention of this use?

Just because the policy is stochastic, it doesn’t mean that we aren’t trying to take a maximum over the Q function.

@DavidIreland, thanks for your reply! What you mean is that actually all algorithms have the issue of the maximum overestimation, just differ in degree?

More or less. Algorithms like SAC and TD3 use the Q-function as the objective for the policy to maximise, so this problem is inherent even though it may not be explicitly obvious. SAC and TD3 use two critics to avoid overestimation bias. If you sample an action from your and the Q-function gives an overinflated estimate of the returns, then your policy will be updated too optimistically, and so using two Q-functions and taking the minimum over these two for the policy update is a way of trying to control the overestimation bias.

Of course, 2 is just chosen to trade-off between computational efficiency and dealing with the problem. You could choose any reasonable number (choosing a high number would give a pessimistic value, which would potentially slow down learning too, but I think has a less dramatic effect than overestimation bias).

@DavidIreland That means there are two sources of overestimation bias, one is from the Q function maximum (like in DDPG and TD3), the other is from the policy maximum? The latter seems like the main stream of such a bias in SAC, since SAC uses a stochastic policy, which chooses action randomly, rather than deterministically.  If this holds, however, can I infer that the overestimation bias of policy maximum is not as severe as the Q function maximum?


text:
Using "softmax" (non-linear) vs "linear" activation function in Deep Reinforcement Learning
I am following the tutorial in this video: https://youtu.be/cO5g5qLrLSo which implements deep reinforcement learning (DQN) to balance cart pole in OpenAI default environment.
The DQN model looks like as follows:
model = Sequential()
model.add(Flatten(input_shape=(1,states)))
model.add(Dense(24, activation='relu'))
model.add(Dense(24, activation='relu'))
model.add(Dense(actions, activation='linear'))

Full code is available here: https://github.com/nicknochnack/TensorflowKeras-ReinforcementLearning/blob/master/Deep%20Reinforcement%20Learning.ipynb
The implemented code is also available on Google Collabs for you to run/test here: https://colab.research.google.com/drive/1oQILItVu6Y8jOCprzwMGzwlztYmVKK-F?usp=sharing
I do understand the concept and mathematics behind using "linear" vs "non-linear" (softmax) activation function in the output layer.
But, what I am struggling to understand is that why in this target application linear activation function is used in the output layer instead of softmax? Can someone specify how to realize that which type of activation function will be the best for what type of target application using DQN?
P.S. I tried to change the activation function to softmax instead and got completely different result. Hence, I am confused as to why changing the activation function in the output layer could generate completely opposite results.

The normal use case for softmax in the output layer is for a classification problem, where the output is an array of probabilities for each class.
The normal use case for a linear output is for a regression problem, where the output is an array of floating point numbers that are estimates for some measurement.
In a DQN, the desired output for the neural network is the action values (or Q values) for each action. The action value is the expected sum of future rewards given the starting condition of state and action choice, plus following the current target policy from that point onwards (this is a little self-referential in DQN in that the target policy is defined by taking the action with the highest action value in all states).
It is entirely possible for this predicted action value to be greater than $1$ or less than $0$, and there is also no reason to expect a the set of action values for all actions possible in a specific state to sum to $1$. This is a regression problem, and you should generally use a linear output for such problems.
Some Deep RL will learn a policy function, either instead of, or in addition to the value function - for example REINFORCE or A3C. In that case, if the action space is discrete, then the policy function will behave a lot like a classifier (it classifies the preferred action for a given state), and you would use a softmax output for that network. Other action spaces may be described differently though, so you are not guaranteed to see softmax used in output layer of policy function networks.

Hello Neil, thank you for your answer on this. Will it be possible to elaborate why we would use softmax with an example of REINFORCE or A3C? Because we can also use linear in that case as well, isn't it? Or am I missing something?

You would not use linear activation to define a policy for REINFORCE or A3C, at least not for discrete action space. However, some action spaces can be described using linear outputs - the output might be the mean and variance for a Normal distribution for example.

@SomdipDey I don't understand sorry. The phrase "the RL has a policy on it" does not make sense to me. In general, you should understand first what the neural network is doing inside the agent before choosing the activation function, because it makes a big difference. Different RL methods, and different action spaces will affect this. If you want to experiment by changing the activation functions, that is fine and might help you learn what they do. Mostly, inside RL agents, you will discover the activation function is constrained by what it is modelling, and changing it will break it.


text:
Deep RL reward design for neuron centerline extraction task
As part of a bigger scope project, I'm training a RL agent that attempts to reconstruct, pixel by pixel, the trajectory of a neuron on a segmented image. To give a better insight on the task that I'm trying to achieve, here is an example of an attempt on simple vessel ground truth image (red : vessel/ground truth, green : agent trajectory, yellow : overlap between the two) :

Details on the environment and the model :
At each step, the agent is picking an action based on the output of a Deep Q-Network and chooses the next pixel to move the cursor to. It can move in any of the 8 directions (in diagonal too). The goal is to match the red curve with the green one by updating each step the position of the cursor. At the moment, I'm training the agent with a few different such ground truth images and for a fair amount of episodes and it's really struggling at sticking close to the red line. I set a maximum number of pixels to stop the agent after a while in case it doesn't approximately match the red line (which it does not). I'm used to applying RL to games like Atari etc, but here it's a bit more abstract to determine what would be a smart way to design the reward system.
Question :
At the moment, I'm using the completeness score that compares what fraction of the pixels of the dilated red line are covered by the agent's path as part of my reward system. If the completeness increases as compared to the previous step, I give a reward of +1, if it doesn't, I give a reward of -0.1 (to penalize the agent adding useless pixels). I've tried a few other things like directly using the completeness score as the reward, but this technique is the overall best I can achieve, but I believe that my system is not very optimal which might cause the poor performance, as I think that the design of the reward system is essential to the good performing of a RL agent. I'm therefore open to any suggestions on how to improve or re-think the reward system by anyone that's got a bit more experience than me in designing RL agents when it's not as obvious as for some simple games.
Thanks in advance for your help !

Can you please put your specific question in the title? "Reward system design for Deep RL task" is not a question and it's also not specific. Thanks.

Have you tried to base the rewards on the distance to the correct trajectory? For example, when the agent deviates a lot from the desired trajectory as it does in the picture you gave, it should learn that this is unwanted behaviour. In a pixel world, you could use the Manhattan distance to compute this reward.
What are the observations provided to the agent? Does it see the trajectory it should learn to follow?
You can also experiment with various functions to shape the reward. You want to do this because in many cases a binary reward doesn't provide enough information to the agent about what it did right or wrong. I have had great success by using an exponential reward which increases as the agent gets closer to the goal or target trajectory (see this paper on reward shaping)
Furthermore, you also want to prevent the agent from learning a positive feedback loop where the agent never attempts to reach the goal. Essentially, the positive feedback loop is one where the agent learns that it can gain more reward by taking a series of action that does not reach the goal. One way to prevent this is by giving a larger reward when the agent reaches the goal. In your case, if I understand correctly, the goal is not the end of the trajectory but reaching some satisfiable % level of completeness.

I'm feeding the binary image as an input to the convolutions of the DQN, so not directly the ground-truth path, as it will ultimately be used on a distance map output of neuron imagery. This means that I will not necessarily have access to the location of the ground truth pixels but probabilities. Thanks for the article, it looks like I should indeed scale the extent of the reward to how far the agent has gone into the correctly segmenting the line (as I do not always have a stopping condition).


text:
Why does my regression-NN completely fail to predict some points?
I would like to train a NN in order to approximate an unknown function $y = f(x_1,x_2)$. I have a lot of measurements $y = [y_1,\dots,y_K]$ (with K that could be in the range of 10-100 thousands) which come out from either a simulation or a measurement of a system. I've built a feed-forward NN for solving this problem by using a MSE loss-function, i.e.
$$\mathcal{L} = \frac{1}{K}\sum_{i=1}^K(y_i-\hat{y}_i)^2$$
where I defined as $\hat{y}$ the prediction of the NN. As per activation function I used a ReLU. The network topology is fairly simple with input layer having two neurons $(x_1,x_2)$, three hidden layers with 10 neurons each and finally the output layer (single output neuron).
After the training process expires I've obtained a very curious result. The loss function assumes very small values hence (apparently) indicating that the training is successful. However, if I analyse the squared-error point-by-point, i.e. the quantity
$$ \boldsymbol{\epsilon}_y = [(y_1-\hat{y}_1)^2,\dots,(y_K-\hat{y}_K)^2] $$
I find that for the vast majority of points said quantity is basically zero, with exception of some "outliers" where the error is huge. It looks like this event happens where the gradient of $f()$ is rather big, which might be a reasonable assumption maybe.
I would like for this to not happen anymore. I'd rather accept a slightly bigger error throughout the whole function domain then have the majority of points with null error but some local points that are completely off. As a requirement, the network topology shall be kept rather "easy" so I would not like to increase the number of layers and/or neurons per layer. As a side node, I've also tried to increase the topology complexity a little bit (i.e. 15 neurons per hidden layer and adding a 4th hidden layer) obtaining slighlty better results but still unacceptable error around the steepest points of the function.
I've got two ideas for now:

Use a different loss-function $\mathcal{L}$
Sample the dataset $(x_1,x_2)$ more frequently around steepest regions, and less frequently where the function is rather smooth and flat

First option
A different loss-function to be adopted. I'm not very familiar with loss-function that might help solve my problem, some rather quick research yielded no good results and I found no significant literature highlighting this kind of problem. I initially thought of something of the likes of
$$ \mathcal{L} = \frac{1}{K}\sum_{i=1}^K(y_i-\hat{y}_i)^2 + \mu\, max\{\boldsymbol{\epsilon}_y\} $$
where $\mu$ is some tuned parameter that penalizes the maximum error. I'm not sure if this makes sense and if it increases the training complexity to the extent that the training process runs in no convergence territory.
Second option
I am not sure how to formalize the undersampling where the function is smooth and flat, I imagine that some kernel used for image processing (edge detection kind of kernels of some sort?) might be helpful but I'm in completely unknown territory.
Conclusion
I'm looking for some insight and ideas (literature results or analogous cases linked are a huge plus!) for solving this curious problems.
Thanks for the help!
Edit
I realize that without having graphical indsights of what I'm doing, it's going to be rather difficult to have a clear understanding. The function $f(x_1,x_2)$ is plotted here below (let alone the red line, it is just there to highlight what happens when $x_1=\text{M}=1.2$ is fixed to a certain value). The missing points of around the variable $x_2=f_n=1$ represent a singularity in my model and I don't care about them being predicted since they physically have no pratical use. I removed them completely for this reason.

Once the training of my NN is complete, I plot the $L2$ error (MSE) between predicted and true value

As you can see, it looks like the points where there is a significant gradient change cannot be predicted correctly.

Can you please put your specific question in the title? "Neural network for regression problem: inaccuracies" is not specific and it's also not a question.

@nbro I changed the title, maybe it is a little more specific. It is difficult to put in few words

You could try to replace MSE with the recently proposed General and Adaptive Robust Loss Function, a generalization of several losses among which l2 norm (which aside from a square root is equal to MSE). The neat property of this loss is that it uses a dynamic parameter $\alpha$ which makes the loss behaving differently, that can be learned during training. In this way the loss itself adapt to specific outliers in your data (the authors made a video with great visuals to explain this point).
I suspect at first glance (i.e. without data examples) that the problem your dealing with is that steep points are treated by your network as outliers, i.e. your model learn to generalize a smooth function cause the function is smooth most of the time, and surely also cause the amount of parameters of your network is really small (so it has a hard time fitting local irregular patterns). If it true that your function is smooth most of the time than oversampling steep points would be a mistake, cause that would mean training your network on a non representative sample of points, hence if this adaptive loss doesn't improve much your results I would just make the network larger.

This looks like a nice trick. Anyways, it looks like it makes your regressed model "outlier independent" . I read the paper very quickly, hence I might be misunderstanding the majority of their proposal. I will definitely try this trick! Tyvm


text:
Using conditional probability as an estimate in a loss function
I have a rather large ML framework that takes multiple conditional probability terms that are computed via classifiers/neural networks. This arbitrary loss function is computed via a function:
loss_value = arbitrary_loss(probability1, probability2, ..., P(Y|Z))

I wish to have an end-to-end framework that computes everything and trains everything together. So I do not want to have an independently trained classifier. Say at some point I develop some intermediate values (embeddings) Z from the input samples X. I wish to model the conditional probability P(Y|Z) via an MLP softmax layer.
This term P(Y|Z) is then estimated and plugged into the final loss which is the sum and product of other probabilities.
P(Y|Z) = MLP(input_Z) #probability given input Z over labels

My issue is that if I simply take the value of the softmax layer to estimate this probability and plug it in, at no point are the true labels taken into account for a supervised machine learning problem.
How can I fix this without modifying the final loss function?
TLDR: I need a probability term P(Y|Z) modeled via an MLP softmax layer to be used in a complex arbitrary loss function. How do i ensure this term is accurate via the true label values, so that it can be used in the final loss?

Thanks for the update.  Can you tell me for which variable you have access to the true labels?

of course, lets say i have a variable 'labels,' corresponding to the true labels of inputs. Lets say inputs go through a sequence of multiple deterministic mappings until they become 'input_Z'. the variable P(Y|Z) corresponds to the output of the MLP given the intermediate variable 'input_Z'. I hope this clarifies things, thank you for your time!

You can simply add CE loss to your existing arbitrary loss function.

Thank you @cinch ! I should have updated the answer. Yes this is what I have resorted to and implemented. It works!

your question is rather vague. A trained classifier should already estimate conditional probability of Y given input Z.
if your question is how to train such a classifier the answer is via CrossEntropy loss. so you have the output probability from your (untrained) classifier network and also the grountruth. CrossEntropy(prediction,grountruth) would be your loss. Here you can find more explanation
https://medium.com/swlh/cross-entropy-loss-in-pytorch-c010faf97bab
if you have a different question please edit your question.

Hi @saleh, I have edited the question.


text:
What are the specific differences between vision transformers variants?
I have tried 4 different types of attacks on vision transformers (ViT small and tiny, DeiT small and tiny) but the attack successes on smaller versions are higher than the tiny versions. My understanding is that the smaller versions are stronger than the tiny ones. I expect the results to be higher on tiny than smaller versions.
I am really confused. Is there any explanation for this?
What are the specific differences between the vision transformer models apart from the number of parameters and the embedding dimension?
Edit: I am loading these models from timm


text:
Possible improvements to WGAN-GP output images
I am mapping rather complex data into what essentially amounts to a greyscale image to take better advantage of GANs for generative means. Here is an example of some real data:

All real data is of the same shape (108 across x 12 high), with an enforced border region of two pixels along each edge - so the actual data is captured by the central 104x8 pixel data. There is significant coupling between individual pixels, so I applied a CNN in the network to try and learn these dependencies, with unequal kernels because the image is rectangular. A rough description for each row of pixels is as follows (note that each row is normalised between 0 and 1 for the network, with 0 corresponding to border region):

Integer values -1 -1 -1 -2 -1 -1 -1 -2... etc.
Very small float values (~0.03) with a break in the middle (~0.7)
Integer values -2 -2 -3 -3 -2 -2 -3 -3...
Float values
Integer values -3 -4 -4 -4 -3 -4 -4 -4 -3...
Float values
Integer values (one hot of 24, moving in blocks of 4)
Integer values (one hot of 4, repeating in blocks of 4)

I have applied the WGAN network with gradient penalty implemented in PyTorch. The general generator network has this architecture:

ConvTranspose2d
LeakyReLU(0.2)
BatchNorm2d

With a sigmoid at the end. The critic looks like:

Conv2d
InstanceNorm2d
LeakyReLU(0.2)

With no activation at the end of the critic. I have a dataset size of 1795, sadly I don't really have any conventional means to increase this. I train the critic for 5 rounds for each round of generator training. After 900 epochs, an example of the generated output is as follows:

So looks quite close, but converting this back into the actual form of my data results in some weird results because of the variation in pixels along the float rows.
There are also some weird bits on the side (it doesn't seem to have properly learnt the border). The first few pixels in the rows are also unusual in the real data, with the first few for some of the rows basically being a border region - this behaviour seems to have been learnt for some rows (e.g. row 5), but not others. This may be because of my unequal kernels, I'm not sure.
Some current improvements I'm considering are to add gaussian noise to the data (this may help as it removes the integer aspect of some pixels), augment the training set both with noise and with conventional methods such as rotating the image 180 degrees, adding extra border regions and translating the images etc.. If anyone has any insight into how I can further improve output I would really appreciate it. This image is actually a simplification of the dataset, and eventually, I want to have less regular sequencing in the pixels. Thanks!
(Edit)
Here is an image of the critic (D) vs generator (G) loss over the 900 epochs used to generate the fake data:

do you have some plots of the losses? it would be helpful to know at the very least if the model already converged or not (gans can take days to converge even for small datasets like your) and if the generator vs discriminator losses are correlated with each other or behaving randomly. A plot of the gradient penalty would be insightful as well, in my experience sometimes it can blow up to insane values and take over the whole adversarial loss.

Hi, thank you for your comment! I don't have a plot of the GP right now but I will try and recreate one (I didn't know it could blow up like that actually), I've included a plot of the generator and critic losses.

The purpose of the generator is to generate images. You don’t add sigmoid to your generator architecture


text:
Why CNN inference works on larger images
I have been reading up on 'regular' CNN's such as Mask R-CNN, and as far as I understand it they rely on a fully connected layer in the end to classify pixels. FCN's (such as U-Net) which do not use these layers are able to effectively process images of any size. I have been wondering why it is that Mask R-CNN can still process images of larger sizes. Doesn't that mean that a lot of data in the image will be unused in the fully connected layer?
For example consider a Mask R-CNN model trained on 512x512 images, that then does inference on a 2048x2048 image. I would expect that to result in an error or at least very poor performance, but according to my results this is not the case.
I also noticed that for some CNN models the image dimensions have to be some power of 2, is that due to implementation or also something that has to do with the connected layers?

Edit:
I've now found that the answer regarding image dimensions is here, it is due to the downsampling and upsampling.

Your question is not about all CNNs, but about FCNs, so I would suggest that you change the title to be more specific and precise.

My question actually is about CNNs, as FCNs can process larger images by design. If there is any other term that you can suggest instead of CNNs, I am happy to change it!


text:
Conceptual clarification on Value-based Reinforcement Learning
In slide 27/66, it states that "This is the maximum value achievable under any policy". My understanding of Value-based RL is that the optimal Q-table/function will be learnt. Afterwards, the implicit optimal policy can be "derived" by greedily picking the actions based on this learnt Q-value.
Does the implicit policy not matter here?

Probably related: https://ai.stackexchange.com/questions/40540/how-do-we-get-the-optimal-value-function

I have not idea what you're asking here. The implicit policy, which you don't define, is supposed to matter to what? Reformulate the question in a more specific and clearer way, please.

This is the maximum value achievable under any policy

The statement says that $Q^*(s,a)$ is the maximum possible state-action value that is achievable (at most) by any policy, $\pi$: including the random policy, and the optimal policy. It is also explained in slide 23:

"An optimal value function is the maximum achievable value"
"Once you have $Q^*$ we can act optimally"
"Optimal value maximizes over all decisions"

Indeed, the optimal policy $\pi^*$ would always achieve $Q^*$. It does not mean that the implicit policy (the one that can be derived from it: $\pi^*(s)=\arg\max_aQ^*(s,a)$) is not important, neither that it cannot be computed from $Q^*$.
You have to think about this in terms of policy evaluation (or prediction, in classical RL terminology): you have the optimal action-value table, that you can use to compute the value (or return) that some policy achieves, as well as to derive the optimal policy by taking the action that maximizes $Q^*$ for a given state.

The slide 22 in your link says:

Once we have $Q^*$ we can act optimally:
$$\pi^*(s) = \arg \max_a Q^*(s,a)$$

That is the optimal policy, which is greedy wrt $Q^*$.

Yeah, but there comes my confusion with the statement in slide 27. It then says "... under any policy". Does this mean this derived optimal policy is irrelevant? Does "any policy" here mean that we don't need the optimal policy to obtain the expected maximum value?


text:
How to make a proper approximation of Sine function with Neural Networks?
TL;DR;
How to build a neural network that properly approximates the sine function with different ranges?
Context and Question:
From this question I decided to use the Sergey's answer, however I used a different range of values for $x$ which yielded a very poor estimation of the sine function. I noticed that for large values of the $x$ range it will provide incorrect results.
I plotted the r$^2$-score for different values of the $x$ range obtaining the following:

Am I missing someting? Why is this happening?, how can one build a neural network for approximating the sine function?
The Code:
Here is the code I used to create the plot in case you want to reproduce it
from sklearn.neural_network import MLPRegressor
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import r2_score
import numpy as np
import matplotlib.pyplot as plt

sineLengths = []
scores = []

for sineLength in np.arange(40, 60, 0.1):
    sineLengths.append(sineLength)
    X = []
    x = np.arange(0, sineLength, 0.1)
    y = np.sin(x)

    for i in range(len(x)):
        X.append([x[i]])

    X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=1, test_size=0.2)
    sc_X = StandardScaler()
    X_trainscaled=sc_X.fit_transform(X_train)
    X_testscaled=sc_X.transform(X_test)

    reg = MLPRegressor(hidden_layer_sizes=tuple([64]*5), activation="relu" ,random_state=1, max_iter=20000).fit(X_trainscaled, y_train)
    y_pred=reg.predict(X_testscaled)
    r2score = r2_score(y_pred, y_test)
    print("The Score with ", r2score)
    scores.append(r2score)

If I am reading this correctly, your feature for X is five different x input values in a strict sequence. Why? It would mean you are not approximating $sin(x)$ but some other related function $f(x,y,z,a,b)$ but constraining the valid inputs to $f$ whilst expecting $f(\cdot,\cdot,\cdot,\cdot,b) = sin(b)$

@NeilSlater, Thanks for your comment. I'm working in predicting a signal value taking into account the immediate previous historic values, I figured out that I could expect $f(x,y,z,a,b) = f(x) + f(y) + f(z) + f(a) + f(b)$, then, in this specific example $f(x) = f(y) = f(z) = f(a)=0$ and $f(b) = sin(b)$. However I took your comment into account and updated the code in the post as well as the scores and the range. I also scaled the $X$ with fit_transform and transform, and obtained the samples with the same step across ranges. It still doesn't work. Am I forgetting something?

@NeilSlater, I also updated the plot, btw :)

In the previous code, you were not providing the 5 previous values of $y$ as a sequence to predict the next value of $y$, but the 4 previous values of $x$ plus the current value of $x$, which is very different from predicting the continuation of a sequence. Either way, thanks for making the edit and including code that matches the question better.

@NeilSlater, haha, you're right, my bad. Thx for pointing that out, and I agree that now the edition reflects the question more clearly

Two things are happening here:

As you increase the number of $sin$ cycles that the neural network needs to approximate, the problem becomes harder for it to learn. The network - being 5 layers with 64 neurons each - actually has plenty of capacity to learn up to the approx 10 cycles you are trying to teach it. However, training it becomes progressively harder.

scikit-learn's MLPRegressor has a lot of hyperparameters, which are mostly set with sensible defaults, but it looks like* it is not set to try hard when training begins to stall and become more stochastic. The main issue with that is how quickly it will give up when cost function is not improving epoch-by-epoch, and it is common for measurable improvements to take multiple epochs, especially with a small data set (you only have a few hundred samples) when progress epoch-to-epoch can become quite noisy.

I managed to fix this and make training more robust by . . .
Increasing training set size slightly
  x = np.arange(0, sineLength, 0.05)

You could probably do a lot more here, the larger the set, the less noisy each improvement per epoch will be. But I wanted to use other hyperparameters to show what you might need to do if training is tougher and you cannot afford to easily increase the training set size.
Adding hyperparameters that made the fit routine try harder
    reg = MLPRegressor(hidden_layer_sizes=tuple([64]*5), activation="relu",
      random_state=1, max_iter=2000, early_stopping=True,
      validation_fraction=0.2, n_iter_no_change=200,
      tol=1e-5).fit(X_trainscaled, y_train)

Notes:

I added using validation for early stopping, to avoid over-fitting. You should always use a validation set (in any non-toy exercise), although you could use other regularisation approaches. I did not check whether any of your bad scores were due to overfitting, but I don't think it was likely. This consumes more of the training data, which is another reason why I increased the size of the training set.
I increased n_iter_no_change significantly, to overcome the noise in the improvements per epoch. Small data sets may need many epochs in order to reveal the real trend in the training process.
I decreased tol, to allow for almost any improvement to be considered worthwhile.
I reduced max_iter to prevent training taking too long on this toy problem, and because I probably over-compensated with n_iter_no_change and tol changes. That meant I did get quite a few warnings of Maximum iterations (2000) reached and the optimization hasn't converged yet but the scores were nearly always very good after 2000 epochs, so I did not feel any need to increase that. Probably your original code was giving up after 100 or even less epochs because loss was not improving, and that was what gave you the bad r2 scores.

The above changes resulted in this graph for me:

. . . almost all r2score values were above 0.99, just one scored badly at 0.22, likely due to random chance. I could probably tune the hyperparameters further to make it perfect, or maybe double the size of the training set, but I will leave that as an exercise for you.

* This is the first time I have used MLPRegressor, hence "looks like". I am not certain what all the consequences are for how they have set defaults.


text:
How to define actions on a list of values?
For a DQN algorithm, where my state is a list of values, say:
[5, 3, 4, 7, 8, 2, 6]
How can I define an action space that allows me to move a value in the list from one position to another? For example, the action should tell to move the '7' to the first position in the list. Here assume that the reward is a function of the order of values in the list.
This is likely similar to how chess would be played, where the action is a combination of picking a piece and choosing a position to put it in.
Note: this is an overgeneralization of what I actually want to do, but conceptually this is representative enough.

I had tried working on a problem similar to this using combinatorial scoring games. I ran into other issues with the players competing, but I think I can give some advice to how I handled this.
In my scenario, I had an array with game pieces for each (two) player. They had to pick a piece to play on, and a direction to move it. I was able to teach the policy to pick its own piece by:

reward of +1 when it picked a moveable piece
reward of -1 if it chose an unmovable piece, and forced it to choose a new piece to move

The policy was able to learn "I need to pick a playable piece to optimize the reward".
For your case, your action space could be two indices (move a value from the first index to the second). If you have arrays of varying sizes/dimensions, possibly that could be an observation? By forcing the policy to choose pairs of indices until one is valid, you will hopefully be able to teach it to pick viable pairs immediately.

Thank you for the answer! So am I correct in assuming that the action space would be a discrete space? If so, do you have any suggestion which RL method to use? In my understanding, DQN is probably not the best way since the Q-network's output is a scalar value for each action, which would likely be telling us "move this piece" but won't tell us "where to move the piece". In other words, how to define a tuple of actions using a neural network? A common answer I found was to have the number of neurons in final layer equal to the product of number of actions and dimensionality of the actions.

Yes the action space would be discrete of size 2. The ranges would be equal to the dimension of your array, since we can move any piece to any other location. Not sure which algorithm would be best for this, you could try and integrate your algorithm with OpenAI's Gym environments and try some open source packages for RL algorithms (stable-baselines, for example). Good luck!


text:
How Does The Scaled Dot Product's Dimensions Work Out In Mult-Head Attention?
I don't understand how self-attention works with batched values for the $Q \times K^T $ step. According to the diagram below (assume 1 head), once we get past the first 3 linear steps, we arrive at the equation. $$softmax(\frac{Q \, K^T}{\sqrt{d_k}})V$$
Assuming Q and K have dimensions $[\text{batch}, \text{sequence}, \text{key_dim}]$, if the tensor contraction goes like $$Q \, K^T = [\text{batch}_q, \text{sequence}_q, \text{query_dim}_q] \times [\text{batch}_k, \text{key_dim}_k, \text{sequence}_k] = [\text{batch}_q, \text{batch}_k, \text{query_dim}_q, \text{key_dim}_k]$$
there is an extra batch dimension in the result.
I can avoid this extra batch dimension by doing a for-loop over the batch dimension and compute the formula 1-to-1 for each query and key. But one of the benefits of self-attention is that it's parallelizable so there must be a way to do this without a for-loop. How do I achieve this?

With Eigen Tensor I have these incorrect duplicate results
Eigen::Tensor<double, 3> query(2, 2, 3);
query.setValues({{
                         {1, 6, 9},
                         {8, 7, 11}},

                 {       {1, 6, 9},
                         {8, 7, 11}}});

Eigen::Tensor<double, 3> key(2, 2, 3);
key.setValues({{
                         {4, 5, 0},
                         {10, 15, 7}},

                 {       {4, 5, 0},
                         {10, 15, 7}}});

// contract along last dim of query and key
Eigen::array<Eigen::IndexPair<int>, 1> matmul{Eigen::IndexPair<int>(2, 2)};

Eigen::Tensor<double, 4> qk_t = query.contract(key, matmul);

std::cout << "QxK_T: dimensions" << qk_t.dimensions() << "\n" << qk_t << "\n";

QxK_T: [2, 2, 2, 2]
[[[[ 34, 163],
   [ 34, 163]],

  [[ 67, 262],
   [ 67, 262]]],

 [[[ 34, 163],
   [ 34, 163]],

  [[ 67, 262],
   [ 67, 262]]]]

Say you have 3-dimensional tensors Q & K of shape (B, T, C) (for Batch, Time, and Channels).

K is transposed. 'Transposing' a 3-d tensor in this case means switching around the last two dimensions. Hence shape(K.T) = (B, C, T).
Q and K.T are matrix multiplied. Matrix multiplication is done on a per-sample basis. Logically, samples are not mixed. Hence we get matmul(Q, K.T) gives the matrix multiplication (B, T, C) matmul (B, C, T) -> (B, T, T)

Think of it like this. Take the batch apart into individual samples, and simply matrix-multiply the individual samples.

I was hoping there'd be an Eigen equivalent of numpy and TF's einsum, specifically something like "abc,adc->abd". They have a merge request open for it (https://gitlab.com/libeigen/eigen/-/merge_requests/52/commits?commit_id=3b1dcc4a15b7e60b2f4d991719950b8027c1adc2) but it's a big feature and, according to discussion, they're putting it off util after version 3.4.

I cant help you with imementations in Eigen unfortunately. I can pnly describe the theoretical answer to your question, which i hope I have done. For implementation questions, the DS SE is a better place to start.

The theoretical portion is simple, it's just the implementation in Eigen that I'm having difficulty. I have opted for a for-loop with omp parallel instead. For large enough tensors, I found that a for-loop with omp parallel and chip/slice is also nearly as fast.

The question "How Does Self-Attention Query and Key Multiplication Work For Batched Inputs?" and i get the idea that that is answered. If you'd want advice on implementation the DSSE is a better place to open up a question.

Thank you Robin. I have arrived at a solution already.

Great! Please guve the answer a checkmark & upvote if its helped solve the question, then future visitors wil have a better time finding what they are looking for


text:
Why Is There The Term 1/m In Backpropagation
In backpropagation the gradients are used to update the weights using the formula
$$w = w - \alpha \frac{dL}{dw}$$
and the loss gradient w.r.t. weights is
$$\frac{dL}{dw} = \frac{dL}{dz} \frac{dz}{dw} = (\frac{dL}{da} \frac{da}{dz} \frac{1}{m}) \frac{dz}{dw} $$
Why is the there the $\frac{1}{m}$ term? Does batch size matter and what if it's 1?

could you post a link to the book/blogpost from which you got the formula? m should be just the total amount of weights, but the notation is not consistent across literature so to be sure I should check from your source.

Sorry I forgot to mention that m is the number of features in the last layer's output. Here's the video link https://youtu.be/yXcQ4B-YSjQ?list=PLkDaE6sCZn6Ec-XTbcX1uRg2_u4xOEky0&t=780 but he writes it in an abbreviated way


text:
How Does Convolution Backpropagation Work?
Assume in a convolutional layer's forward pass we have a $10\times10\times3$ image and five $3\times3\times3$ kernels, then $(10\times10\times3) *( 3\times3\times3\times5)$ has the output of dimensions $8\times8\times5$. Therefore the gradients fed backwards to this convolutional layer also have the dimensions $8\times8\times5$.
When calculating the derivative of loss w.r.t. kernels, the formula is the convolution $input * \frac{dL}{dZ}$. But if the gradients have dimensions $8\times8\times5$, how is it possible to convolve it with $10\times10\times3$? The gradients have $5$ channels while the input only has $3$.
Since during the forward pass the kernel window does element-wise multiplication and brings the channels down to $1$, do the gradients propagate back to each of the $3$ channels equally? Should the $8\times8\times5$ gradients be reshaped into $8\times8\times1\times5$ and broadcasted into $8\times8\times3\times5$ before convolving with the layer input?

I think this article will help you with this question.

I figured it out a while ago and double-checked my results with TensorFlow, so I'm fairly confident with the implementation. Here is what I did using Eigen Tensor and the im2col method:
If a [N,10,10,C] image is convolved with a [F,3,3,C] kernel (both in NHWC format, F = # kernels), using stride & dilation 1 and valid padding, the output is [N,8,8,F] image.
Therefore the gradients coming back to this layer is also Nx8x8xF which are element-wise multiplied with the activation gradients to get dL/dZ, also [N,8,8,F]. The kernel gradients formula is $input * \frac{dL}{dZ}$ so this is a [N,10,10,C] dimension image convolved with [N,8,8,F] dimension gradient.
Convert the gradients tensor into an im2col tensor

Shuffle the gradient $\frac{dL}{dZ}$'s dimensions [N,8,8,F] into [F,N,8,8]
Reshape [F,N,8,8] into [F,N,8,8,1] and broadcast into [F,N,8,8,C] to match the input image's channels, divide it by the number of times broadcasted, C
Reshape [F,N,8,8,C] into a 2D tensor as [F,N*8*8*C]

Step 2 addresses my question on convolving 2 tensors with mismatching channels dimensions.
Then extract patches from the input tensor and convert it to an im2col tensor

Extract image patches from the input image, using the same amount of padding during forward pass (following the convention that if uneven padding, the extra goes to the bottom & left) and the same stride and dilation (but the latter 2 are swapped), with the gradients from the previous step 2 playing the role of kernel, resulting in the 5D tensor [N,P,8,8,C], P = # patches = 9, the # times the kernels (gradients) slid across the input image
Shuffle the 5D tensor's dimensions from [N,P,8,8,C] to [P,N,8,8,C]
Reshape the image patches [P,N,8,8,C] into the 2D tensor [P,N*8*8*C]

Now that we have the gradients as [F,N*8*8*C] and [P,N*8*8*C], we can multiply the two

Do a contraction (matrix multiplication) along the first dimensions of both, the resulting tensor dimensions are [F,N*8*8*C] x [P,N*8*8*C] = [F,P].
The tensor is reshaped from [F,P] to [F,3,3], then to [F,3,3,1]
Broadcast C times on the last dimension, divide by batch size N, then we get the kernel gradients [F,3,3,C] which match the kernel used during the forward pass, and can be fed to your optimizer of choice during the weights update

With Eigen Tensor, if you wrap all of this as a function with the return type auto to keep everything as an operation, you can lazily evaluate it into a 4D tensor using Eigen::ThreadPoolDevice with 2+ threads for improved speed (2-3 times faster than a single thread on my machine).
Edit: Here's my implementation on GitHub (1 2)

Yes, you are right that you just zero-pad to get the right dimensions. The operation in color space is just a scalar product, so that you could get the backwards operation in that dimension also just per the formulas of the matrix-vector product.
Note that the convolution operations forward and backwards are different.

The forward pass of the convolution layer has two elementary steps: first the convolution operation and then the cutting out of the fully convolved center sequence.
\begin{align}
z&=c*_{rev}x\\
y&=P_Nz
\end{align}
Let's just consider the one-dimensional case, higher dimensions proceed similarly. Then $P_N$ cuts out the finite sequence with support $N$, $y_n=z_n$ if $n\in N$, else $y_n=0$. Similarly for the projections to the support $K$ of the coefficient sequence $c$ and $M$ of the input sequence $x$.
The convention for CNN is to not use the convolution as used for polynomial multiplication, where $[a*b]_n=\sum_ka_kb_{n-k}$, but to have it look more like a scalar product,
$$
[c*_{rev}x]_n = \sum_{m\in M} c_{m-n}x_m=\sum_{k\in K}c_kx_{k+n}=[rev(c)*x]_n,
$$
$rev$ for "reverse", where $[rev(c)]_k=c_{-k}$.
Note that this version of the convolution product is not commutative.
These all are linear operations, so in principle the gradient back propagation has the same structure as the usual matrix multiplication.
The structural principle of gradient computation is that gradients $\bar u^T=\bar L\frac{\partial L}{\partial u}$ act as linear functionals on tangent vectors $\dot u$, and that the scalar value $\langle \bar u,\dot u\rangle$ of it is independent on where in the graph the pairing of gradient and tangent is carried out,
$$
⟨\bar y,\dot y⟩=⟨\bar z,\dot z⟩=⟨\bar x,\dot x⟩+⟨\bar c,\dot c⟩.
$$
The tangent propagation is, using the product rule,
\begin{align}
\dot z&=\dot c*_{rev}x+c*_{rev}\dot x\\
\dot y&=P\dot z
\end{align}
Evaluating the defining relations gives
\begin{align}
⟨\bar z,\dot z⟩&=⟨\bar y,\dot y⟩=⟨\bar y ,P\dot z⟩=⟨P_N^T\bar y,\dot z⟩  \\
⟨\bar x,\dot x⟩+⟨\bar c,\dot c⟩&=⟨\bar z,\dot z⟩=⟨\bar z,(\dot c*_{rev}x)⟩+⟨\bar z,(c*_{rev}\dot x)⟩
\end{align}
The first operation $\bar z=P^T\bar y$ is just zero-padding of the sequence as $(0_{k_1},\, y^T,\, 0_{k_2})^T$. If one thinks of all sequences continued indefinitely by zero, then $P_N^T$ does nothing, as the zeros for the padding are already present.
To analyze the combination of scalar product and convolution, let's go into the details
\begin{align}
⟨\bar z,(\dot c*_{rev}x)⟩
&=\sum_{n\in N}\bar z_n\sum_{k\in K}\dot c_kx_{n+k}  \\
&=\sum_{k\in K}\dot c_k\sum_{n\in N}\bar z_nx_{k+n}  \\
&=⟨(\bar z*_{rev}x),\dot c⟩   \\
\implies \bar c &= P_K(\bar z*_{rev}x)
\end{align}
as $\dot c$ always only has support $K$, so the support of the gradient to it needs to have the same support.
For the second term one gets similarly
\begin{align}
⟨\bar z,(\dot c*_{rev}x)⟩
&=\sum_{n\in N}\bar z_n\sum_{m\in M}c_{m-n}\dot x_m  \\
&=\sum_{m\in M}\dot x_m\sum_{n\in N}\bar z_nc_{m-n}  \\
&=⟨(\bar z*c),\dot x⟩   \\
\implies \bar x &= P_M(\bar z*c)
\end{align}


text:
What does linear regime of nonlinearity mean in normalisation?
In section 3 paragraph 2 of Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift paper (https://arxiv.org/abs/1502.03167) they say that normalizing a layer's input may change what it represents, I understand this. But what do they mean in the bolded part?

For instance, normalizing the inputs of a sigmoid would constrain them
to the linear regime of the nonlinearity.

The normalization makes the signal small enough to remain in the region of the sigmoid that can be well approximated by a straight line.
The idea is the same as in electronics: https://en.wikipedia.org/wiki/Small-signal_model

Oh so near the range of around [-1, 1] on the horizontal axis where the slope is close to being a straight line?

Yes, that's it. The sigmoid is also almost linear, flat, for large absolute values. The hard sigmoid is a piecewise linear approximation to the sigmoid based on these three regions: large negative, around zero, large positive.


text:
permutation importance shuffling
When calculating permutation importance for a certain feature, that feature is shuffled randomly and predictions with the shuffled feature are compared to predictions with the feature in its original order.
What are the advantages of this over filling the feature up with a certain value, randomly generated values, deleting the feature and retraining + predicting, and other possible alternatives?
Note:
all the alternatives that I mentioned like filling the feature with randomly generated values would be used to predict several values and then those values would be compared with the standard ones

Could you please clarify whether you are asking "Why use permutation importance?", or "Why use random shuffling to measure permutation importance?". Some of your suggested alternatives have nothing to do with measuring anything about permutations, and it is not clear whether that is because you have misunderstood what permutation importance is measuring, or whether you are questioning why someone might be interested in it.

@NeilSlater it is likely because I have misunderstood and am attempting to clarify with this question.

Distributional mismatch, the value that you pick might be out of distribution, and if inside the distribution, might be that it correspond to the correct one for some samples
Again, distributional mismatch
Might be expensive

Shuffling guarantees you that the noisy value still comes from the same distribution, thus if it comes from the distribution and doesn't create a lot of change then $D_{KL}(p(y|x)||p(y))$ it's very small, which means that the two distributions are similar and that $x$ is irrelevant
As for all ML algorithms, you have tradeoffs, and there is no specific reason why you should use a method over the other, if two are theoretically sound, then it's up to your experience and use case to use one over the other


text:
Unsupervised pretraining on the supervised learning training data
Is it ok to pre-train and train (fine-tune) the neural network on the same training data?
Here is the specific context:
I am using the TabNet model on a tabular dataset. The dataset is fully labeled. I read about the performance and optimization benefits of pre-training the network. Can I pre-train it using the same training dataset on which I would eventually train my network? Or one is not supposed to re-use the data that was used for pre-training.
If I had to guess, I would say pre-training will use my training data to learn the dependencies between the features, which will help the training model in the supervised learning portion. I'm not sure if this is statistically sound or correct.
Is what I am about to do also called self-supervised learning?

As long as the first training is unsupervised and the second one supervised it does fall into the category of self-supervised learning.
I would say pre-training will use my training data to learn the dependencies between the features
This sentence is quite vague so let's clarify it a bit. Supervised and unsupervised losses are much different from each other statistically speaking. Supervised losses learn to predict conditional probabilities $p(y|x)$, i.e. the best predictions given an input. Unsupervised losses have no labels to compute such probabilities so they try to learn $p(x)$ directly, i.e. they try to learn the prior distribution of you data.
Of course between the two tasks the latter is much harder. Usually you would leverage labels to compute some distance or difference between predictions and labels but since you have no labels the only thing that can be done is to compute distances or differences between predictions and the same input that was fed to the model. So what an unsupervised model learn is not per se dependencies or correlations of features, is more generic than that, it's about learning to return similar predictions, usually embedding vectors, for similar inputs. Some approaches like contrastive learning try to go beyond that by imposing a margin between different predictions, so learning not only embedding vectors similar for similar inputs, but embedding vectors that are also dissimilar for dissimilar inputs.
This key aspect of unsupervised learning is the reason why it can be useful sometimes for pretraining supervised models. Supervised losses (almost all) are not designed to learn good embeddings. Problems like shift invariance and miss calibration are examples of consequences of that.


text:
Deep Q-Learning Model Effectiveness Improves then Crashes
I am implementing a Deep Q-Learning Algorithm. The model appears to improve but after awhile it just crashes and does just as well as if an agent was making random decisions. Shouldn't the behavior get better and better as time goes on? I tried training 3 times and the same thing happened each time. Letting the model run longer never saw the model improve again and basically stayed random. I'm new to data science/machine learning and if someone could point me towards some resources to prevent this behavior I would very much appriciate it.
For some more context, I'm trying to solve the cart pole problem (inverted pendulum) using Q-Learning. The graph I've included below shows how long the model is able to balance the pendulum. The x-axis represents the episodes and the y-axis represents time balanced.

EDIT 1:
When the graph was generated, this is how the model was created. I never set a learning rate.
model = keras.models.Sequential()
model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
model.add(tf.keras.layers.Dense(64))
model.add(tf.keras.layers.Dense(32))
model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))

model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])
return model

I've since changed it to this.
 model = keras.models.Sequential()
 model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
 model.add(tf.keras.layers.Dense(64))
 model.add(tf.keras.layers.Dense(32))
 model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))

opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
return model

I have a 'target model' that gets updated every 5 'episodes.' An episode ends after 15s or when the angle is greater than 45 degrees.
Since I've changed it to use the tf.keras.optimizers.Adam optimizer, the model progress looks very noisy. Since changing the optimizer, when I evaluate it while training, sometimes the agent will be very good then drop the to very bad the next time it is evaluated. (The agent is evaluated after training on 20 episodes. For an evaluation, the agent runs through 10 episodes and the time is averaged). Here is a plot of the training progress. I've tried changing the learning rate to be lower but the model doesn't appear to improve at all.

Would an evolutionary type algorithm be better because it takes the best of each generation? I feel like that wouldn't allow a model to drop like it does now. However, I would still like to know why what I'm doing now isn't working.
EDIT 2:
I've since changed things to use a learning rate of 0.00003 (3e-5) as suggested in the comments. I've also changed the target model to update every 1000 episodes. I'm still finidng similar results. The training progress graph is below (like before it has episodes on the x-axis and average time survived on the y-axis). I am training the model on every step of the episode, should I only be training after an episode completes?

The green vertical lines represent when the target model was updated. The red horizontal lines represent the min and max times. Ignore the x-axis. Model progress was recorded every 20 episodes so the x-axis should be scaled by 20.
EDIT 3:
I'm solving the cart pole problem. I want to reward behavior when the pole's angle is closer to upright (theta=0). I also want to reward behavior when the cart itself is closer to the center (x=0). I take away 150 from the reward when it tips over.
def get_reward(self):
    score = 0.0
    fail = False
    complete = False

    if abs(self.cart.theta) > CartEnv.THETA_MAX: # THETA_MAX = 45 degrees or pi/4 radians.
        fail = True
        complete = True
        score -= 150

    if not fail and self.time >= 15:
        complete = True

    # Take away score further from center
    score += 30 - (abs(self.cart.theta) * RAD_TO_DEG) # [30, -15]
    score += .5 * max(10 - abs(self.cart.x), -20) # [5, -10]

    return (score, complete)

Please, edit your post to provide more details about the hyper-parameters that you're using (e.g. learning rate, number of layers, etc.). Those might be useful to answer your question.

@nbro , I've added an edit. I'm new to the world of data science/machine learning so any advice would be appriciated!

I would choose a learning rate way smaller. Most of implementations that I have seen assume something around 3e-5. Also, the rate that you're updating the target net seems too short. DQN are know to be unstable and the idea of use a target net aims circumvent it. Perhaps try to use something above ~1000 and let me know if your agent improved.

@HenDoNR, under 'EDIT 2' you can see the results from the changes you suggested. The model appears to make changes after the target model is updated. Should I update the target model only when the model appears to be doing well?

Thanks to give more details. You definitely shouldn't train your agent after the end of each episode. The idea of Experience Replay/Replay memory is store the experiences and shuffled them to break the correlation between data. It's always hard to put parameters in numbers, but try to increase substantially the amount of experiences stored before train your agent. I would say the same for you target net update rate

@HenDoNR, thank you for your help! But ok good, I have been training my model on each step of the episode, not just the end. I wait for my replay memory to have >1,000 transitions before doing any training whatsover. Should I wait until I have >10,000? Is that a good number? Everytime I train I select a random sample of 64 samples from my replay memory. Should I be using more than 64 samples? Could another issue be my reward function? I've posted my reward function under EDIT 3.

Is that the actual model you are using? If so, you have another issue that will significantly hinder its ability to learn.  If you add a layers.Dense(32) without specifying an activation, the default activation is linear.  Given that, creating multiple layers will have no benefit for your model since it will trivially collapse to a single linear transformation.  You likely want to specify something as your activation.  Perhaps start with relu?

I just checked an implementation and your parameters seems okay. Indeed they used a smaller update target rate and higher learning rate than I suggested you. Other things that might help you is ReLu in dense layers and set the discount rate gamma =~0.99.  Furthermore, check if your epsilon decay is progressing towards fully rely on Q-function (and this is quite important!)

@JoshMajors if you are just asking "why", the answer is that in RL "learning collapses" are very common. Even when using algorithms like TRPO and PPO (which are specifically designed to prevent learning collapses), they are still rather common! Welcome to RL

@DavidHoelzer , that makes a lot of sense with the activation function. That was a huge question I had with machine learning in general: how can we get nonlinear behavior when the system is just a bunch of linear equations? And the answer seems to be the activation functions, right? Right now I'm training with 'relu' as my activation for the hidden layers. I made my activation 'tanh' for the final layer. Will update with results.

@profPlum , I've seen some things on "catastrophic forgetting" but it felt like I was doing all of the things to combat it. I have my replay memory with things from the start of my training, I tried a really low learning rate of 3e-5, and wasn't really sure what was going wrong. In practice do people just train a model until it reaches a peak, save the peak, and retry with changing hyperparameters until you keep getting higher peaks? Or is there someway to completely prevent it so it never drops out?

@HenDoNR , right now I'm trying things with a learning rate of 3e-5 and and the target model is being updated every 500 iterations. It seems like adding the activation function is helping. Will report back with the results. Thanks so much for all your help!

I've converted my comment to an answer.

You report that your model is configured as follows:
model = keras.models.Sequential()
model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
model.add(tf.keras.layers.Dense(64))
model.add(tf.keras.layers.Dense(32))
model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))

opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
return model

It is important to note that the TensorFlow/Keras documentation reports regarding the activation keyword parameter:

[The activation parameter allows you to specify the] Activation function to use. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).

Currently, your network is using the default of None, which is equivalent to linear.  Having multiple layers but only using a linear activation is akin to performing multiple linear transformations in series; the entire series is always collapsible to a single transformation matrix.
In other words, since all of your layers are linear, the network does not really gain complexity by adding additional layers.  In effect you have only one hidden layer.
You might try simply adding a non-linear activation.  Perhaps:
model = keras.models.Sequential()
model.add(tf.keras.layers.InputLayer(input_shape=(4,)))
model.add(tf.keras.layers.Dense(64, activation='relu'))
model.add(tf.keras.layers.Dense(32, activation='relu'))
model.add(tf.keras.layers.Dense(CartEnv.NUM_ACTIONS))

opt = tf.keras.optimizers.Adam(LR)        # LR = 0.01
model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
return model


text:
How to convert prediction probabilities of 2D images (initially 3D image) to 3D image predictions?
Classification: binary
Model: CNN (ResNet50V2)
During our research we've had 91x109x91 images (3-dimensional). We've used 2D CNN to train and evaluate our images and make predictions on labelled cases, thus we had to convert 3D to 2D this way (slices (n*x), y, z, #color_channels):
maindata = maindata.reshape(n * 91, 109, 91, 3)

What we did here effectively is merge a number of images (n) with dimension x (in our case 91). So we basically trained our model on slices (2D images) of this 3D image. Thus far everything was good. We've received great results.
However, now we need to create prediction probabilities of n images (3D images) and not n x 91 slices (2D images).
Can we take the prediction probabilities we've received (2D) and for every 91 prediction probabilities calculate the mean or is there a better way of interpreting 2D results to 3D images?
Both train (labelled) and test (unlabelled) data are shaped the same way, so the only part missing is interpretation to 3D.

two questions: you say you have images of size 91x109x91 but then you say you reshaped them this way (n * 91, 109, 91, 3), introducing channels, so are your images normal 3d images or 4d? Also it's not clear to me why did you have to reshape to 2d? 2d convolutions don't need reshaping, it's called 2d only cause the kernel move along the x and y axis (2d) but the kernel can be 3d, actually that's the standard procedure when processing images.

We were unable to find Conv3D in Tensorflow library (https://www.tensorflow.org/api_docs/python/tf/keras/applications/resnet).
Answer to questions:

I have an array of 3D images (4 dimensions), I need to rescale the dimensions to 2D for each image in order to use any of the Conv2D they use (Tensorflow).
Keras models only support 4 parameters, that is (#images, x, y, #color channels), not the depth dimension(z axis) that we would require.

Sorry but I'm still confused cause saying 3D images with 4 dimensions is a contradiction. You talk about depth, is that the extra 4th dimension you have along with the 3D RGB image? Or is the 4th dimension just the batch size?

Moreover if i look at the documentation of resnet50 they explicitly write the allowed input dimension, either (224, 224, 3) or (3, 224, 224), which follow the standard resnet paper architecture. So again I don't see why is necessary to reshape your input images. Unless they are 4D, in which case I wouldn't advice to use models pretrained on 3D images anyway.

just to be extra clear, what are the 91, 109, 91 numbers you mention? Are they 91=height, 109=width, 91=channels? or 91=height, 109=width, 91=extra dimension + 3=channels (RGB)?

Sorry for late response, 91x109x91 are height, width, depth, in this order. We have 3D models (cubes - MRI scans of the brains)


text:
What happens if MIN plays suboptimally and unpredictably?
The following quotes are an extract from AIMA, 3ed.

The definition of optimal play for MAX assumes that MIN also plays optimally—it maximizes the worst-case outcome for MAX. What if MIN does not play optimally? Then it is easy to show (...) that MAX will do even better.

Consider a MIN node whose children are terminal nodes. If MIN plays suboptimally,
then the value of the node is greater than or equal to the value it would have if MIN played optimally. Hence, the value of the MAX node that is the MIN node’s parent can only be increased. This argument can be extended by a simple induction all the way to the root. If the suboptimal play by MIN is predictable, then one can do better than a minimax strategy.
For example, if MIN always falls for a certain kind of trap and loses, then setting the trap guarantees a win even if there is actually a devastating response for MIN.

What happens if the MIN plays both suboptimally and unpredictably? What happens if MIN plays randomly, for example?
Is it ok to say that MAX will do better playing against a suboptimal and unpredictable MIN than playing against an optimal MIN?


text:
How do I improve the reward of policy gradient network when multiple states and actions exist per time step?
I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets.
The details are as follows.

Each time step is a random batch from the dirty dataset.
Actions: the cleaning methods (KNN imputer, median imputer etc.)
loss = sum(-log(selected cleaners' probabilities)) * reward
reward = predictor score - baseline score
Predictor score: The score obtained on a clean validation set, after the target task model trained on the cleaned batch with chosen cleaners by the PC network.
Baseline score: The validation score obtained by training the target task model on ground truth clean dataset.

After the training is completed, the policy gradient network does not always converge to the best cleaner / combination of cleaners.
I know that policy gradients are problematic when there are negative rewards, and in my case the reward is mostly negative, as the baseline score is obtained on the ground truth clean dataset.
I'm also feeding multiple feature vectors of erroneous rows as state vectors to the PG neural network, which could lead to ignoring the appropriate cleaners for single samples -or even penalize them- when a low reward is obtained for the current batch.
In DVRL, they also use batches to evaluate the samples at each iteration (all at once). I believe, in such cases, the minimum loss may not mean the maximum reward. But what could be done to improve the final reward?

Could you clarify what's are the actions? Is the network outputting one cleaning method for the whole batch, or potentially a different cleaning method per input in the dataset?

@RaphaelLopezKaufman Sir, the network outputs one cleaner per input in a batch. I didn't tell about the input states, but I've used the feature vectors generated by error detection methods or some custom features that I created related to metadata.


text:
How might AI analyze abusive discussion using natural language grammar?
Opening thoughts
This does not only apply to SE comments, but the idea in general.
This is not a Question for Linguistics.SE; those Questions might come later, after AI analysis. Example Linguistics Quesions:

What grammar categories might AI use to research for an analysis of abusive vs helpful discussion? (before the AI research, after this OP Question is answered)
What grammar patterns can we identify from AI researched that analyzed abusive vs helpful discussion? (after the AI research)

This is a Question about how AI might be useful in the real world, thus helping AI programmers decide where to effectively focus energies.
AI analyzes comments and discussion
Many web apps and sites (including Facebook, YouTube, and Stack Exchange) analyze posted content using what some people call AI algorithms.
Presuming this is used also for comments on posts on sites such as these...
AI may take many factors into consideration, viz buzz words (type 'COVID' on a post and watch the info-notice pop up), profanities, bigotous phrases, etc.
I'm curious about the results if AI analyzed just the grammar of a history of comments that were deemed "abusive" juxtaposed against a history that was not deemed abusive.
Why ask?
Creative-analytical thinkers like Steven Levitt (viz Freakonomics) and Malcolm Gladwell like to discuss counter-intuitive findings from research. Levitt says that this is "economics" (nothing to do with money). Even the video game League of Legends has stats on how often certain gaming choices (items, champion, etc) win and lose. But, we need the data. I want to know if "grammar" is a good place to dig.
I would be curious if there were any grammar patterns that might indicate abuse, as might be found by AI research from past comments. Pardon the grammar lingo, but for example:

Complex subjects
Imperatives
Subjunctives
Passives
Verbal pauses (Bothering to type out "Um..." in "Um... No.")
Direct Objects vs Indirect Objects vs "Raised Objects"
Prepositions

...Say research finds that comments containing "at" were 60% more likely to be flagged as abusive. That would be great content to ask on Linguistics to see if there were other patterns to analyze.
I don't know what should be analyzed, nor do I know what all grammatical categories would go into such an analysis. That would be my next Question for Linguistics.
Scope of my question
I'm trying to ask for open-ended answers, not binary (yes/no) answers, while also narrowing scope. So, let me put it this way...
Can AI be used to analyse abusive vs non-abusive discussions through grammar patterns and categories?
If so, which models or algorithms can be used to achieve that? References are also appreciated.

Let us continue this discussion in chat.

Part 1: Identifying and or Scoring Posts
There are various levels and types of analysis one might seek, depending on the output desired. At perhaps the simplest level, text classification could place the text into binary buckets of pass and fail. This level is similar to basic spam filtering. More complex classification could involve more buckets and or give a numerical rating.
A perhaps closer term for the task in question is sentiment analysis. From Wikipedia:

Sentiment analysis (also known as opinion mining or emotion AI) is the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information.

Natural language processing (NLP), itself, can be sub-divided into three broad approaches: symbolic, statistical, and neural. These days, and particularly in the context of AI, the last option is likely given the most attention.
The main reasons for choosing neural networks over traditional hard-coded logic are flexibility, often less coding, and support for elusive, perhaps heuristic logic that may otherwise be difficult to systemise. Conventional statistical methods may resolve some complexity over manual logic, yet possibly at the expense of inferential depth. With deep neural networks, obtaining deeper inference, for better classification and analysis, sometimes can be as simple as training on a network with more parameters, for more neurons.
Neural-based language models like GPT-3 and GPT-NeoX (third-party playground available) highlight the functional yet elusive reasoning sometimes expressed by trained neural networks. Their logic is often heuristic and hidden, a phenomenon termed black box. For those new to language models, I recommend learning more (ie. by searching "GPT-3" on YouTube) and perhaps using a playground, like the one linked above.
No doubt, the specific details of implementation can vary greatly. One could, for example, run the input through a grammar classifier (neural or otherwise), followed by feeding the resultant parts of speech (plus tense) into a sentiment analysis network. Doing so could hypothetically be cheaper to train, possibly, though not necessarily, at the expense of quality. As neural computing gets cheaper, the trend is simply using a bigger network with more training data, hopefully giving a better result with minimal coding.
The amount of time required for development depends on available data -- especially tagged data -- and computing. A person with experience in sentiment analysis could possibly design or adapt a network in days. But preparing the training data could be a challenge, particularly for a large network. If you had access to, say, the set of flagged posts for a popular site, the data may already have the desired level of tagging and already be enough for training.
Indeed, if insufficient data is available, a grammar pre-processor, as described earlier, may help. In general, less data means smaller networks and more coding. With the right skills, off-the-shelf pre-trained language models, like those mentioned earlier, may be able to support or provide text classification, perhaps even for this purpose.
Part 2: Finding Trends in Language Usage
Whether done manually or though AI, once the posts have been categorised, many options exist for identifying trends in grammar and word usage.
At its simplest, basic word frequency could be analysed using conventional coding. For example, a sorted word-frequency mapping could be made and compared between each population of tagged posts, with the most frequent words first.
If context-sensitive properties, such as grammatical mood, tense, or part of speech are of interest, then a grammar pre-processor is likely in order. Such a parser could be neural or conventional. The resulting output could be hierarchical, a simple array, or perhaps a map. Analysis of the results could be simple, like described above for word frequency; or the results could be trained through a neural network to establish more complex inference.
Since language is generally one-dimensionally arranged, sequential pattern mining may be applied in search of unknown patterns. Presumably each population of tagged posts would be analysed separately, followed by taking the difference between the result sets. Those patterns of greatest discrepancy might be insightful. This step could be applied to either the raw tokens (ie. words), or the parsed grammar output.
Other options include cluster analysis and self-organizing map, although special pre-processing, from a broad set of possibilities, may be recommended for these.
Assuming the set of pattern types is open-ended, the set of programming paths is rather unbound.