text:
Why use a fully connected layer for attention?
In the paper Neural Machine Translation by Jointly Learning to Align and Translate, attention is used with a single fully connected layer. Specifically, in the auto-regressive set up (equation 4), the context vector c_i is computed using attention  from all the annotations (equation 5).
I'm curious why a fully connected layer is used here and not some other architecture that captures "locality" better: the relationship between adjacent states should be similar independent of where in the sentence they occur. I.e. the relationship between state 1 and 2 should be analogous or similar to the relationship between states n and n+1? However, the fully connected layer treats these two relationships as completely independent weights: a[1,2] vs a[n,n+1].


text:
Are goal-reaching and optimizing the utility function special cases of performance measure?
In AIMA, performance measure is defined as something evaluating the behavior of the agent in an environment.
Rational agents are defined as agents acting so as to maximize the expected value of the performance measure, given the percept sequence they have seen so far.
Goal-based agents are those acting to achieve their goals. Utility-based agents are those trying to maximize their own expected "happiness".
Now, can we say these two design approaches induce performance measures?
What I suggest is that in goal-based agent design we want to find a point satisfying some conditions, so it's an optimization problem with a zero objective function and either this function is the performance measure or performance measure is optimized if and only if we find a solution to this optimization problem with zero objective function. In the utility-based agent design, we have an objective function (as a performance measure) that we want to optimize, and the agent has its own utility function, which it wants to optimize, and this utility function is optimized if and only if our objective function is optimized.

Does anybody have any idea?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?
I am reading the Sutton & Barto (2018) RL textbook.
On page 59, it derives the recursive property of the state-value function as below.
Could you explain the steps of third and fourth equality?
Here is what I have.
\begin{aligned}
E_\pi[R_{t+1}+\gamma G_{t+1}|S_t=s] &= E_\pi[R_{t+1}|S_t=s]+E_\pi[\gamma G_{t+1}|S_t=s] \\
&= \sum_a \pi(a|s) \sum_{s',r} p(s',r|s,a) \times r + ???
\end{aligned}

This post asks the same question: https://stats.stackexchange.com/questions/243384/deriving-bellmans-equation-in-reinforcement-learning

To expand $\mathbb{E}_\pi[\gamma G_{t+1}|S_t=s]$, you can take the same expectation over next state and reward as for $R_{t+1}$ (in fact this is normally shown without separating the two terms as you have done, but as it is the expansion of this part where you want help, we can do it separately).
The key thing is to move forward one time step - choosing the action using the policy, and the reward and next state using the state transition function - and express the expectation as a sum of the probabilities for the next step (also noticing that this changes the condition from $S_t=s$ to $S_{t+1}=s'$):
$$\mathbb{E}_\pi[\gamma G_{t+1}|S_t=s] = \gamma \sum_{a}\pi(a|s) \sum_{r,s'} p(r,s'|s,a) \mathbb{E}_\pi[\gamma G_{t+1}|S_{t+1}=s']$$
Then, we can notice that $\mathbb{E}_\pi[\gamma G_{t+1}|S_{t+1}=s']$ is $v_{\pi}(s')$ and get
$$\mathbb{E}_\pi[\gamma G_{t+1}|S_t=s] = \gamma \sum_{a}\pi(a|s)\sum_{r,s'} p(r,s'|s,a) v_{\pi}(s')$$
Now we can recombine this with the other expression for expected immediate reward that you have already resolved, because the summation is the same in both parts, and get result 3.14 from the book.
It is more common to resolve both parts of the expectation the same way though, and not split them up to later recombine. The tricky part is perhaps realising that you cannot resolve the expected return fully when looking ahead one time step, only express it as a sum of expected returns from all possible next states.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Given embedding vector A and vector B, how to find top k embedding vectors such that they are similar to vector A and dissimilar to vector B
Which would be better approach for getting top k embedding vectors such that they are similar to embedding vector A and dissimilar to vector B.
Approach 1:

calculate f(V) = cosine_similarity(A,V) - cosine_similarity(B,V) for each vector V
sort vectors by f(V) value in descending order
take first k of them.

Approach 2:

calculate
f(V) = cosine_similarity(A,V) , g(V) = cosine_similarity(B,V)
for each vector V
sort vectors by f(V) value in descending order
take first k of them
sort selected k vectors by g(V) in ascending order.

Approach 3:

calculate f(V) = cosine_similarity((A - B),V) for each vector V
sort vectors by f(V) value in descending order
take first k of them.

Also, suggest better approach if you have other than above two.
Note: embedding vector was calculated using word2vec algorithm

As the objective is to find the most similar to A and disimilar Vector to B approach 2 would be the most appropriate.
Why not Approach 1:
It can lead to confusing results. If you look at the example below multiple scenarios lead to same final value. This may lead to few problems :

For same output how do you know which vector should be preferred
Subtracting with each other may cancel out impact of dissimilarity on one with other

Why not Approach 3:
Vector Subtraction is not commutative. So you will get different results based on whether you use A first or B first. Moreover lets see an example where it fails completely :
Suppose we have 2 dimension vector A and B, and we want to vector V which is
A = 6,0
B = 0.01,-0.01
V = 3,-1
Cos(A,V) = 0.94
Cos(B,V) = 0.894
Cos(A-B,V) = 0.94
Second Secnario:
A = 6,0
B = -0.01,0.01
V = 3,-1
Cos(A,V) = 0.94
Cos(B,V) = -0.894
Cos(A-B) = 0.94
Though cos A and cos B are two extremes in scenario 1 and scenario 2. Cos(A-B) gives the same result. Hence i will not advice to use Cos(A-B)
Approach 2 will work expected in your case.

This is just a heuristic, but how about f(V) = cosine_similarity(A,V) * min(1, 1 - cosine_similarity(B,V))? I'm applying the min here so that the multiplier won't get larger than one. Up to you whether you want to include it or not.
This could be tuned further as f(V) = max(0, cosine_similarity(A,V))^a * min(1, 1 - cosine_similarity(B,V))^b, but you'd need to experiment to see which values of a and b works the best for you.


text:
Is there a metric to compare BOW vs TFIDF results?
I am working on a document search task and have used Bag of Words (BOW) and TFIDF vectorization techniques. My observation after going through some sample searches are -

Both of them seem to provide similar results when we look at top X results for a given search term.
However, in some cases BOW might give a slightly better top X results compared to TFIDF and vice versa.
The cases in which TFIDF is slightly better is comparatively more than cases in which BOW is slightly better.

I wish to select either of the two and based on above eyeballing I decided to go with TFIDF. But this is not explainable since the decision is based on individual perspective after looking at some sample cases. I would like to know if there is some kind of metric that I can make use of to arrive at a decision? Since eyeballing can lead to biased decision.


text:
A technique of aggregating many input images to a single representation of the relevant features within
I have a few thousand images and I would like to generate a representation of the foreground patterns within - a composition of all of its features, so to speak. In simple terms: take 10000 images of a dog and then draw the archetypical dog.
Does this task have a name, and is there a method out there specifically for such purposes?
The images have different sizes and neither scale nor rotation invariant, so simple averaging algorithms wouldn't work. I would guess that deep learning techniques could be capable - e.g., extracting the features from the first layers of a neural network - as hinted at here: ..."The original network can't be used to classify new identities, on which it wasn't trained. But, the kth layer may provide a good representation of faces in general....".
I just don't necessarily need a model for prediction afterwards, just the aggregate representation will do.

It sounds similar to eigenface. Here, PCA was used to extract principal components where each of them looks like an "average" of faces.

This idea is more or less what driven the development of Google Deep Dream.
They framed the problem in a much different way though, i.e. as a reversed task compared to classification. So given a trained network, we choose a final activation, and we modify a given input image till the chosen activation is maximized trough gradient ascent (n.b. not gradient descent).

You don't specify your definition of "archetypical" so maybe that's not exactly what you're looking for, but conceptually this is more sounded than combining feature maps. Reason being that a feature map is just an image modified with some filters, so the average of a feature maps is basically the same as the average of images themselves. Deep dream instead modify an image in such a way that this image will react to specific, already learned filters. So it's a generative approach. You can see it in the example below, where a sky image is modify to produce a "face" activation. All those faces are not training images nor part of the original sky image.

Note that if you're looking for a model that given thousands of images produce always the same output one, then the problem is simply ill posed. Even unsupervised energy based models like restricted boltzmann machines simply learn a mixture of features, not a single "archetypical" image (see pic below).

yeah I had thought about deep dream, but I guess I was thinking more of "statistically reproducing" the content of a set of images without being generative. "archetypical" might be too a humanizing - basically, I meant that I want a model to redraw the features or components that it learned to common features in the images I supplied before. basically you are right with it somewhat being of a "reversed classification" task, but I guess I will need to think more about the criteria that I want to apply here


text:
Why do we lose detail of an image as we go deeper into a ConvNet?
I was reading this research paper titled 'Image Style Transfer using Convolutional Neural Networks' which as the title suggests was based on Neural Style Transfer. I came across this line which didn't make immediate sense to me.

Here's how it went -

We reconstruct the input image from from layers ‘conv1 2’ (a), ‘conv2 2’ (b), ‘conv3 2’ (c), ‘conv4 2’ (d) and ‘conv5 2’ (e) of the original VGG-Network. We find that reconstruction from lower layers is almost perfect (a–c). In higher layers of the network, detailed pixel information is lost while the high-level content of the image is preserved (d,e).

The line that is italicised; Why does that happen?

The excerpt you're quoting is not saying that you lose detail of an image as you go deeper, but that using a certain output of a certain layer rather than another leads to "better" reconstructed images. So, you may want to edit your post to clarify that.

It says "detailed pixel information is lost". How does that mean you have better images as you go deeper in the network ?

Yes, but because you're trying to reconstruct the pixels of the images, as far as I understand.

Ohh okay. But at the end of the CNN are the pixels reconstructed better right?

Does this answers your question? https://stackoverflow.com/questions/28953622/do-convolutional-neural-networks-suffer-from-the-vanishing-gradient

@sam2611 No, this is different from vanishing gradients

A convolution layer has a Kernel  that is matrix  smaller that the image (in many papers 3*3), so this Kernel have learnable values.
The kernel is applied pixel by pixel (and dimensions, for color images), by applied I mean take the kernel centered in the pixel and multiply element-wise with the image and sum all results. This is the output for that pixel, a value that takes information from pixel's neighborhood.
When you apply this many times you lose specific pixel's information, but it gains information about pixel's neighborhood. And this is good because maybe this neighborhood is a corner, a line, curve, etc. (this is what learned the kernel to identify value content, for that reason the network in d and e have more basic forms and color without detail) . if you continue applying more convolutions you will find a abstract representation of the all the input, even you can get only one number that represent that input (if you don't avoid dimension reduction with padding or other tool) and say something like "oh it's a five and therefore the input is a house" (in classification case).
NOTE: This is very general and many details were omitted, if you want to learn more I recommend the book deep learning (Goodfellow et al., 2016) chapter 9 convolutional networks.

The point of a convnet, or many kinds of neural networks in general, is to go from a lot of data down to a small piece of data. In classification tasks, for example, the input is all the pixels that make up a picture of a house, and the output is just the word "house" (or rather a number representing the word "house").
Obviously this process loses information. If you go from the word "house", back to a picture (i.e. you tell it "draw a house") you're probably going to get a completely different house!
In the style transfer task we have many numbers to describe the house picture with, not just the word "house", but we still have less than the full pixel data. Imagine that the intermediate representation represents something like "yellow wooden house with three windows and one window above and a red brick basement with 4 windows and the house is drawn at a 30 degree angle and there's a pink house to the left with two small windows below and one window above and a red roof is visible behind and above the pink house and ....."
If you had that representation, you could try to draw the original picture again, and the more information you have, the more accurately you can draw it. Early layers of the convnet contain information like "there's a vertical line at pixel coordinates 123,456" and later layers contain information like "there's a yellow house at pixel coordinates 123,456", and if you get to dense layers, they may just say "there's a yellow house".


text:
What are your "current parameters" in Minibatch Stochastic Gradient Descent?
I was reading a book on Deep Learning when I came across a line, more like a few words that didn't make apparent sense.

Thus, we will often settle for sampling a random minibatch of examples every time we need to compute the update, a variant called minibatch stochastic gradient descent.
In each iteration, we first randomly sample a minibatch B consisting of a fixed number of training examples. We then compute the derivative (gradient) of the average loss on the minibatch with regard to the model parameters. Finally, we multiply the gradient by a predetermined positive value η and subtract the resulting term from the current parameter values.

We can express the update mathematically as follows ($∂$ denotes the partial derivative):

$(w,b) \leftarrow (w,b) - {η\over |B|} {\Large \Sigma}_{I \in B} \partial_{(w,b)}l^{(i)}(w,b).$
The set cardinality $|B|$ represents the number of examples in each minibatch (the batch size) and $η$ denotes the learning rate.
What does "current" parameter values mean in this context?
You can find the book here -
https://d2l.ai $\rightarrow$ Chapter Linear Neural Networks, Part 3.1.1.4

The current parameter values are the values of the weights $w$ and the biases $b$ in the neurons in each layer.  What you are calculating is the gradient of the average loss of the batch relative to the weight and the bias.  Since the gradient points "up", or toward greater loss, you multiply by the learning rate and subtract it from the current values.


text:
Are All the Target Q Values in DQN same?
So I am trying to understand and make a DQN. But I didn't understand a part. So basically state's Q values computed with the network and the target Q values will also compute with a target network according to the next state. Then the results will pass into the Bellman equation that is (R(t+1) + Q_max(s(t+1),a'). But this taking the maximum thing makes all actions' Q values same, right? And target's Q values being same means the network try to make predictions for different actions (in other words network's outputs) very similar. Am I missing somewhere or is there something I didn't know?

please consider checking the structure/spelling of the question, it's very hard to get what's your point

Well, you can ask the same question without the target network. What happens in tabular Q-learning? There's a max there too.

First, the problem you're raising also applies to tabular Q-learning or DQN without the target network because the max operation exists in those algorithms too.
Second, note that the Q-network should be initialised randomly, so that means that the initial $Q(s, a)$ should be different for all states and actions pairs (at least at the beginning).
Let's ignore the target network then.
Now, you should look at algorithm 1 in the DQN paper
You're right the max operation selects only one value. However, as you can see from the algorithm, we don't use this target value to update all state-action pairs, but just one, the one at time-step $t$, denoted by $(s_t, a_t)$ in tabular Q-learning, and $(\phi_t, a_t)$ in DQN. The target would be denoted by $y_t$ in DQN.
Note also that the target is not just the max, but also the reward. The reward you observe may be different for each state-action. So, no, the target value is not always the same, even if you keep the target network frozen forever, which is not usually the case - we update the target network every $C$ steps.

I know the reward(prediction of the target network) will change in time. I understood mostly you said thanks but just to make sure one more question: So, the prediction network's "target output" for the third output will be the value of the third output of the target network even if it isn't the maximum Q value between the predictions, right?

The reward and the prediction of the target network are not the same thing. The target network estimates the return. The return is not the same thing as the reward. The return is the (cumulative/discounted) sum of rewards. The reward that I meant was what you denoted by $R(t+1)$. In the paper's algorithm, it's denoted by $r_t$. I am not sure what you mean by "prediction network". In any case, the target value is calculated using $r_t$, which depends on the action that you take while exploring (see the algorithm), and the maximum value at the state $s_t$.

In the same state $s_t$, the maximum Q-value is always the same if the Q-network does not change. However, it might be different across states, of course. The reward $r_t$ in the same state $s_t$ may also be different. This depends on the reward function (which is not the reward, return or Q-network - it's a function that also defines the MDP, problem, and gives you the reward) because you may end up in the state $s_t$ because of different events.

Yes sorry for the stupid mixing the reward and target Q value, when I am writing it, in my mind it was Q value but I wrote it as reward. Silly mistake. I am asking again and again sorry but I want to understand it well. To be more clear, in my DQN I am using MSE loss function. All I am asking is when calculating the MSE's derivative for the gradient decant, I am subtracting the predictions from target values for all output values(O1-o1, O2-o2, O3-o3...) and should I do that like this also in DQN. I am confused because at DQN loss is calculated according to max of the Q values beside the normal

If you're doing some kind of batch GD, you should calculate the maximum value and reward, so the target, for each different $(s_t, a_t)$. They might not be the same. But, tbh, I'm not familiar with the typical DQN implementations.


text:
Is It Reasonable to Get NaN Outputs With Identity Activation Functions
I made my own Neural Network from scratch in unity with C# and I am using it as DQN. I set up my network which has 4 layers: 9 input values, 20 nodes in the second layer, 15 nodes in the third layer, and 8 outputs. I set all the layers' activation functions as Identity (which has no impact on the value), and the loss function as MSE I scaled my inputs and rewards, so my input values are between 0-1, and targets are between -1 and 1.
But after the first training, the weights become NaN. On another project I have a much simpler network which is 5-4-3-2 nodes in layers. And there it works perfectly. But when I increase the nodes for layers it immediately makes the weights NaN. If I add a coefficient factor to Identity something like 0.1 it doesn't give NaN but this time the 8 output values become very similar to each other.
I debugged the delta values and after 6 or 7 epochs later the weights go to very large numbers like 2345221. Than after that they become NaN, I think due to large values.
With other activation function(TanH, sigmoid) they don't give that NaN values but the outputs are become very close to 1(I understand that because the network has very nodes and the outputs are larger than 3 or 4 and this functions converge to 1 with larger values than 2). But this NaN thing happens with the ReLU functions too.
My question is: Is it normal to have this numbers or something is wrong with my code? If it is normal what can I do because the other functions also give bad results, I am trying to predict the cumulative reward here.

Having NaN might be because of the exploding gradient problem. This happens when your model learns very slowly and perhaps the training stagnates at a very early stage just after a few iterations.

I don't think it is because of slow learning because when I made the learning rate 0.001, it stopped giving NaN values. I think it is the opposite.


text:
Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?
As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit.
My action vector is perfectly described as a vector, where all values are between 0 and 1, and which should sum up to 1. This is perfect for a Softmax function. But these values are not probabilities of discrete actions. Each value in action vector should be a percentage from the whole portfolio to be invested in a certain stock.
But I cannot figure out, if it would be mathematically fine to use Softmax as an activation layer in TD3 or SAC?

You should check out discrete SAC. Original SAC was designed for continuous action spaces (the code was, at least — I believe theoretically the paper works with both but was motivated by continuous action spaces) but I believe they developed a discrete version. I don’t have the reference to hand but a Google search should turn up results.

@DavidIreland, my action space is not discrete actually. I did not imply probabilities by these values. Each value in action vector is actually a percentage from the whole portfolio to be invested in a certain stock.

Ah, okay, I apologise. In which case then you can just use SAC with a soft max activation. I believe tanh is usually used because they have symmetric intervals for the action space, it is not something it relies on theoretically.

@DavidIreland, thank you for the answer. I also assumed so, but I was not sure whether it's mathematically fine or not.

Yes, it should be fine :)


text:
Best algorithms/approaches for data sets of binary (1/0) features
I am working with a dataset with about 400 features, all binary (1 or 0). What approach would you recommend? Data set is about 500k records.

What's wrong with treating the 0s as 0s and the 1s as 1s?

Nothing wrong with it, just didn't get me anywhere, so was wondering if maybe there are some rules that say "Binary 1/0 features need to be treated differently than 0-1 range features"

I think it has more to do with the model you're working with, than with how you input the features.

What do you want to do with the data in the first place?

more details was provided in the comments under Andre Goulard's answer

@Darko Please, edit directly your post to include the details of your problem (because comments are temporary). It's not even clear if you're trying to solve a classification problem or any other problem. In this type of question, it's also important to show us what you have found or tried so far. So, I would suggest that in your edit you also include these details.

Most standard algorithms will work well on binary data, like:

Decision trees (and random forest)
Nearest Neighbors
Neural Networks
etc

But your choice depends on many other things, like:

What is the expected output?

Are you doing classification?
Regression?
Is it deterministic (same features should always give the same outcome) or stochastic (random factor).

What is the nature of the database and relationship between the features?

A 20x20 black-white image.
A phrase embedded as 20 sequences of 20-size-token.
A 400 questions true/false exam.

They can all have the same shape, but are very different in nature and would perform better with different algorithms.
How disperse / smooth is your data?
Do all 400 features have the same importance?
How independent are they?

How complex the problem really is?
How much performance do you really need?
How much work and tuning are you willing to put on this?

Classification - end result is true/false, deterministic
A 400 questions true/false exam is the best description. Unclear on importance of the features.
False positives are not an issue, as long as true negatives are minimal to non-existent.
How much work and tuning are you willing to put on this? 1-2 weeks full time

If you could just describe the problem, it would make it easier to recommend something.

If it's like an exam, with a previously expected answer for each feature, a simple neuron network with a threshold would perform fine. If each feature impacts on another, the network must be deeper. If features are highly correlated, you might want to reduce them... There are too many IFs to provide a straight answer.

The features are unrelated. The closest to the scenario is: a user selects if they like or don't like 400 products, and based on that i am trying to determine if they will like the label product.

If the data is about preference (like or don't like), I would consider a neighbor approach (like KNN). If you have no idea and low previous knowledge, I'd also consider some AutoML approach for setting a quick baseline.


text:
Should I label static objects on video dataset?
I'm using nvidia Transfer Learning Toolkit to detect cars in some video frames.
I found some dataset (for example https://www.jpjodoin.com/urbantracker/dataset.html and https://www.kaggle.com/aalborguniversity/aau-rainsnow) and I noticed that usually parked cars are not labeled, and covered under a mask.
Why shouldn't I add also their labels? It would be easy to label them because they are static objects and I could copy-paste in all labels set. So why in video dataset they are not labelled?

Model architecture:

In machine learning, static image detectors can be is very different from video detectors, as movement plays a big role on the task. So, even when comparing frames the objects are similar, when digesting the video, a model can learn very different things. Maybe adding parked cars to the database increased false positives, mistakenly labeling other static noises as cars.

Business goal:

Why are you labeling cars from security camera, in the first place? What is your goal here? Maybe if you want to know car density in a parking lot, than labeling parked cars is very useful. But if you just need to know traffic flow, than parked cars will be just a distraction, noise from the data.
So maybe whoever built the dataset had a different goal in mind.

On  further reflection, I noticed that there is a black mask aviable covering area of "no interest", including parked cars. I'm going to use something similar: the user select the "important" area and the model will detect all cars inside.
I just need some YOLO behavior, so my business goal is to find all cars.
However, including parked ones may over-represent one red Ferrari that will be labeled in all frames.


text:
What would be the advantage of making channel dimension first in TensorFlow Keras implementation?
I was reproducing the findings of a research article in which I discovered that they had switched the Channel dimension from last to first. To clarify this concept, I went through A Gentle Introduction to Channels-First and Channels-Last Image Formats . The author of this link stated:

When represented as three-dimensional arrays, the channel dimension
for the image data is last by default, but may be moved to be the
first dimension, often for performance-tuning reasons.

There are two ways to represent the image data as a three dimensional array. The first involves having the channels as the last or third dimension in the array. This is called “channels last“. The second involves having the channels as the first dimension in the array, called “channels first“.

Channels Last. Image data is represented in a three-dimensional array where the last channel represents the color channels, e.g.
[rows][cols][channels].
Channels First. Image data is represented in a three-dimensional array where the first channel represents the color channels, e.g.
[channels][rows][cols].

.
We are aware of when the channel was last used and the manner in which kernels were applied, theoretically. However, I'm curious as to when the dimensions of the channel come first. How kernels will process. More precisely:
Assume we have [rows, columns, channels] -> [2,4,3] image dimensions. We may say we have three data channels, each with two rows and four columns, correct?
Alternatively, if we assume that channel dimensions are first means [channels, rows, columns] -> [3,2,4]. In other words, we now have four data channels, each with three rows and two columns, am I correct? If I am taking correctly than this is quite confusing because we are completely modifying our image.
Question:
What is the benefit of shifting the channel dimension first, and how will the kernels move on it?

For more detail check the code:
input_layer = tf.keras.Input(shape=input_shape, name="Time_Series_Activity")
con_l1 = tf.keras.layers.Conv2D(64, (5, 1), activation="relu", data_format='channels_first')(input_layer)

Summary of Code
Layer (type)                 Output Shape              Param #
=================================================================
Time_Series_Activity (InputL [(None, 1, 30, 52)]       0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 64, 26, 52)        384
_________________________________________________________________


text:
How do I use GPT-2 to summarise text?
In section 3.6 of the OpenAI GPT-2 paper it mentions summarising text based relates to this, but the method is described in very high-level terms:

To induce summarization behavior we add the text TL;DR: after the article and generate 100 tokens with Top-k random sampling (Fan et al., 2018) with k=2 which reduces repetition and encourages more abstractive summaries than greedy decoding. We use the first 3 generated sentences in these 100 tokens as the summary.

Given a corpus of text, in concrete code terms (python preferred), how would I go about generating a summary of it?

I think the referenced article explains clearly how to do it: http://www.aclweb.org/anthology/P18-1082

In short: It depends.
Where will you run it?

On Premises: You may want want to run in your own environment.
IaaS: GPT models are often too big, so people might prefer to setup a different server for that, serving your API.
PaaS: If it's more experimental, I recommend running it on Google Colab.
SaaS: Or even use some external API, so you don't need to worry about this setup and just use it as a service. (easiest)

Each approach demands a different architecture and a different code.
Once you've setup the environment / API, you'll run it by providing an initial prompt and some parameters:
GPT
GPT-2 (any GPT model) is a general, open-domain text-generating model, which tries to predict the next word for any given context.
So, setting up a "summarize mode" is not just flagging a parameter. It's a non-deterministic process and requires tries and errors.
The GPT setup is experimental:

You use a sandbox.
Create an initial prompt.
Set some parameters (Temperature, Top-P, Top-K,...)
Evaluate your results
Adjust the prompt
Adjust the parameters
Until you consistently achieve the desirable results.

The prompt
1. Simple way (no shot learning)
A very simple way to do it is something like:
prompt = text+"\nTL;DR:"

It should work specially well if your text is simple and small.
2. Explicit Introduction (no shot learning)
You may have a better result if initially prompt some context, for example:
"Here is a text and it's respective summary.\n"+
"#Full text:" + text + "\n"+
"#Summary:"

3. Add examples (few shot learning)
Another approach is filling your prompt with a few (high-quality) examples before your original prompt:
prompt = sample_1+"\nTL;DR:"+summary_1
prompt += "\n###\n"
prompt += sample_2+"\nTL;DR:"+summary_2
prompt += "\n###\n"
prompt += your_input+"\nTL;DR:"

And you realize the next logical thing for the generator to make is a summary of your input.
The results will greatly depend on your text size and style. So you should find what prompt template best suits your needs.
Execution
Keep in mind that GPT will not learn from previous executions.
It has no memory or learning in between executions, which means that each input will require the whole prompt again.
So your main program should run a loop and prompt GPT once for every text file. Something like:
for file in glob(./files.txt)
  text = open(file, "r")
  prompt = text+"\nTLDR:\n"
  result = GPT(prompt, parameters)

Other tips to keep in mind:
If you have no examples:

Results may be different from desired (Too big, too short, too informal, omit something you consider important)
Be less stable. (Nail it sometimes and ruin other times).

If you have some examples, it should replicate your example style. But:

It may wrongly make references to the example (instead of the desired text).
Gets confused if your text does not match the example(s) lenght / style.
Prompt may get too big.

If the original text is too big:

It will require more computational power.
It might extrapolate the limit of tokens it can digest. (So you might need to split the text into chapters).
The model might get confused, like: Summarizing only the last paragraph (so you need to make it clear when the text starts and ends).

You wrote TD;DR each time you meant TL;DR


text:
Different ways to produce the same network in NEAT
I have an interesting example for the NEAT and want to clarify what behavior is correct from NEAT's perspective and why (why the opposite is wrong, what are the consequences of choosing the different one).
So let we have an initial network of 3 nodes and 2 edges:
Initial Condition
Nodes: [A, B, C]
Edges: {
1: A->B
2: B->C
}

1st Gen
Then in the 1-st generation we get 2 mutants:
Mutant 1 (edge 1 got split)
Nodes: [A, B, C, D]
Edges: {
1: A->B DIS
2: B->C
3: A->D
4: D->B
}

Mutant 2 (edge 2 got split)
Nodes: [A, B, C, E]
Edges: {
1: A->B
2: B->C DIS
5: B->E
6: E->C
}

2nd Gen
In the second generation 2 if we mutate Mutant 1 (by splitting edge 2) and mutate Mutant 2 (by splitting edge 1) which result should we get?
Hypothesis 1: the same result:
Nodes: [A, B, C, D, E]
Edges: {
1: A->B DIS
2: B->C DIS
3: A->D
4: D->B
5: B->E
6: E->C
}

or...
Hypothesis 2: Two new mutants:
Nodes: [A, B, C, D, F]
Edges: {
1: A->B DIS
2: B->C DIS
3: A->D
4: D->B
7: B->F
8: F->C
}

and
Nodes: [A, B, C, E, G]
Edges: {
1: A->B DIS
2: B->C DIS
5: B->E
6: E->C
9: A->G
10: G->B
}

In case the second hypothesis is correct, how does it deal with crossover in the next run?
Say these 2 mutants are breeded. We get :
Breeding in 2nd Hypothesis
Nodes: [A, B, C, D, E, F, G]
Edges: {
1: A->B DIS
2: B->C DIS
3: A->D
4: D->B
5: B->E
6: E->C
7: B->F
8: F->C
9: A->G
10: G->B
}

Looks like a too complicated genome for the 3-rd generation, doesn't it?
In case the first option is correct then actually innovation numbers are somewhat redundant in NEAT and can be done differently.
We can have node list as a list of strings (node names).
Then instead of assigning the innovation number to an edge we can use string value calculated like HASH(fromNodeName + toNodeName).
That way whenever the new link is created in any generation between 2 nodes it gets the same innovation number name for it.
When the node is created (by splitting an edge) its name can be taken right from the edge getting split and the innovation names of 2 new edges can be calculated like HASH(fromNodeName + splitEdgeName) and HASH(splitEdgeName + toNodeName).
That way the algorithm has no global variables, no shared list of all innovations and can be simply parallelized

So let's first draw the network in each case, for better visualizing the problem:
Initial:
┌─┐
│A│
└┬┘
┌▽┐
│B│
└┬┘
┌▽┐
│C│
└─┘

1st Generation:
1st Gen
Mutation #1   Mutation #2
┌────┐        ┌────┐
│A   │        │A   │
└┬──┬┘        └┬───┘
┌▽┐ ┆         ┌▽───┐
│D│ ┆DIS      │B   │
└┬┘ ┆         └┬──┬┘
┌▽───┐        ┌▽┐ ┆
│ B  │        │E│ ┆DIS
└┬───┘        └┬┘ ┆
┌▽───┐        ┌▽───┐
│ C  │        │ C  │
└────┘        └────┘

2nd Generation
1st Hypothesis (Same Result)
┌────┐
│A   │
└┬──┬┘
 ┆ ┌▽┐
DIS│D│
 ┆ └┬┘
┌───▽┐
│B   │
└┬──┬┘
 ┆ ┌▽┐
DIS│E│
 ┆ └┬┘
┌───▽┐
│C   │
└────┘

In order to converge the 2 mutations like this, we would need to:

Somehow make topology accessible globally.
Make one mutation search for the population, looking for an occurrence of a similar split.

Both items go against the principle of evolution, as only 2 mating individuals should trade genetic information.
2nd Hypothesis (Two Mutants)
┌────┐   ┌────┐
│A   │   │A   │
└┬──┬┘   └┬──┬┘
 ┆ ┌▽┐    ┆ ┌▽┐
DIS│D│   DIS│G│
 ┆ └┬┘    ┆ └┬┘
┌───▽┐   ┌───▽┐
│B   │   │B   │
└┬──┬┘   └┬──┬┘
 ┆ ┌▽┐    ┆ ┌▽┐
DIS│F│   DIS│E│
 ┆ └┬┘    ┆ └┬┘
┌───▽┐   ┌───▽┐
│C   │   │C   │
└────┘   └────┘

Yes. This 2nd approach is the most usual. Each mutation generates a completely new gene.
┌────┐
│A   │
└┬──┬┘
┌▽┐┌▽┐
│8││D│
└─┘└┬┘
┌───▽┐
│B   │
└┬──┬┘
┌▽┐┌▽┐
│u││F│
└─┘└┬┘
┌───▽┐
│C   │
└────┘

Looks like a too complicated genome for the 3-rd generation, doesn't it?

And yes. A 3rd generation could theoretically come up like this complicated genome.[1]
┌───────┐
│A      │
└┬──┬──┬┘
┌▽┐┌▽┐ ┆
│G││D│DIS
└┬┘└┬┘ ┆
┌▽──▽───┐
│B      │
└┬──┬──┬┘
┌▽┐┌▽┐ ┆
│F││E│DIS
└┬┘└┬┘ ┆
┌▽──▽───┐
│C      │
└───────┘

But keep in mind that more complex structures are naturally harder to train and therefore will not survive if the performance does not justify the complexity.
[1] - Usually disjoint genes and excess genes (those that do not match) are chosen from the fittest parent. However, in (very specific) cases where 2 parents have the same fit, it could be chosen randomly.
Main source: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.28.5457&rep=rep1&type=pdf

In the end, you propose an interesting method, to use hashes instead of integers for identifying genes. This is a bold suggestion, and I'm not an expert, so this part is mostly opinionated:
I believe original genes numeration idea is:

Inspired on the natural genes sequencing
For assuring uniqueness genes identifications
And tracking history changes, just like Git.

But it could also work using a hash.
On one side, that could be a great feature, for allowing distant relatives to breed with consistency.
On the other side, distant relatives tend to have a very different structure and the gene might not be so relevant to them.
I encourage you to test it. And if it works better than the vanilla, you should release a paper!

Thanks for the answer! Specifically 'tracking history changes, just like Git' was what inspired me to think about hashes :) What really changes with hash-based approach is what genes are the same. In my approach more genes are treated as the same (as you see in hypothesis 1). Definitely I have to test both approaches now, I will put results here for you (but do not expect I will do it quickly - it is just a hobby for me)

Great! I have no previous academic experience on Machine Learning, but if you want to publish a paper and want some help, I think I could join you. Anyway, I'm excited to see the results! :)


text:
How to incorporate action information in the state input of a DQN?
I am working on an RL problem that I am trying to solve using a Deep Q-network. The problem concerns choosing drivers to take specific taxi orders. I am familiar with most of the existing works and that they use RL to determine which orders to take. I specifically look at the situation where we want to determine which drivers to take.
This means that action space concerns the various drivers we can choose. Initially, we assume a fixed number of drivers to ensure a fixed action space.
My question is about defining the state space. First of all, the state space consists of information about the next order we are trying to assign to a driver from our action set. Besides that, we also want to incorporate state information about the different drivers (e.g. their location). However, this would mean we include state information about the actions as input of the DQN. The reason is that the state of the drivers is the main thing that changes when choosing a different action and therefore determines the choice we want to make at the next timestep. I am for example thinking about creating a list of size |drivers| with element i defining the location of agent i.
I tried to find existing work that uses a similar setting (so that incorporates action states in the state input), however, I did not succeed in this yet. Therefore I am wondering:

Is this a logical/reasonable approach to the problem?
If yes, is someone familiar with existing works that use a comparable approach?

I am familiar with works that take (state, action) as input, which describes the full pair of the state s and the action a, and then produce a single Q(s,a) for each specific pair of state + action. This is an approach we do not want to take, given that it leads to |A(s)| passes through the network instead of a single pass (as explained here).

Drivers are not actions in this case, they are objects that are part of the state space, your state vector would look something like this
\begin{equation}
\mathbf{x} = [x_{o}^T, x_1^T,\ldots, x_N^T]^T
\end{equation}
where $x_{o}$ is a vector that contains information about location (e.g. starting location, destination location, etc.) and $x_i$ is a vector that represents state of driver $i$ (e.g. location of the driver, do they have an active order, etc.). The action in this case would be assigning the order to one of the drivers not the drivers themselves. For instance if you assign the order to driver $1$ the order vector $x_o$ would change and the vector $x_1$ would also change, so you would change your total state $\mathbf x$

Thanks for your reply! It's a good point that the action is assigning the order to one of the drivers, instead of the drivers themselves. I do grasp that vector x_1 would change, since the state of the driver changes.
In case vector x_0 only includes information about the location, the change to this vector is clear. However, it is preferable to include e.g. the order price in vector x_0. How does then the order vector x_0 change? For each order appearing, we do a forward pass through the network. The next order is therefore not yet known when assigning an order to a driver.

The way it is posed, the price of the order would not be affected by the action nor would it affect other state variables. It would be a completely random element, it would  lead to no benefit (it may even make things worse). If you want to add the price, you would need to relate it to other state variables somehow. For instance adding something like gas tank value and then reducing it with respect to order value and distance passed with respect to location of the order and location of driver.

Fair point! The thing is mainly that we would like to make the action dependent on factors such as the price (assign a lower-value order to a specific driver) or the destination. Also in the case of including the order destination, I foresee problems in including them in the state space. What would then be the value of 'x_0' in the next state? We cannot choose the next order, since that is not dependent on the action.
My main struggle is that we want to incorporate the characteristics of the specific order and base the action on this, but that these cannot be incorporated in a new state.


text:
How to deal with infinite loops in the MCTS search of AlphaTensor when using a transposition table?
In the published version of the AlphaTensor algorithm, there are two mentions of a transposition table:

In addition, a transposition table is used to recombine different action sequences if they reach the exact same tensor. This can happen particularly often in TensorGame as actions are commutative

and

After simulating $N(s)$ trajectories from state $s$ using MCTS, the normalized visit counts of the actions at the root of the search tree $N(s, a)/N(s)$ form a sample-based improved policy. Differently from AlphaZero and Sampled AlphaZero, we use an adaptive temperature scheme to smooth the normalized visit counts distribution as some states can accumulate an order of magnitude more visits than others because of sub-tree reuse and transposition table.

I don't understand how this use of a transposition table do not lead to infinite loops in the MCTS search, or how to avoid them.
For instance, say that a state $A$ has many children $B_1, \cdots, B_n$. Each of these child is associated to a certain prior given by the Neural Network. Let us assume that the child maximizing the UCPT score is $B_1$. The MCTS search will then go to $B_1$ and expand it if it's a leaf node. Now, say that $A$ is one of the children of $B_1$. It may very well happen at some point that $A$ is $B_1$'s child maximizing the UPCT score. So at some point, the algorithm will:

Start from $A$, reckon that $B_1$ maximizes the UCPT and go to $B_1$
Reckon that $A$ maximizes the UCPT and go to $A$
But at that point, $A$ is not a leaf node, its children are known, and in particular, the one maximizing the UCPT score is $B_1$.

Hence the infinite loop. How to deal with this situation? You want to treat $A$ as a child of $B_1$, so that you don't have to query the neural network for its priors once again, to update the visit count, etc. But at the same time, considering it as a non-leaf node may make this loop appear.
One possible solution would be to create a "copied node", such that all of its characteristics are shared with $A$, but its children are treated as new nodes (though they can also be in the transposition table). That way, each time we get at a leaf node, we will expand the tree instead of circling back. But his seems rather artificial. Is this the best way to deal with this problem?

I would like to acknowledge the excellent work by Nebuly in creating this implementation of AlphaTensor. My answer to your question will be based on his implementation, and I will reference some of his code accordingly. If you are keen on understanding the finer details of AlphaTensor, I highly recommend reading through their code.
Just to clarify, the "infinite loop" you mentioned isn't technically an infinite loop. When executing MCTS, you always need to specify the number of simulations you want the algorithm to perform. Moreover, theoretically, as models are expected to improve with each iteration, such loops should occur less frequently. This is because the model learns from its mistakes, and repeating the same states worsens the returned rewards. Hence, the use of a transposition table in AlphaTensor doesn’t exactly address the repetition of states in a loop but rather serves as a tool to economize the inference cost of the Neural Network.
def monte_carlo_tree_search(
   model: torch.nn.Module,
   state: torch.Tensor,
   n_sim: int,
   t_time,
   n_steps: int,
  game_tree: Dict,
  state_dict: Dict,):

  #More Code

for _ in range(n_sim):
    simulate_game(model, state, t_time, n_steps, game_tree, state_dict)
# return next state
possible_states_dict, _, repetitions, N_s_a, q_values, _ = state_dict[
    state_hash
]
possible_states = _recompose_possible_states(possible_states_dict)
next_state_idx = select_future_state(
    possible_states, q_values, N_s_a, repetitions, return_idx=True
)
next_state = possible_states[next_state_idx]
return next_state

Here's an excerpt from the MCTS function which demonstrates the use of transposition tables, represented by two dictionaries: game_tree and state_dict. The process begins with a for loop that runs n_sim times (the limit of simulations per Monte Carlo play), executing the simulate_game() function. Essentially, this function plays the TensorGame and updates the dictionaries.
def simulate_game(
model,
state: torch.Tensor,
t_time: int,
max_steps: int,
game_tree: Dict,
states_dict: Dict,
horizon: int = 5,
):
"""Simulates a game from a given state.

Args:
    model: The model to use for the simulation.
    state (torch.Tensor): The initial state.
    t_time (int): The current time step.
    max_steps (int): The maximum number of steps to simulate.
    game_tree (Dict): The game tree.
    states_dict (Dict): The states dictionary.
    horizon (int): The horizon to use for the simulation.
"""
idx = t_time
max_steps = min(max_steps, t_time + horizon)
state_hash = to_hash(extract_present_state(state))
trajectory = []
# selection
while state_hash in game_tree:
    (
        possible_states_dict,
        old_idx_to_new_idx,
        repetition_map,
        N_s_a,
        q_values,
        actions,
    ) = states_dict[state_hash]
    possible_states = _recompose_possible_states(possible_states_dict)
    state_idx = select_future_state(
        possible_states, q_values, N_s_a, repetition_map, return_idx=True
    )
    trajectory.append((state_hash, state_idx))  # state_hash, action_idx
    future_state = extract_present_state(possible_states[state_idx])
    state = possible_states[state_idx]
    state_hash = to_hash(future_state)
    idx += 1

# expansion
if idx <= max_steps:
    trajectory.append((state_hash, None))
    if not game_is_finished(extract_present_state(state)):
        state = state.to(model.device)
        scalars = get_scalars(state, idx).to(state.device)
        actions, probs, q_values = model(state, scalars)
        (
            possible_states,
            cloned_idx_to_idx,
            repetitions,
            not_dupl_indexes,
        ) = extract_children_states_from_actions(
            state,
            actions,
        )
        not_dupl_actions = actions[:, not_dupl_indexes].to("cpu")
        not_dupl_q_values = torch.zeros(not_dupl_actions.shape[:-1]).to(
            "cpu"
        )
        N_s_a = torch.zeros_like(not_dupl_q_values).to("cpu")
        present_state = extract_present_state(state)
        states_dict[to_hash(present_state)] = (
            _reduce_memory_consumption_before_storing(possible_states),
            cloned_idx_to_idx,
            repetitions,
            N_s_a,
            not_dupl_q_values,
            not_dupl_actions,
        )
        game_tree[to_hash(present_state)] = [
            to_hash(extract_present_state(fut_state))
            for fut_state in possible_states
        ]
        leaf_q_value = q_values
else:
    leaf_q_value = -int(torch.linalg.matrix_rank(state).sum())
# backup
backward_pass(trajectory, states_dict, leaf_q_value=leaf_q_value)

The states_dict will contain values such as q_values, N_s_a (the visit counter of the state), actions, children's states, and a repetition map to address the commutative nature of multiplication, which results in many repeated states.
Conclusion
In conclusion, if you are interested in examining the implementation of AlphaTensor, I would urge you to read through the repository. If an infinite loop occurs, whether it's within the transposition table or not, it will eventually terminate. Based on this implementation, the simulation maintains a trajectory of the state hashes with each iteration. If a repeated state is found, it will append a reference to the state dictionary and append a copy to the state in the game tree. Therefore, your initial idea of creating a copy is valid. I hope this information is helpful to you.


text:
Can NeuralHash be used as a loss for an Autoencoder?
I've recently read about NeuralHash, and immediately thought that it might be used as a loss for an autoencoder. However, it only seems to preserve "structure" from what I've read, not actual pixel values (which makes sense, given its purpose). Thus, how likely it is that an autoencoder performs well given a loss that compares the NeuralHash of its output with the NeuralHash of its input?
I feel like, assuming that NeuralHash is secure, it should either work well, that is produce an image similar to its input (because the hash is approximately unique) or not work at all (otherwise we would have found a collision), no middle-ground. Is there any thoughts/research on this?


text:
In what situation would you want to use NEAT over reinforcement learning?
NEAT is an evolutionary algorithm. When would you want to use NEAT over more traditional/common RL algorithms like PPO or SAC etc. What advantage does it give you?

It seems that you know what NEAT can be used for. Do you know what RL is used for? If yes, can you give an example of when you think you could use NEAT rather than RL (which doesn't mean it is necessarily a better approach than RL)?

Sure, in the NEAT python repo they have examples of applying NEAT to OpenAI lunar-lander and cartpole problems. Are you asking a rhetorical question (actually just asking so i dont sound dumb lol)?

I was just making sure that you know what RL is used for, and if you know how NEAT could be applied to RL problems. Maybe you can briefly state in your post what NEAT can be used for. To find a policy? To evolve neural networks that represent the policy that you want to find in RL?

Yes, I'm very familiar with the de-facto RL like using PPO, Q-Learning etc. NEAT can be used to find a policy through "evolution" of both the neural net architecture and the weights in the neural net. I'm wondering in what situation would NEAT be better than policy gradient RL.

In my opinion, this shouldn't be an either/or question.  Both NEAT and rl with fixed network topology has their own advantage when solving decision problem.
NEAT is good to solve a simple problem fast with minimum network topology and without local optimum issue.
While RL with fixed topology suffers local optimum but learning more directly, and policy-gradient based Rl would be even more adaptive to an environment with stochasticity in the context of statistics.
Then, why not just combine them? At present, the best paper on this topic is this： https://dl.acm.org/doi/10.1145/3205455.3205536
In this paper, it is suggested that doing NEAT at first, then KEEP DOING RL forever, though I don't agree with this special routine.
I think, both NEAT and RL should do interactively during the WHOLE training process.
The problem is, how to combine these 2 in an effective way. One problem I met is for RL, like SAC, which has 2 outputs(one for policy and one for Q-value), And the topology of the Q-value output has no contribution when doing NEAT. Then, how to deal with the Q-value topology? If both the 2 outputs shares some layers at first, then the learning would become more unstable, since doing NEAT would dramatically change these 2 outputs.
The paper above takes a fixed output layer for both 2 outputs, which I think is just a workaround, not the best way.


text:
Where to start with reinforced learning on actions and rewards sampled from slow ongoing real life system
I would like some pointers, possible projects that solve conceptually similar goals, code examples or tutorials.
I am trying to achieve a system that is able to start or stop ventilation of a given space based different outside and inside metrics such as humidity, temperature, time, etc. to achieve a decrease of relative humidity.
Actuating the system based on simple physics gave me questionable results, this is because I am not able to model the whole dynamic of the system.
I was thinking if reinforced learning could help me learn a good policy. The system should learn on real life data, actuating real ventilation, with the obvious slowness of such system.
I am quite new with AI, able to comprehend and create a simple OpenAi Gym. I am not even sure if and how something like this is achievable with so limited data flow.
I am currently recording and analyze all possible data I can measure, together with some more or less random ventilation sessions. I am sure there are better ways to do this.

RL is very bad at learning in real time, it basically only works through simulators and requires a massive amount of data unfortunately. Generalization is a fundamental problem.

First you'd need to mathematically model your real environment. Probably use some differential equations.
Once you have a good model, you still won't have your real case parameters. So I can see 2 different approaches:

Theoretically + Experimentally: Empirically measure real data to try to find those parameters. (Make a simple PID controller)
Make a robust and general policy that adapts to any parameter.

The first alternative is very straightforward, you basically just have 3 parameters to adjust and there are clear methods to adjust it depending on your data behavior.
Once you are in AI.exchange, I assume you're choosing the 2nd way. I consider it's way harder to implement, but also more fun.
So you'll need create a highly parametrized environment, make a reinforcement learning model, train it on all different kinds of parameters (just make sure your real parameter are somewhere inside that range).
If you do it all right, you should have a robust general policy that can behave well in any general condition.


text:
Is it possible to solve a linear programming problem using reinforcement learning? (DDPG algorithm)
I'm trying to solve a linear programming problem using reinforcement learning.
The linear programming problem is:
\begin{array}{ll}
\text{maximize}_x  & C* x \\
\text{subject to}& A*x \le b\\
                 & x_i \in [0,1],\ where \ i=1,2,3,...
\end{array}
For instance:
\begin{array}{ll}
C &=  [1 \; 2 \;3  \;4]\\
x &=  [x1; x2; x3; x4]\\
A &=  [2 \;3 \;4 \;5]\\
b &= 10\\
\end{array}
I've tried to use the DDPG algorithm to train in MATLAB but the result is not good. Any suggestions for this problem, and is it possible to do so, thanks?

How are you changing a linear programming problem into a markov decision problem? What are the state, actions, and rewards?

In my design,
The state is equal to $C.x$, I mean $[1x1\ 2x2\  3x3\ 4x4]$.
The action is the matrix x $[x1;x2;x3;x4]$.
And the reward is equal to $Cx$, I mean $1x2+2x2+3x3+4x4$, and the constraint (the isdone signal) is A*x<=b. Do you have any comments on that?

Straight theoretical answer:
In theory, yes, it is possible to model this problem as a Reinforcement Learning. But in practice, RL is not the most suitable approach for a simple linear maximization with a boundary. For instance, you could use a Lagrangian.

Practical analysis on your specific problem
In this specific example, you have 1 single constrain: $\sum_{i} a_i x_i \le b$, for an $n$ degree equation (n = size of $X$).
So you might also want to add another boundary, like: all $X > 0$. Otherwise your solution will diverge:

$C = [1 2 3 4];$
$X = [x_1; x_2; x_3; x_4];$
$A = [2 3 4 5];$
$b = 10$

Simple example of divergent solution:
$X = lim_{k=\infty} [-3k, 0,0, k]$
Gives you: $C*X= -3k + 0+0+4k = k$ ✅ Maximum possible reward for $lim_{k=\infty}$
Constrained by $A*X = -6k + 0 + 0 +5k = -k \le 10$ ✅ Minimum possible boundary for $lim_{k=\infty}$

Edit after adding $x_i \in [0,1] $ constraints:
You have described the simplest version of Knapsack Problem, where we can split items in fractions.
For this problem, the greedy solution is very simple and effective:
Calculate a new weight vector: $W = C/A = [ c_1 / a_1, c_2/a_2, ... ]$, which represents the ratio of value $c_i$ $/$ cost $a_i$ for each index $i$.
Now, to have the best value $C$ for a limited cost $A$, you just need to greedy select the $i$ from the largest ratio $w_i$ and "fill your Knapsack" (by increasing continuously $x_i$) until some boundary is filled:

If $x_i\le1$ is reached (you have exhausted all available $x_i$), than proceed to the next best $w_i$.
If total boundary $B$ is reached, than you've finished the algorithm and that's a guaranteed best solution.

Yes, exactly. I forgot to mention that the problem has lower and upper bound constraints for solution x. I've updated it above.

So now you got the Knapsack Problem. I've updated the answer to cover that! ;)

Thank you for your answer, Andre. This problem can be solved easily by using the Knapsack algorithm. But I'm a new learner with Reinforcement Learning and just trying to solve this problem with RL.


text:
How to reduce variance in F1 scores of GAT across multiple runs while using PU Loss?
I am training GAT using a custom loss function(PU Loss) on the Cora and Citeseer dataset. My training file looks like
f1_scores = []
N_ITER = 10
seeds = np.random.randint(1000, size=N_ITER)

for i in range(N_ITER):
    seed_value = seeds[i]
    np.random.seed(seed_value)
    random.seed(None)
    torch.manual_seed(seed_value)
    model = GAT().to(device)
    # train it
    # find f1 score
    f1_scores.append(f1)

print(np.mean(f1_scores))

When I run this file multiple times by doing
 for i in `seq 1 10`; do python train.py; done

I am getting high variance in the values (for e.g 0.43 and 0.76). I don't understand why this is happening even after taking the mean.

Is this the right way to take the mean of the model's F1 scores?
How to reduce this variance?

I have followed the steps mentioned here. I must use a NN. I increased the weight decay (L2) values without any success.

I don't know about PU-learning, but from glimpsing into the introduction of the paper you provided it sounds like this loss is designed for binary classification. Cora/Citeseer contain 6/7 classes and are not binary classification tasks. Did you account for this in your script?

Yes, I chose one class as positive and the rest as negative, as described here.

Hmm, okay then I have no clue but I also don't know about the PU training. Sadly, I currently have no time to read the paper you provide. Maybe you could give a brief overview on that method to make it less effort for the community to provide ideas.

The seeds themselves were randomly sampled. So each time I run the script, I get a different value of the average F1 score.
Now I am not sampling the seeds each time. I have fixed those values.

The high variance is because of the model's sensitivity to the training data. In PU-Learning, we sparsely label(~1%) the positive data for training. Especially in the case of graph datasets, the model performance is sensitive to which nodes have been labelled.

So, variance is at least fixed for the given set of seeds. Now I get identical values each time I run the script.


text:
Policies for which the policy improvement theorem holds
According to Reinforcement Learning (2nd Edition) by Sutton and Barto, the policy improvement theorem states that for any pair of deterministic policies $\pi'$ and $\pi$, if $q_\pi(s,\pi'(s)) \geq v_\pi(s)$ $\forall s \in \mathcal{S}$, then $v_{\pi'}(s) \geq v_\pi(s)$ $\forall s \in \mathcal{S}$.
The proof of this theorem seems to rely on $\pi$ and $\pi'$ being identical for all states except $s$. To my best understanding, this is what allows us to write the expectation $\mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = a, A_t = \pi'(s)]$ as $\mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1})|S_t = a]$ in line 2, which is central to the proof (re-produced from the book below).
\begin{aligned}
v_{\pi}(s) & \leq q_{\pi}\left(s, \pi^{\prime}(s)\right) \\
&=\mathbb{E}\left[R_{t+1}+\gamma v_{\pi}\left(S_{t+1}\right) \mid S_{t}=s, A_{t}=\pi^{\prime}(s)\right] \\
&=\mathbb{E}_{\pi^{\prime}}\left[R_{t+1}+\gamma v_{\pi}\left(S_{t+1}\right) \mid S_{t}=s\right] \\
& \leq \mathbb{E}_{\pi^{\prime}}\left[R_{t+1}+\gamma q_{\pi}\left(S_{t+1}, \pi^{\prime}\left(S_{t+1}\right)\right) \mid S_{t}=s\right] \\
&=\mathbb{E}_{\pi^{\prime}}\left[R_{t+1}+\gamma \mathbb{E}_{\pi^{\prime}}\left[R_{t+2}+\gamma v_{\pi}\left(S_{t+2}\right) \mid S_{t+1}, A_{t+1}=\pi^{\prime}\left(S_{t+1}\right)\right] \mid S_{t}=s\right] \\
&=\mathbb{E}_{\pi^{\prime}}\left[R_{t+1}+\gamma R_{t+2}+\gamma^{2} v_{\pi}\left(S_{t+2}\right) \mid S_{t}=s\right] \\
& \leq \mathbb{E}_{\pi^{\prime}}\left[R_{t+1}+\gamma R_{t+2}+\gamma^{2} R_{t+3}+\gamma^{3} v_{\pi}\left(S_{t+3}\right) \mid S_{t}=s\right] \\
& \vdots \\
& \leq \mathbb{E}_{\pi^{\prime}}\left[R_{t+1}+\gamma R_{t+2}+\gamma^{2} R_{t+3}+\gamma^{3} R_{t+4}+\cdots \mid S_{t}=s\right] \\
&=v_{\pi^{\prime}}(s)
\end{aligned}
Does this mean that the proof is merely proving the special case of the policy improvement theorem for when the policies are identical except at $s$? I am having trouble seeing why the proof holds for the more general case of the two policies being potentially different for all states. In that case, line 2 would not hold and the theorem would not hold for all states as it claims to do.


text:
Why must the value of a state under an optimal policy equal the expected return for the best action from that state?
The Sutton and Barto reinforcement learning textbook states that

the value of a state under an optimal policy must equal the expected return for the best action from that state.

That is,
$$v_*(s) = \max_a q_*(s, a).$$
I am having trouble gaining intuition for this. Since state values can be written as an expectation of the action values under a given policy, I am not sure I see how
$$v_*(s) = \sum_a \pi_*(a|s)q_*(s,a) = \max_a q_*(s, a).$$
I'd appreciate any insights!

I was also confused when Sutton and Barto claimed in chapter 3, as you stated, that

the value of a state under an optimal policy must equal the expected return for the best action from that state.

However, in chapter 4, they present and prove the "Policy Improvement Theorem" and deduce that

It is a natural extension to consider changes at all states, selecting at each state the action that appears best according to $q_\pi(s, a)$. In
other words, to consider the new greedy policy, $\pi'$, given by
$\pi'(s) = \underset{a}{argmax} (q_\pi(s, a))$

The fact that $v_∗(s)=\underset{a}{max} \ q_∗(s,a)$ is immediate from there, given that $v_*$ is the optimal value function.


text:
What should you consider when splitting timeseries data into train and test sets?
My goal is to classify windows of multivariate timeseries data into a positive and a negative class. When I construct sliding time windows and randomly shuffle them into the train and test sets, I get surprisingly high test scores, leading me to suspect data leakage somewhere in my pipeline. Should I first be splitting the data at a chosen timestamp into train and test sets and then building sliding windows for the two sets separately? My understanding is that the rule-of-thumb states never to train with timeseries data from the "future", which is what happens with the way I currently do it.

A timeseries is time dependent so randomly shuffle maybe is not the best option (same for a random split). Check this question

Thanks! Would this be true if my windows are non-overlapping though?

I'm not expert, but probably you can not assume independence between windows, despite beeing disjoint.

@Robert Why is shuffling here bad? It seems like he's trying to train on random independent windows of time.

@JobHunter69, that's right. I am training on random independent windows. This problem also made me curious about the i.i.d. assumption in general. It makes sense to me that randomly split time windows may not be i.i.d. since the underlying process generating the data is identical, and, therefore, adjacent time points are correlated. But then is that not true in all ML? Since we are always assuming the training and validation/test data comes from roughly identical distributions, "adjacent" points would also not be i.i.d.

@bonzo_pippinpaddle but shuffling makes the adjacent points iid right?

Still thinking that "time" is key.  Probably the solution here is in time-series or stats theory. Anyway the assumption of i.i.d. is always curious, many times done in stats but few times happens on real life.


text:
How to incorporate the probability threshold for binary classification into scikit-learn GridSearchCV?
How would I perform grid search in scikit-learn including over the probability threshold for binary classification? In my search, most answers suggest first fitting a model and then performing a loop over different threshold values. However, doesn't the model fitting itself assume a threshold of 0.5? My "best" model doesn't predict any positive classes due to class imbalance. How can I trust the corresponding hyperparameters when playing with the threshold later?

I don't think that tuning the probability threshold in a binary classification model can help improving the model accuracy.
That said, if your best model is "bad" in the sense that it doesn't predict any positive classes then you should change your metric for evaluating the model. In your case it's probably useful to try an F1 metric, you can find more details on how to choose a good metric in imbalanced datasets here: https://www.kaggle.com/code/marcinrutecki/best-techniques-and-metrics-for-imbalanced-dataset#2.3-Choosing-metrics.
When you have a good metric, you can loop over various hyperparameters to find the best combination.


text:
What do equations 1 and 3 describe in the "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" paper?
This paper uses image augmentation to improve RL algorithms. It contains the following paragraph -
"Our approach, DrQ, is the union of the three separate regularization mechanisms introduced above:

transformations of the input image (Section 3.1).
averaging the Q target over K image transformations (Equation (1)).
averaging the Q function itself over M image transformations (Equation (3))."

I do not understand how part 2 and 3 (Equation 1 and 3) and would highly appreciate some detailed elaboration on it.
Here are the equations -

If possible, you can update with equations mentioned in the paper along with notations for completeness.

Equation 1
In normal Q-Learning your target is defined as $y_t = r_t + \gamma \mathrm{max_a}Q(s_{t+1}, a)$. Since you're training a regularized version, you construct the estimated value of the next state via averaging your estimations for each image augmentation. To turn this into the expected value over all $k$ transformations for the given state we need to average it by dividing the summed targets by the number of transformations ${k}$.
Equation 3
Here the Q-Function is updated with respect to all the image transformations. $f(s_i, v_{i,m})$ is the transformed image, i.e. it is the same as $s_i$ but its brightness is increased by 0.5. We fit our action value network on the mean squared error between the output of the net and the Q-Target $y_i$ averaged by the number of images transformations and states.


text:
REINFORCE differentiation on sum or single value?
I'm currently learning Policy-gradient Methods for RL and encountered REINFORCE algorithm. I learned from this site : https://towardsdatascience.com/policy-gradient-methods-104c783251e0 that the gradient of the objective function is calculated as follows:

From what I understand $\sum_{t=0}^{H}\nabla_{\theta}\log{\pi_{\theta}(a_{t}|s_{t})}$ is the sum through the entire trajectory and $\pi_{\theta}(a_{t}|s_{t})$ is the policy of the agent at time step $t$. However in Suton's book the gradient objective is defined differently.

There is only $\nabla \ln{\pi(A_t | S_t)}$ at time step $t$ and no sum of all time steps. So does the algorithm not consider the policy for the whole trajectory when updating? Only a single-step policy?
Furthermore, there is $\gamma^{t}$ (discounted reward) term in the latter and not the former. What is the reason for that?
Hopefully, someone can help me clarify this.

the site you learn from accumulates the gradient at each action taken in the episode via the sum and performs a single gradient ascent step for all these actions (this is analogous to taking the loss over a batch before performing a gradient update in supervised learning), whereas Sutton and Barto perform a gradient ascent step for every action taken (analogous to performing a gradient update for every single data point individually in supervised learning).

@norbertk thanks, I missed the bit of the question about the second discount term.
OP, essentially the second discount term is needed because you also need to discount your state distribution. This is rarely done in practice but it is theoretically incorrect to omit it, and
this paper shows without the second discount factor the gradient is not actually a gradient at all.

Hello. Welcome to AI SE. Rather than writing "Help with understanding the REINFORCE algorithm", please, just put your specific question in the title.


text:
Reinforcing Learning when action has no effect on the environment
I am trying to get my head around a problem where the action by the agent can not change the environment. Without going into details, my problem is about error correction in an stochastic environment.
So, here the action by agent can not change the environment that causes these error and all we can do is to smartly correct as the errors happen. I am currently thinking about using Reinforcement learning for this agent who could correct the errors.
Now my questions are:

Would reinforcing learning be an overkill since the agent can not  influence the environment?
How do RL, LSTM, and even random forest compare in such scenarios?

Thank you.

Could you clarify what the action is to "smartly correct"? Is it a numerical value (or vector of values) that adds to some other prediction, or is it a selection from different types of correction (e.g. choose between method "A" vs method "B" for correction)?

@NeilSlater. Here, by "smartly correct" I mean, if my machine learning approach (whatever it is which is not implemented yet) is successful and it comes up with a strategy then it can "smartly correct" the detrimental effect of the noise. And it is an action. for example, if noise flips a binary bit, the agent should be able to fix that by flipping it again.

Sorry I don't understand that. Could you please state what form your actions take? Are the actions numerical (e.g. "correct by +1.345") or categorical (e.g. "correct using choice 5 out of 10")? Or is the action literally a vector of bits to flip? I may have further questions to clarify details of your reward/loss function and how you are establishing ground truth, but best to start with details of action. What I am trying to understand is what the supervised learning version of this would look like, in order to figure out the asnwer to your second question.

Thanks@NeilSlater  The action is one of few operations the agent can take, let's say, integer from 1 to 4. For example, 1 is for turning on switch 1, and so on. To be more specific, I few sensors that give me some values, like temperature, how crowded the area is , and so on. Now I would like to find the best strategy, which is turning on and off some switches based on the information I got from sensors to correct the disturbances caused by the environment. As it is obvious, I can not do anything to change the environment that causing these errors, I can not even reduce them.

OK, so your error correction choice is discrete, one of a few options. How do you then know whether the action is correct? Do you have some ground truth for historical data? Do you have any way to assess an action taken in either simulation or in reality as being the correct action?

The party who creates that system knows what state the system is supposed to be at all time. In ordinary operation that knowledge is hidden from us and all we can do is just keep correcting for errors cause by the environment. However, in training we can start with a state that we create ourselves and have perfect knowledge and the RL agent is supposed to preserve it by error correction. So, creating training set is easy.

I am interested in your use of the word "state" here. You imply that there is hidden state, and that the agent's actions do in fact impact it? Could you clarify what the agent's actions do? The implied behaviour from your question is that the environment produces some data output, depending on a known state, which is consumed outside of the environment, but the output can be wrong (again depending on the known state) and the agent's job is to correct that output. Your comment above implies instead there is hidden state and it is the agent's job to somehow maintain it?

Short Intro
It's very common for people to think that Deep Learning is a "superior form" of Neural Network, a "smarter model". And then they try to use DL for solving simple tasks and they'll find more problems than solutions.

We might think of plane as superior to a bike. But when we need to buy some bread for breakfast, taking a plane is not even an overkill. It's just nonsense.

Your problem
In a similar way, I've seen people thinking Reinforcement Learning as somehow superior to Supervised or Unsupervised Learning. So let's establish a baseline:

The choice between Reinforcement Learning or Supervised Learning is not about superiority, but rather the nature of the task and your available dataset.

Reinforcement Learning
Great when your problem can be modeled as an agent interacting with an environment. The agent will sense (input), process (policy) and act (output). The policy is learned based on the reward of each action.
In most RL tasks (like strategy games), you can't objectively rate each individual action. Instead, it takes multiple actions before the outcome is obtained. But you can't train your model when you can't measure it's performance. So how to train a RL model?
Policy:
A policy is like a function that maps states into actions. When you can't measure the performance of each individual action, you create a policy.
You let the agent interact with the environment (play the game) until you have a score. If the environment is stochastic (the game depends on luck), you might want several rollouts (play lots of times) before evaluating the policy.
So, for acquiring a single data sample, you might need to let an agent to play a game several times.
But if you can directly measure the agent's perform every each action, congratulations! You can probably save lots of computational power by modeling the problem as a:
Supervised Learning
The model is learned based on the input-output pairs. A training dataset.
LSTM
Great when your input is sequential and your output is a function of the previous state.
Random Forest
Great when your input is not sequential, but you have a lot of features.
Unsupervised Learning
Great when your input is not sequential and you want to discover hidden structure in your data.
I hope this helps you with both questions. :)

Thank you for the insightful categorization. Since my data is sequential, is it correct to say that if I know the best policy for every predicted future ( Let's say coming from LSTM), that would be the same as RL predicting the future and finding the best policy? How important is it that the agent in RL is able to affect the environment?

I've edited the answer, adding brief description about policy. I hope it answers your question.

Thank you, Andre Goulart. It was a very informative answer.


text:
How to decide which column has more weightage to output

As per Image we can see Column_A value is directly proportional to output,
While Change in value of Column_B has no effects in output.
So basically I want to know is there any algorithm where I can get weightage of columns which is affecting more to output than another.
In this case we can see Column_A has more weightage as change in Column_A value will lead to change in value of B.
but in case of Column_B it will be no change.
Expected Output:
so is there any algorithm to detect this change something like it will give 100% to Column_A and 0% to Column_B

Hi! I recommend searching for articles about "feature selection". This is what this process is usually called.

Scikit-learn also has a submodule specifically for this: https://scikit-learn.org/stable/modules/feature_selection.html


text:
Skip-Gram Model description in word2vec explanation article
In his article word2vec Parameter Learning Explained Xin Rong says (page 7):

Each output is computed using the same hidden->output matrix:
$$ p(w_{c,j} = w_{O.c}|w_I)=y_{c,j}=\frac{exp(u_{c,j})}{\sum_{j'=1}^{V}exp(u_{j'})} \ \ \ \  (25) $$

Looking into the word2vec source code I don’t see any “panels” or “output layers”. With this most of the terms in the equation above in totally unclear for me. Could you please help me realizing how this mathematical description intended to work (I understand the source code, but mathematical description is another question here).
Am I missing something here? With the description above from the article, since “the output layer panels share the same weights”, how the results on output panels (even if they existed could be different)?

The right hand side is called a softmax function, it turns a set of numbers, eg. [10, 3, 1, 0] into a probability distribution [0.99, 001, 0, 0] which always totals 1. It means the probability of the 1st token/word appearing next is 0.99, the probability of the 2nd token appearing next is 0.01, and there is no chance of the 3rd of 4th token appearing next.

@James, thank you for the input. I know what Softmax is and more than that I know how hierarchical softmax works here. May be my wording of the question is vague, but I am concerned move with C-indexes here, since in the code which is described by this formula there are no panels at all. Meanwhile I noticed that the actual code uses "window" of words in the text so maybe the panels are "virtual" in the implementation. That is my question here. And if we have the same input word and same weights on the panels, how the outcome could be different. I know how it works in code, but the formula...

I skimmed through the pdf but the index notation seems a lot of work to wade into. Conceptually though as I understand it the skipgram is a simple tallying of surrounding words relative to the current word, eg. for sentence "am i missing something here", the skipgram for the word "missing" should read [0, 0.5, 0, 0.5, 0] for just this one sentence. The probabilities gets muddied where there are multiple occurences of the word "missing" throughout the learning text.

@James, thanks again. And again, I know what is the skip-gram and how exactly it works. I was interested in one thing, how math describes the existing code to learn they way to cover things. The author of the paper here uses "panels" which are not represented explicitly in code which was my first concern. Then he says "the output layer panels share the same weights", if this is true than for the same input word they must produce the same output. Again, I know how this works (correctly in code), I am struggling with the "math" description of this.


text:
What are strategies for data driven weights initialization?
I am beginner in deep learning and currently training a few neural networks (Pytorch) for problems in audio and speech. For my tasks, simple feed-forward networks are working well enough. I use basic layers like Linear, ReLU and Softmax with nll loss. I have tested a few initialization schemes provided by Pytorch and noticed that initialization has significant (but not high) effect in the speed of training and the final accuracy of the model. I am currently using torch.nn.init.kaiming_uniform_ for initialization.
In my understanding, all these are data independent initialization schemes. I would like to try something that is data dependent. I saw a few pre-training strategies with unsupervised learning followed by supervised learning, but they seem overly complicated.
I am looking for something simple where I can use (preferably a fraction of the) training data to 'tweak' weights to better positions before the training. Are there any such strategies?
Addendum-1:
Current initialization schemes (AFAIK) are mostly random values with control over range or energy to prevent values from dying down or blowing up. My aim is to further improve the starting point of training by taking account training data (or at least some of it). I am thinking of something like this. We pass a few batches of training data through the initial network and collect statistics on neuron outputs. Based on this, we identify the misbehaving neurons and tweak the weights and biases to reduce such issues so as to improve the training speed or accuracy. Is there anything of that kind?

Hello. Can you clarify why you want to do this? Wouldn't this basically then bias the training process based on the data? Maybe you could use a separate dataset (from the training dataset) for initialising the weights: in this case, it would probably be less biased, but I still don't understand why you want to do this.

@nbro, kindly see Addendum-1.

Based on your addendum, there doesn't seem to be a big difference between what you seem to want and just training the neural network.

What you describe seems just like training (where tweaking problematic weights correspond to taking gradient descent steps) as @nbro has pointed out. Regardless, there's something called meta-learning, and it seems related to me. In meta-learning, you're trying to initialize weights in a way such that, the model can be adapted to different downstream tasks quickly.

@nbro, The key difference is that we are not using back-propagation to minimize a 'differentiable' cost function. Instead, we can tweak the weights using a bunch of heuristics that may or may not be differentiable.


text:
How to use the tf.image.SSIM function
Can anyone please help me understand how to use this SSIM function (https://www.tensorflow.org/api_docs/python/tf/image/ssim)? The filter_size parameter has a default value of 11, according to the documentation information, "the image sizes must be at least 11x11 because of the filter size." If I then use an image of size 8x11, how should I adjust this, and if applicable, other function parameters? Thanks.

The filter_size parameter in the tf.image.ssim function determines the size of the Gaussian filter used to smooth the image before calculating the SSIM.
Option 1: Adjust Filter Parameters
When using this function with an image smaller than 11x11 pixels (the default filter size), adjust the filter_size parameter to a smaller value. For an 8x11 image, a filter size of 3 or 5 would be appropriate.
While you don't necessarily need to adjust the filter_sigma parameter, it's worth experimenting with slightly smaller or larger values than the default (1.5) to see how it affects the results. Generally, a smaller filter size might work better with a slightly smaller sigma to avoid over-smoothing.
The best approach is to try different combinations of filter_size and filter_sigma to find the optimal values that balance accuracy and detail preservation for your specific image.
Option 2: Pad the Image

Padding to your image, bringing it to a size of at least 11x11 pixels.
Be aware that the added padding will be included in the SSIM calculation, potentially affecting the results.

Choosing the Best Option:

Simplicity: If you prefer a straightforward approach and don't mind the potential impact of padding on the SSIM results, padding the image is a good choice.

Control and Detail Preservation: If you want more control over the calculation and need to prioritize detail preservation in small images, adjusting the filter parameters might be a better option.


text:
Different ways to calculate backpropagation derivatives, any difference?
I'm studying error backpropagation in neural networks. I am interested in why we use only one path on the computational graph to get the value of the derivative for a weight? I ask the question because there are several paths in the computational graph to get the  derivative for a particular weight. Why do we only use a one value? Why don't we combine the values from all possible paths?
Schema:

Fromulas:
Normal path:
$$\frac{\partial E}{\partial w_{1,1}} = \frac{\partial E}{\partial Out} \cdot \frac{\partial Out}{\partial a_{1,1}}\cdot  \frac{\partial a_{1,1}}{\partial a_{0,1}}\cdot  \frac{\partial a_{0,1}}{\partial w_{1,1}}$$
Alternate path:
Normal path:
$$\frac{\partial E}{\partial w_{1,1}} = \frac{\partial E}{\partial Out} \cdot \frac{\partial Out}{\partial a_{1,2}}\cdot  \frac{\partial a_{1,2}}{\partial a_{0,1}}\cdot  \frac{\partial a_{0,1}}{\partial w_{1,1}}$$
Why don't we consider both derivatives or the sum of them?

All of them are calculated and added together.


text:
Is there something like person-specific sentiment analysis?
Sentiment analysis, as we know, measures "Cake sucks" as say -0.4, and "Cake is great" as 0.7.
What I'm looking for is something a bit different like so:

Given input text data written by 1 person (say a blog)
Predict how they (the person who wrote the text) might react to a certain piece of text

What might something like this look like?

Let's suppose that Person A with a blog has written in his blog posts thousands of times about how much cake is the best thing to happen to humanity.
The system should probably infer that if that person read something like "Cake is the WORST food ever", they would react negatively to it, if say, they also believe that there is such a thing as 'objective taste' somehow (aesthetic absolutism).
Or if Person A has made anti-racist statements, that racist statements would be strongly negative.
If Person A reads the statement "I hate lawyers" and in their blog they have written about how they don't care either way about law, it should probably be 0.
Finally, if Person A reads the statement "iPhones are better than Android" and there is zero data about either iPhones or Androids, or even related data about Apple or Google, then it should probably be 0, with an additional "confidence" metric at 0 (since there is no data, this confidence metric will let us know whether there is any data to support the measurement or not).

This model would need to be able to somehow inductively 'infer' a value system of some kind, and assign intensities of probable reactions based on the frequency of an expressed view, as well as pick up on nuances (such as philosophical assumptions, (for example in the above cake example: aesthetic absolutism) etc.) that may inform that measurement.
In other words, I'd like to create a model (or find a pre-trained model to fine-tune), that would be able to, given text data from that 1 person, predict their sentiment in response to a new piece of text.
Would love any help whatsoever regarding:

What types of pre-trained models I should look at
Any ideas of any kind whatsoever you might have on how to achieve this
What sorts of architectures/resources/concepts may be relevant to look at


text:
How do convolutional layers of basic Graph Convolutional Networks work?
I was reading the following article on Towards Data Science (here) and it says the following, regarding the calculation of convolutional layers:

So the overall steps are:

Transform the graph into the spectral domain using eigendecomposition
Apply eigendecomposition to the specified kernel
Multiply the spectral graph and spectral kernel (like vanilla convolutions)
Return results in the original spatial domain (analogous to inverse GFT)

Question: How can we visualize the convolutional layer working for a graph neural network?
For example, for a CNN we can imagine the following (source: Stanford CS231n YouTube lectures, Lecture 5: Convolutional Neural Networks (here)). What is the analogous image for a graph convolutional filter?

Note that there are several graph neural networks and, from what I remember when I was studying them, they can be quite different, so it may not be possible to give an explanation that applies to all GNNs. Moreover, it seems to me that now you changed the question to ask for a diagram that illustrates how the convolutional layer for some specific GNN works. If that's the case, you may also want to change the title to reflect that.

Thanks @nbro, I'll make the updates in the morning. Do you have any links to resources you used when you were studying the topic?

I didn't study geometric deep learning or graph neural networks thoroughly. I had read a few papers and articles online, so my knowledge of the topic is not very solid, so I don't really have one resource that I would recommend. Back then (i.e. about 2-3 years ago), there weren't many good resources on this topic. Maybe this answer is useful. Don't forget to upvote the question and answer there if you find them useful ;). In any case, it seems to me you're already aware of this book.

Actually, the given pipeline was used in the old days of Graph Neural Networks.
Canonical paper on the subject is Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering.
You start from arbitrary graph with adjacency matrix $A_{ij}$ (let us assume that graph is undirected), such that:
$$
A_{ij} =
\begin{cases}
1 & \text{if there and edge between vertices $i$ and $j$} \\
0 & \text{otherwise}
\end{cases}
$$
Then one constructs graph Laplacian:
$$
L = D-A
$$
$D$ is the degree matrix (number of edges entering the given vertex). There are also different normalizations in the literature, like $I - D^{-1/2} A D^{-1/2}$ is called normalized Laplacian.
This matrix has several properties:

It is symmetric
Non-negative definite

From the first statement it follows, that the matrix can be diagonalized due to the Spectral theorem,
Therefore, it makes sense to perform the eigendecomposition of this operator. And the eigenvectors form the graph Fourier basis.
Note, that in the special case, when the graph is a regular square grid, graph Laplacian just the discrete Laplace operator:
$$
\begin{pmatrix}
0 & -1 & 0 \\
-1 & 4 & -1 \\
0 & -1 & 0 \\
\end{pmatrix}
\qquad
(\text{for 2d case})
$$
And the Fourier basis consists of plane waves:
$$
\sim e^{i (k_i i + k_j j)}
$$
The next important fact is the Convolution theorem that states that convolution of two signals can be done as inverse Fourier transform of the dot product of their Fourier transforms:
$$
f * g = \mathcal{F}^{-1} [\mathcal{F}[f] \cdot \mathcal{F}[g]]
$$
These operations correspond to steps 3 and 4 in the pipeline in the OP.
I am not aware of the simple geometrical intuition in this setting. But in the following research, full eigendecomposition was truncated to Chebyshev polynomials, and then up to the first term in the decomposition, which gave rise to Graph Convolutional Networks by T.Kipf.
They allow for more intuitive and visual interpretation. Given adjacency matrix $A$, input feature map on the graph $H^{(l)}$ one defines the output of the layer to be:
$$
f(H^{(l)}, A) = \sigma(\tilde{D}^{-1/2} A \tilde{D}^{-1/2} H^{(l)} W^{(l)})
$$
$\tilde{D}$ is the graph Laplacian for adjacency matrix with self-loops (arrows $i \rightarrow i$).  $W^{(l)}$ is the matrix of learnable parameters.
In essence, one transforms the feature vectors $ H^{(l)}$ by a pointwise linear transformation, and aggregates the information from the neighborhood with some coefficients. Equivalently the function above can be rewritten as;
$$H_{i}^{l+1}=\eta\left(\frac{1}{\hat{d}_{i}} \sum_{j \in N_{i}} \hat{\boldsymbol{A}}_{i j}{W}^{l} H_{j}^{l}\right)$$
From the formula above one can see, that graph convolution is the summation of the features in the neighborhood vertices which are transformed by the weight matrix ${W}^{l}$ and division by the normalizing constant.

You may also explain or at least link to some other external resource that motivates your claim "it makes sense to perform the eigendecomposition of this operator" (i.e. the normalized Laplacian). In other words, why if a matrix is symmetric and non-negative semi-definitive, then we can find the eigendecomposition? What also does "graph Fourier basis" mean? Why do we care about this basis? And why does the eigenvectors of the normalised Laplacian form this basis?

You don't need to explain this completely in your answer, but it may be a good idea to link to an external resource that explains this for people not familiar with these concepts. Finally, to make this a better answer, I would also explain the relationship between your diagram and the last sentence. For people that are more or less familiar with GNNs, that last sentence makes sense, but it may not be clear how it is related to the diagram.

@nbro yeah, thanks. I've added links and made some fixes

Thank you very much @spiridon_the_sun_rotator! This is very clear and aligns with what I have learned since making this post. Apologies for not responding sooner, but have just logged back on.