text:
a loss for binary step function data
I have some data with ground truth that looks like a binary step  function, where part of it is 0 and part is one.
An example for the GT can be like 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0
or something like this

I have a hard time to come up with a loss function that can optimize this problem, the simplest option would be something like CrossEntropy or BinaryCrossEntropy, but I am wondering if there is any other loss that I try.
Something that can take into account the property that when the GT is one (1) it is continuous 1 and when it is zero it is continuous.
To give a little more information, for example, I will never have a GT that be like this 0 0 1 0 1 0 1  also I will never have a GT like this  0 0 0 0 1 1 1 0 0 0 0 1 1 1 0 0 0. In other words, I will have one time ones in a continuous way (it can be at the start or middle or end) but it wont be two discontinuous 1s.
Is there any loss function that take into account this properties?


text:
Calculating state-value functions in Markov Decision Process
I am watching David Silver's lectures on RL available on YouTube. My question here is with regard to Lecture 2 (Link to Video). At 1:11:00, I could not understand how he is calculating the state-value functions for C1, C2 and C3 (nodes with values 6, 8 and 10 respectively) in the student MDP example, starting from C3 and working backwards. Can someone please explain this?

I couldn't find any states labeled with C1 C2 or C3 at the given timestamp.

Hello. It may also be a good idea to provide at least a diagram or a screenshot of it that illustrates the problem, if available.

@tnfru Sorry about the lack of clarification! C1, C2 and C3 are the three nodes in the center of the diagram! (nodes marked with values 6, 8 and 10 respectively at the mentioned timestamp!)

I looked at this not long ago.  You need to understand that the slide is referring to an optimal (not expected) value function & optimal Action-Value function.  Let's look at his diagram.  From left to right you have state C1 = 6 for its optimal value, C2 has 8 and C3 has 10.

If we take C10, we can see there are 2 choices for the transition, 1) to the pub for a reward of +1 or 2) to study for +10 reward.  There are no probabilities assigned to our decision, so we will take the action that maximizes our action-value.  So, being in C3 and deterministically choosing to study gives a reward of 10.  The action is studying and the reward is 10 so the action-value is 10 + the undiscounted value of the next state.  From there you end up in the sleep state (with value 0) with no further reward.
If you are in C3 and deterministically choose the action to go to the pub, you receive a reward of +1 with 3 possibilities of states/values.  Given this action value of C3 & going to pub, you can apply an expectation over values of where you end up (not in the choice you make from C3).  Since you can end up in C1, C2 or C3 with their own probabilities and values, you end up with an action-value of 8.3 by choosing to go to the pub.
Likewise for calculating optimal action values for C1 and C2.  You see which deterministic action gives the maximum sum of reward and next state.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Training a RL agent using different data at each episode
I am training a RL agent whose state is composed of two numbers, ranging between 4 ~ 16 and 0 ~ 360. The action is continuous and between 0~90. In real life, the states can be any I am training a TD3 agent using the stable baselines library. In real life, the state may be any pair of numbers in the aforementioned range. Hence, I am generating random numbers for training. Leading to different data at each episode. I have realized that the trained agent is predicting actions just in the boundaries of the action range. Could this issue be caused by using different data for the different episodes of the training?.
edit:
In the real application of my algorithm the pair of numbers will be arbitrary, given that they are in the corresponding range. By random numbers  for training I mean that I am generating pairs of numbers using uniform distributions between the given boundaries. And using them to train the system. The reward is a function of these numbers, however it does not have an analytic expression.

Can you describe in more detail what you mean with training it on random numbers? Do you mean that the target is non stationary (chances each episode)?

This could be a problem related to your implementation So, it might also be a good idea to provide the link (not the code) to the source code for people that would like to look at it, although programming questions (so debugging programming issues) are off-topic here.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to reduce the dimensionality of the actions in RL
I have a single-agent RL model in which the dimension of the dimension of the action space is $70$. This action space is too big and the deep RL agent is not learning properly. The boundaries of the action space are $-1$ and $1$.
My question is, how can I reduce the dimensionality of the action space?
I have tried to use auto-encoders with random vectors of dimension $70$ between $-1$ and $1$, but it is not working properly. I am training the encoders using a hidden layer with 10 neurons. However, comparing the original action with the result of encoding and decoding it I can see that the average difference between the components is $0.2$ when the action is in the range $[-1,1]$

Can you provide more details about 1. how you're using and training auto-encoders and 2. what you mean by "it's not working"?


text:
How to deal with variable action ranges in RL for continuous action spaces
I am reading this paper on battery management using RL. The action consist in the charging/discharging power of the battery at timestep $t$. For instance, in the case of the charging power, the maximum of this action can be given by the maximum charging speed $c^{\max }$ or by the state of charge of the battery, since it cannot be charged more than $100\%$. Therefore, the charging action has the following range:
$$
0 \leq c_{t} \leq \min \left\{c^{\max }, \frac{B^{\max }-B_{t}}{\eta_{c}}\right\}
$$
In some timesteps the maximum of $c_{t}$ will be $c^{\max }$ and for others $\frac{B^{\max }-B_{t}}{\eta_{c}}$. What would be the best way of implementing a variable action range? I have thought in using a range $[0,1]$ for the action, scaling it to the suitable range. Is there any standard way to deal with variable ranges?.

Here is a related question, but your question seems to be in the context of continuous action spaces, so I would recommend that you change your title to emphasize that. See also this.


text:
How to build neural network that detects changed signal firing pattern and is trained on positive patterns only?
Let's have a set of n devices firing signals. Devices are firing in the same cycles, but each device can fire in different phase of the cycle. More, the exact firing point can fluctuate, for example 20%
off the phase to both sides.

For me, this repeated firings can be seen like some kind of time-based pattern.
The main goal is to detect if some device changed it's phase, so the timings in the pattern are changed.
I generated dataset with rows for each second. On each row there is actual time and  another n-1 columns representing difference between firing time of this device and first device.
Example: let's have 4 devices - D0 to D3. t1 means difference of firing times between D1 and D0.
time     t1 t2 t3
13:00:00  5  2  7

That means that D1 fired 5s after D0, D2 fired 5+2s after D0 and D3 fired 5+2+7s after D0.
Also there can be negative numbers since the allowed fluctuation - the D2 could fired earlier than D0.
To amateur me this seems that this numbers create patterns that some kind of NN should be able to learn and filter out the noise.
My questions:

which kind of network should I use? I tried to do simple feed forward network and do a classification, but since I am generating the dataset, I have no negative data and I so I am unable to categorize it.
is this kind of dataset a good approach, or should I use other properties?

Why don't you generate negative data? What makes negative data in this case negative?

The negative data is with at least one device fires out of phase (accounting the fluctuation), disrupting the pattern of firings in the cycle. It is not easy to generate that firing randomness using same tool that generates the correctly timed pulses.

If there are no negative examples, the neural network will always predict yes because it is always true. You very likely need to find a way to generate negative samples.

Taking that I am unable to generate negative samples (for now), is there any other approach? I tried feed the data as time series (one row for every pulse) and set the lag of the window to number of devices. I've trained the network and got some results, but they are not satisfactory - the prediction of the next element is sometimes a way off.

Yes you can formulate it as a time series problem, which makes more sense. The problem without negative samples still exist. You can try predicting the next element instead of classifying if the phase changed. If you're interested in this I can formulate an answer on that basis.

Yes, my classification was: if the prediction is close enough to real firing time, than the device is in phase. But the predictions were off sometimes. Yes I am interested in any correct solution :) I am using encog-java library if that matters.

I'm not sure if the way you classify will work out. You're inducing a really big bias - after consideration I don't think my proposed methodology (RL or transformer) will solve your issue.

I can't really understand the real problem here. I believe you don't even need to use Machine Learning here. Please try to include a sample of the real input and the output expected, so I can help you further.


text:
Multiple GRU layers to improve a text generation
I am using the model in this colab https://colab.research.google.com/github/tensorflow/text/blob/master/docs/tutorials/text_generation.ipynb#scrollTo=AM2Uma_-yVIq for Shakespeare like text generation.
It looks like this
class MyModel(tf.keras.Model):
  def __init__(self, vocab_size, embedding_dim, rnn_units):
    super().__init__(self)
    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
    self.gru = tf.keras.layers.GRU(rnn_units,
                                   return_sequences=True,
                                   return_state=True)
    self.dense = tf.keras.layers.Dense(vocab_size)

  def call(self, inputs, states=None, return_state=False, training=False):
    x = inputs
    x = self.embedding(x, training=training)
    if states is None:
      states = self.gru.get_initial_state(x)
    x, states = self.gru(x, initial_state=states, training=training)
    x = self.dense(x, training=training)

    if return_state:
      return x, states
    else:
      return x

I look for strategies to improve the model, under the hypothesis that there is a reliant why to assess the goodness of my model; how could adding one or more GRU layers improve the model, e.g., number of rnn_units, number of layers, for such a stacked model ?
For instance, this gives extremely bad results
class MyModel(tf.keras.Model):
  def __init__(self, vocab_size, embedding_dim, rnn_units):
    super().__init__(self)
    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
    self.gru = tf.keras.layers.GRU(rnn_units,
                                   return_sequences=True,
                                   return_state=True)
    self.gru_2 = tf.keras.layers.GRU(rnn_units,
                                   return_sequences=True,
                                   return_state=True)
    self.dense = tf.keras.layers.Dense(vocab_size)

  def call(self, inputs, states=None, return_state=False, training=False):
    x = inputs
    x = self.embedding(x, training=training)
    if states is None:
      states = self.gru.get_initial_state(x)
    x, states = self.gru(x, initial_state=states, training=training)
    if states is None:
      states = self.gru_2.get_initial_state(x)
    x, states = self.gru_2(x, initial_state=states, training=training)
    x = self.dense(x, training=training)

    if return_state:
      return x, states
    else:
        return x

Please limit posts to one question, as it stands there are three here: "How can GRU improve my approach?", "What guidance exists for RNN hyperparameter tuning", and "How do I assess the performance of my model?"

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

A single recurrent network layer $l=1$ in the original example carries over its own state $S^{\mathrm{out}(l=1)}_{t-1} \to S^{\mathrm{in}(l=1)}_{t}$ from one minichain/minibatch to the next along the increasing sequence's time[like] (w.l.o.g.) discrete coordinate¹ $(t-1) \gets t$ equidistantly² increasing from one invocation to the next, via the states argument to the MyModel.call() function.
THE INVARIANT $(\mathcal{I})$: Each recurrent layer $l$ has its own carry-over state
$$S^{\mathrm{out}(l)}_{t-1} \to S^{\mathrm{in}(l)}_{t} \mathcal{\tag{I}}$$
This is not what's going on in the second code example, and is indeed the root cause of the very expected, miserably invalid backpropagation gradient approximations. The code takes in $S^{\mathrm{out}(1)}_{t-1}$ and correctly passes it as the input state to the GRU layer $l=1$, receiving a new state $S^{\mathrm{out}(1)}_{t}$ as part of the layer's return tuple. But then the horrible happens: the code passes this state to another layer $l'=2$ as if it were $S^{\mathrm{in}(2)}_{t}$, which is obviously a no-no-no-no-no! These states are unrelated, because they belong to different internal spaces: each one is that of its own respective layer! The tensor[-likeness] $S^{*(l)}$ lives on an entirely different manifold than $S^{*(l')}$ does for $l \ne l'$!
What you should do instead is collect every $S^{\mathrm{out}(l)}_{t}$ for each layer $l$ separately, and pass them to their respective layers as $S^{\mathrm{in}(l)}_{t'}$ on the next iteration and in the same states variable, after the discrete co-ordinate value $t' \gets t$ is advanced by the framework, to ensure that the invariant $(\mathcal{I})$ is conserved.
The code for a model with two recurrent layers, having being corrected for the inconsistency noted above, and including the initialization of the states $S^{\mathrm{in}(l)}_{0}$ for the initial time[likeness] $t=0$ and both layers $l=\{1,2\}$, may look like the following:
class MyModel(tf.keras.Model):
    . . .
    self.gru1 = tf.keras.layers.GRU(rnn_units_1,
    . . .
    self.gru2 = tf.keras.layers.GRU(rnn_units_2,
    . . .

  def call(self, inputs, states=None, return_state=False, training=False):
    . . .
    (s1, s2) = states or (self.gru1.get_initial_state(x),
                          self.gru2.get_initial_state(x))
    x, s1 = self.gru1(x, initial_state=s1, training=training)
    x, s2 = self.gru2(x, initial_state=s2, training=training)
    . . .
    return (x, (s1, s2)) if return_state else x

The False-ity of the None literal value together with the or construct may be conveniently used in this manner to make the code arguably³ more readable.
In the case of packing more recurrent layers into the model, the tuple may be an awkward structure to work with; whether storing the references to all RNN layers in, and passing their respective internal states via a list, tuple or another data structure, is your code readability judgment call.
____________________________
¹ Indices, as do all contravariant tensor bases, transform covariantly. We indicate this fact by the using of a reversed arrow.
² We're using the increment of $1$ w.l.o.g.
³ But please don't: this is far beyond the scope of the question. There's CodeReview.SE for that! :)


text:
How to use TPU for real-time low-latency inference?
I use Google's Cloud TPU hardware extensively using Tensorflow for training models and inference, however, when I run inference I do it in large batches. The TPU takes about 3 minutes to warm up before it runs the inference. But when I read the official TPU FAQ, it says that we can do real-time inference using TPU. It says the latency is 10ms which for me is fast enough but I cannot figure out how to write code that does this, since every time I want to pass something for inference I have to start the TPU again.
My goal is to run large Transformer-based Language Models in real-time on TPUs. I guessed that TPUs would be ideal for this problem. Even Google seems to already do this.
Quote from the official TPU FAQ:

Executing inference on a single batch of input and waiting for the
  result currently has an overhead of at least 10 ms, which can be
  problematic for low-latency serving.


text:
What is the computational complexity in terms of Big-O notation of a Gated Recurrent Unit Neural network?
I have been digging up of articles across the internet in context of computational complexity of GRU. Interestingly, I came across this article, http://cse.iitkgp.ac.in/~psraja/FNNs%20,RNNs%20,LSTM%20and%20BLSTM.pdf, where it takes the following notations:
Let I be the number of inputs, K be the number of outputs and H be the number of cells in the hidden layer
And then goes on to explain the computational complexity of FNNs, RNNs, BRNNs, LSTM and BLSTM computational complexity is O(W) i.e., the total number of edges in the network.
where

For FNN: $W = IH + HK$ ( I get this part as, for fully connected networks, we have connections from each input to each node in hidden and subsequently for hidden to output nodes)

For RNN: $W = IH + H^2$ + HK ( The formula is pretty same as is it for FNN but where does this $H^2$ come into picture?)

For LSTM : $W = 4IH + 4H^2 + 3H + HK$ (It becomes more difficult as it comes down to LSTM as to where the 4's and 3's come into the equation? )

Continuing with these notations, can I get a similar notation for GRU as well? This can be very helpful for understanding.

You can use latex on this site. So, I suggest that you edit your post to format your equations/formulas with latex.

Did you get the BigO for LSTM?

authors: Руслан Тургунбаев


text:
How to model the inputs and outputs of the neural network for the Splinterlands card game?
I have recently just completed a course on deep learning and I feel like an intermediate, but I still don't know how to structure this problem.
I'm looking to create a NN to play the card game Splinterlands.
I have the history of battles, the cards played, who won, and the battle rules and constraints.
I'm struggling with how best to model the inputs. I think that standard feature engineering and the encoding of all the variables affect the game, like manor, cards chosen, rules, etc.
How best to constrain the outputs? For example, the model needs to select cards I have available, etc., while the history contains cards I don't.
I know it's a long shot, but just looking for some inspiration :)
The basic rules of splinterlands are as follows:

Start battle (you are given mana and set of rules, like no
shields etc)
Then select a summon (some rules say it has to be
of certain, you also want to select your highest level summoner)
Then select 6 monsters ( each monster has a position and
stats like attack, speed, health and mana cost)
Press battle

Can you summarize the rules of Splinterlands? Most of us have not played it.

you need to use a reinforcement learning approach.

hey i have edited and added some rules.. i think it helps paint a picture :)


text:
NLP problem Phrase/Token labeling
Looking for suggestions on how to define the following NLP problem and different ways in which it can be modeled to leverage machine learning. I believe there are multiple ways to model this problem.  Deep-learning-based suggestions also work as there is a good amount of data is available for training.
Will evaluate different approaches for the given dataset. Please share relevant papers, blogs, or GitHub repos. Thanks!
Input: Given a sentence S having words W1 to W10.
S = W1 W2 W3 W4 W5 W6 W7 W8 W9 W10
The sentence has some syntactic and semantic patterns, but it is not exactly freely written natural language but it's in English. These are words, can be punctuation
Output: should be something like this.
Label1 - W4
Label2 - W3
Label3 - [W2 W1] continuous // semantically related. Means words [W2 W1] in-order are assigned a Label3. Also okay with solutions that don't output in-order.
Label4 - [W6 W8]
Label5- W10
Noise- W7, W9. Means words W7 and W9 independently are assigned a
Label3.
Label7- W5
Need to solve the problem. Looking for research/thoughts on how this problem can be defined in different ways to exploit different patterns in the structure of sentences. Looking for similar tasks which are already defined in NLP such as token labeling, parsing which can be used.
Would be really helpful to get the suggestions to the latest research on solving/defining this problem.

Hello. Could you please put your specific question in the title?

What about parsing French sentences? See also RefPerSys

I think you can solve this problem with models trained for Named Entity Recognition. In that case, your entities are labels. To do this you can use spacy to train a NER model or more easily you can fine-tune a Distil-BERT for your task.


text:
Deep Clustering Approach for Unsupervised Video Anomaly Detection
I'm working on Unsupervised Video Anomaly Detection, and I've tried implementing the Generative Cooperative Learning method, with the help of this paper.
The method uses a fixed backbone (ResNext-101) for video feature extraction. The videos are divided into segments of 16 frames, and a feature vector is computed for each segment. A generator (A simple Autoencoder) provides pseudo labels (based on the reconstruction error) for the discriminator which is a simple fully connected classifier. Pseudo labels from the discriminator are used to improve the generator using a process called negative learning, and in this fashion, the Generator and Discriminator are put in a collaborative learning loop, and the loss eventually converges.
I've recently come across the Deep Clustering paper , and was wondering if we can use a clustering method instead of the autoencoder as part of the generator. I think we can use the cosine distance as a good distance metric. The troublesome part however, is thinking of a good criteria for generating the pseudo labels.
With the autoencoder, the reconstruction error is a pretty intuitive criteria for pseudo labelling. Since anomalies are sparse, the autoencoder will not be able to reconstruct them properly and so they will have large(r) reconstruction errors.
What can be a similar criteria that we can use for pseudo labelling if we use a clustering method instead of an Autoencoder?

What can be a similar criteria that we can use for pseudo labelling if we use a clustering method instead of an Autoencoder?

Maybe the amount of Data points inside a Cluster. Like you said, an anomaly is a sparse occurrence, so there should be fewer data points in an anomalous cluster.
This would only work for known anomalies, which you can put in a known anomalous cluster, though.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
Correctly input additional values into CNN
I understand that in order to add additional inputs to a CNN, e.g. in self driving, I can append the data to a flattened layer after the convolutions and before the fully connected layers.
However, a few things confuse me. In a paper the authors want to feed speed measurements into the driving network. Instead of just appending a normalized speed value, they first feed it into several FC layers. Why would they do that? What kind of features could you extract from a single real value? Could there be another reason?

(p. 3, paper 1)
Part two of my question is: In another paper, information about the turn signal is appended to a layer as one-hot encoding. The authors talk about how that didn’t work due to vanishing weights (not gradients). So they scaled weights by a constant factor. What do they mean by vanishing weights and how do I scale weights (e.g. in PyTorch)?

(p. 6, paper 2)

Where exactly those are mention in the papers? Could you give page or equation numbers?

@serali I have added screenshots and page numbers.

I don't see anything obvious to answer question 1.  i have to wonder if this is a matter of, "We used some dense layers and it worked, so that's the way you should do it."

Hello. Please, ask only one question per post. If you have 2 questions, create another post where you ask the second question. Otherwise, this post could be closed as too broad.

I would expect the dense layers to be able to detect certain speed ranges. This neuron activates for 0-10, this one for 10-20, this one for 20-30, this one for 20-50, this one for 47.6-89.2...
Of course a later layer could also do that, but it looks like there aren't many layers after this one.

Paper 1-
If I'm understanding the paper correctly, the "Measurements " just represents a collection of auxiliary information.  It's not necessarily a single speed measurement, but any auxiliary information, so perhaps gas level, engine temperature, etc. I think them choosing to show a speedometer as a measurement might not be the best choice considering they are predicting the speed value.
Paper 2-
They said in the paper

The information of both turn indicators (on or off) is mapped to the values 1.0 and 0.0, respectively. However, the turn indicators do not contain any structural information  convolutional layers can make use of. For this reason, this additional  information is directly fed into the second fully connected layer after the  feature extraction layer (which then contains 102 hidden units).

So perhaps feeding a value of 1 into the network at that level is just too small of a value to have any effect at the layer they are injecting the information. Perhaps the layer they are injecting has a average weight magnitude that is high enough that injecting a single 1 would be close to injecting a 0.  This is my best guess.  I would expect the network would be able to adjust the weights by themselves, but there are many things that could go wrong here.


text:
Appropriate ML algorithm to solve a cutting pattern problem
I have a rectangular area, where I need to place some 2 dimensional geometrical shapes - like a square or circle or a little more complicated shapes. And after the arrangement these shapes should be cut out.
Requirements to the disposal of shapes:

These shapes are not allowed to intersect
And also they must disposed on the recatangular area
They must have at least a minimum distance
The waste should be minimized
When more than one shape is arranged on this area it is desirably that the shapes have a certain quantity (e.g. shape A: 50 %, shape B: 30 %, shape C: 20 %)

After the arrangement I get the coordinates of the single shapes so that I can cut out my shapes...
To solve this I thought of (deep) reinforcement learning but because I'm new to ML I'm not sure if there is a more appropriate method to solve this problem.
I hope that you can give me some hints or simply confirm my assumption that (deep) reinforcement learning is appropriate. And perhaps you can also offer me some useful links...
Many thanks in advance for your help!
And lastly a little picture which is showing a possible bad result because shape A and shape E intersect. And probably there is to much waste.

Have you looked at https://en.wikipedia.org/wiki/Knapsack_problem ?

@OliverMason: I know the Knapsack problem, but I‘m not sure how I can solve my problem with this. For me it‘s necessary to know the distinct position of the shapes on the rectangular area. After a proper positioning I want to cut the shapes out. I edited also my question for hope of being more precise...

You might want to start on googling the "Irregular Cutting Stock Problem". I think your problem formulation is similar to Irregular Cutting Stock Problem. Some cool papers are up in the results such as this heuristic method which is tested on real-world based problem instances.
By browsing the existing heuristic/metaheuristic methods, you may get inspiration on how to represent the solution, how to evaluate immediate shapes placement, and the existing local search operators. By then, you can try ML-based adaptive operator selection (AOS) method so that given the current state of the sheet and the existing shapes, you can choose the "best" local search operator to improve the placement. On the other hand, if you can embed the current state of the sheet as well as the current considered shape, you can predict the action of placing that shape (x,y,rotation degree) and train your model with RL methods assuming you have defined the appropriate reward function for the action you've taken.

Hey Sanyou, thanks a lot for your response. I wanted to create my own environment with OpenAI Gym with the action space select one of the three shapes, rotate it when you want and place it in the area. Positiv rewards are given for placing a shape, considering the minimum distance of the shapes. When there is an overlap between shapes or the edge of the area then the reward is negativ. These rewards are given after every disposal of shapes. At the end (agent cannot place a further shape) I think I have to give also rewards concerning the quantity of shapes and the waste...

I think setting up the AI gym is straightforward as it is returning "raw" state and action choice(?) The challenge though is the reward function. The gym can be helpful, similiar to what you said, by giving penalty or do nothing when constraint is violated e.g.. shapes are intersected, or even terminate early. I said helpful because careful reward function design can help the model learn better. As for the final reward i.e., number of shapes and wasted area, you can aggregate them by linear/nonlinear combination, or you can treat them as separate reward and turn this into Multiple Objective RL


text:
Can we learn a policy network via a sequence of manually determined actions?
In policy gradients, is it possible to learn the policy if the chain of actions is selected and performed manually/externally (e.g. by myself or by someone else who I have no influence over)?
For example, we have four actions, and I choose in the beginning an action 2, and we end up in a given state, then I choose action 4 and we end up in another state, etc. (the actions can follow some logic or not but the question is general; some of the actions will end up with positive rewards).
Can we learn any meaningful policy network from such a chain of actions?

Hello. It may be a good idea to explain why you would like to do this, and how exactly do you plan to choose the actions.

It seems that what you're proposing is related to imitation learning, but maybe I am wrong, as imitation learning is really not reinforcement learning but supervised learning applied to an RL problem. You're suggesting that humans provide/specify the behavioural policy. I don't remember now all the details of policy gradients to answer this question, but, in the case of Q-learning (which is off-policy), you could in principle use any exploratory policy that explores enough the environment, so even a policy decided by a human.


text:
Among N documents, how to summarize the most unique content in each document?
I now have $N$ documents, which share common content and they have special unique content.
Say I have $3$ legal documents related to the same person. Document $A$ is about land law, document $B$ is about company law and document $C$ is about marriage law.
How can I extract the land, company and marriage content from each document respectively and skip the common personal information?
It sounds like text-summarization but with a very different nature. Any idea is welcome.
Edit: In my situation, $N$ varies and the nature of the unique content is unknown.

It might be worthwhile to try TF-IDF and see if that works for you.
Score each term in each document proportional to how often it occurs in that document, but inversely proportional to how often it occurs across multiple documents. Then look at the the terms that have the highest scores for each document. You can use scikit-learn's TF-IDF Vectorizer to help you with this, if you are using Python.
Presumably, words that are highly specific and relevant to each of your three documents will stand out, and words that relate to personal information (as well as generic legal terms and non-specific English words) will be common to multiple documents and get filtered out.
Note: This will get you the specific words that are particular to each document. If the type of "content" you are seeking to extract goes beyond the word level, then you might have to take a different approach. Perhaps one way is to use the words obtained from TF-IDF to highlight the places in each document where the desired content might be found.

This may be a good base to obtain an extraction-based summarization. Thanks.

I think the best task for your purpose is name entity recognition (NER) rather than text summarization.
The logic is the following: if the three classes of documents are truly specific, there would be specific entities for each of them, but since the documents are linked by information about a single individual, all entities related to that individual and not to the specific domain would be shared.
So the most obvious shared entity in all documents, the name of the individual, could be identified and then pruned in all document, same holds for every other shared entity (can't came up with more clever examples right now).
If you work with python, SpaCy offer pretrained models that do a great job already also for NER, and in several languages as well. But you might consider to train your own model as well, maybe retrain on top of spacy models, cause for these type of tasks, the most information you can provide about which entities belongs to which class, the best the performances, and unfortunately, generic use models can account for many entities, but they can't associate them directly to specific domains of interest.

In my situation, N varies and the nature of the unique content is unknown. How can we make NER work in this case?

The models in spacy are pretrained on billions of documents, so you don't need to know before hand every name or entity. And you said that one document is law related, the second is about marriage and the third about company stuff, so to me sound like the content domains are known?


text:
Implementation of MDP in python to determine when to take action clean
I am trying to model the following problem as a Markov decision process.

In a steel melting shop of a steel plant, iron pipes are used. These pipes generate rust over time. Adding an anti-rusting solution can delay the rusting process. If there is too much rust, we have to mechanically clean the pipe.

I have categorized the rusting states as StateA, StateB, StateC, StateD with increasing rusting from A to D . StateA is absolute clean state with almost no rust.
     StateA -> StateB -> StateC -> StateD
     ∆  ∆ ∆       |        |         |
     |  | |       |        |         |
   Mnt Mnt Mnt    |        |         |
     |  | |_clean_|        |         |
     |  |_______clean______|         |
     |_______________________________|
                          clean

We can take 3 possible actions:

No Maintenance
Clean
Adding Anti Rusting Agent

The transition probabilities are mentioned below:
The states degrades from StateA to StateD. State degrades with rusting with certain amount of rust denoted by transition probabilities. Adding Anti Rusting Agent decreases the probabilty of degradation of state
The transition probabilities from StateA to StateB is 0.6 with No Maintenance
The transition probabilities from StateA to StateB is 0.5 with adding an anti-rusting agent.
The transition probabilities from StateB to StateC is 0.7 with No Maintenance
The transition probabilities from StateB to StateC is 0.6 with adding an anti-rusting agent.
The transition probabilities from StateC to StateD is 0.8 with No Maintenance.
The transition probabilities from StateC to StateD is 0.7 with an anti-rusting agent.
Action clean will move any state to StateA with probability 1
Rewards for StateA is 0.6, StateB is 0.5, StateC is 0.4, StateD is 0.3
Clean action lead to Maintenance (Mnt) state which has 0.1 reward. The Maintenance state will lead to increase in productivity after cleaning which is good, but there will be shutdown while Maintenance, so there will be loss of production. So reward is less.
I am new to MDP. It will be helpful if anyone can help me in getting the decision about when should we clean through MDP through a python implementation (python codes)? Shall we Clean at StateB, Clean at StateC, Clean at StateD?

Hello. Welcome to Artificial Intelligence Stack Exchange! Did you choose the transition probabilities and rewards arbitrarily? To me, it's also not clear what your question really is. Are you asking which methods you should use to find a policy to this defined MDP? Are you aware of value iteration or policy iteration? Please, put your specific question in the title to clarify what your question is.

Your MDP definition is not complete. For instance, you ask about a "decision about when should we clean", but there is no action to clean defined anywhere, and no transition probabilities go to the Clean state. In addition there seems to be no other consequence to adding anti-rusting agents, so not at all clear why you would ever not use them. Finally "Clean" looks like an end state, as if nothing happens afterwards, which seems odd. I think you probably need help with how to construct the MDP for your problem (which should be a different q) before you can ask a question about solving it.

You have improved things, but it is still incomplete. There seemsto be no reason to choose between adding the anti-rust agent or doing nothing - one or other will always be optmnal. The optimal policy appears to always be "do nothing" since the agent will arrive in the highest rewarding StateD the fastest, then stay there (unless there are other transitions from StateD that you are not showing).

@NeilSlater To answer your query, addition of Anti Rusting Agent will delay the rusting process. So the probability of moving to next degrade state decrease with addition of Anti Rusting Agent, compare to doing nothing.

@nbro I have edited the question to clarify the objective. The Markov Decision Process should determine whether to do the clean action at StateB, StateC, StateD. It will be helpful if I can get help in terms of python codes with MDP.

@shan To clarify, the MDP is just a mathematical model that represents your problem. The solution to this problem (the MDP) is a policy. So, "The Markov Decision Process should determine whether to do the clean action" does not make sense. What you want is an algorithm that is able to find a policy for the MDP. So, that's why I asked you: are you familiar with, e.g., value iteration?

@shan: I understand the probabilities are changed, that is clear from your description. However, the rest of the MDP so far makes "do nothing, always" the clear optimal policy, because landing in StateD gives the largest reward. The agent will want to get to StateD as fast as possible, then stay there.  I think some further description about the rewards, and how you determined them might help clarify that. I still think the issue you face right now is understanding how to turn your problem into a MDP, not yet how to optimise the agent.

@nbro: Thanks for the response. Sorry for not making myself clear. The solution to the above problem can be a optimum policy generated by MDP through value iteration, that will help us taking the decision whether to do clean action at StateB or StateC or StateD

@NeilSlater: I have edited and accommodated your suggestion. My mistake. StateD is the most degraded state. The reward for that is lowest or negative. StateC, StateB are better states. Mnt state have less reward because of shutdown. Eager to have python implementation of the above problem, which give solution regarding optimum decision.


text:
What knowledge is required for understanding the AlphaZero paper?
My goal is to understand AlphaZero paper published by deepmind. I'm beginning my journey trying to get the basic intuition of reinforcement learning from the book by Barto and Sutton.
As per my background, I'm familiar with MDPs, value iteration and policy iteration.
I wanted to ask until what chapter of Barto and Sutton's book is one required to read in order to fully comprehend AlphaZero's paper. Monte-Carlo Tree Search is discussed in Chapter-8 of the book. Will it be enough till that? Or would I be needing more resources apart from this book?

Start reading the paper and look up the things that you don't understand or ask a question about it.

After completing the relevant chapter of the book, I would suggest you take a look at the cited references in the paper (especially those parts that you feel you need more materials).

@Brale yes, this worked for me with GPT-2... however, the tree of unknowns is quite deep!

The more you read, the more deeply you can understand any paper, but given your stated background, reading the Monte-Carlo Tree Search chapter of Barto & Sutton, plus Gerald Tesauro's TD-Gammon paper (which is pretty accessible, and which is the basis for the other technique used in AlphaZero) should be enough to get a pretty good idea of what they did.


text:
How to improve the performance of my model trained with NEAT for a drone to learn how to fly?
I am working on a project in which a drone needs to learn how to fly. I am using NEAT.
For the first experiment, I want the drone to learn how to hover inside a $3 \times 3 \times 3$ meters box. My input is 6 sensors for each direction.  The output is the same as in a drone, so thrust (normalized to 0-1), aileron, rudder, and elevator.
Initially, I just used the time as the fitness; and, after many generations, it hovers inside the box.  However, it only really learns to use the thrust in the function of up and down sensors, but it never learns to react to input from other sensors because they are not directly connected to the fitness.
So, what can I do to improve the performance of my model? Should the drone be put to fly a track with obstacles? Should I have some more input data? Should I define fitness to better reflect good reactions for input sensor data?
NOTE: the drone uses relatively accurate physics, I am able to do tasks using a controller.

So, if I understand the question properly, fitness evaluation is currently restricted to maintaining vertical position (up/down thrust)?

No, fitness is just time. The problem is that that way it favours the units that only use thrust (the spawn point is in the center of the cube), the result is that it never learns to react for input from other sensors because any species that would go out of center would need to learn to react to side sensors and counter steer but those species extinct because they crash early (much quicker than ones that hover in the middle of the box using just thrust).

Seem like time is a good fitness, though you need it to engage into learning side sensor inputs and side movement.
I would consider adding a bit of randomness to the environment. How about adding some random mild forces that might sway it left, right, front and rear a bit so that that bots are forced to use other sensors and inputs to stay in the center.
In cases when drone is not simulated, this task is a little harder but adding randomness to the environment can still be done. For example, tilt your drone a tiny amount in a random direction at random intervals. This will force your drone to learn to correct for being jostled by wind, without you having to actually produce wind.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What does "unknown search spaces" mean in the context of Evolutionary Algorithms?
In the article Multi-Verse Optimizer: a nature-inspired algorithm for global
optimization (DOI 10.1007/s00521-015-1870-7), it's written

The results of the real case studies also demonstrate the potential of MVO in solving real problems with unknown search spaces

where MVO stands for Multi-Verse Optimizer.
What does "unknown search spaces" mean in the context of Evolutionary Algorithms and, especially, in the context of the Multi-Verse Optimizer?

(Disclaimer: although I have never seen a formal definition of unknown search space, here is my attempt to define it based on my knowledge of search and search algorithms in machine learning and evolutionary algorithms; I am aware of a definition of unknown environment (see chapter 2, p. 44, of Norvig and Russell's AI book), but that definition is different from the one below.)
An unknown search space is a search space (i.e. a set of objects of interest with some relationship between them) where you do not know anything about

where the best/worst solutions/objects are (i.e. where the local/global minima/maxima are) and/or
how these objects are related.

A typical example where the search space is known is the search space associated with a path-finding problem where you know the start and goal locations, the intermediate locations that you can go through from the start to the goal locations, and the distance between locations. If you have this information, you can find the optimal path from the start to the goal location by just exploiting your knowledge of it (you can use e.g. depth-first search). Think about finding the shortest path from one city to another.
In other (most) cases, you may not know anything about the relationship between 2 or more solutions/objects, unless you explore the search space (i.e. take random actions or some other more informed actions).
For example, let's suppose that you want to search for a function of the form $f : [0, 1]^{28 \times 28} \rightarrow \{0, 1\}$ from a set of functions $\mathbb{F}$, i.e. $f \in \mathbb{F}$. In this case, you can think of a function $y = f(\mathbf{x}) \in \mathbb{F}$ as a classifier of greyscale images, where $\mathbf{x} \in  [0, 1]^{28 \times 28}$ is a $28 \times 28$ greyscale image and $y \in \{0, 1\}$ is the class (name) of the object in the image $\mathbf{x}$ (in this case, we assume there's only one main object in the image $\mathbf{x}$). In machine learning, $\mathbb{F}$ could be represented by a set of neural networks (with a specific architecture, e.g. number of layers), and $f \in \mathbb{F}$ would be one of these specific configurations. In this context, usually, we do not know anything about the relationship between these neural networks, so the search space is unknown. So, you need to explore the search space in some way, for example, by changing some of the parameters of one neural network $f \in \mathbb{F}$ to get another neural network $f' \in \mathbb{F}$ and see how that affects, for example, your accuracy at classifying images.
This definition also applies to evolutionary algorithms (and, although I have not read that paper, I assume it also applies to that context). In fact, evolutionary algorithms are usually used to solve problems with unknown search spaces. For example, test case selection or finding a policy for an agent.

See also Bayesian Optimization with Unknown Search Space by Huong Ha et al. It seems that their definition is consistent with the one above that I gave. However, this paper, which cites the previous one, seems to suggest that an unknown search space is a search space where you really don't know what it contains (i.e. which objects) and that you may need to define a search space arbitrarily and this search space that you chose may not contain the solution at all.

They write "When the search space is unknown, one heuristic solution is to specify it arbitrarily. However, there are two problems: (1) an arbitrary search space that is finite, no matter how large, may not contain the global optimum (2) optimisation efficiency decreases with increasing size of the search space.". This still seems consistent with my definition above, but here they put an emphasis on knowing which objects it contains. This is still a bit vague. It's possible that the authors of the paper mentioned by the OP had in mind a different definition of unknown search space.

I sent your answer to the author of the article and he accepted your answer.
please edit your answer and remove the disclaimer section.

@Commander Hello. It may still be a good idea to leave the disclaimer because, even though the author(s) of that article agrees with my definition (which is not completely precise in a mathematical sense, which I acknowledge), this definition may not be applicable to all cases, i.e. some other author may have something else in mind. People reading this answer, hopefully, will also read these comments. Anyway, thanks for confirming that this definition is what the original authors of the paper had in mind.


text:
Transferring a Q-learning policy to larger instances
How do I best transfer and fine-tune a Q-learning policy that was trained on small instances to large instances?
Some more details on the problem:
I am currently trying to derive a decision policy for a dynamic vehicle dispatching problem.
In the problem, a decision point occurs whenever a customer requests delivery. Customers expect to be delivered within a fixed amount of time and the objective is to minimize the delay. The costs of each state are the delays realized in that state (i.e., the delay of the customers that were delivered between the last two states.
Some details on the policy:
I used a Q-learning policy (Dueling Deep Q Network) to estimate the discounted future delay of assigning an order to a vehicle. The policy was trained on a small-scale instance (5 vehicles ~100 customers) using epsilon-greedy exploration and a prioritized experience replay. I did not use temporal difference learning (as the cost of a decision are only revealed later in the process) but updated the policy after simulating the entire instance.
My problem:
As it is, the policy transfers well to instances of larger sizes (up to 100 vehicles and ~2000 customers) and I could do without fine-tuning. However, there certainly is room for the policy to improve. Unfortunately, when I try to fine-tune the initial model on the larger instance, the retrained model becomes worse over the training steps with regards to minimizing delay.
I suspect that large gradients play a role here as the initial q-values, trained on the small instance, are obviously way off for the large instances (due to the increase in customers).
Is there a standard approach to deal with such a transfer problem or do you have any suggestions?


text:
What is my mistake in applying the AC-3 algorithm on this problem?
I want to apply AC-3 algorithm to the following CSP:
There are two variables $A$ and $B$.
Domain of $A: \{0, 1, 2, 3, 4, 5, 6, 7, 8, 9\} $
Domain of $B: \{0, 1, 2, 3, 4, 5, 6, 7, 8, 9\} $
The constraints:
$$A < B $$
$$B = A^2 $$
I'll apply the AC-3 algorithm depicted in Artificial Intelligence: A Modern Approach, 4th edition (figure 6.3)
Step 1. By converting constraints to arcs, our arc queue will be as following:
$$A < B $$
$$B > A $$
$$B = A^2 $$
$$A = \sqrt{B} $$
According to the algorithm, when processing an arc such as $(X_i, X_j)$, if the domain of $X_i$ was changed, all the arc that their destinations are $X_i$ and their source are a neighbor of $X_i$ must be add to the end of the arc queue except for $X_j$. Since we have only two variables, therefore I think we do not need to add any new arc to the queue.
Step 2. processing the arc queue:

Applying $A < B$ reduces the domain of $A$ to $ \{ 0, 1, 2, 3, 4, 5,
   6, 7, 8 \} $.
Applying $B > A$ reduce the domain of $B$ to $ \{ 1,
   2, 3, 4, 5, 6, 7, 8, 9 \} $.
Applying $B = A^2$ reduces the domain of $B$ to $ \{ 1, 4, 9 \} $.
Applying $A = \sqrt{B} $ reduces the domain of $A$ to $ \{ 1, 2, 3 \} $.

So, at the end, the domain of $A$ is $\{1, 2, 3\}$ and the domain of $B$ is $\{1, 4, 9\}$.
But this answer is not correct. Since for $1$ belongs to the domain of $B$, there is no element in the domain of $A$ such that $ B < A $ satisfies.
It is obvious that the domains of the arc consistent CSP is as following:

Domain of $A: \{2, 3\} $
Domain of $B: \{4, 9\} $

What is my fault?
I think adding arc $(X_j, X_i)$ as well as other neighbor arcs $(X_k, X_i)$, when the domain of $X_i$ is reduced by applying arc $(X_i, X_j)$, leads to correct answer.


text:
Why does an action cost function dependes on result state in search problems?
In the famous AI book Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig (4th edition), in chapter 3, the action cost function of a problem solver agent denoted as $c(s, a, s')$, that gives the numeric cost of applying action $a$ in state $s$ to reach state $s'$.
But according to the text, in that chapter, it is assumed that the environment is deterministic. So, given the current state and the action, the next state is known.
My question is that, as the next state can be determined by the current state and applying action, why does the cost function depend on the next state?

Actually your source only gave a deterministic environment example (ToZerind) which may not definitely mean they only consider deterministic environment for the general action cost function which is a concept rooted from control theory similar to (expected) reward from RL and loss function from statistics as the author wrote somewhere else in your reference.

It prevails not only in AI, but also in control theory, where a
controller minimizes a cost function; in operations research, where a policy maximizes a sum of rewards; in statistics, where a decision rule minimizes a loss function

In Sutton's RL book the reward (cost) function $\mathcal{R}$ is a random variable and they never wrote or needed $\mathcal{R}(s_t, a_t, s_{t+1})$, but instead, its expected value $r(s_t, a_t, s_{t+1})$ is defined on page 49 due to possible stochastic environment unlike a normal grid world game:

We can also compute the expected rewards for state–action pairs as a two-argument function $r : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$:
$$r(s,a) = \sum_{r \in \mathcal{R}} \sum_{s' \in \mathcal{S}} rp(s',r|s,a)$$
and the expected rewards for state–action–next-state triples as a three-argument function $r : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$,
$$r(s,a,s') = \sum_{r \in \mathcal{R}}r\frac{p(s',r|s,a)}{p(s'|s,a)}$$

In fact in Sutton's book the only place the 3-argument expected reward is explicitly used appears in a nondeterministic finite state diagram for a recycling robot example on page 52:

You see in such a diagram the expected reward function needs the explicit dependence of the entering state to conveniently express its value based on its entering state for each of the possibly branching state transition due to its random environment characterized by Markovian transition dynamics $p(s',r|s,a)$. Therefore sometimes the 3-argument cost (reward) function in the sense of expectation may be needed even assuming the usual MDP environment in the most general conceivable cases.
There's a very similar question recently and you can refer to my related answer there.

You are correct that $s'$ is redundant information in $c(s,a,s')$ for a deterministic environment.
However, using the full form of the cost function, that also covers non-deterministic environments, saves the authors from redefining it later. The form $c(s,a,s')$ is generic enough to cover a much wider range of problems, and some of the equations and methods derived using it can be re-used in more complex environments, whilst if it was specialised to deterministic environments only, then the same equations would have to be derived again.


text:
How do automatic high-beam headlights work on cars?
Modern cars can operate high-beam headlights automatically:

They automatically switch from high-beam headlights to low-beam ones (less intense) when you enter a town or there is a car in front of you either going in the same or opposite direction so you don't dazzle other drivers or people in the street.

Oppositely, when you are in almost complete darkness and there aren't any other drivers at sight, the system automatically sets the high-beam headlights.

I am aware that in the front part of these cars there is a camera or sensor and also imagine that the automatic switching when entering or exiting a town is achieved by just having a threshold ambient illumination.
But I am unable to imagine how the recognition of other cars works. It might be that an image recognition program is used to detect pairs of front car lights (white) and rear lights (red). However, how do you deal with:

pairs of street lamps (far from the road and not illuminating it) that could be identified as a car coming in the opposite direction,
pairs of lights coming from the reflectors of the crash barriers,

many other random pairs of lights that could be interpreted as cars.

Is this technology based on AI software that after intense training is able to deal with these points? Or is it a less complex image analysis program that takes into account that moving lights outside the car (with respect to the road) move differently than static lights (with respect to the road) when seen from the car?
Edit: I've seen this technology working on Audi A4 and A5 cars.

I don't have domain knowledge on how these cars perform this detection, but between my EE and ML degrees, I'm quite confident they are most likely not using any AI software to perform this prediction.  It is similar to using AI prediction with IR bathroom sensors for a paper dispenser.
What they are probably doing is just simply looking at light intensity. The reflectors you provide in the image will have a luminosity much lower than a car light, so from a threshold perspective I would assume is easy to detect car lights from a reflector.
Here is a video from a guy talking about it. youtube.com/watch?v=teilITzPjPU
He shows an example where the light intensity is so high for some signs that the high beams turn off. I would guess that at 300 meters a directed car light would have a higher average area luminosity, than most street reflectors, but i can't verify if that's true.
if very_bright_light:
    turn_off_highbeams()
else:
    turn_on_highbeams()

I'm sure there is more complexity to it, as the time frame of exposure, or perhaps car lights release light with specific frequencies that are easily detectable.
Hope this helps in the event that a domain expert doesn't show up.

Thank you @JohnRothman You're right that the reflectors are less luminous than most cars' lights but, on straight roads, I've seen the technology detecting car lights at least 300 m away and those lights were less intense than pair of lights of nearby reflectors or street lights.
Also, I considered the frequency trick taking into account that street lights work with alternating current at 50 Hz (typically) while car lights use direct current from batteries. That way you could differentiate car lights from street lights but the reflections from your car are still continuous signals.

Here is a video from a guy talking about it. https://www.youtube.com/watch?v=teilITzPjPU.  He shows an example where the light intensity is so high for some signs that the high beams turn off.  If what you say is true, then I would guess that at 300 meters a directed car light would have a higher average area luminosity, than most street reflectors, but i can't verify if that's true. Sorry.

Thank you again! The video definitely clarified the point. Would you mind making an edit to your answer so I can mark it as the correct one? Regarding the distant cars and nearby reflectors, I imagine that it might cause some sporadic problems but somehow it works.

Glad I could provide some help. I've edited the original answer.


text:
Why does research on faster Transformers focus on the query-key product?
A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism:
$$\text{softmax}\left(\frac{Q K^T}{\sqrt{d}} \right)V,$$
As I understand it, the runtime, assuming $\{Q, K, V\}$ are each of shape $(n, d)$, is $O(n^2 d + n d^2)$. In general, the issue is the $n^2 d$ term, because the sequence length $n$ can be much bigger than the model dimension $d$. So far, so good.
But as far as I can tell, current research focuses on speedups for $Q K^T$, which is $O(n^2 d)$. There's less focus on computing $A V$, where $A = \text{softmax} \left(\frac{Q K^T}{\sqrt{d}} \right)$ -- which also has complexity $O(n^2 d)$.
Why is the first matrix product the limiting factor?
Examples of these faster Transformer architectures include Longformer, which approximates $QK^T$ as a low-rank-plus-banded matrix, Nystromformer, which approximates $\text{softmax}(QK^T)$ as a low-rank matrix with the Nystrom transformation, and Big Bird, which approximates it with a low-rank-plus-banded-plus-random matrix.


text:
Negamax: how should you avoid the horizon effect in the connect four game?
I'm trying to implement a quiescence search in the negamax algorithm, for a connect four game.
The algorithm is as follow for a chess game:
int Quiesce( int alpha, int beta ) {
    int stand_pat = Evaluate();
    if( stand_pat >= beta )
        return beta;
    if( alpha < stand_pat )
        alpha = stand_pat;

    until( every_capture_has_been_examined )  {
        MakeCapture();
        score = -Quiesce( -beta, -alpha );
        TakeBackMove();

        if( score >= beta )
            return beta;
        if( score > alpha )
           alpha = score;
    }
    return alpha;
}

I get the idea, but unfortunately there is not much more details in the article. I don't get how the sentence "until( every_capture_has_been_examined )" could be transposed to the connect four game. How would one evaluate silent move in such a game?
Also, there is no depth parameter, does that mean that quiescent search only applies to a single depth? As far as I understand, it seems so.
Here is an example output of my connect four AI game, where the horizon effect occurs (if I understand correctly):

AI player is YELLOW
Depth is 1 (obviously)
AI player wrongly chose to play c5 in the -300 cell, considering letters a,b,c, ... for the y axis. Thus, AI adds a third connected chess men and improves his is score (c3 to c5)
However, AI doesn't see that by doing so, it gives a winning move to the RED player. Indeed, RED now sets a connect-four in the line just below (d3-d6, playing d6) and wins the game.

Please note that MIN is actually MAX, because I use negamax and not minimax.

Quiescence search relies on a definition of quiescent. In chess, a quiescent state is apparently one where no capture is possible. Perhaps for Connect-4, it could be defined as one where each player can't make a longer line than their current longest.


text:
How to build an expert system similar to ES-Builder Web
I made a simple expert system using ES-Builder. Please click the link to view it. ES-Builder is a web-based expert system shell. There is a tree-based knowledge representation. In ES builder, User Interfaces are also automatically designed. They generate a link as I have shared above and anyone can access it and can use it.
But when I try other ES shells such as JESS, CLIPS & PyKE, I only noticed that there we have to write facts and rules and the program is run on command line upon consulting the Expert System. There is no UI like in ES-Builder.
My question is, is there any way to build UI to the expert systems created by CLIPS/JESS? Or else should I create a web application using another framework like Spring, DotNet, and integrate it with the knowledge base created with CLIPS/JESS?
(I am a bit confused, because according to what I have learned: if we use an expert system shell then we need not program it using languages (such as Prolog). Because the User Interfaces and Inference Engine is already there. What we are remained to do is just to build the knowledge base. Similar to ES builder UI is auto built.)
Thank you very much for the support! If the question is confusing, I am happy to modify it in a more understandable manner.

Hello. It may be a good idea to provide the link to JESS and CLIPS. Personally, I had never heard of the term "ES shell", but I am also far from an expert in ES. So, it may be a good idea to tell us what you learned about ES shells. Having said that, this question seems to be about specific software, so I am afraid it's off-topic. However, given that it's about ES and no other place probably covers this topic, I will leave it open for now.

@nbro Thank you for your concern. Actually, ES shells are developed with the purpose of building expert systems without the knowledge of a specific programming language. This is the definition I learned from my professor. ES builder perfectly matches the definition. but in my research, I found there are many others like CLIPS, JESS, PyKE where programming has to be done manually. This is contradictory to what I have learned. That is why I'm seeking help from an expert in this community. I will provide more information and necessary references in my question ASAP.

@PawaraSiriwardhane did you find a proper solution to this question? I'm having the same exact problem and ES builder seems to be the feasible solution to get a decent UI without too much ground work.

Most old school Expert System tools are text based and based on If/Then rules going forwards or backwards (chaining)
Some have graphical front ends (such as VisiRule (https://www.visirule.co.uk/), XpertRule, etc)
Some have now repositioned themselves as Business Rules tools and most of these have GUIs (InRule, Actico etc)


text:
What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?
In PyTorch, transformer (BERT) models have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to output fully connected layer for the encoder just after adding the residual connection.
Why is there an intermediate layer within an encoder block?
For example,

encoder.layer.11.attention.self.query.weight
encoder.layer.11.attention.self.query.bias
encoder.layer.11.attention.self.key.weight
encoder.layer.11.attention.self.key.bias
encoder.layer.11.attention.self.value.weight
encoder.layer.11.attention.self.value.bias
encoder.layer.11.attention.output.dense.weight
encoder.layer.11.attention.output.dense.bias
encoder.layer.11.attention.output.LayerNorm.weight
encoder.layer.11.attention.output.LayerNorm.bias
encoder.layer.11.intermediate.dense.weight
encoder.layer.11.intermediate.dense.bias
encoder.layer.11.output.dense.weight
encoder.layer.11.output.dense.bias
encoder.layer.11.output.LayerNorm.weight
encoder.layer.11.output.LayerNorm.bias

I am confused by this third (intermediate dense layer) in between attention output and encoder output dense layers

Welcome to SE:AI!  I did a slight edit to draw attention to the question posed.

Hello. Can you please provide the link to the PyTorch model that has an intermediate layer that you're referring to?

Hello thank you for your response. in fact each pre-trained model on hugging face has the same architecture. you can look at the following for example 'bert-base-uncased' at https://huggingface.co/bert-base-uncased (three fully connected layers. one in attention one intermediate and one output. for all encoder blocks)

I am confused by this third (intermediate dense layer) in between attention output and encoder output dense layers.

Feedforward layer is an important part of the transformer architecture.
Transformer architecture, in addition to the self-attention layer, that aggregates information from the whole sequence and transforms each token due to the attention scores from the queries and values has a feedforward layer, which is mostly a 2-layer MLP, that processes each token separately:
$$
y = W_2 \sigma(W_1 x + b_1) + b_2
$$
Where $W_1, W_2$ are the weights, and $b_1, b_2$ - biases, $\sigma$ - is the nonlinearity ReLU, GeLU, e.t.c.

This is kind of a pointwise nonlinear transformation of the sequence.
I suspect, that intermediate here corresponds to $W_1, b_1$ and the output is about $W_2, b_2$.

Hi, thank you for your response. I understand that the attention sublayer in the encoder has a fully connected dense layer followed by another output fully connected dense layer. but all pre-trained transformers at hugging face have three fully connected layers in each encoder block. one in attention one intermediate and one output. just for confirmation, are you suggesting that the MLP within attention (before skip connection) has two layers one of which is being referred to as the intermediate layer?  thanx again

The language "take a moment to think and process this information" is a seriously unhelpful and obfuscating way to describe a matrix multiplication. I would discourage people from unnecessary anthropomorphism of simple operations just because they are in the context of AI.

@mohammadaliHumayun The multihead-attention block has four linear layers - $W_Q$, $W_K$, $W_V$ and $W_O$. The layer $W_O$ is used to combine the concatenated output from the multiple heads. It is called attention.output.dense in the model that you are using. After the attention block you have a multi-layer perceptron (MLP). As explained in the answer, the MLP usually has two layers $W_1$ and $W_2$. So $W_1$ is called intermidiate.dense and $W_2$ is called output.dense.

So I came across this myself when trying to explore why models require intermediate layers in general. BERT is fundamentally different than the previous transformer architectures. The intermediate layer acts as a way to ensure the different multi attention heads are able to utilize pre-trained information. So for example if you examine the block directly you will notice they upscale and descale in my experience something like 768 dim to 3072 back down to 768, I am approximating dimensions from memory. This does not correspond to the KQV in the attention layer weights. Now it is still a point of research concern why need the scale and descale of the embedded information to allow for fine tuning of the models. When we remove the intermediate layers we have considerable drop in transfer learning performance and therefore pre-training experience is lost. You will also notice more recent models have this intermediate layer as opposed to the original transformer paper.

documentaion of Bert on Hugging Face
you will find that

intermediate_size (int, optional, defaults to 3072) — Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.

so intermediate layer is feed-forward layer.


text:
Does there exist functions for which the necessary number of nodes in a shallow neural network tends to infinity as approximation error tends to 0?
The Universal Approximation Theorem states (roughly) that any continuous function can be approximated to within an arbitrary precision $\varepsilon>0$ by a feedforward neural network with one hidden layer (a Shallow Neural Network) with sufficient width.
I remember stumbling upon articles showing that for some functions the necessary number of hidden neurons tends to infinity as $\varepsilon$ tends to zero (similar to this one by A closer look at the approximation capabilities of neural networks), but I have not had success finding them again. Is my memory incorrect or is it merely my searching skills that are insufficient?

This is obviously not true for some functions, e.g. the ones produced by neural networks with N hidden neurons :) For an example of one where it is true, try something like the Weierstrass function.

@user253751 You  are right, it is of course not true for all functions! I have tried to make the question more precise: I am looking for certain functions for which it is not true. Regarding the Weierstraß function: I am not certain that the number of neurons tends to infinity, since in Universal Approximation with Deep Narrow Networks it is shown that for some networks with continuous activation functions that are nowhere differentiable, they still have the uniform approximation property.

I would suspect the need for infinite neurons to reach 0 error is for most functions. user253751 gives one obvious exception - i.e. functions that are naturally expressed by a neural network architcture that you are allowed to use in the approximation. However, even $f(x) = x$ cannot be approximated with zero error by a finite neural network once you add a hidden layer with a non-linear activation function.

@NeilSlater You are probably right that this applies for most functions, but I am still hoping to find a paper giving a proper example or proof, since I would prefer not having to either state it without a reference, come up with a proper example/ proof or just using handwaving. Looking at Lemma 4.1 in Universal Approximation with Deep Narrow Networks I think $f(x)=x$ can actually be approximated arbitrarily well by a shallow neural network with a nonlinear activation function and just a single hidden node.

Yes, I missed that sigmoid is approximately linear at $x=0$, so you can magnify that by having a tiny weight to the hidden layer followed by a gigantic weight to the output. However, "arbitrarily well" does not equate to "zero error" - it is just that you don't gain anything by adding more neurons, the approximation can be improved by manipulating the weights for a single neuron.

@NeilSlater True, but I am looking for a function that forces me to add more nodes, if I want to decrease the approximation error.


text:
Is there some kind of "weighted maximum" that allows the gradients to backpropagate?
I was wanting to add a maximum in my neural network, but this seems a bad thing to do since it kills the gradients to all but one of the inputs.
Is there some kind of "weighted maximum" that allows the gradients to backpropagate?
Edit: I had a two dimensional tensor (correlation matrix) I wanted to reduce to one dimension.

"wanting to add a maximum in my neural network", do you mean as an activation function? But how? It's not clear where you would use this "max" function. Edit your post to clarify that.

The maximum function is not smooth, since it's first derivative is not continuous.
Having non-smooth functions is generally a bad thing for neural networks, since they don't work nicely with gradient decent.
So what you want is a smooth approximation to these functions.
Logsumexp is the smooth approximation to the maximum function and so it is what you should use in a neural network, just like softmax is a smooth approximation to the argmax
https://en.wikipedia.org/wiki/LogSumExp

People have been using ReLu functions just fine. Differentiable almost everywhere is probably fine for most applications.

@StefanPerko SoftPlus(x) = log(1 + exp(x)) is a smooth approximation to relu which unlike relu would allow the gradients to propagate backwards for x<0, but I doubt there would be any benefit since gradient decent doesn't get stuck with relu anyway, which is interesting


text:
Attention: Isn't it redundant to apply a linear layer to both the keys and values?
Transformer attention is calculated $Attention(X) =X W^V\times \text{columnwise-softmax} (Att(X))$ where the attention attention matrix is $$Att(X) = Q \times K = {X W}^Q \times ({X W}^K)^T = {X W}^Q (W^K)^T X^T $$
But then couldn't one of $W^Q$ and $W^K$ be absorbed into the other? So one of them is redundant say $W^K$ and so attention only needs the parameters from $W^Q$ and $W^V$.
$$Attention(X) = {XW}^V \times \text{columnwise-softmax}({XW}^Q  X^T).$$
Diagrams I use for thinking.

In multi-head attention the Keys, Values and Queries are chunked along the "channel" dimension, you need to apply a linear layer before you do this.
What I wanted to do wouldn't work because you would be chunking $X$. The chunks of $Q,K,V$ depend on all the channels of $X$, where as the chunks of $X$ do not.
There is code here to see exactly what is happening with multi-head attention:
q = rearrange(self.w_qs(q), 'b l (head k) -> head b l k', head=self.n_head)
k = rearrange(self.w_ks(k), 'b t (head k) -> head b t k', head=self.n_head)
v = rearrange(self.w_vs(v), 'b t (head v) -> head b t v', head=self.n_head)


text:
Can you train a neural network by simply giving it ratings each time it runs?
I am currently trying to train a bot for a game I am creating. It is a 2d game with a complex map made of various shapes. The bot and character shoot bullets that are capable of ricocheting. The neural network outputs a vector in which the bot will turn and then fire. I myself cannot calculate the correct trajectory and find the Loss of the network. But, I can give a rating on how well the neural network performs when it fires. Can I train it by simply giving it a rating, and if, how so?

@foreverska, can I just rate it on the damage the bot deals to the character? Higher damage = lower loss and vice-versa, then just back-propagate?

No, it's more complex than that, although there are simplified reinforcement learning approaches. If your action choices are discrete, you might like to look into the Cross Entropy Method (CEM), which is where you run e.g. 100 episodes, take the top 10 performing ones and treat every action taken in them as being correct to train a multi-class classifier. Clearly that's not true, but the small degree to which it is true (some of the actions will be better choices) can be enough to train the neural network.

Yes, this is a stereotypical reinforcement learning problem.  Instead of trying to calculate the dynamics of the environment the agent is given a reward or punishment for its behavior in the environment (shooting an enemy, etc).  The training process tries to find a policy to maximize the reward signal.  It’s a rather deep field and a bit much for a single post.  Maybe try reading one or two papers on Deep Q Network (DQN) or Proximal policy optimization (PPO) and see if either can be formulated in a way to work for your problem.

If the neural network outputs a sequence of actions, and we only find out at the end whether that entire sequence was successful, it is a RL problem.  The problem as stated does not appear to have that form.  Instead, the neural network appears to make a single choice (the angle to shoot at) and then we learn whether that was a good choice.  That is a simple supervised learning problem -- no need for the machinery of reinforcement learning.

It would probably be helpful to spell out the full names of DQN and PPO (Deep Q-Network and Proximal Policy Optimization?).

@D.W. I can see your argument there.  I took there being a bot and a human to mean  that it might be competitive and round based.  A supervised learner trained to make a locally optimal choice may miss a broader strategy.

Yes, but we're talking about a lot of ratings, like, millions of ratings. You have to automate rating generation, human feedback would just get way too expensive.

One alternative would be to have the AI play versus another AI in a reinforcement learning scenario, which is a fairly common strategy. (You can do this in parallel for many hours.) Of course, there are some drawbacks. It gets more complicated to code and they don't always behave like you want them to. If your simulated AIs start taking advantage of edge glitches or decide it's far better to not compete at all, it wouldn't be very useful for normal gameplay.

@NotANumber Sure. You've suggested a very reasonable means of automating it. :)


text:
Non-face "deepfakes" in videos
Instead of changing faces (like James Bond to Putin) what if, given sufficient training data, I wanted to:

Remove or add some windows from a brick house?
Convert a glass of red wine to a glass of white wine?
Remove the infamous the Starbucks cup from Game of Thrones?

Although deepfakes are notoriously hungry to train, the pre-trained weights are plug-and-play.
Does there exist any equivalent for non-face objects?

Hello. Welcome to Artificial Intelligence SE. If you're looking for a pre-trained model, your question is off-topic here. If you're looking for the name of a model (pre-trained or not) that you could use to solve this task, then your question becomes on-topic. Please, take the time to read our on-topic page. So, please, edit your post to clarify what you mean by "plug-and-play".

Hi @nbro. I am not looking for a pre-trained. I am looking for the name of this model, the family of this model, maybe a few paper outlining this model. I have my own training and testing data on tap. Sorry if that was unclear. Cheers!

Ok, thanks for the clarification. Can you please include this info in your post directly (so that I can delete these comments to avoid clutter)? Thank you.


text:
Why does the number of input tokens to an LSTM have an impact on the convergence of Integrated Gradients?
Background
I am computing the attribution scores for a simple LSTM model using Integrated Gradients. This method defines the contribution of a feature to a model prediction by integrating over the gradients along a path between the input and a fixed baseline:
$$IG_i(x) = (x_i - x'_i) \cdot\int_{\alpha=0}^1 \frac{\partial F(x'+\alpha(x-x'))}{\partial x_i}d\alpha$$
A common way of measuring the quality of the generated attributions is via the completeness axiom, which states that:
$$\sum_i IG_i(x) = F(x) - F(x')$$
The key to computing the IG scores is the approximation of the path integral, which can be approximated via Riemann sum, or a similar interpolation method. In section 5 of the IG paper, it is stated that, in practice, between 50 to 300 interpolation steps are sufficient to obtain IG scores that converge to satisfy the completeness axiom.
Issue
I am now testing the IG attributions on a simple LSTM model (1-layer, 16 hidden units). For shorter inputs (<20 tokens), convergence is reached in a reasonable number of steps, and the approximation of the integral is stable. However, when the length of the input increases, I find that the integral approximation diverges when the number of interpolation steps is increased! This can be seen in the following plot (N.B. the y axis is logarithmic):

Question
My question is: why does the number of input tokens to the LSTM have an impact on the convergence of integrated gradients?
It is stated in footnote 1 of the IG paper that the completeness axiom depends on whether the model satisfies Lebesgue's integrability condition. It would surprise me, however, that increasing the number of input tokens would dissatisfy this constraint: would it be possible that the model has become too nonlinear for numerical integration to still work? If so, are there alternative numerical integration methods that could be used here, instead of Riemann approximations or the Gauss-Legendre quadrature?

Hello. Welcome to Artificial Intelligence SE. This seems to be an interesting question, even for people not familiar with IG. But can you clarify what you mean by "LSTM steps"? Do they refer to "interpolation steps"? If that's the case, what exactly do you mean by "interpolation steps" in this context?

By LSTM step I mean a single forward step of an LSTM cell: so processing a single token and up to the next. Concretely: an input string consisting of 10 tokens would amount to processing 10 LSTM steps in order to obtain the final hidden state that is used to create a model prediction.

By interpolation steps I mean the steps in the sense of how a Riemann sum splits up the calculation of a path integral into smaller steps. The higher this number of steps is the more accurate the approximation of the integral will become.

I have updated the question a little bit to make the use of "steps" less confusing. Thanks also for your edits to the question, looks nice now :)


text:
how to decide the optimum model?
I have split the database available into 70% training, 15% validation, and 15% test, using holdout validation. I have trained the model and got the following results: training accuracy 100%, validation accuracy 97.83%, test accuracy 96.74%
In another trial for training the model, I got the following results: Training accuracy 100%, validation accuracy 97.61%, test accuracy 98.91%
The same data split is used in each run.
Which model should I choose, the first case in which the the test accuracy is lower than the validation? or the second case in which the test is higher than the validation?

Do you have the same data split for each run - i.e. exactly the same examples in train, validation and test sets? Also, can you give size of each set? Are your numbers exact or have you simplified them? There would be a difference in analysis if your values were 98.3% vs 98.7% instead of 97.6% vs 99.2% for instance. What matters is the ratio of error rates - whilst 98% vs 99% appears as a ratio of 2, if you have rounded nearest then the ratio could be anything from 1.1 (not really meaningful) to 5.0 (impressive).

Testing each time on a test set is against the point of a train-val-test split. The reason test is important, is that you are only supposed to test on it when you think your model is good and ready with all final model and hyperparameter decisions made.
A good description can be found in this article: https://machinelearningmastery.com/difference-test-validation-datasets/
but to sum it up, test should be unbiased. The more you test against it, the more you bias the result.

More techincally for the last paragraph, the testing does not create bias, but the selecting does.  If OP were to select a model using test, they could no longer trust the measurement, especially for really close values as in their question. I have other concerns with the question too though, where I have asked for clarification from the OP.


text:
Minimax evaluation function for games with score instead of loss/draw/win result
I am trying to create minimax evaluation function for the Ms Pacman game. The goal of the player is to maximize score.
I have some idea about the features that I would like to use in my evaluation function (which is weighted sum of all features):

avg distance to the ghosts
score
is pacman close to pacdots
etc.

This can work in some way if AlphaBeta search considers only non-terminal nodes. However evaluation for the terminal nodes will be substantially different from evaluation of non-terminal nodes because we are returning true evaluation which is weighted score in this case. Hence once search reaches some terminal nodes, it may discard them completely or focus too much on them (depending on how our evaluation changes).
In games with decisive result we can directly evaluate loss/draw/win with some very small/big value to overwrite the effect of evaluation function during search and properly backpropagate evaluations.
However what is the correct approach for the games with score?
I was thinking about adding another feature such as is game finished and using it in evaluation but this does not seem like an optimal solution.
I will be thankful for any tips or references.

Is the Ms Pacman game a 2-player game, or are you training both the main character and the enemies in your version? I'm not sure that I see the relevance of minimax if not?

@NeilSlater It is turn based 2-player zero sum game. I am using same state evaluation for pacman and ghosts. Also all ghosts are moved in single turn (same for pacman).

So, just to be clear, the ghosts will be AI-controlled too? You said zero-sum game, so the score of the Ms Pacman, and score of the ghosts are equal and opposite? Or are the scores a separate quantity (not zero-sum) and highest score wins the game?

Yes, ghosts and pacman are controlled by the same minimax algorithm. There is no winner or looser in this game (which is the reason for my post; if there would be winner I would just return +infinity or -infinity from terminal nodes based on the fact whether pacman or ghosts won). The goal of the pacman is to gain as much score as possible. Goal of the ghosts is to minimize the pacman's score. So yes, I consider this game as a zero-sum game.


text:
Reasoning about 2d spatial square configuration
I am trying to use RCC8 algebra relations and axioms in order to tackle the following problem :
We consider the below configuration and square $x$ such that "$x$ NTPP $a$" and "$x$ TPP $c$"

and we must prove that :

a) $x$ DC $b$
b) For $y$ such that "$y$ EC $a$", prove that it is not possible to have "$y$ EC $x$"

using the tree

My difficulty is that I do not know how to "formally" prove the above statements using the tree.
For instance, regarding question (a), since "$x$ TPP $c$" and "$c$ DC $b$", by the composition table we have that "$x$ DC $b$". But, this argument does not use "Interior" and "Boundary" statements as the tree suggests. Could you please assist with this ?

According to the tree:

"x TPP c" implies "x IntInt c" and "x BndBnd c" and "x BndInt c" and "not x IntBnd c".
"c DC b" implies "not c IntInt b" and "not c BndBnd b".

From "x IntInt c" and "not x IntBnd c" we can say "not x IntExt c". This relation together to "not c IntInt b" proves that "not x IntInt b".
In similar way, we can infer "not x BndBnd b".
Following tree for "not x IntInt b" and "not x BndBnd b" we conclude "x DC c".
In other words, you need to use the composition of the relations "IntInt", "IntBnd", ... . Look for "9-intersection model", "4-intersection model" and/or "3-intersection model".

@ pasaba por aqui thank you for the above.. I found the 9-int and 4-int models here https://web.mst.edu/~chaman/home/pubs/2011CAINE_HawaiiHonolulu.pdf in page 2. But still it is not clear how you used the 9-int matrix to conclude on the above.. Could you elaboarte a bit more ? For instance how to use the matrix to prove that "From "x IntInt c" and "not x IntBnd c" we can say "not x IntExt c" " or that " "not x IntExt c" along with "not c IntInt b" proves that "not x IntInt b" "


text:
How to fix high variance of the returns on a 2d env?
I'm trying to train an agent on a self-written 2d env, and it just doesn't converge to the solution.

It is basically a 2d game where you have to move a small circle around the screen and try to avoid collisions with randomly moving "enemy" circles and the edge of the screen. The positions of the enemies are initialized randomly, at a minimum distance of 2 diameters from the enemy.
The player circle has $n$ sensors (lasers) that measure the distance and speed of the closest object found.
The observation space is continuous and is made of concatenated distances and speeds, hence has the dimension of $\mathbb{R}^{n * 3}$.
I scale the distances by the length of the screen diagonal.
The action space is discrete (multidiscrete in my implementation) $[dx, dy] \in \{-1, 0, 1\}$
The reward is +1 for every game step made without collisions.
I use PPO implementation from Stable Baselines, but the return variance just gets bigger over the training. In addition to that, the agent hasn't run away from the enemies even once. I tried even setting the negative reward, to test if he can learn the suicide behavior, but no results either.
I thought maybe it's just possible for some degenerate policies like going to the corner of the screen and staying there to gain big returns, and that jeopardizes the training. Then I increased the number of enemies, thinking that it will enforce the agent to learn actually to avoid the enemies, but it didn't work as well.
I'm really out of ideas at this point and would appreciate some help on debugging this.

What diagnostics have you already done? What behaviour are you observing for the failing agent? Do you have unit testing around the base environment, so you know for sure that the sensors are working as intended? Are the sensors wide - so anything within the sensor angle is detected? Or are they narrow and might miss unaligned enemies?

Could you please elaborate on "diagnostics"?
Right now he moves to the lower part of the screen and then moves up and down for some time until he hits the enemy. I'd say the best he's got so far was that he learned not to collide with the border. This was observed in different policies.
The sensors have been tested and are working properly.

Sensors are basically just rays. They can miss some enemies that are far away, but the problem is that the agent doesn't even learn to flee the enemies standing very close.

By diagnostics, I mean collecting data about training progress and outcomes in detail. E.g. graphing loss, reward, also being able to spot check an episode and view  behaviour with visualisation of the environment. I was wondering about this because of your concern over "degenerate policies" - have you actually observed the agent behaving according to such policies?

Have you calculated the coverage of the sensors - what percentage of enemies are detected at any given range? WIthout knowing more, that would be my first concern, that there was too much hidden state due to small movements making enemies visible then invisible again. Unless the agent has a memory this could be a major problem.

Thanks a lot for the advice! I found a bug in how my observation was made and it started working. What helped me was that I visualized the collected state, which involved inverse transformation and a step-by-step review of my env. As a side note, the variance was still getting over the training, but it is related to the env dynamics.


text:
Which introductory courses (preferably video lectures) could I use to learn ML for applying ML to black hole simulations?
I am a Ph.D. candidate in High Energy Physics and my research involves numerical simulations and data analysis. I am interested to learn Artificial Intelligence and Machine Learning from the basics so that I could implement the same in my research. Due to the huge popularity of AI & ML one could easily find a large number of articles on the internet on the topic, but it seems to me that a search for a proper course (that would teach the basics at the beginner level) is nearly impossible for me.
It would be helpful if someone suggest me an introductory course (preferably video lecture) on 'machine learning' with the goal of applying the concepts to simulations of black hole environments.

Hello. Please, reformulate the question so that it's not too broad. If you're looking for courses (with video lessons) on a specific subject (e.g. "introduction to machine learning"), then that should be your specific question. Asking for resources on AI and ML is too broad (because there are many types of resources, many subjects in AI and ML, etc.), so this post could be closed as such.

@nbro Thanks for the suggestion. I had reformulated the question. My interest is on implementation of AI in numerical simulations of black hole environments that include hydrodynamics around black holes.

Ok, thanks for trying to improve this post. I would also suggest that you put your specific question in the title. Right now, you still have a quite broad title and you still left "any introductory course, lecture notes or lecture videos on 'machine learning' suitable for the beginners" in your post, so the post is still a bit broad.

It may be a good idea to reformulate your question as something like "Which resources (in particular, courses with video lessons) could I use to study machine learning for the ultimate purpose of applying ML to simulations of black holes" (if this is your question, I don't know for sure)

In addition to that, this type of question has been asked a lot here in the past. For example, here, here, here, or here. As you can see, one person already provided an answer, which can be useful, but shows that your post is too broad.

I have edited your post to focus on 1. ML, 2. courses and 3. beginners, so that your post is more focused and thus acceptable now. In any case, I still think your question is a bit too broad and partially a duplicate of the linked ones. It may also be a good idea to provide an example of a course that you found and that is "not suitable for beginners" or more specifically you, in order to understand which courses you're really looking for.

@nbro Thank you for the clarification. Actually I am completely new to the subject and I am yet to learn ML. I had started with Andrew Ng's course.

@nbro on what grounds this is not an opinion-based question (hence closable as such)?

@desertnaut The OP is looking for one course that meets certain requirements (i.e. for beginners, that covers machine learning, preferably with the "goal of applying the concepts to simulations of black hole environments.". So, it seems to me that this is the type of question is similar to a question like "Which model could I use for this problem X?". In other words, the OP is looking for one possible solution to their problem, not necessarily the "best", which would make it opinion-based, as "best" is ambiguous or relative. We had many similar questions in the past...

In many cases, this type of question doesn't lead to "good" answers. So, in a way, you're right, this might lead to opinions, but, hopefully, people will provide references that meet the requirements rather than based on their preferences or opinions. If you want, we can discuss this topic more on meta. Feel free to create a meta post so that we can discuss there, and see what the community thinks about this type of question.

Cornell University offers lecture notes and videos of over thirty machine learning related courses in this link. CS4780 Introduction to Machine Learning in particular is a great resource (lecture notes, recordings).
The Visual Computing Group of TU Munich also offers lecture notes and videos of modern computer vision related tasks in this link. Introduction to Deep Learning in particular is a great resource (lecture notes, recordings). There are also courses on advanced topics of deep learning, image segmentation, 3D scanning, as well as a lot of publications on scene reconstruction, simulations and more.