text:
NN to find arbitrary transformation
Problem description
I'm creating a clock with 4 seven-segment LED displays. In an effort to get more familiar with tensorflow, I figured I should try to drive this clock with use of a Neural Network.
The input of the network is the Unix time.
Initially I wanted to make the network output a UINT32 value, which I can then bit-shift into the shift-registers I use for driving the LEDS. Because this proved to be unsuccessful, I removed the last step, and instead went with 32 booleans for every led-segment as output.
Current status
This last action was unfruitful as well, the best I get my network is a loss of about 0.48, indicating to me that it's best effort is to guess what the output could be.
Input
print(x_validate[0])
print(y_validate[0])
3116360099
[0 1 1 0 0 1 1 0 1 1 1 1 0 0 1 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 0]

The input is the unix time code. I tried normalizing this by dividing by 2^32, but this didn't make any significant difference.
Validation is an array with either 0 or 1 based on whether the LED needs to be on or off. The first 8 bits represent the first 7-segment display, etc. (The 8th bit is never used here because that's connected to the dot on the display.)
Data generation
# Number of sample datapoints
SAMPLES = 2000000

# Generate a uniformly distributed set of random numbers in the range from
# 0 to uin32 max
x_values = np.random.uniform(
    low=0, high=2**32, size=SAMPLES).astype(np.uint32)

# Shuffle the values to guarantee they're not in order
np.random.shuffle(x_values)

print(x_values)

# Time helper function
def to_utc(x):
  return datetime.utcfromtimestamp(x).replace(tzinfo=timezone.utc).astimezone(pytz.timezone("Europe/Amsterdam")).strftime('%H%M')
to_utc_batch = np.vectorize(to_utc)

y_values = to_utc_batch(x_values)
print(y_values)

# translate to bitstream
def lookup(number):
  switch = {
      0: [1,1,1,1,1,1,0,0],
      1: [0,1,1,0,0,0,0,0],
      2: [1,1,0,1,1,0,1,0],
      3: [1,1,1,1,0,0,1,0],
      4: [0,1,1,0,0,1,1,0],
      5: [1,0,1,1,0,1,1,0],
      6: [1,0,1,1,1,1,1,0],
      7: [1,1,1,0,0,0,0,0],
      8: [1,1,1,1,1,1,1,0],
      9: [1,1,1,1,0,1,1,0]
  }
  return switch.get(number)

print(y_values)

def compile_output(value):
  f = []
  for i, c in reversed(list(enumerate(value))):
    f = f + lookup(int(c))
  return f

output_values = []

for y in y_values:
  output_values.append(compile_output(y))

y_values = output_values

After, data is distributed in 3 sets:
Training: 60%
Validation: 20%
Testing: 20%

Model
model = tf.keras.Sequential()
model.add(keras.layers.Dense(32, activation='relu', input_shape=(1,)))
model.add(keras.layers.Dense(64, activation='relu'))
model.add(keras.layers.Dense(64, activation='relu'))

# Final layer
model.add(keras.layers.Dense(32, activation='sigmoid'))

model.compile( optimizer='adam', loss='binary_crossentropy', metrics=["accuracy"])

I went with binary_crossentropy and sigmoid as I figured the case is essentially a multi-label setup.
I have tried the following already, but did not succeed:

Add more layers
Make dense layers wider (to max of 512)
Add a Dropout layer -> so to have it try out more things to find the relation
Use Softmax activation
Normalize input data by dividing Unix time by 2^32
enlarge sample data to 20.000.000
do anything between 10 and 50 epochs ( I usually quit after 15 epochs, when absolutely no change was observed between the last 5 epochs)

Question

Why is this model not successful in finding a relation between the data?
How can I improve this model or the data so it will be able to succeed?

Bonus
When successful, ideally the output of the model would be a UINT32 number instead of this array of booleans. Any tips on how to get there would be appreciated as well.
Edit:
Really sorry, left out this particular line:
# Train the model
history = model.fit(x_train, y_train, epochs=10, batch_size=64,
                    validation_data=(x_validate, y_validate))

What is your batch size? What is your learning rate? Very generally, those two are good parameters to play around with first, and you don't mention them neither in code nor text (unless I'm very blind). I don't think you need more layers or bigger layers. Why are your inputs so high? If I'm not mistaken, you have 24*60=1440 different options for clock displays, and if not you should be able to do modulo to bring it into this range, as you're only interested in the time, right? Please clarify if I'm misunderstanding your idea.

@Mafu, Thank you for your comment! Sorry I forgot to specify the batch size, added it now. Learning rate I didn't specify, I guess it's using the default for Adam?
Do you mean high numbers as input? That's because the Unix time code is a 32-bit value, and the idea is that the model will be able to deduce the time-setting from any arbitrary time code.

Ok, I think you should really try varying the learning rate. That's one of the most important tuning factors. According to the Tensorflow Documentation, the default is 0.001, which in many of my applications has been too high. Try going with 1e-4, 1e-5, also 1e-2, see how it changes the learning curve and the results. From that you should be able to deduce useful conclusions.


text:
Does the order of data augmentation and normalization matter?
What is the preferred order of data augmentation and normalization? Is it the former followed by the latter?

Although I'm not really an expert in this topic, I would say that normalization should be applied after data augmentation, given that the latter can change e.g. the range of values of the inputs, and neural networks should generally learn better when the inputs are normalized.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Can I train a DQN on the same dataset for multiple epochs?
I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a generator to feed the data into my DQN. I've been running some automated tuning on the hyperparameters and seem to have found some good values.
Now I am wondering if I should be training on the dataset more than once or whether that would cause the network to simply memorize past experiences and cause overfitting. Is there a standard practice when it comes to training on historical data in regards to the number of epochs?
Edit: I'm not nessesarily looking for an answer to how many epochs I should be using, rather I'd like to know if running over the same data more than once is okay with DQNs

The samples(experiences) store inside the experience replay those are different at each run despite of the data being same(historical data). So I don't it will matter.

@SwaksharDeb Thanks, I think I understand. Does this mean that it won't hurt the network to train more than once or that it won't benefit the network? ...or both?

I think it will not hurt the performance and your dataset is also pretty big. If your dataset is for particular company and we are collecting experience based on past 20 years data then it is obvious thing to do.

Okay, that makes sense, I'll try it out and see how it performs, thanks. It is just for one company, IBM. The dimensions for each example are pretty large, 354 parameters, and it's got a little over 2 million examples so running through the dataset takes several hours even with a decent GPU.

I'm assuming according to your question that you have a fixed batch, or in other words, there's no possibility of further exploration in your settings. If this assumption is true, you have what's known as Batch/Offline Reinforcement Learning.
First, let's check some aspects about this: in Offline RL once that there's no possibility regarding further exploration, your dataset must contain a brunch of situations to leads your system to learn a robust policy. Imagine that you're working with robots, and training them using a fixed batch extracted using real-world interactions. If this batch was collected from a system that works pretty well, it might not contain samples about non-desirable situations. Once this system is deployed in the real world, facing a non-desirable situation, your system could not "know" which action should be taken to "escape" from these non-desirable states, once it never saw this during the learning phase. To summarizing, your dataset should be large and representative.
Now we assume that your dataset is large and representative. So, another problem arises: fixed batch shift the distribution of samples, creating a kind of bias that can be extrapolated by many epochs.
Finally, your dataset is composed of expert demonstrations? In other words, do you have the best actions on those samples in your dataset for each state?
The good thing about a fixed batch with real-world data is that it can booster the learning process once it contains the true dynamics of the real system. But be careful regarding the distribution of samples and training hyperparameter. Resuming, there's not a recipe for all problems, it depends on your case. I'll let two references that can be useful and where this answer is based:
References:
Nair, A., Dalal, M., Gupta, A., & Levine, S. (2020). Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359.
Fujimoto, Scott, David Meger, and Doina Precup. "Off-policy deep reinforcement learning without exploration." International Conference on Machine Learning. PMLR, 2019.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is it possible to do face recognition with just the eyes?
Assuming the input photo is focused on a person's face, if the person is wearing a surgical mask, most face recognition software fail to identify the subject's face.
Most facial landmark models are trained to identify at least the eyes and the tip of the nose (for example, dlib's 5 point landmark).
Is it possible to construct a model that is trained to identify a face based on only the eyes?
Edit: Sorry for my broken english, but by "eyes" I mean the periocular area. I am terribly sorry because english isn't my first language.

Try transfer learning from a network already proficient in face recognition on the entire face, and see how it performs. My guess would be yes, it is possible (but wont be as accurate).

It could be possible. It might be inaccurate. But in some countries it might be illegal... See in Europe the GDPR

@exius To clarify the terminology, facial recognition is accomplished using a photo or video of the face.  When you say 'just with the eyes' you are no longer talking about facial recognition.  Facial recognition is a biometric using an image of the face.  Iris and retina recognition are biometrics using an image of the eye.  A biometric is used to identify a person.

Yes, it must be possible as retina scanners have been used as a method of personal identification for some time. The difference is, if you have a retinal scanner you are probably controlling the focal distance of the picture you are taking and using suitably high resolution. Your mileage may vary as these things decrease in quality.

The two main eye biometrics are iris recognition and retina recognition (aka retinal scan).  These are not going to work from an ordinary photo of someone's face.  I have used iris recognition at about ten feet away and this article claims it can be done at 40 feet!
Eye recognition, or identification of a person from an image of their eyes alone (i.e., without seeing their iris or retina), has such a high error rate that it is not done.  You may find the following paper of interest:

Nawaz Ripon, K. S., Ershad Ali, L., Siddique, N., & Ma, J. (2019). Convolutional Neural Network based Eye Recognition from Distantly Acquired Face Images for Human Identification. 2019 International Joint Conference on Neural Networks (IJCNN), Neural Networks (IJCNN), 2019 International Joint Conference On, 1–8. doi:10.1109/IJCNN.2019.8852190

For more information on iris recognition see pp 10-11 of this tutorial, and pp 12-13 for retinal scan.
An excerpt from Retinal vs. Iris Recognition: Did You Know Your Eyes Can Get You Identified? by Danny Thakkar:

Retina recognition
The posterior portion of human eye forms retina. It
is made of a light sensitive tissue. When light passing through cornea
and lens reaches retina, neural signals are generated and transferred
to the brain via the optic nerve. Retina is a thin layer of tissue
formed by neural cells. Capillaries responsible for blood supply of
this layer forms a pattern that can be used for personal
identification. This pattern of blood capillaries is believed to be
unique in each individual due to huge possibility of variation how
these capillaries run on the surface of retina. Since retina is
located at the posterior portion inside the human eye, special
equipment is required to scan this pattern. Retina recognition is one
of the least deployed biometric methods because of high cost of the
implementation and its highly invasive nature that may cause some user
discomfort. Still, it is used is very high security applications like
military and high level government access due to its accuracy and high
level of security.
Retina recognition systems make use of low energy infra-red light to
scan the retinal pattern. Blood vessels absorb infrared light while
surrounding tissues reflect it. This reflection is detected by the
retina recognition system and image of this pattern is captured. This
image is further enhanced to make is usable for the recognition
algorithm. Retina template is generated once the image is taken
through recognition algorithm; this template is associate with a
subject’s demographic data and stored. The process so far is called
enrolment. The subject’s identity can be verified anytime by scanning
a new retinal sample and matching it against the stored template.
Iris recognition Iris is the ring shaped colored portion in a human
eye and is visible from outside with naked eye. It is made of muscle
tissue that adjusts the size of pupil and controls how much light can
enter the eye. Amount of melatonin pigment in iris is responsible for
different colors that human eyes take. Folds in iris muscles
throughout the ring create a pattern with great amount of details.
Formation of this pattern is completely random and there is no rule
how it will turn out in an individual’s eye. However, once this
pattern is created during the foetal development, it stays the same
throughout the life. An individual’s irises are unique and
structurally distinct, even iris of same individual does not match.
All these attributes make them good enough for personal recognition.
Details of iris can be captured with any high quality digital camera,
however, modern recognition systems make use of near infrared (NIR:
700–900 nm) instead of visible light to capture details. Since iris
recognition can be established with high quality camera and
recognition software, it can be setup on any computing device;
however, dedicated recognition systems are more common due to
performance and security reasons. Iris recognition systems use a
camera to capture details of the iris and this image is enhanced by
the image enhancement algorithms. Once the image is usable enough, it
is processed by the recognition algorithms, which extracts unique
features to generate a biometric template. Associating identity data
with this template establishes identity of the subject in question,
which can be used for identity verification in future.

Thankyou for your answer. However I have some questions about your claim that eye identification is not done due to it having high error rate. In this paper and this paper having both around 90% accuracy at best. Why did you claim such a way or am I horribly missing something ? thankyou very much.

My response is about real world applications and not research.  90% at best is very poor when compared to over 99% for facial recognition.  For the very unusual, and temporary, situation we are in now (i.e., the COVID-19 pandemic) it sounds like eye recognition would be a solution but for what?  Would you risk a 10% or higher error to use it at an ATM machine?  What application could you possibly deploy this in before the pandemic is over?  I can certainly see eye recognition being used to supplement a human in identifying people in special situations.  One example would be for detective work.


text:
The MLP output of a neural network can be written as $\|x\|\|w_l\|\cos(\theta_l)$: why is the norm easier to maximize?
The MLP output of a neural network is a dot product between the weights and the input and therefore can be written as $\|x\|\|w_l\|\cos(\theta_l)$ (see this for more details), where $x$ is the input, $w_l$ is the weights of layer $l$ and $\theta_l$ is the angle between them.
I read this in a paper: Angular Visual Hardness. The paper stated that it's much easier to maximize the norms $\|x\|$ and $\|w_l\|$ than the cosine similarity. Why is this the case? Just because $\cos(\theta_l)$ gives less weight because it's bounded between $[-1,1]$? Or is it due to the gradient? So, why is the norm easier to maximize?


text:
Is $\min(h_1(s),\ h_2(s))$ consistent?
If $h_1(s)$ is a consistent heuristic and $h_2(s)$ is a admissible heuristic, is $\min(h_1(s),\ h_2(s))$ consistent?

You can easily find a counterexample. Suppose that there are three nodes $s$, $p$, and $goal$ such that $s \rightarrow p \rightarrow goal$. The real cost of going from $s$ to $p$ is $c(s,p) = 10$ and $c(p, goal) = 10$. Also, $h_1(s) = 18$, $h_1(p) = 9$, $h_1(goal) = 0$, $h_2(s) = 17$, $h_2(p)  = 1$.On the other hand, $h^*(s) = 19$ and $h^*(p) = 10$.
Now, $h_1(s) \leqslant c(s,p) + h_1(p)$ and $h_1(s) \leqslant c(s, goal) + h_1(goal)$ that satisfies the consistency constraint of $h_1$. Also, $h_2(s) \leqslant h^*(s)$ and $h_2(s) \leqslant h^*(p)$ that approve $h_2$ is admissible.
However, for any node $n$, if we define $h_3(n) = \min(h_1(n), h_2(n))$:
$$h_3(s) = \min(h_1(s), h_2(s)) = 17 \not\leqslant c(s,p) + \min(h_1(p), h_2(p)) =
$$
$$c(s,p) + h_3(p) = 10 + 1 = 11. $$
It means that $h_3$ as a heuristic function is not consistent.


text:
Can predictions of a neural network using ReLU activation be non-linear (i.e. follow the pattern) outside of the scope of trained data?
Training on a quadratic function
x = np.linspace(-10, 10, num=1000)
np.random.shuffle(x)
y = x**2

Will predict an expected quadratic curve between -10 < x < 10.

Unfortunately my model's predictions become linear outside of the trained dataset.
See -100 < x < 100 below:

Here is how I define my model:
model = keras.Sequential([
      layers.Dense(64, activation='relu'),
      layers.Dense(64, activation='relu'),
      layers.Dense(1)
  ])

model.compile(loss='mean_absolute_error', optimizer=tf.keras.optimizers.Adam(0.1))

history = model.fit(
    x, y,
    validation_split=0.2,
    verbose=0, epochs=100)

Here's a link to a google colab for more context.

It isn't too surprising to see behaviour like this, since you're using $\mathrm{ReLU}$ activation.
Here is a simple result which explains the phenomenon for a single-layer neural network. I don't have much time so I haven't checked whether this would extend reasonably to multiple layers; I believe it probably will.
Proposition. In a single-layer neural network with $n$ hidden neurons using $\mathrm{ReLU}$ activation, with one input and output node, the output is linear outside of the region $[A, B]$ for some $A < B \in \mathbb{R}$. In other words, if $x > B$, $f(x) = \alpha x + \beta$ for some constants $\alpha$ and $\beta$, and if $x < A$, $f(x) = \gamma x + \delta$ for some constants $\gamma$ and $\delta$.
Proof. I can write the neural network as a function $f \colon \mathbb R \to \mathbb R$, defined by
$$f(x) = \sum_{i = 1}^n \left[\sigma_i\max(0, w_i x + b_i)\right] + c.$$
Note that each neuron switches from being $0$ to a linear function, or vice versa, when $w_i x + b_i = 0$. Define $r_i = -\frac{b_i}{w_i}$. Then, I can set $B = \max_i r_i$ and $A = \min_i r_i$. If $x > B$, each neuron will either be $0$ or linear, so $f$ is just a sum of linear functions, i.e. linear with constant gradient. The same applies if $x < A$.
Hence, $f$ is a linear function with constant gradient if $x < A$ or $x > B$.
$\square$
If the result isn't clear, here's an illustration of the idea:

This is a $3$-neuron network, and I've marked the points I denote $r_i$ by the black arrows. Before the first arrow and after the last arrow, the function is just a line with constant gradient: that's what you're seeing, and what the proposition justifies.

Am I misunderstanding this, or do different activation functions lead to the same result because they just activate the y = weight*x + bias linear function?

Well, you'd get slightly different results with other activations. As an example take a layer of neurons with sigmoid activation: you'd essentially have a function which is a sum of (scaled and repositioned) sigmoids. I haven't thought about it very deeply but I expect that as $x \to \infty$ you'd see each neuron saturate (either at zero or one) so the function would be "almost constant" after a certain point. Play around on a graph plotter with sums of logistic functions to see what I mean, it's a fun exercise.

Short answer: Yes.
Consider a non-linear regression on that dataset. Using a model of degree two, it would fit a quadratic exactly to your perfect data here. But I suppose you're asking about neural networks. You can have neural networks set up that are exactly equivalent to this kind of regression, so even with neural networks, yes you can get this non-linear extrapolation. Of course as you probably realise, you would have to know in advance what kind of behaviour you expect in this extrapolation before really trusting any extrapolated predictions.


text:
Why do I have better RMSE when I don't scale the target?
I use PyTorch for training a simple neural net for a regression task on a dataset with 12 numerical features + target (target is the 13th column) + 2 categorical features
Before training, I execute
# numeric_columns = numeric_columns[:-1]
scaler = StandardScaler()
scaler.fit(df_train[numeric_columns]])

Also, in my custom torch.util.data.Dataset I scale the data using my scaler object.
After each epoch, I evaluate the RMSE("reversed scaled" prediction, non-scaled target), like the following:
y_pred = (y_pred * self.scaler.scale_[13]) + self.scaler.mean_[13]
loss += self.criterion(y_pred , y_true).item()

RMSE if I don't scale the target (the first comment would be uncommented and the y_pred row would be commented) is around 0.95 (I tried multiple hyperparameters)
RMSE if I scale the target is 1.7
The target has mean 3.3 and standard deviation of 2.
What am I doing wrong? I thought scaling the target is a must when dealing with neural networks.

The score should be better with scaling, or not worse at least.
Check the indexing. The 13th column has the index 12, because index is zero-based. Also, if you want to leave out the 2 last categorical columns it should be columns[:-2]

Good catch and thanks! I used the proper mean and standard deviation and I got much closer RMSE scores


text:
Are there any benefits of adding attention to linear layers?
Is attention useful only in transformer/convolution layers? Can I add it to linear layers? If yes, how (on a conceptual level, not necessarily the code to implement the layers)?


text:
What practically makes a good architecture of ANN?
When we take a look at the literature there are so many opinions.
I was wondering what are some generally good practices to design an architecture, like how much depth would you prefer and how much width would you prefer.
Should the number of training data influence your decisions of designing the architecture.
What should the number of parameters be ? etc.

Can you please provide 2-3 examples of "literature" that are contradicting each other by citing them and quoting the relevant parts?

@nbro https://stats.stackexchange.com/a/223637/272525 and https://arxiv.org/abs/1605.07146

Can you please edit your post to quote (provide 2 sentences) that are contradicting themselves?

I am not sure if there really are contradicting opinions on this matter. CNNs, RNNs, LSTMs all have specific types of data they are good at predicting. Depth and width, or in general the size of the neural network mostly depends on the size of your dataset. You don't want to build a too large network that will overfit the available data, which can usually be understood to be the case after running a few epochs but in general, number of trainable parameters should be smaller than to total number of independent data points you have to avoid memorizing the data definitely.
Width and depth of the network depends on the size of the data available (as mentioned before), but can simply be thought just as another hyper-parameter that is to be decided by training itself. It can be defined as a part of cross validation process of building a neural network.


text:
Find the expected reward in an expectimax-based dice rolling game?
I have this question that I'm kinda stuck on.
It's a game scenario in which we set up an expectimax tree. In the game, you have 3 dice with sides 1-4 that you roll at the beginning. Then, depending on the roll, the player can choose one of the dice to reroll or not reroll anything. Points are assigned like so:

10 points if there's 2 of a kind
15 if there's 3 of a kind
7 if there's a series like 1-2-3 or 2-3-4
Otherwise, or if the sum is higher than the rewards from above, the score = sum of the rolls

For additional context, this is an example expectimax tree I came up with, for the case that the player rolled a 1,2,4 and is considering rerolling or not:

Now lets introduce a new agent -- a robot that's supposed to help the human player. we assume

the human player choses any action with uniform probability regardless of the initial roll
there's a robot that, given a configuration of dice and the human's desired action, actually implements the action with probability 1-p and overrides it with a "no reroll" order with probability p>0. It has no effect if the human's decision is already to not reroll.

For that scenario, I came up with this expectimax tree:

Now for the part I'm actually stuck on -- lets define A, B, C, and D as the expected reward of performing actions "reroll die 1", "reroll die 2", "reroll die 3", and "no reroll." How do we find $R_H$, the expected reward for the human acting without the robot's help, and $R_{AH}$ the expected reward for if the robot helps? We can't use p in the expression, we only have access to A,B,C,D and we're supposed to write it in the form $X + Y_p$
*EDIT: I asked again and the question was worded weirdly. They said we should definitely use p. What was meant by not using p is X and Y themselves can't contain p. But Y will be multiplied by p in the final simplified form.
For $R_{H}$ I think the answer should be $\frac{(A + B + C + D}{4}$ because of uniform distribution over A-D.
I'm supposing that $R_{AH}$ would be $\frac{(A + B + C)(1-p) + D + Dp}{4}$? Because the robot doesn't override with probability $1-p$, but he can only override A-C and does with probability $p$.
I think something feels slightly wrong about my answer but I'm not sure what.

Are you sure you are not allowed to use p? The expected reward R_AH is a function of p.

Hi. Could you please put your main specific question in the title? "Expectimax World Problem — Dice Rolling Game" is not a question, but the title of an article.

@Cohensius I asked the instructors and they clarified that you can use p. What was meant by not using p is X and Y themselves can't contain p. But Y will be multiplied by p in the final simplified form. It was a slight wording confusion. I added my best guess at a solution. However, I'm not 100% sure, and would appreciate confirmation :)

@nbro done! sorry

If I understand,

With probability $p$ the robot select no re-roll (action $D$).
With probability $1-p$ the human uniformly select an action between $A,B,C,D$.

Thus the expected reward for if the robot helps is
$$R_{AH}= p\cdot D+(1-p)\frac{A+B+C+D}{4} = \frac{(A+B+C)(1-p)+D+3pD}{4}$$


text:
What approach to use for selecting one of the category according to short category text?
I need some tool to classify articles based on short category text which consists of two or three words separated by '-'. The RSS/XML tag content is for example:

Foreign - News

Football - Foreign

I created my own categories and now I need to classify categories from parsed RSS of this news source, so it fits news categories defined by me.
I would for example need all articles containing category "football" to be identified as a category Sport but sometimes those categories XML tags contains exact match like Foreign - News should belong in the DB to category defined by me as Foreign.
I can of course also use longer description text if that would be needed but I think for this simple problem that would not be even necessary.
Since I used only trained decision trees frameworks so far for another project, I would like to hear advice about approach, AI technique or particular framework I can use to solve this problem. I don't want to get into a dead-end street by my own poor in this field not experienced decision.

Hi. Asking for tools (software) is off-topic here. So, if you don't want me to close this post, I would suggest that you edit this post and ask for an "approach" (using AI techniques) to solve your problem.

For this application, you can frame it as text classification. Look at SpaCy. You just need to create embeddings for your text and put a Softmax in the end. You can get those embeddings from BERT or anything else out there. You can in fact just use GLOVE vectors and others like it, concatenate them and then train a classifier.


text:
Is there an AI that can extract proper nouns from free text?
I have some free text (think: blog articles, interview transcripts, chat comments), and would like to explore the text data by analysing the proper nouns it contains.
I know of many ways to simply look up the text against a 'list' of proper nouns. The problem with the approach is many false positives and false negatives, as well as inaccuracies where one proper noun (e.g. "John Allen") is identified as two proper nouns ("John" and "Allen"), as well as other problems, mostly to do with long or unusual proper nouns (e.g. "the Gulf of Carpentaria" - a single proper noun containing the word "of", and long names like "Joost van der Westhuizen"). These kinds of longer, non-conformist proper nouns tend to really trip up grep-style proper noun identification models.
Does anyone know if any AI available to the public can more accurately identify proper nouns in free text?

This is a hard problem, unless you have a list of proper nouns you want to recognise. If John Allen is in this list, then you can easily use a longest match to prefer it over John or Allen. The same applies to the other examples you give.
Capitalisation on its own is not very reliable, as words at the beginnings of sentences are also capitalised, and sometimes technical terms or emphasis are expressed By Capitalising Them In Mid-Sentence. It's not what I would do, but you really have to expect anything in free text.
You could go some way by looking for sequences of proper nouns with of, de or van etc between them. There is probably a reasobaly small list of those connectors.
You can set up a grammar to capture complex names, but by far the most reliable way is to have a list. Unfortunately there is no easy solution for this. I would approach it iteratively, ie process a segment of text, tidy up the results, generate a list from them, and repeat with more text. Everything that is in the list you accept, other candidates you vet before adding them to the list. Eventually you should get fewer candidates that are not in the list.
You can probably kick-start your list with a gazetteer, ie a list of names and places. There are several of those on the web.
In general this procedure is referred to as Named Entity Recognition (NER). There are many ways to solve it, and my suggestion here is a pragmatic approach, which works reasonably well with not too much effort.

This does not appear to be an AI answer, but more of a hack-ish approach. You're correct that small words like "van" are an indication of proper nouns, as is capitalization, but neither is perfect. AI's are however very good in statistics and context. Our biggest problem might be deciding whether "Ford Transit Van" is a proper noun ! Two AI's might have different conclusions, and I couldn't fault either.

You might want to mention that this problem is called Named-Entity Recognition (NER).

@MSalters I would prefer to call it a "pragmatic" approach -- nothig hack-ish about it. There are several heuristics that one can make use of, as it's a non-trivial task there is no clear-cut 100% answer.


text:
Is it possible to transform audio with neural networks to make it sound like 3d sound
so the idea is to feed neural network data like
input: mono audio(extracted from existing 3d audio) output: 3d audio
after training it should convert mono audio to 3d sound
do you think it is possible? does it already implemented?(I didn't found)
P.S
it should sound like https://www.youtube.com/watch?v=kVH_y0rOyGM
not like usual 3d youtube.com/watch?v=QFaSIti5_d0

I have not heard of ASMR in the context of a technical sound quality ("high quality 3d sound") but in context of audio content type - https://en.wikipedia.org/wiki/ASMR - it involves low sound levels because it involves recording quiet sounds like whispering, and may require accurate noise-free stereo recording to have desired effect for some kinds of source, so may be recorded in high quality for that reason. So could you clarify - are you trying to improve quality on recordings of ASMR content (as per the Wikipedia link), or have some other goal.

Perhaps I am wrong and ASMR is also the name of an audio quality standard? In which case please link to a definition and explain which kinds of content you need to enhance.

so, the basic idea is to make sound feels natural, and you should hear the position of it
I want to feed my neural network dataset consists of [x, y, monosound] -> 3d sound(supervised learning), the monosound is extract from existing 3d sound
when neural network trained, it should be able to convert
x, y, monosound -> 3d

to see difference between regular 3d sound and asmr take a look on this
https://www.youtube.com/watch?v=kVH_y0rOyGM&t=53s - asmr
https://www.youtube.com/watch?v=QFaSIti5_d0 - 3d sound
(put on headphones)

so, the idea is pass mono audio + (x, y, z coordinates) -> get 3d sound, it can be not neural network, any technology, I just want to know, which will be the best to do it

OK, that makes sense. I do not think you are asking about ASMR at all. Instead, it happens to be the case that ASMR fans like high quality binaural recordings. Please could you update your question to make it clear what you are asking? Use [edit] to make changes

This is generally done with softwares with static programming. Why do you need a learning algorithm for this?

could you give an example?


text:
Clarifying representation of Neural Nerwork input for Chess Alpha Zero
In the Alpha Zero paper (https://arxiv.org/pdf/1712.01815.pdf) page 13, the input for the NN is described. In the beggining of the page, the authors state that:
"The input to the Neural Network is an N x X x (MT + L) image stack [...]"
From this, I understand that (for one training example) each input feature is an 8x8 plane. (Technically speaking, every value of every 8x8 plane is a feature, but for the purpose of the question let's suppose that a plane is an input feature).
In the description of the table on top of the image, the following statement is made:
"[...] Counts are represented by a single real-valued input; other input features are represented by a one-hot encoding using thespecified number of binary input planes. [...]"
I understand how they convert the P1 and P2 pieces to one-hot encodings. My questions are:

When they say single real-valued input, since every input feature should be an 8x8 plane, do they mean that they create an 8x8 plane where every entry has the same single-real value? For example, for the 'Total move count' plane, if 10 moves had been played in the game so far, it would look like the one below?

  move_count_plane = [[10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10],
                      [10, 10, 10, 10, 10, 10, 10, 10]]

For the 'Repetitions' plane, is it the same case as above? They mean a plane where every value is the number of times a specific board setup has been reached? For example, if a specific position has been reached 2 times, then the repetitions plane for that position would be

  # for a specific timestep in the T=8 step history
  repetitions_plane = [[2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2],
                       [2, 2, 2, 2, 2, 2, 2, 2]]

? Also, why do they keep 2 repetitions planes? Is it one for every player? (8 repetition planes for the past T=8 moves for P1, and more 8 repetition planes for the past T=8 moves for P2?)
Thanks in advance.

For anyone wondering, I believe to have found the answer:

Yes, it will be an 8x8 plane where all the entries are the same, the number of moves (or mpves with no progress).

There are two repetitions planes (for each position from the most recent T=8 positions):
a) The first repetition plane will be a plane where all the entries are 1's if the position is being repeated for the first time. Else 0's.
b) The second repetition plane will be a plane where all the entries are 1's if the current position is being repeated for the second time. Else 0's.


text:
What is the difference between model setup, model configuration, and model customization?
In the context of research papers related to deep learning models, the authors usually mention these terms in the experiment section when they are talking about the model: configuration, setup. For example: Akbik et al. 2018.
For example:

"We utilize the BiLSTM-CRF sequence labeling architecture proposed by Huang et. al (2015) in all configurations of our comparative evaluation."

"Baselines. We also evaluate setups that involve only previous word embeddings."

What is the difference between the terms? Is the model architecture the same with different hyperparameters?
Thank you in advance.

Using the search tool of my browser, I cannot find any of those terms in the paper that you're mentioning, so maybe you should explain more in detail in which section and/or page those terms can be found.

@nbro could you check the question now? I added two examples


text:
How to improve prediction performance of periodic data?
I have a 1 column dataset of $50 000$ points where 95% of the values equal $-50$. The data looks like  the following: $$\begin{matrix}
\text{time} & \text{value}\\
1&-50 \\
2&-50 \\
3&-50 \\
4& -50 \\
5&3 \\
6&-50\\
7&-50\\
8&5
\end{matrix}$$
As an addition, I know the exact time instance in which I will get value $\neq -50$ (as in the example above these are instances $5$ and $8$). The data is somewhat periodic, so the values which are different from $-50$ are chosen from a finite set $\mathcal{S}$.
To predict the values I use a 3 layer LSTM network with l2 regularizer where along with the values I input another column that looks like that:
$$\begin{matrix}
\text{time} & \text{value} & \text{expect a change}\\
1&-50 & 0 \\
2&-50 & 0\\
3&-50 & 0\\
4& -50 & 1\\
5&3 & 0\\
6&-50& 0\\
7&-50& 1\\
8&5 & 0
\end{matrix}$$
which identifies the change one time instant in advance, so the LSTM will know to expect a change. However, the prediction performance is quite poor, it always predicts the change in the value but is far from real one and usually takes values out of the set $\mathcal{S}$. Any idea of how this could be improved?


text:
Validation set performance increasing even after seemingly overfit on training set
I am training a semi-supervised GAN network using data from multiple subjects. I separated the labeled and unlabeled data based on my subjects, so there is no leakage, while having much more unlabeled data than labeled data. After few epochs training accuracy hits 100% which normally indicates overfitting, however the performance on the validation and test sets keeps increasing for 200-300 epochs. Is this considered overfitting and is there an explanation for this behavior?

There is a lot of variables here which can help answer this. The main one: How are you measuring accuracy? For example, if you measure a correct prediction as above 0.5 when positive and below 0.5 when negative, the accuracy could be 100% well before overfitting, as the network still has a lot of room to approach 1 and 0.

I am in a multiclass setting with k=12 classes and I use argmax on the output logits, and compare them to real labels using sklearn's accuracy function. I am also computing the loss, for example in one epoch i had 0.4 for validation and 0.01 for training, and later training loss drops to 1e-5 while validation loss oscillates around 1, with no noticable impact on the accuracy score.

How does it do on the test set? This does sound like it's overfitting based on the loss. I would triple check you are definitely running the same version of the network that's producing this loss when computing accuracy before continuing, because it does sound quite strange that loss would increase by over 100% with accuracy also increasing at the same time

Same on the test set, performances kept increasing even after signs of overfitting on the training set. For example in a recent model, I obtained 100% acc, 0.0016 loss for training, around 93% acc and 0.4 loss for both validation and test sets that came from different subjects.

Can you maybe post a graph of the loss and accuracy for training and validation? That could help see what's going on here more exactly.

I just added a graph.

If you ruled out leakage completely read this observation about double decent https://openai.com/blog/deep-double-descent/
This Blogpost from openAI shares the observation that the validation loss can decrease again even after initially increasing (which is typically a sign for the start of overfitting, e.g. in early stopping).

You did a great job at this...
You can use the Tensorflow’s LogSumExp built-in function to avoid numerical problems. This routine prevents over/under flow issues that may occur when LogSumExp encounters very extreme, either positive or negative values.
You have sorted out this:
There need to be Images from the generator. To these ones, the discriminator learns to classify as fake.
Real images with labels. These are image label pairs like in any regular supervised classification problem.
Real images without labels. For those, the classifier only learns that these images are real.
I would recommend you visit this link:
Semi-supervised learning with Generative Adversarial Networks (GANs)
Good luck.


text:
Advice required for identifying bone fragments in CT-scans using STL Files (3D image segmentation)
I am working on a project related to automating the procedure of manually segmenting some bones in CT scans and hopefully if everything goes alright in this stage, move on to do something more with them - like bone reconstruction etc.
I have been doing extensive research regarding this - and CNNs were something in my target as a ML method that could be used here. Emphasis is more on using Deep learning for this project.
So, what I have - the data: CT scans of chest/shoulder and for each of the CT scan, I have 4-6 STL files of the individual bone fragments or segments located in the shoulder or near shoulder region. I am a tad uncertain as to how to use those individual STL files.
Target: To label/classify/identify these fragments in the CT scan - automate it.
My MOA (Method of Approach) or what I understand - I believe it is object (bone fragment being the object) detection and feature (of those bone pieces that I need to lock on in the CT-scan) extraction using CNNs. I am looking at Mask R-CNN etc, use a pre-trained CNN for this.
But I am not entirely sure if my understanding is correct. This is my first time with this stuff, but hoping to learn more. CT-scans are in nifti format.
I could provide more info if required, would gladly appreciate any insight or help or advice with what could be the way forward and if I am thinking along the correct lines.
Thank you.

I am also working on this project, can you please share your dataset


text:
How to pathfind with volatile probabilities (Slay the spire)
I'm attempting to write an AI for the game Slay the Spire.  One of the tasks it will need to do is navigate the map.  The map is a directed acyclic graph with the same start and end node.
Each node (including the end node) will have 2 values associated with it: Expected value and death probability.  The goal of the AI should be to maximize the expected value without dying.  So far, none of this seems tough.
The twist here is that the death probability changes over time.  Some nodes (elites) have high volatility:  The death probability may go up or down drastically as we move through the map.  The pathfinding algorithm would need to consider adjacent alternatives.  A path that allows me to switch to a low-volatile node if things are getting tough is important.
As an example, the following map has two major routes.  Both routes have an elite (the creature with horns), but the one on the right is a forced elite, while the one on the left can be skipped if death probability is too high.  The ability for me to be flexible mid-route is an attractive feature, and one I'd like to take into account when pathfinding somehow.

How can my path-finding algorithm take into account adjacent paths/path flexibility?  Is this even a job for pathfinding at all?

I don't know Slay the Spire. Can you explain how your AI will get access to the map? E.g. is it always the same and someone (maybe you) has coded it up outside the game. Or do you have an API that will allow your bot to inspect game state - including the map - to help make decisions?

I essentially have an API to inspect the game state, including the game map.  The values "expected value", "death probability", and "volatility" are calculated by a separate part of my AI, and fed into my path-planning algorithm.  All paths are visible from the start, and every node will have those 3 values.

I've added a picture, hopefully that helps.


text:
Which RL algorithm would be suitable for this multi-dimensional and continuous action space?
Is there an RL approach/algorithm that would be suited for the following kind of problem?

There is a continuous action space with an action value $A_{a,t}$ for each action dimension $a$.
The objective function is a non-linear function of the satisfaction factors $S_{s}$ for each satisfaction dimension $s$ and some other random & independent factors. This objective function can be known to the agent.
Each satisfaction factor depends on an independent variable $\Delta^S_{s,t}$ and the effect $\delta^S_{a,s,t}$ of each action: $S_{s,t}=\Delta^S_{s,t} +\sum_a A_{a,t} * \delta^S_{a,s,t}$.
Each action can further have an effect $\delta^R_{a,r,t}$ on the inventory factors $I_{r,t}$ for each resource dimension $r$, with inventories being kept between time-steps and a factor $\Delta^R_{r,t}$ that is added or removed from the inventory at each step independent of the actions: $I_{r,t+1}=I_{r,t}+\Delta^R_{r,t} + \sum_a A_{a,t} * \delta^R_{a,r,t}$
The agent is constrained by each of these resources (i.e. the inventory has to remain positive).
The agent should be able to deal both with $\delta$ and $\Delta$ factors that are visible (states) and invisible (have to be learned).
A trained agent should be able to know how to adapt to changes of the $\delta$ and $\Delta$ factors, as well as the introduction or removal of activity dimensions.

EDIT: I have adapted the problem description after some feedback.


text:
Bias gradient of layer before batch normalization always zero
From the original paper and this post we have that batch normalization backpropagation can be formulated as

I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with respect to $b$, where $\{X_i\in\mathbb{R}, i=1,\dots,m\}$ is a network input batch and $\sigma$ is some activation function with weight $w$ and bias $b$.
I'm using Adam optimizer so I average the gradients over the batch to get the gradient $\theta=\frac{1}{m}\sum_{i=1}^m\frac{\partial l}{\partial x_i}\frac{\partial x_i}{\partial b}$.
Further, $\frac{\partial x_i}{\partial b}=\frac{\partial}{\partial b}\sigma(wX_i+b)=\sigma'(wX_i+b)$.
I am using ReLu activation function and all my inputs are positive, i.e. $X_i>0 \ \forall i$, as well as $w>0$ and $b=0$. That is, I get
$\frac{\partial x_i}{\partial b} = 1\ \forall i$. That means that my gradient $\theta=\frac{1}{m}\sum_{i=1}^m\frac{\partial l}{\partial x_i}$ is just the average over all derivatives of the loss function with respect to $x_i$. But summing all the $\frac{\partial l}{\partial x_i}$ up is zero, which can be seen from the derivations of $\frac{\partial l}{\partial x_i}$ and $\frac{\partial l}{\partial \mu_B}$.
This means that my bias change would always be zero, which makes no sense to me. Also i created a neural network in Keras just for validation and all my gradients match except the bias gradients, which are not always zero in Keras.
Does one of you know where my mistake is in the derivation?
Thanks for the help.

Hi. Could you please put your main specific question in the title? Thanks.


text:
What's $\mathbb{V}[\gamma]$ in Return-based Scaling: Yet Another Normalisation Trick for Deep RL?
In the paper Return-based Scaling: Yet Another Normalisation Trick for Deep RL,
$\sigma$, the scaling factor to normalize the TD error, is computed as:
$\sigma^2 = \mathbb{V}[R] + \mathbb{V}[\gamma]\mathbb{E}[G^2]$
What does the variance of $\gamma$ means? I believe $\gamma$ is the discount rate for the discounted return.

From the paper, note 3 of section 3.1.

Note that, perhaps unconventionally, we treat $\gamma$ as a random
variable here, because it is zero at the final step of an episode (even
when it is constant throughout the episode).

Later, it is stated that $\mathbb{V}[\gamma]$ is calculated based on the entire set of observed data. So the distribution of $\gamma$ is I imagine, categorical with two possible values; $\gamma$ or $0$, with probability being normalised total steps vs. number of trajectories.

So say there are 3 trajectories, so 3 times $\gamma=0$, and say each trajectory has 5 steps, so $4\times3$ times $\gamma$. $12\gamma$ and $3\times0$, is that it?

@Sanyou having your same doubt, I just think that $\gamma \sim Be(p)$ where $p$ is just the ratio between the steps and final-step in your MDP (so say you have a trajectory of 5, gamma is 0 only in the last step), so say you gave 10-step long episode, $E[\gamma] = 0.9 \gamma + 0.1 \cdot 0 = 0.9 \gamma$, thus the variance is $V[\gamma] = (0.9 - \gamma)^2 * 0.9 + (0.9 - 0)^2 * 0.1$


text:
Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?
For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to van Hasselt from his book, this discretization changes the problem into a partially observable MDP (POMDP), and this is understandable. And I know POMDPs require special treatment from the vanilla Q-learning we are used to (observation space, belief states, etc).
But my question is: is there a specific technical reason why a discretized-state problem (which is now POMDP) should be solved using POMDP algorithms, instead of plainly constructing a vanilla Q-table using the discretized states (i.e. the buckets from discretization)? In other words, is there a disadvantage in not using POMDP algorithms to tackle the discretized-state problem?

I don't see why would state discretization that you propose cause your MDP to be partially observable, do you mind clarifying your conclusion ?

When you discretize, several true states will be observed as the same. For instance, if we discretize all numbers from 0 to 0.1 as state 0, then 0.05 will be observed as state 0, and so will 0.097.

That's more of a problem of insufficient discretization than the method itself being inherently flawed. You can always refine your discretization until your policy is "constant" over your discretization tile, so that you don't have an issue where sparse discretization might be a problem.


text:
How is the padding mask incorporated in the attention formula?
I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention?
The attention formula taking into account a causal mask is:
$Attention(Q, K, V) = softmax(\frac{QK^{T} + CausalMask}{\sqrt{d_{k}}})V$
But how do we add the padding mask? The aim of a padding mask is to mask the padding positions as they're used just to make the batching feasible. But I don't know how this mask is added in the Attention formula.
Does it make sense if we do element-wise multiplication of the Attention matrix with a tensor of ones of shape (batch size, sequence length, $d_{model}$) and whatever sentence $s$ in that batch and wherever position $p$ is a padding token then the tensor[s, p, :] is zeros?
Thank you in advance for your help!

Entries of an attention mask are typically either $0$ or $-\infty$.
So, adding such a mask gives either the original entry of $QK^T$ or $-\infty$.
The issue with entrywise multiplication with a binary matrix, is that $0$ values still contribute to softmax.
$$softmax(z)_i = \frac{e^{z_i}}{\sum_k e^{z_k}}$$
$e^0$ is $1$, so element-wise product before softmax is not really a mask. After softmax doesn't work either, since the output is no longer a probability distribution. To ensure the masked elements do not contribute at all, you need them to be $-\infty$, which mask addition does.
If you know that certain indices of the input of are padded, that is, they are all zero vectors, then the padding tokens are already ignored by matmul (since they are zero). The issue is the rows of $QK^T$ corresponding to the pad tokens are zero and not $-\infty$.
To ignore pad tokens, you would create a mask with $-\infty$ along columns corresponding to the location of padding, and add this to $QK^T$ before softmax. So it's just the same attention formula, different value of $M$.

Thank you for your answer. You're totally right about the multiplication didn't give it much thought. But I have a question about this type of masking for the padding tokens. If we put $-\infty$ in rows corresponding to the pad tokens, the softmax will take in rows that are all $-\infty$ and will then output a row of nan values. I reckon that we can turn those nan values into $0$ but I wonder if it is how it's being done in deep learning frameworks like PyTorch etc.

@Daviiid In the case of TensorFlow, they mask for infinite values (floats with full exponent and empty mantissa) in implementing exp. The other frameworks probably do the same thing.

thank you for your answer. Can I ask you if that is equivalent mathematically to replacing those rows where we should have nan values, in a normal implementation, with 0? I'm trying to understand the effect of having a row full of $-\infty$ (the rows corresponding to padding tokens) on the multiplication of the values $V$, by the softmax, since having such rows will lead in a "normal" setting to an undefined value output from the softmax because of $0/0$

Well, you will only see a $0$ in the denominator if all entries are $-\infty$, not just some rows. That is, your entire sequence would have to be padding. In this case, the framework will return some NaN symbol, and the network evaluation will not work. I think the formula I used for softmax in the answer could be clearer. Edited.

Yeah you're right. Well, we do apply the softmax per row right? And we put $-\infty$ in the rows corresponding to pad token positions right? So we get a whole row of $-\infty$ that's fed to a softmax per row. Maybe this last part is wrong. If $Q$ is of shape (bsz, seq len, d_m) and the same for $K$, we get $QK^{T}$ of shape (bsz, seq len, seq len), will the padding mask, of shape (seq len, seq len) have $-\infty$ column-wise? E.g. if all sentences have a pad token at the end then will the last column of the padding mask be $-\infty$, or, will the last row be all $-\infty$?

Let us continue this discussion in chat.

For a given sequence x you calculate the attention scores using the formula:
\begin{equation}
A = \frac{(xQ K^Tx^T)}{\sqrt{d_k}},
\end{equation}
where $Q, K$ are the query and key matrices of the attention layer.
The result is a square matrix of size $T \times T$ where $T$ is the length of the sequence $x$. The entry $A_{i,j}$ gives the attention score between $x_{i}$ and $x_{j}$ (note $A_{i,j} \neq A_{j,i}$). So basically row $i$ gives you the attention scores for token $x_i$, which other tokens of the sequence it should attend to. However, you actually want to use these scores to perform a weighted average over the value encodings given by $xV$ ($V$ is the value matrix of the attention layer). But the scores can be arbitrary real numbers, and you want to have positive weights that sum to $1.$ That is why we apply a softmax layer to convert the scores into attention weights.
Now, if your sequence contains pad tokens you don't want $x_i$ to attend to them. So you want to "remove" the attention between $x_i$ and the pad tokens. You could set the attention weights directly to $0.$, but then the sum will not be $1.$ You actually want to mask the attention before applying the softmax function. Thus, you set the attention scores between $x_i$ and the pad tokens to a large negative number. Now applying the softmax will produce $0.$ attention weights. You can set the scores to -float("inf"), but I think setting them to $-1e9$ is more than enough.
For a concrete example see this github repo. Also here is an extensive blog post that I wrote about the Transformer, you might like it.

Sorry for the late reply. Thank you for your answer!


text:
What does $v(S_{t+1})$ mean in the optimal state-action value function?
In Sutton & Barto's Reinforcement Learning: An Introduction page 63 the authors introduce the optimal state value function in the expression of the optimal action-value function as follows: $q_{*}(s,a)=\mathbb{E}[R_{t+1}+\gamma v_{*}(S_{t+1})|S_{t}=s, A_{t}=a], \forall s \in S, \forall a \in A$.
I don't understand what $v_{*}(S_{t+1})$ could possibly mean since $v_{*}$ is a mapping, under the optimal policy $\pi_{*}$, from states to numbers which are expected returns starting from those states and at different time steps.
I believe that the authors use the same notation to denote the state-value function $v$ that verify $v(s)=\mathbb{E}[G_{t}|S_{t}=s], \forall s \in S$ and the random variable $\mathbb{E}[G_{t+1}|S_{t+1}]$ but I'm not sure.

I am not sure if it is standard notation, but Sutton & Barto use a convention that a function of a random variable is a new random variable that maps between values of the old variable to values of the new one using the function, and without affecting probability distribution (other than the function could be one-way hence probabilties may effectively combine e.g. if there were several states with $v_*(s) = 5$).
Given this convention then $v_*(S_{t+1})$ is a random variable of the optimal state value functions of possible statues at time step $t+1$. That is, it has the same probability densities, based on policy and state transition rules, as $S_{t+1}$, but has the associated values of $v_*(s)$ for each possible $S_{t+1}$.
The actual distribution of $v_{*}(S_{t+1})$ will vary depending on the conditions in the context where it is evaluated.
If you resolve the expectations in the first equation, which has conditions on $S_t$ and $A_t$:
$q_{*}(s,a)=\mathbb{E}[R_{t+1}+\gamma v_{*}(S_{t+1})|S_{t}=s, A_{t}=a]$
$\qquad\quad= \sum_{r,s'} p(r,s'|s,a)(r + \gamma v_*(s'))$
. . . which expresses action value $q_*(s,a)$ in terms of the state transition rules, immediate reward function and the state value $v_*(s')$ one half-step ahead. That is, at the next state, but before the next (optimal choice) action is taken.

I'm not sure if I understood well so let me please ask a question in general terms. If $X$ is a random variable from $\Omega$ to $\mathbb{R}$ and $f$ is a function on $\mathbb{R}$ then according to the convention, $f(X)$ is a random variable from $\Omega$ to $\mathbb{R}$ and the values of this random variable are $f(x)$ for $x$ in $\Omega$ but $P[f(X)=f(x)]=P[X=x]$ right ?

@Daviiid Yes I think that has correctly put my first two paragraphs into notation. Sort of, I think you have some typos, i.e. $X$ is a random variable in $\Omega$ (the "to $\mathbb{R}$" after that looks like a typo), and $f(x): \Omega \rightarrow \mathbb{R}$

Thank you for your answer. Can I ask please if there is a particular reason for going with this convention ? I think we can still get the same formula with the normal association of functions and random variables; $\mathbb{E}[v_{}(S_{t+1})|S_{t}=s,A_{t}=a]=\sum_{s'}{p(s'|s,a)v_{}(s')}=\sum_{r,s'}{p(r,s'|s,a)v_{*}(s')}$ since $\sum_{r}{p(r,s'|s,a)}=1$, or am I missing something ?

@Daviiid I cannot see any difference between what you have written and I have done in the question, in terms of needing to interpret $v_(S_{t+1})$. You have just resolved the expectation to the sum more conventionally (the double-variable sum $\sum_{r,s'}$ is a different conventon I have borrowed from Sutton & Barto). But I don't think it uses a different interpretation of what $v_(S_{t+1})$ is? Sorry I am not a notation expert, and may have missed some subtlety.

I'd like to apologize first for not being able to add your name with the @. There is no difference between what I've written and your answer I just wanted to rewrite it as a clarification to my question. I just wanted to know if there is a particular reason for the authors for choosing their convention for functions of random variables. Because normally a function of a random variable would change the distribution of probabilities, we just imagine a scalar multiplication of a Gaussian random variable and how the distribution changes.

@Daviiid: OK, I think I understand the concern. With discrete probability distributions then the problems with the convention are minor and do not impact any of the sums. For continuous distributions you have to be more careful, although I think most of the manipulations of the expectations etc in RL would still be fine.

I understand now. Thank you for the explanations and all your clarifications.


text:
Why my Fully Connected Neural Network outputs the same prediction?
I have a relatively small data set comprised of $3300$ data points where each data point is a $13$ dimensional vector where the $12$ first dimensions depict a "category" by taking the form of $[0,...,1,...,0]$ where $1$ is in the $i-th$ position for the $i-th$ category and the last dimension is an observation of a continuous variable, so typically one data point would be $[1,...,0,70.05]$.
I'm not trying to have something extremely accurate so I went with a Fully Connected Network with two hidden layers each comprising two neurons, the activation functions are ReLus, one neuron at the output layer because I'm trying to predict one value, and I didn't put any activation function for it. The optimizer is ADAM, the loss is the MSE while the metric is the RMSE.
I get this learning curve below:

Eventhough at the beginning the validation loss is lesser than the training loss (which I don't understand), I think at the end it show no sign of overfitting.
What I don't understand is why my Neural Network predicting the same value as long as the $13-th$ dimension takes values greater than $5$ and that value is $0.9747201$. If the $13-th$ dimension takes for example $4.9$ then the prediction would be $1.0005863$. I thought that it has something to do with the ReLu but even when I switched to Sigmoid, I have this "saturation" effect. The value is different but I still get the same value when I pass a certain threshold.
EDIT: I'd also like to add that I get this issue even with normalizing the 13th dimension (substracting the mean and dividing by the standard deviation).
I'd like to add that all the values in my training and validation set are at least greater than $50$ if that may help.

two hidden layers each comprising two neurons

From your description it looks like that you only have 6 parameters for your inner layer (2x2 weight matrix + 2 biases). The whole network should be easy to interpret: you've got two 13-dimensional weight vectors $\vec{w}_1,\vec{w}_2$ that are dot-multiplied with the inputs, plus two biases $b$ and activation $\sigma$:
$$ l_1 = \sigma\left(\vec{w}_1\vec{x} + b_1\right)$$
$$ l_2 = \sigma\left(\vec{w}_2\vec{x} + b_2\right)$$
Then these two values are multiplied by 2x2 matrix + biases, then activation and linear combination.
I'd look at how $l_1$ and $l_2$ are distributed. The fact that the outputs don't change is most likely due to the first layer getting saturated somehow. Look at the 13th dimension of $\vec{w}_i$ - it is likely to be large compared to other dimensions.
First thing I'd try - standardizing your input 13th dimension, so it is distributed closer to $[0,1]$ ( or $[-1,1]$ ) range.

Thank you for your answer. For $w_{1}$ the largest value is at the 4th dimension then the 13th one. For $w_{2}$ it is the same but we can't say that the value of 13th dimension is really greater than the others, sometimes it's just the double. I get this issue while I have standardized my 13th dimension input sorry for not precising it in my post, I'll edit it. Do you have any other idea for solving this issue ? I'd like to add that I get null outputs from the beginning. I've taken the initialized weights and computed the outputs of the first layer. I got $0$s after applying the ReLu.

By the way can I ask you please how is the loss decreasing with each epochs while the outputs from the first layer are from the beginning of the training null ? Since the biases are initialized to zero too, it means that all outputs are zero. No weights would be updated am I wrong ?

@Daviiid (For future: I'd suggest you avoid using the word "null" when talking bout zero values.) It seems that you are dealing with the "dying ReLU problem". Since you have only two neurons both of them likely get dead. In this case, the biases in 2nd layer and the output layer's linear coefficients would still able to train and converge to some average values, not accounting for your inputs.

you're right I shouldn't use "null" when talking about zero values. I don't understand quite well how the other layers' coefficients will train. From the first layer we get a $[l_{1} l_{2}]$, then if $W_{2}$ denotes the weights matrix for the second layer and $b_{2}$ the corresponding bias vector we get $W_{2}*[l_{1} l_{2}] + b_{2}$ but since the biases are initialized to zero and $[l_{1} l_{2}]=[0 0]$ we get zeros outputs through the second layer as well am I wrong ?

@Daviiid  The outputs of the second are going into the final layer, which, as I understand, is another linear function. So, you'll have two more non-zero initialized weights there. (If I understood your description correctly)

(I'd like to apologize first because I don't know how to fix the way your name isn't displayed after the @) The second layer contains 2 neurons, if we denote $W_{3}$ the matrix of weights for the final layer, $b_{3}$ the bias of the corresponding layer and $s_{1}, s_{2}$ the outputs of the second layer, we have for the final layer $W_{3}*[s_{1} s_{2}]+b_{3}$ but since $s_{1}, s_{2}$ are zeros we still get no update. Well I think so but I may be wrong.

@Daviiid You are right - if all the biases are 0, then you have to have 0 at the output. Are you sure your biases are 0 at the initialization? I'm afraid I'd have to look at your code to figure it out. I suggest you make a question focused on that.

Thank you for the advice, I shall take it and make a question focused on it right now.


text:
Closed networks vs Networks with a removed delay to predict new data
I've come across two types of neural networks to predict, both from Matlab, the closed structure and the net that removes one delay to find new data.
From Matlab's app generated scripts we see:

% Closed Loop Network
% Use this network to do multi-step prediction.
% The function CLOSELOOP replaces the feedback input with a direct
% connection from the output layer.

netc = closeloop(net);
netc.name = [net.name ' - Closed Loop'];
view(netc)
[xc,xic,aic,tc] = preparets(netc,{},{},T);
yc = netc(xc,xic,aic);
closedLoopPerformance = perform(net,tc,yc)

% Step-Ahead Prediction Network
% For some applications it helps to get the prediction a timestep early.
% The original network returns predicted y(t+1) at the same time it is
% given y(t+1). For some applications such as decision making, it would
% help to have predicted y(t+1) once y(t) is available, but before the
% actual y(t+1) occurs. The network can be made to return its output a
% timestep early by removing one delay so that its minimal tap delay is now
% 0 instead of 1. The new network returns the same outputs as the original
% network, but outputs are shifted left one timestep.

nets = removedelay(net);
nets.name = [net.name ' - Predict One Step Ahead'];
view(nets)
[xs,xis,ais,ts] = preparets(nets,{},{},T);
ys = nets(xs,xis,ais);
stepAheadPerformance = perform(nets,ts,ys)

My question is: What is the real difference between them?
Can one uses them equivalently? If yes, why? I mean, even tho the structure or how they are equipped, could be very very different, e.g. one is apple, the other is grape?
As far as I understand both can return new data if one codes them for that. For example, taking the closed net, one can predict 10 new values. Taking the net that removes one delay, one can predict one new value, but if one does this recursively 9 times, one can get the new 10 data. Is there a problem in using this last net in that way?
On another side, running both codes, as they are now (this changes depending on the code one works on), yields very different performances. Why?
Update:
I've checked this page https://www.mathworks.com/matlabcentral/answers/297187-neural-network-closed-loop-vs-open-loop, and in the answer by Greg Heath, we see
[...]

OPENLOOP: The desired output, AKA the delayed target, is used as an additional input. The OL net will produce output for the common time extent of the input and target.
CLOSELOOP: The delayed target input is replaced by a direct delayed output connection. The CL net will produce output for the time extent of the input.

[...]
"The desired output, AKA the delayed target, is used as an additional input." how is this?
"The OL net will produce output for the common time extent of the input and target." and this?
"The CL net will produce output for the time extent of the input." What does this mean?

Closed Loop Network
Step-Ahead Prediction Network

The function CLOSELOOP replaces the feedback input with a direct connection from the output layer.
Step-Ahead Prediction Network also known as removedelay function helps to remove delay to neural network’s response

In Closed loop networks, its own predictions become the feedback inputs.
targets with a delay were used as feedback input

Network used to do multi-step prediction
Network used to predict one step ahead

Highly helpful to turn the network into parallel configuration
Can't be used for parallel configuration

Output is not shifted one timestep
the output is shifted one timestep.

Closed loop network continue to predict when external feedback is missing, by using internal feedback
Open-loop and Remove-delay use external feedback

Real time Configuration
Not real time configuration

Closed loop network will produce output for the time extent of the input.
will produce output for the common time extent of the input and target.

Open-Loop Network
In open-loop networks, targets were used as feedback inputs. Open loops are primarily used when future outputs are not known. Below is how open loop network looks like
Note:

The typical workflow is to fully create the network in open loop, and only when it has been trained (which includes validation and testing steps) it is transformed to closed loop for multistep-ahead prediction

UPDATED Response to newly added questions

"The desired output, AKA the delayed target, is used as an additional input." how is this?

This is with reference to the Step-Ahead prediction network. Here, the delayed target is used as an additional input. Kindly refer to diagram in point #9. when the target is y(t+1) the additional input is the delayed target, i.e., y(t). Whereas in open-loop target is used as additional input this is the main difference between OL and step ahead prediction.

"The OL net will produce output for the common time extent of the input and target." and this?

As you can see, the open-loop network  will produce output y(t) as long as open-loop has two inputs x(t)and the targety(t)` for a certain time

"The CL net will produce output for the time extent of the input." What does this mean?

In closed loop based on the input x(t) the output y(t) will be produced for a certain time
Reference Link:

Official Documentation
Discussion on Closed Loop(multi step ahead prediction)
Multi-step ahead prediction using neural networks
Example of Closed loop, open-loop and remove delay
Multi-step Neural Network prediction
Difference between Open loop, Closed loop
Additional difference information

Some of the text here appears to have been copy-pasted from an external source, without attribution -- e.g., "continue to predict when external feedback is missing, by using internal feedback" appears to have been copied from https://www.mathworks.com/help/deeplearning/ug/multistep-neural-network-prediction.html or another source.  The images also appear to have been copied, e.g., from the same article, and https://www.mathworks.com/help/deeplearning/gs/neural-network-time-series-prediction-and-modeling.html, and so on.

Plagiarism is not cool.


text:
About the choice of the activation functions in the Multilayer Perceptron, and on what does this depends?
I've read in this: F. Rosenblatt, Principles of neurodynamics. perceptrons and the theory of brain mechanisms that in the Multilayer Perceptron the activation functions in the second, third, ..., are all non linear, and they all can be different. And in the first layer, they are all linear.
Why?
On what does this depends?

When it is said "the neural network learns automatically", in colloquial words, what does it mean?

AFAIK, one first train the NN, then at some point NN learns. When does the "automatically" enters then?
Thanks in advance for your help.

Hello Veronica. Could you please focus on 1 question at a time (so that people can focus on 1 problem at a time)? So, I would suggest that you remove the second question and ask it in a different/separate post. Please, try also to put your specific question in title to give immediately the idea of what your post is about to the readers.

@nbro Hello moderator nbro. Ok, but notice that nxglogic has answered both questions, if I edit my question, that would affect their answer, so?. I've modified the title to make it more specific.

Helpful -> https://ai.stackexchange.com/questions/7088/how-to-choose-an-activation-function and their respective answers.


text:
How much research, approximately, is done in ANNs?
Does someone know where can I find information about how much research, nowadays, is done in ANNs?
I've checked in this document Redes Neuronales: Conceptos básicos y aplicaciones, Universidad Tecnológica Nacional, México (2001) by D. J. Matich, that "nowadays research is uncountable" but that was in 2001.
I found nothing else on my further google search. Then, I've consulted Google Scholar, and by clicking on the option "Since 2021" it displayed 38,800 results, but, AFAIK, it includes a lot of different types of documents, e.g. books.

I've no idea but checking the proceedings of top AI/ML conferences such as NeurIPS, ICML, ICLR, AAAI, IJCAI etc. might be useful.


text:
How to choose an activation function for the hidden layers?
I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm dealing with probabilities, a ReLU when I'm dealing with positive values, and a linear function when I'm dealing with general values.
In hidden layers, I use a leaky ReLU to avoid dead neurons instead of the ReLU, and the tanh instead of the sigmoid. Of course, I don't use a linear function in hidden units.
However, the choice for them in the hidden layer is mostly due to trial and error.
Is there any rule of thumb of which activation function is likely to work well in some situations?
Take the term situations as general as possible: it could be referring to the depth of the layer, to the depth of the NN, to the number of neurons for that layer, to the optimizer that we chose, to the number of input features of that layer, to the application of this NN, etc.
The more activation functions I discover the more I'm confused in the choice of the function to use in hidden layers. I don't think that flipping a coin is a good way of choosing an activation function.

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU).
You may want to look at ELU (exponential linear units) and SELU (self-normalising version of ELU). Under some mild assumptions, the latter has the nice property of self-normalisation, which mitigates the problem of vanishing and exploding gradients. In addition, they propagate normalisation - i.e., they guarantee that the input to the next layer will have zero mean and unit variance.
However, it would be incredibly difficult to recommend an activation function (AF) that works for all use cases, although I believe that SELU was designed so that it would do the right thing with pretty much any input.
There are many considerations - how difficult it is to compute the derivative (if it is differentiable at all!), how quickly a NN with your chosen AF converges, how smooth it is, whether it satisfies the conditions of the universal approximation theorem, whether it preserves normalisation, and so on. You may or may not care about some or any of those.
The bottom line is that there is no universal rule for choosing an activation function for hidden layers. Personally, I like to use sigmoids (especially tanh) because they are nicely bounded and very fast to compute, but most importantly because they work for my use cases. Others recommend leaky ReLU for the input and hidden layers as a go-to function if your NN fails to learn. You can even mix and match activation functions to evolve NNs for fancy applications.
At the end of the day, you are probably going to get as many opinions as there are people about the right choice of activation function, so the short answer should probably be: start with the AF of the day (leaky ReLU / SELU?) and work your way through other AFs in order of decreasing popularity if your NN struggles to learn anything.

If my input is a length 4 vector containing only positive number, and my output is also a length 4 vector containing only positive number. Then, ReLU, leaky ReLU would always be a good choice, am I correct? not the tanh, sigmoid, softmax, etc.

***Take my answer as a side note to that given by cantordust:
If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems.
Thus, by testing activation functions on a few different problems, one can often infer how well (or badly) it will perform on most problems.
The following video shows how different activation functions perform in different problems:
https://www.youtube.com/watch?v=Hb3vIYUQ_I8
One can verify that an activation function usually perform well in all cases, or the other way around: it does it poorly in all cases.
As cantordust says, I would recommend always starting with leaky ReLU: it is simple, efficient, and generally produces nice results in a wide variety of problems.
It also evades the dying ReLU problem, and does not suffer from the vanishing gradient problem.
The only thing to keep in mind is the exploding gradient problem if the neural network is too deep, or if it is a recurrent neural network, which are essentially the same concept.
The video shows that other activation functions worth trying (in addition to leaky ReLU) are Gaussian, Sinusoid, or Tanh.

May I ask, what do you mean by  "It also evades the dying ReLU problem, and does not suffer from the vanishing gradient problem."?

leaky ReLUs take values different than zero for negative inputs. The derivative is different than zero for those values. Then, your gradient will not "vanish", and your gradient descent/ascent algorithm will be able to change the weight matrix in zones where ReLUs would be "dead" (because their derivative would be zero).

I don't know what kind of neural networks you are working on. But one should also consider tanh activation functions when dealing with recurrent neural network. The why is to avoid exploding gradient issues since the tanh function is bounded at the difference of the RELU function for instance.

In the question I stated that I use tanh and sigmoid, not only ReLU. Also, to keep it simple I'm referring in general to classical hidden fully connected layers. If you think that the fact that we are dealing with a recurrent neural network is significant for the choice of the activation function please state the reason for that. The exploding/vanishing phenomena could happens in non recurrent neural network as well.


text:
What is the correct notation for an operation that applies to each element of an array independently?
I am looking for the standard notation to define element-wise / Hadamard-style functions, if there is one.
That is to say, if the operator I am looking for were represented by a hexagon ⬡, I could use it as such:
$$A(x) = \underset{i}{\Large{⬡} } f(x_i)$$
$$A : \mathbb{R}^n \rightarrow \mathbb{R}^n$$
$$f : \mathbb{R} \rightarrow \mathbb{R}$$
It is very convenient to define such functions explicitly because I want to manipulate them: $B \circ A$ . It seems to me that the following notation is correct: $A_i(x) = f(x_i)$ but I worry it is nonstandard and confusing.
My functions are non-linear so I cannot simply apply them directly to the array as a vector.
As stated in an answer, this is unnecessary when a function is strictly scalar because it is implied to apply element-wise. There are still some situations I would hope to have it:
$$\underset{ij}{\Large{⬡} } e^{M_{ij}} \ne e^M$$
The answers suggest to me the best option would be something like these:
$$A(M) = \text{for each } i,j: e^{M_{ij}}$$
$$A(M) = \text{element-wise}: e^{M_{ij}}$$
The question is now closed in the negative, but I would welcome a new answer. Would be nice to find something like $\forall$.
Related:

https://math.stackexchange.com/questions/159005/notations-for-array-operations
https://math.stackexchange.com/questions/20412/element-wise-or-pointwise-operations-notation

Based on my experience, I would say that the standard notation is just to have a regular function, and specify that it applies element wise. For example, a common notation for activation functions is $\sigma$, so e.g. you could represent the activations of a regular dense layer as $\sigma(W x + b)$ where $x, b$ are vectors and $W$ is a matrix. I've never seen a special notation for specifying that the function $\sigma$ is applied element wise.
As you suggest in the question, if the function to be applied element wise is linear, then you can use either hadamard product, e.g. $a \circ x$ or the diag function $\text{diag}(a)x$.

Thank you, great answer, and this does match what I have seen in the literature: a scalar function applied to a vector is implicitly element-wise. Thinking about it, I can now see why there isn't an operation like the one I asked for. There are still some cases where it would be useful (updated question), so I'll leave the question open on the slim chance there has been a notation for it somewhere.

Thinking more sensibly, I think the "operator" is the just words "element wise".

The mathematical notation for complex tensorial expressions always tries to balance complexity and precision. More precise notation - the one that explicitly spells all the indices - becomes extremely convoluted very quickly. My favorite example illustrating it is from physics -- the Standard Model Lagrangian is written shortly on T-shirts and coffee mugs as:
$$ \mathcal{L} = - \frac{1}{4} F_{\mu \nu} F^{\mu \nu} + (i \bar{\psi} \hat{D} \psi + \bar{\psi}_i y_{ij} \psi_j \phi + h.c.) + |D_\mu \phi|^2 - V(\phi) $$
But if you try to expand all the indices in all the objects above - then it barely fits on a page.
On the other hand, more succinct notation always leads to ambiguities in interpretations. Your example $f(x_i)$ can be read as:
$$f(x_0, x_1, \dots, x_N)\quad \text{or as}\quad f(x_0),f(x_1), \dots, f(x_N)$$
One way to implicitly resolve this ambiguity is to show that the index $i$ "escapes" the argument brackets:
$$a_i = f(x_i)\quad \text{or e.g.} \quad \sum_if(x_i)$$
This can only be interpreted as $f(x_i)$ being element-wise. Also, at least in my opinion, using $x_i$ with index and $x$ without index in the same expression is extremely confusing.
And, of course, the best way to resolve these ambiguities is to state them explicitly. For example, I've seen authors using square brackets $f[x_i]$ or capital letters $F(x_i)$ the vector-argument functions.

Hi Kostya thank you for your answer. Sorry, I think my question may not be clear, f(x_i) is not a vector function, it is a simple scalar function. f: R->R ; a: R^n->R^n . Unless I miss something, I have to have a variable index and unindexed in the same expression, because I am relating behaviour an the variable to its components? If I just write a_i = f(x_i) it looks like x is a free variable.

I have edited my question to follow your advice regarding capital letters.


text:
What's the best way to take a list of lists as DQN input?
I have my own environment for the DQN algorithm. In my environment, the state space is represented by a list of lists, where each sublist can be of different lengths. In my case, the length of the global list is 300 and the length of each of the sublists varies from 0 to 10. What is the best way to use such state representation as a DQN input if I want to use the PyTorch platform?
#exapmle state with only 4 sublists and each sublist length can be highest 5
state = [[1,2,3,4], [1,20,20], [10], [20,4,5,6,7]]

I am thinking of using the raw data with zero(s) at the end of every sublist to make them all of the equal lengths.
state = [[1,2,3,4,0], [1,20,20,0,0], [10,0,0,0,0], [20,4,5,6,7]]

Then I can convert them to torch.tensor (and maybe flattened) and take that as input in DQN. However, I am wondering - is there a better approach?

Note that programming questions are off-topic. To me, this is a "conceptual implementation" question (i.e. it seems to me that you are looking for a general approach to solving this problem), so it's more or less fine, so I will leave it open here for a while.

@nbro, you are correct. That's exactly what I meant. I have slightly edited my question.


text:
Video Analysis: Providing a success score for a of a student carrying out a specific task
I have an AI/ML challenge in relation to video analysis and am unsure where to start.
I am investigating an application that will grade students performance of carrying out a task, based on analysis of a video of them carrying out the task.
The problem has not been sufficiently defined yet but to get a high level idea, imagine  video showing close up of a trainee doctor performing stitches to close a wound. the AI model would be trained using many videos of someone performing the stitches correctly and score the trainee on a number of criteria.
Most frameworks will allow detection of objects but taking a video of a person carrying out a task and assessing their success using an AI/ML model feels a step above regular object analysis.
Assumption is we will create the training material, having professionals video themselves carrying out the task successfully, which will also be graded by other professionals to provide a rubric of scores.
I understand this is not something that can be simply answered but an idea of where to start would be very helpful.

are there specific areas of AI i should investigate?
are there frameworks that can actually do this ( i have not found any)?

Appreciate any advice.
Thank you

You have not described exactly what the tasks will be, but there are some open source libraries for real time pose tracking. For example, OpenPose is one that can be configured to track the body, the hands and the face. However, this is only going to give you predicted pose information for each frame. If the subjects are meant to be doing specific tasks, e.g. picking up objects and moving them, OpenPose won't help and you may need to do further image recognition/analysis or object tracking in order to interpret what is going on.

Thanks for the reply. the problem has not been sufficiently defined yet but to get a high level idea, imagine  video showing close up of a trainee doctor performing stitches to close a wound. the AI model would be trained using many videos of someone performing the stitches correctly and score the trainee on a number of criteria.
Ive updated the original post with this information.


text:
One hot encoding vs dummy variables best practices for explainable AI (XAI)
When creating artificial columns for your categorical variables there are two mainstream methods you could use:
Disclaimer: For this example, I use the following definitions of dummy variables and one-hot-encoding. I'm aware both methods can be used to either return n or n-1 columns.
Dummy variables: each category is converted to it's own column and the value 0 or 1 indicates if that category is present for each record
one-hot-encoding: similar to dummy variables, but one column is dropped, as its value can be derived from the other columns. This is to prevent multicollinearity and the dummy variable trap.
As an arbitrary example, let's take people's favorite color: pink, blue and green. For a person who's favorite color is pink, the dummy and one-hot-encoded data would look as follows:
dummy variables

person_id
favorite_color_pink
favorite_color_blue
favorite_color_green

xyz
1
0
0

one-hot-encoded variables

person_id
favorite_color_blue
favorite_color_green

xyz
0
0

From a statistics point of view, I would use the one-hot encoded columns to build my model. In addition, I can infer the favorite color is pink, because I encoded the variables.
However, when I'm applying XAI to explain the prediction to someone else and they see the favorite color wasn't blue or green. I'm not so sure they will infer the favorite color was pink unless it's explicitly stated. So using dummy variables might serve explainability better, but brings other risks..
Are there any best practices on this?

Pasting an answer here from a colleague, credits go to Amelie Groud
"we encountered the same issue during our PoC, not only with encoding but also with normalization (seeing "age=0.18" wasn't super meaningful).
If you are using model-agnostic techniques (such as Lime, PDP or ANCHOR), you should be able to apply XAI on your original data, before applying the encoding.
With these techniques, usually we need 2 main elements: 1. an input dataset and 2. a "predict()" function (calling your trained model).
From there you have 2 possibilities:

use your already transformed data and the predict function of your trained model (which seems to be what you are describing)
or, use the original, unchanged data and have a custom predict function which first apply the transformation needed (encoding, normalization, etc.) and then call the predict() function of your trained model.

If you choose the 2nd option, then you are free to choose any encoding strategy you see fit. Plus, most XAI techniques have some embedded capacity to deal with categorical data so instead of seeing "pink=1, blue=0, green=0", you will see "color=pink".
Here is an example with LIME but it works similarly for other XAI techniques: https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html
with the predict function defined as predict_fn = lambda x: rf.predict_proba(encoder.transform(x))"
Please feel free to add more answers/views if you have a different way of dealing with the question at hand
Regards,
Koen

Personally I would chose one hot encoding as it is statistically correct/model friendly. Moreover, you can always provide additional help/tools to aid explainability. Lastly even if you add the nth column, you still need some idea about the working of model(and the boundaries it created while training) to interpret the result.


text:
CAPTCHA based on text comprehension and random tokens
I developed a novel type of CAPTCHA based on text comprehension and random tokens. Given a task Pick the first pair of adjacent letters and a random token 8NBA596V, the user has to provide the solution NB. It offers basic protection and an attacker can solve individual tasks with specific effort. I am curious, whether contemporary AI can solve it generically?
You can access more example tasks here:
https://www.topincs.com/manual/captcha
There is a task database and at every attempt a new task is presented with a new random token. They always have a solution of varying length and pure guessing thus has limited chances of success. It is easy to attack an individual task by writing a small piece of code, thus a large task database is essential. What intrigues me is the question whether natural language processing or machine learning at its current state can attack the CAPTCHA generically by building a model of the meaning of the task
– essentially a predicate in a tiny universe of discourse – and then applying it to the random token.

Have you tried feeding this task into any of the available GPT models to see how it answers? Such as GPT-2 or GPT-J-6B

I tried it on GPT-J-6B and it repeatedly failed this task. They can sometimes do previously unknown tasks just based on a description of the task, but apparently, not this one.

(Assuming English.) In your specific example, there would be $26^2$ combinations of capitalized letters. Also assuming a fixed-length token of eight, that gives you $26 * 26 * 7 = 4732$ possible combinations. My intuition is that the key space is too small.
It may actually be simpler for a machine to solve the CAPTCHA than that. Let's say that both your system and a generic one obscure text in an image with equal ability. The generic one is a string of length six composed of capital letters and numbers. So the attacker needs to make an informed guess from $(26 + 10)^6 = 2,176,782,336$ possibilities. That keyspace is $460,013$ times larger than your system with the longer string. It gets worse though.  An attacker could use a probabilistic approach, determining a "letter probability" for each of the eight positions, taking the two adjacent positions that maximize the probability, and then choosing the letter with the highest probability for each of those solutions. It would not be a guaranteed win for the attacker. But it would be guessing at a higher probability than $\frac{1}{4732}$.
Also, it may be easier for an ML system to distinguish certain letters from others. B/P may be confused more easily compared to M/T (I'm hypothesizing here). So the attacker's ML could be more confident in some letter pairs than others. This is analogous to the idea of weak keys in cryptography.
I like where your head is at. But this may not be the result you're looking for. In theory, you could have multiple rules beyond just "Pick the first pair of adjacent letters." But each new rule would add a negligible benefit on top of what is a fundamental problem in the design.

Sorry for being unclear. I will augment the original question with more detail.