text:
Training strategy on continuous video stream with CNN-LSTM
I have videos that are each about 30-40 mins long. With the first 5-10 mins (at 60fps, can be down-sampled to 5fps) are one type of activity that would be categorized by label-1 and the rest of the video as label-2. I started off by using CNN-LSTM to do this prediction (Resnet-50 + LSTM + FC-classifier).
Using Pytorch...
For training my initial approach was to treat this as an activity classification task. So, I split my videos into smaller segments with each segment having a label.
video1.mp4 (5 mins) --> label-1
Split into 30 seconds -->
video1_0001.mp4 --> label-1
.
.
.
video1_0010.mp4 --> label-1
But, with this strategy even after 100 epochs the network does not train. I can at the most fit about 40 frames on the 2-GPUs, but a 30 second segment of video @5fps has about 150 frames. Any further subsampling seems to not capture the essence of the video segment.
I also tried training without shuffling and with single thread, so a single stream is loaded continuously. But perhaps its not the right strategy.
I wanted to request some help on how to tackle the problem. I would really appreciate some insights into a training strategy for this problem.

Is using CNN-LSTM the right strategy here?

=== Update ===
After reading a few other posts on similar topics,

data stream for a LSTM
action recognition on video stream

I feel that to get the network to see a larger part of the sequence, I will have to use more GPUs or resize images. However, since the pretrained Resnet accepts 224x224, I will need more GPUs. But I am curious; is there another strategy? Because the question could also be about what is the ideal segment length that would enable the network to learn.
From my perception, a 30 second video sampled @5fps at the bare minimum captures the context. From observation so far, going below this number hasn't allowed the network to learn.

Using CNN into an LSTM is definitely a valid option for your task. There are also papers on integrating the LSTM mechanism directly into the convolution layers (like  RCNN), which would be an alternative to try out. As you already identified, these architectures require a lot of memory, because your classifier will depend on the full sequence of images and you have to store the gradient for each of them.
Ways to attenuate the problem with memory are:

Resizing the images
Choosing a smaller sequence length
Smaller networks
Preencoding the images (only works if the convolutional encoder isn't trained)
...

Smaller networks might be an option for you. ResNet50 ist quite large, so you might want to explore some smaller convolutional encoders like EfficientNet. Below is a figure plotting number of parameters vs ImageNet top-1 accuracy for multiple architectures. Complementary, you can save even more parameters by substituting the LSTM with a GRU. The number of parameters in the GRU scales quadratically with layer size, so reducing layer size of LSTM/GRU is worth experimenting with.
Another option when you are using pretrained models is to freeze the weights of the pretrained model and preencode the images images in your dataset. So first, apply your ConvNet to each frame of your video dataset and store the embedding. This gives you a dataset with just the embeddings on which you can then train the LSTM independently.
Having solved the problem, you may be able to experiment with longer sequences and experiment a bit to find out what works.
For more help you may want to include a bit more information about your data. It sounds like you are doing human action classification? If so you can explore paperswithcode a little to get inspiration.

Image Source

Thank you @Chillston, appreciate the insight. I agree with you regarding the options, Resizing the images - depends on the pretrained network.
Choosing a smaller sequence length - This option hasn't yielded much benefit.
Smaller networks - Will try it, I didn't think about it.
Preencoding the images - This option cross my mind, but I was not sure if it would provide much benefit. My image domain is not natural image (such as imagenet). Using a GRU - I will read up about it, I was focused on LSTMs because they are the state-of-the-art.

My domain is not exactly action recognition, I would like to apply them to medical images. Hence to continue training the backbone feature extractor.

Do you think Resnet + LSTM is able to capture the temporal context sufficienty? From the link you suggested (thanks again for that suggestion), I came across spatio-temporal Resnets that include an additional temporal context from optical flow. My original consideration was to use the output of Flownet as an input to the LSTM network for further classification. What do you think?

Regarding GRU vs. LSTM: GRUs can outperform LSTMs, depending on the data (e.g. this paper). If you need SOTA temporal processing performance, then the self-attention is the way to go (e.g. Transformer). A general piece of advice is to keep things as simple as possible until you have something that works :)
To give you my opinion on the Resnet + LSTM architecture, a few more details about the problem will be very helpful. What exactly is depicted in the images and whats their temporal relation?

Thanks. The data I am looking at are endoscopic images. And I am trying to determine the direction of motion in these videos.

In that case, Optical Flow estimation might be a nice keyword for you to look into. Such models output a vector for every pixel, indicating where it will be in the next frame.  Here is a model that uses self-attention for that :)
CNN+LSTM might work as well though. What might be a pitfall in this task is sampling rate. I would experiment with higher sampling rates ~20fps and rather reduce the number of images in the time series, so that the motion happens somewhat slower. What do you think?

Agreed, my problem has been with the sampling rate. There is no (trivial and) meaningful way to sample the videos. What you are suggesting is to keep the fps high and sparsely sample frames from this. Is that correct? The issue I foresee with this is that (in the worst case), the batch will end up with a bunch of blurred or water filled frames.

Regarding the backbone feature detector, I am wondering if using an optical flow might be an overkill, as optical flow has to find match for every pixel between frames. Perhaps I could get away with using a shallow CNN network (untrained). What do you think? Of course, I have to try it, but thinking out aloud. :)

I agree, your use case is the slimmed down version of the optical flow problem. So I think a CNN+LSTM or CNN+Self-Attention approach should generally work. However, what might be difficult is that visual features move in multiple directions from one image to the next, right? E.g. if you move the endoscope forward you get a radial zoom effect.
I'd actually try to increase sampling rate and sample subsequent frames (not skipping frames), because this increases the resolution of the movement, thus making it smoother.

In this case ResNet50 really seems like an overkill, but you have to find out empirically.
You may want to start with a really small network to get a baseline and see if performance increases when you add more layers. I'd start with only few CNN layers (3 to 5 maybe) - It will train much faster and provide you with a lower bound for performance. And from there it probably is a lot of trial and error to find a nice architecture :)
Is there any dataset on the internet that compares to your task?

So yes, I think this is a great idea :)

Unfortunately not, there isn't a dataset to try this out on :(. I will try the shallow network idea and let you know how it works.

Sure, I'm interested - I wish you much success with the project

:) thanks!!I have another question regarding training with lstm. During training, does it make sense to label every frame in the segment or have a single annotation for the segment. The motivation for this question is to understand how to consider the output from a lstm. In other words, should the output for the complete segment be considered and passed on to the loss or the loss from individual frames in the segment be accumulated and backpropagated? When I write it down the former seems obvious, but I would like to have a second opinion :). Thank you!

That's a good question. Without knowing how the videos look like, I would assume that only experiments will show what works best. I'd go for the approach where you have labels for each time step, so you don't have to worry about sequence chunks where both classes occur. Intuitively having a label for every time step should also yield a better gradient, but I might be wrong about that.


text:
When is it appropriate to use information like sex or race in ethical machine learning?
I'm a little confused on best practices regarding ethical ML. Specifically, I've seen in some courses that when building a model that affects people, it's helpful to have sensitive personal information related to sex, race, etc., to help ensure that the model distributes its error evenly across populations and avoids disparities.
I've also seen (in practice on the job and in recommendations) that this kind of information should be entirely omitted from your data to ensure the "blindness" of the model to protected and sensitive classes.
These directives seem at odds with each other. Has any kind of rubric been developed to help modelers determine when it is and is not appropriate to use this kind of information in a model?

Those suggestions are actually not in contradiction with each other.
Having data for testing and using them for training are two completely different things. Let's take the infamous example of Amazon automatic recruiting algorithm. The model was trained on real people curricula tp rank them and suggest this way the best candidates to hire. After deploying it  it became clear pretty soon that the model was gender biased, cause it was not ranking female curricula high even when all other requirements were met.
Why was the model biased? Because humans are also biased and we are those who labelled the data from which the model learn. So if a human recruiter compare the same curricula with a male name and a female name chances are it will select the curriculum with a male name.
So far so good, we spotted a potential cause for the bias, so we can just retrain the model without the gender variable and it should be fine. But can we trust the model will not be gender biased anymore? Well not 100%, there are many correlations between gender and other variables, and the model might still learn a biased behavior by "learning" the gender variable from other variables.
So that's why we want to keep the gender variable in our data, but only to use it in test phase, not as an extra feature to feed to the model (this time it was trained without) but as a variable to use to compute statistics out of the model predictions, i.e. how many males and females does the model choose after being trained without information about gender?

This is a great answer. Thank you! Good teachers make answers seem obvious.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Object Classification: How to decide which detected region is a RoI for classification?
I am working on a project where I am working on the Flickr-47 dataset to do logo detection and classification. My approach is to first finetune a YOLO v5 model with high recall to detect as many "logos" as it possibly can and then to classify the detected regions using some sort of feature extraction.
Here is an example of the actual image and what the model detects as logos.
Ground Truth Annotations:

YOLO detections:

My question is because there are instances as can be seen above where the number of regions  detected by the model can be less or more than the actual regions to be detected in the ground truth image, how do I decide which ones to keep for classification?
P.S. I know I can apply non-max suppression to remove overlaps but the above can still persist if the YOLO detector finds something worth a logo, but isn't. I can also modify the IoU threshold and confidence threshold during detection inference, but I want to be able to extract as much as the YOLO model detects, and keep the filtering at a later stage.
Any ideas?

I am not too familiar with YOLO but from having a brief look at it and the images you posted, they seem to be detecting letters (which is quite ironic because on StackOverflow this is what people usually have trouble with e.g. https://stackoverflow.com/questions/43003369/can-we-use-yolo-to-detect-and-recognize-text-in-a-image )
But anyway one thing I know the way in the past, from my own personal experience with object recognition, is usually an AI / Robot best detects things in terms of shapes, usually a Triangle,.Circle and square then cross-reference that within a dataset of similarities, e.g. four cylinders and a square = stool. You could do a similar thing here perhaps, if object / image contains circle alone, or square alone or triangle alone or combined with the latter 3 shapes then such and such % chance a logo. The issue would come when you have logos that are all letters. I can see its done similar in the images scoring less on the fonts and more on the shapes which makes some sense to what I am saying. If you have any more context to those images I could perhaps answer better in that respect. So basically I would score shapes higher like in the images,that are shapes, and text lower at first but then...
I am not sure if you can train YOLO but that would seem a logical way forward in terms of differing between shapes/logos and fonts, but then you would have to score higher on say bigger sized fonts would be a good way to detect text based logos. So I would train it until it has a good basis for sizing of fonts and shape differences overall.
I am probably oversimplifying it here, never used YOLO but I always start simple before scaling up and it never failed me in the past and before learning something new. Apologies if this is the not the answer you are looking for, I can't comment yet.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Best algorithm for the Word Ladder puzzle
What would be the best performing algorithm to solve the Word Ladder problem, in terms of guaranteed finding of the shortest solution in the shortest possible time? Is it BFS, DFS, A*, IDA* or another one? How are these algorithms compared with each other?
For the Word Ladder puzzle, a starting and an ending word of the same length are given. The goal is to convert the starting word to the ending word by changing only on letter at a time. Intermediate words should be valid words. E.g. silly => sully => sulky => hulky => hunky => funky => funny

What do you mean by "best performing"? Performing in terms of what?

Best performing in terms of guaranteed finding of the shortest solution in the shortest possible time

Ok. Edit your post to include this info! What do you know about this problem and algorithms? For example, can DFS be applied to this problem?

Ok, the question is edited. I know that BFS and A* can be applied, probably also DFS, but I don't know which one will give a guaranteed optimal solution.


text:
What are the advantages and disadvantages of higher order neuron activation functions?
I've been reading about different types of neurons that the traditional linear one. One example I came across is the Sigma-Pi neuron, where the activation function includes higher order terms, such as
$f$" />
where the activation has the additional higher order term on the right, as well as the standard term on the left. What are the advantages and disadvantages of using neurons like that?  Why do we tend not to use them?

Can you please 1. put your specific question in the title, 2. format your question with mathjax, 3. provide a link to a reference that talks about this "sigma-pi neuron" (for people that are not familiar with it)?

My question isn't specific to the sigma-pi neural network, but here is a link to that anyway: https://repository.up.ac.za/bitstream/handle/2263/29715/03chapter3.pdf?sequence=4
The question at heart is fairly straightforward: why are we using linear activations of the form wX+c and not higher order activations? What advantages and disadvantages are there?


text:
How to define a custom layer in Pytorch
I am new to PyTorch and seeking your help regarding a problem I have. I need to add a costume layer to a NN in training phase. Please see the figure which shows a simple DNN with the custom layer. NN is trying to solve a regression problem.
Let's consider the simplest case, $f_1(input)=f_2(input)=f_3(input)=sin(a\times input)$, where $a$ is constant. How can I handle such an NN in PyTorch.
Also, can we treat $a$ as a trainable parameter as well if we don't assume it as a predefined constant?
I appreciate any hint or similar example using PyTorch.

This site is more heavily focussed on theory than related Stack Exchange sites. As you are asking for Python code using a specific framework, then you may have better luck in Data Science stack exchange or even original Stack Overflow (if you show some code so far without the custom layer implemented). Your diagram suggests this is not a full custom layer, but a custom activation function (either combined with or instead of the activation layer from the previous layer). It is worth clarifying that, separately of where you ask.


text:
Does a colour consistency loss in neural networks (cycleGAN) make sense?
My neural network takes an image as an input and outputs another image. It's the generator of a cycleGAN.
I would like to add (to the discriminator loss, the cycle consistency loss and the identity loss) a colour consistency loss i.e. i want the output image to globally have the same colours than the input image.
Why? My problem is that the network "tints" my images too much in my opinion (a standard example is the following: my dataset has women that wear pink clothes more than men do and when i transform men into women, the GAN also transforms the clothes, and sort of adds a pink tint to the whole image). I know that adding this loss will encourage the network to "not change anything" though as any change would be a change in colours. Unless my loss looks at the averages of red, blue and green instead of looking at them pixel by pixel, which is what I'd like to go for.
Not the main question but any thoughts on that are appreciated:
any idea about how to implement it in Pytorch, or if it's already been implemented? Searches for colour consistency on GANs just give me GANs that take black and white photos and add colour to them.

Can you please put your specific question in the title? "Colour consistency loss neural networks" is not a question. If you're asking for code, that would be off-topic here.

I'm asking if it makes sense and if there are some references of people that did it :) I edited my post thanks, sorry for not posting it correctly!

Ok, so you should edit your post to clarify that "Any idea about how to implement it in Pytorch, or if it's already been implemented?" is not your main question.

Done! In the case that I'd want help on that specifically, stackoverflow is the stackexchange I should go to?

Yes, I think Stack Overflow would be a better option for programming questions.

The identity loss should already fulfill for what you're asking for, which means that if the problem is still there even with a strong weight for it then you should consider improving/cleaning up your data to remove the bias you observe.
If you really don't like this idea, something close to what you're asking for are histogram based losses, but be aware that they are mostly used in neural style transfer, so you will probably have to remove the identity loss to avoid conflicting losses and I wouldn't expect them to work as you think in preventing color biases, since they come from the data.


text:
Doesn't the n-step Tree Backup algorithm negatively affect the DQN-Agent by creating inconsistent look-ahead targets?
In the text book of Sutton and Barto on page 152 they introduce the n-step Tree Backup algorithm, where the tree-backup n-step return is defined via
$$
G_{t:t+n} = R_{t+1} + \gamma \sum_{a \neq A_{t+1}} \pi(a | S_{t+1})Q_{t+n-1}(S_{t+1}, a) + \gamma \pi(A_{t+1}| S_{t+1})G_{t+1, t+n}.
$$
Assume we have a sample of the replay buffer and its consecutive
transitions $$ (S_{t}, A_{t}, R_{t+1}, S_{t+1}), (S_{t+1}, A_{t+1},
R_{t+2}, S_{t+2}), ..., (S_{t+n-1}, A_{t+n-1}, R_{t+n}, S_{t+n}). $$
Since in Deep Q-Learning, the target policy $\pi$ is usually the greedy-policy, we have
$$ \pi(a|S_{t+1}) = \begin{cases} 1, & \text{if }  a= argmax_{a}Q(a,
S_{t+1}) \\ 0, & \text{otherwise}. \end{cases} $$
Hence, I think we
would in general look only up to n-steps ahead rather than
always n-steps, since the n consecutive samples from the replay
buffer don't guarantee that the transitions happended according to
the current policy $\pi$. Doesn't this negatively affect the agent's behavior since sometimes the target is made up of only one timestep and sometimes of several?
Thanks in advance for any help!

Could you please put your main question in the title? If that doesn't seem possible (or it becomes "questions about N-step Tree Backup"), it may be a clue that you should split the question up into simpler components. Generally I think that well-presented questions about RL theory are on-topic and appreciated here, and N-step tree backup doesn't have many questions so far, so it won't matter to the site if you post two or three on a similar theme.

@NeilSlater I updated the question. Thanks for pointing this out.

Hence, I think we
would in general look only up to n-steps ahead rather than
always n-steps, since the n consecutive samples from the replay
buffer don't guarantee that the transitions happended according to
the current policy $\pi$.

This is correct in general in n-step off-policy learners: Whenever exploration chooses an action that the target policy has $0$ probability of selecting, then there is nothing to learn about the target policy from that specific choice. The best you can do is take some other estimate - in TD learning that will be a bootstrap estimate taken earlier than n steps for some state-action estimates. You don't have to do this, but a bootstrap estimate is often better than throwing away the sampled experience (in off-policy Monte Carlo control, you would indeed throw away that earlier experience, and the method can still work in practice though).
It is possible to learn from time steps later than exploration, you just re-start from the new $s, a$ pair (and new count up to n). If this changes the estimate of $Q(s,a)$ so that this exploratory action is better in state $s$, then the target policy will change and what was once exploratory will become part of the target policy.
There is no way around this in off-policy value estimators.

Doesn't this negatively affect the agent's behavior since sometimes the target is made up of only one timestep and sometimes of several?

"Negatively affect" with respect to what?
The  main reason for using n-step learning is to find a sweet spot between the bias inherent in bootstrap estimates - worse in TD(0) - and the variance inherent to fully sampled returns from long trajectories - worse in Monte Carlo or TD(1). This reason still applies when the steps that are full samples of experience can vary between $1$ and $n$. There will usually be a value of max $n$ (combined with specific exploration probabilities) between 1 and the full episode length that works better than either of those extremes.
It is worth noting that in the DQN "rainbow" paper, the authors add n-step returns, ignoring the theory regarding truncating with an early bootstrap. They simply store the discounted sum of rewards $r_{t+1}$ to $r_{t+n}$, plus $s_{t+n}$ for e.g. n=4 steps ahead in the experience replay table, regardless of whether the agent took an exploratory action (in addtion, any of the actions might become exploratory with respect to the target policy later because the experience replay table is effectively an offline dataset). In practice this still works better than single-step updates despite the break from theory, at least for relatively low $n$ and $\epsilon$, and the kinds of environments that DQN was developed against (Atari games).


text:
What is MLM & NSP loss function
Two objective functions are used during the BERT language
model pretraining step.
The first one is masked language
model (MLM) that randomly masks
15% of the
input tokens and the objective is to predict the vocabulary
The second objective
is the next sentence prediction (NSP) task. This is a binary
classification task for predicting whether two sentences are
subsequent in the original text.
I am looking for such objective function as mathematical definition.

Cross-posted: https://ai.stackexchange.com/q/40590/1794, https://cs.stackexchange.com/q/160345/755.  Please do not post the same question on multiple sites.


text:
How to increase accuracy for CNN?
I have built one CNN model and applied it to chest-xray Covid 19 pneumonia dataset. I am getting the classification report as follows:

I am surprised to see that it is giving an excellent result on minority class but relatively poor on class 1. Can anybody help me tune my model so that I can build a more robust model?
CNN code
def architecture (_img_dims ):

    inputs = Input(shape=(_img_dims, _img_dims, 3))

    x = SeparableConv2D(filters=64, kernel_size=(3, 3), activation='relu', padding='same')(x)
    x = SeparableConv2D(filters=64, kernel_size=(3, 3), activation='relu', padding='same')(x)
    x = BatchNormalization()(x)
    x = MaxPool2D(pool_size=(2, 2))(x)

    x = SeparableConv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='same')(x)
    x = SeparableConv2D(filters=128, kernel_size=(3, 3), activation='relu', padding='same')(x)
    x = BatchNormalization()(x)
    x = MaxPool2D(pool_size=(2, 2))(x)
    x = Dropout(rate=0.2)(x)

    x = SeparableConv2D(filters=256, kernel_size=(3, 3), activation='relu', padding='same')(x)
    x = SeparableConv2D(filters=256, kernel_size=(3, 3), activation='relu', padding='same')(x)
    x = BatchNormalization()(x)
    x = MaxPool2D(pool_size=(2, 2))(x)
    x = Dropout(rate=0.2)(x)

    x = Flatten()(x)
    x = Dense(units=512, activation='relu')(x)
    x = Dropout(rate=0.7)(x)
    x = Dense(units=128, activation='relu')(x)
    x = Dropout(rate=0.5)(x)
    x = Dense(units=64, activation='relu')(x)
    x = Dropout(rate=0.3)(x)

    # Output layer
    output = Dense(units=3, activation='softmax')(x)

    return inputs, output

INFO ABOUT CLASSES

Class 0:  COVID
Class 1:  Normal
Class 2:  Pneumonia

Image SAMPLE

I have also used Data-Augmentation. Still the result is same
train_datagen = ImageDataGenerator(
        rescale=1./255,
        rotation_range=10,
        zoom_range=0.4,
        horizontal_flip=True,
        validation_split=0.01
        )

Edit
I am trying to increase, the number of class-level 1 images i.e Normal Images by using AC-GAN. Code is taken from here
from math import sqrt
from numpy import asarray
from numpy.random import randn
from keras.models import load_model
from matplotlib import pyplot

# generate points in latent space as input for the generator
def generate_latent_points(latent_dim, n_samples, n_class):
    # generate points in the latent space
    x_input = randn(latent_dim * n_samples)
    # reshape into a batch of inputs for the network
    z_input = x_input.reshape(n_samples, latent_dim)
    # generate labels
    labels = asarray([n_class for _ in range(n_samples)])
    return [z_input, labels]

# create and save a plot of generated images
def save_plot(examples, n_examples):
    # plot images
    for i in range(n_examples):
        # define subplot
        pyplot.subplot(sqrt(n_examples), sqrt(n_examples), 1 + i)
        # turn off axis
        pyplot.axis('off')
        # plot raw pixel data
        pyplot.imshow(examples[i, :, :, 0], cmap='gray_r')
    pyplot.show()

# load model
model = load_model('../input/model111/Xception.h5')
latent_dim = 224
n_examples = 100 # must be a square
n_class = 1 # Normal
# generate images
latent_points, labels = generate_latent_points(latent_dim, n_examples, n_class)
# generate images
X  = model.predict([latent_points, labels])
# scale from [-1,1] to [0,1]
X = (X + 1) / 2.0
# plot the result
save_plot(X, n_examples)

But I am getting error as
ValueError: Layer model expects 1 input(s), but it received 2 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(None, 224) dtype=float32>, <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=int64>]

could you describe more in details what the class represent? (maybe add a couple of images as well). There can be many reasons why class 0 is detected better than class 1 even with less examples, e.g. if class 0 were bananas, class 1 oranges and class 2 mandarins, the color would suffice to get that class perfectly with few examples. So the CNN code is kinda useless here, more important is to inspect the data.

Hello @EdoardoGuerriero , I have edited the question.


text:
Is it possible to train an AI to bring a picture story in the correct order (correct story flow)?
I want to know if it is possible to train a neural network (or some other kind of an AI) to bring a simple picture story in the correct order, if it is in random order, so that the story has the correct story flow.
For example, this simple picture story:

or this one

So, imagine the pictures of these stories are in a random order and the AI has to put them in the order that the correct story is told.
Most 8 year olds would be able to do that. So, can an AI learn it? How would an approach look like? Does anyone know if something like that has been achieved or even tried?
From my research so far, the approach would be first to translate the images into descriptive sentences and then try to order them in a meaningful way. But I will do further research, I found so far this paper: Sort Story: Sorting Jumbled Images and Captions into Stories (2016).
To clarify, this is not a "real problem" for me, I just asked from a philosophical standpoint and from interest. I will not attempt to solve it, because I think if it is possible it would be extremely difficult.
Edit:
I just tested it with ChatGPT-4o it was not even close to solve it. So it seems to be a really good test for AI...

I just was interested if an AI are already that advanced or not. I found related topics, but for continuous images like movements in a image stream like a video. This is no external source it's just a random example for a picture story. I mean a picture storys for childs. For example if the six pictures in my example where in random order, if an ai can bring them in the correct "story flow". So it has to recognize whats the meaning behind each picture and connect (order them) in a meaningful way. Another exmaple: https://i.pinimg.com/474x/c2/41/11/c24111650800563c899c4efd4269ccdd.jpg

This is a really hard problem for statistical AI such as a neural network. The difficulty level is due to lack of grounding and common sense in what a neural network can process.
A neural network could feasibly label all objects in the example scenes, and even do pose estimation, detect activities and guess emotional state for the "actors". It can even create a vector representing the content of the image and convert to/from a caption for the image.
However, so far any structure or embedding that neural networks have produced is not amenable to reasoning, or common sense. Such embeddings can be translated into other representations, but lack "grounding" in the sense of a deeper understanding based on a more general model of the world. It is this understanding - e.g. a parent will be stressed if they think a child is missing (panel 2 from second example), and may then act to find them (panel 4) - that is missing and it is not at all clear how such a world model could be added to neural network training.
There are some neural network models that come close in different ways:

Large language models. Descriptive text often has a narrative structure, and language models like GPT-3 can easily produce stories as sophisticated as the example panels. In theory such a model could be used to analyse the likelihood of different series of static descriptions extracted by an image captioning system, and identify the highest likelihood story based on trying all combinations. I do not know if this has been attempted.

Video activity prediction. In the simpler world of immediate actions and consequences (as opposed to understanding inner state, motivation and narrative), predicting what happens next using a neural network is already possible. These predictions would be short term - In the first panel of the first example for example, a neural network might predict that the fish will go into the bucket. That doesn't mean the neural network models what a fish or a bucket actually are in enough detail to reason about this further, nor that it could extrapolate to the excited child looking at the fish in the bucket on panel 2.

Thanks for this great answer! I think the point is common sense. I also though if you have enough training data it could be trained to guess the likelihood of emotion transitions based on various factors or something. But in fact you would actually train some kind of common sense this way I think.


text:
What would be a good cost function based on both saliency-maps and labels?
I have a number of input samples where: every input sample has both a label and a reference-map. This reference-map gives a score to each location of an input sample. The score defines how much this location of the input sample SHOULD contribute to the model’s decision making w.r.t. the correct label.
A gradient based saliency-map defines how much a location of an input sample ACTUALLY contributes to a model’s decision-making with regard to the correct label.
(Saliency-map: https://arxiv.org/pdf/1312.6034.pdf)
I would like introduce a penalty (based on the difference of the saliency-map and the reference-map) if the model does not focus on the areas that should be used to infer the label.
Hence there are two terms involved here: 1. A penalty based on the difference between the inferred-label and the actual-label (normal approach in deep-learning), 2: difference in saliency-map and reference-map.
I know how to calculate the saliency map, the question is more on how to construct an effective cost/loss function based on both the saliency-map as well as the inferred-label. (Currently I am using the categorical crossentropy without any extra penalty-term)
Does anyone if there has been any research done in this area (cost-function based on both saliency map and label), or some paper suggestions related to this approach?

Can you please describe more in detail what these saliency maps are doing or are? And what do you mean by "calculate the salience map every epoch while the model is being trained"? Isn't this the default behaviour? What loss function are you currently using? What task are you trying to solve? Edit your post to clarify all these points.

@nbro I have edited the question according to your request.

You can use Dice loss between the saliency map and the reference map. You can then add this as a penalty to your regular loss with a weight parameter: CrossEntropyLoss + weight * DiceLoss


text:
What can we learn from AlphaZero in the development towards AGI?
According to DeepMind,

AlphaZero's creative insights coupled with the encouraging results we see in other projects such as AlphaFold,  give us confidence in our mission to create general purpose learning systems

"our mission" being:

[...] to solve intelligence, developing more general and capable problem-solving systems, known as artificial general intelligence (AGI).

What exactly can we learn from AlphaZero that is so important in the development of AGI?
I asked this to a postdoctoral researcher at the Reinforcement Learning Group, Leiden University in The Netherlands, and he found this statement way too optimistic. He thinks there is still a big hole between solving games with AlphaZero and actual artificial general intelligence.
What indication is there that AlphaZero is a big step towards AGI?

General purpose learning is not the same as AGI.  Training AlphaZero on a specific task, it will generally perform well... but you are still producing a narrow AI, not an AGI.

Please, do not ask "What is your opinion?". We have a close reason that specifically says "opinion-based", i.e. questions that ask for opinions are not suitable for our site. Rephrase your question to ask for facts or evidence.

Your question might be a duplicate of this. Maybe not an exact duplicate, but it should partially answer your question. I think your question is slightly different and maybe you can clarify that because it seems that you're implicitly asking "which components or properties of AlphaZero would be useful towards the development of AGI?", which is not the same thing as asking "Is AlphaZero an AGI?". But maybe it's me just extending what was your actual/different question, which might be a duplicate.

@nbro Thanks, I clarified the question. I saw that thread, but it didn't give me the info I was looking for.

Some learnings from AlphaZero:

Self-play, and more generally sandbox training, is effective. This indicates that given the right enviroment and enough computational power we can build highly effective agents.

Human input is not required, the learnings comes purely from the rules of the game.

A bit weaker, but fundamentally any problem can be phrased as search problem (see Levin’s universal search), so improving on search algorithms, as done in AlphaZero, has direct impact on how smart the agents will be.

Regarding human level AGI, the limitations with 1 and 2 are that:

Interesting environments are far more complex than Go, resulting in a phase space explosion so that self-play becomes less effective.
Interesting problems involve and affect humans, therefore humans in the loop are needed. But experimenting with humans is slow, hence it will be harder to build an useful AGI only following the self-play strategy.


text:
How do I use A.I. to analyse & score news articles?
I'm working on a project that would benefit from using A.I. or machine learning to analyse news feeds from a variety of websites and grade each article between 0 and 10. We would manually grade hundreds of articles to train the A.I. on what we like and what we don't like using the scoring range. The A.I. is expected to learn how we grade by identifying similarities between articles. When the A.I. starts to grade similar to humans, then we would go more hands of and leave this task to the A.I.
Not sure where to start with A.I. what tools and approaches would be the easiest to achieve this?

From comments on one of the answers, it appears that you may already have some training data - a bunch of news articles and some manually assigned scores (based on human experts?). It is worth using [edit] to add that detail - if you could also indicate the approximate size of your data (hundreds? thousands? millions? of rated articles) set that would be useful to someone suggesting an answer.

You first need to be clear about what constitutes a 'good' or a 'bad' article. AI is not a magic tool that does that for you. Once you have a clear idea, work out how to tag an article, by doing it by hand a couple of times. That should give you some idea of what to look for. In the simplest case these might be certain key words, or even grammatical structures.
Then you need to work on how to recognise these features automatically, essentially developing a scoring function. You use that to calculate a score for each article.

Good & bad is determined by the score. It would be nice to teach the AI what we consider good & bad by showing articles and grading them between 0-10. The AI would then need to analyse the similarities in what we grade to predict what we would grade new articles. The idea is to be hands-off and allow the AI to manage the grading task in the future. How do I achieve this?

That is a circular definition: "the score for 'bad' is determined by the score"?! What is it that you "consider good & bad"? Where do the scores come from?

I'm not sure I understand. It's just like scoring anything else on a scale. 0 is extremely bad and 10 is extremely good and 5 average. The point is to manually score hundreds of articles and have the A.I. learn what we like and don't like based on the scores we manually give between 0-10. It should be able to analyze and identify what topics, writing style and other factors to consider to be able to score like the humans. when it is close to this point we wound then leave this task to the A.I.

I have looked at film rating systems in the past if you are trying to score between a set number these would be a good thing to look at. A simplified version without machine learning can be done utilising mathematic ratio comparison. I would google "comparing ratios python" and try and do this first with your news articles based on the scoring. Then from there should give you a good logical basis to scale it up into machine learning, this is what I have done many years ago but struggling to find my code (stupidly didn't use git back then) I know its not exactly a machine learning solution, but its a solid foundation that does not have too much complexity.

Thanks, I will consider this approach and get back to you!

Cool, yeah let me know how it goes. I looked hard for my code I don't know what happened to it, it was not an easy thing to do I remember that.

You'll need more than just a few hundred news articles if you intend to train the AI from scratch. Training a model to understand text requires a substantial amount of data. However, you can use these articles to finetune existing embedding models. An embedding model converts a paragraph into a vector that encapsulates its meaning. You can then add a small model at the end of this embedding model to produce a score between 1 and 10. Afterward, simply fine-tune both models.

It's great that you're exploring the use of A.I. for analyzing and scoring news articles in your project. To begin with, you can start by looking into natural language processing (NLP) tools and machine learning frameworks, which are commonly used for tasks like sentiment analysis and article grading.
Here's a simple roadmap to get you started:
Data Collection:
Gather a diverse dataset of news articles that you'll use to train your A.I. model. Make sure it covers a wide range of topics and writing styles.
Preprocessing:
Clean and preprocess your data. This involves tasks like removing irrelevant information, handling missing data, and converting text into a format suitable for analysis.
Feature Extraction:
Identify relevant features in the articles that your A.I. can use for grading. This might include word frequency, sentiment, or key topics.
Choosing a Framework:
Select a machine learning framework that suits your needs. Popular ones include TensorFlow and PyTorch. Additionally, pre-built tools like spaCy and NLTK can be useful for NLP tasks.
Model Training:
Train your model using the manually graded articles as a training set. This will help the A.I. learn the patterns and preferences you've identified.
Evaluation:
Assess the performance of your model using a separate set of articles not used during training. Adjust your model based on the results.
Iterative Improvement:
Continue refining your A.I. by iteratively improving the model based on feedback and additional training data.
Regarding Impressico Business Solution, you might want to explore if their expertise aligns with your project goals. Seeking advice or collaboration from experienced professionals can provide valuable insights during your A.I. implementation journey.
Remember to take it step by step, and don't hesitate to seek guidance from the community or relevant forums as you encounter challenges. Good luck with your A.I. project!

This answer has a very AI-generated vibe, I thought those were forbidden?

There are two main relevant terms from data science and AI methods that apply in your case:

Natural Language Processing (NLP). This is a general description of tasks that involve processing inputs that use human language, such as written text. In order to rate the text of your news articles, you will need to use some NLP techniques so that the language in them is converted to forms that an AI can process.

Supervised machine learning. This is a technique that can be used to learn an approximation of an unknown function by processing examples of known inputs and outputs. In your case, you have a function that can take a (maybe pre-processed using NLP tools) news article and output the rating that your group would give it.

There are a wide range of approaches possible using these two things. You will want to search these terms and read around the subjects. Expect to spend weeks to months learning the basics before attempting your project in full.
As a starting point to get a taste for what is involved, I recommend you look into sentiment analysis. Sentiment analysis is used to label text as having certain emotions, or simply as being "positive" ot "negative", and is usually applied to shorter pieces of text than a news article. This makes it a simpler problem that the one you are attempt, and maybe a useful introduction. Here is a step-by-step tutorial for sentiment analysis of Twitter feeds.
Another similar example might be to classify emails as spam vs not spam. Emails are often longer than tweets, so slightly different approaches may be used to help summarise the content before attempting to classify. Here is a step-by-step tutorial for spam classifiers.
Unfortunately, a few hundred examples of human-rated articles is probably not very much data, in case you are expecting anything sophisticated, such as exploring the quality of journalism. It may be enough to rate interest in different subjects. But the only way to find out for sure is to start your project. Do take the time to research a few simple examples and tutorials first, before you dive straight into your own work. This will help you structure and plan your work plus avoid common pitfalls.


text:
Determine Gridworld values
I am learning Reinforcement learning for games following Gridworld examples. Apologies in advance if this is a basic question, very new to reinforcement learning.
I am slightly confused in scenarios where probability of moving up, down, left and right are not provided or stated. In this scenario, I assume we assume the optimal policy and therefore, you would apply the Bellman equation as:
$V(s) = max_a(R(s,a)+\gamma V(s'))$
Cost for any movement is 0 and an agent can choose to terminate at a numbered grid to collect a reward amount of the grid number. This is why my square closest to the reward takes in the value 8 since it will terminate with the action to the next state to collect the reward.
Would this be the correct way to determine the value for the surrounding grid squares?

Could you please put your specific question in the title? "Determine Gridworld values" is not a question and it's also not specific. Thank you.

Why is "In this scenario, I assume we assume the optimal policy" true? Why would you assume the optimal policy you don't know the transition dynamics? That doesn't make sense to me. Anyway, are you asking if that equation is the right one to update the values?


text:
Denoise autoencoder not training properly
I'm trying to make a denoise autoencoder wherein the encoder part is vgg16 and decoder is opposite of vgg16(encoder) network. My dataset consists of 5K images in grayscale.
Now while training, the loss and accuracy doesn't changes. I can think of reducing filters in the initial decoder layers but i fear that's going to affect the autoencoder. Here, i'm really clueless about what approach to follow.

VGG16 is not trained to be used as an encoder for image reconstruction, it is trained to extract features from an image using which we can do classification task on the image.
This is why, you cannot use VGG16 as the encoder part of your denoise autoencoder.
However, if you want, you can use that architecture of VGG16 as the encoder part of your autoencoder, by just retraining those layers of VGG16,
vggmodel = keras.applications.vgg16.VGG16()
model_encoder = Sequential()
num = 0
for i, layer in enumerate(vggmodel.layers):
    if i<19:
      model_encoder.add(layer)

model_encoder.summary()
for layer in model_encoder.layers:
  layer.trainable=True # Set encoder to trainable, and your autoencoder should work.

Again, there is a problem with your current architecture of autoencoder -it downsamples too much, leading to a significant loss of data. A smaller architecture would work better. For example:
model = Sequential()
model.add(Input(shape=(224,224,3)))
model.add(Conv2D(64, (3,3), activation='relu', padding='same'))
model.add(Conv2D(64, (3,3), activation='relu', padding='same'))
model.add(MaxPool2D((2,2)))
model.add(Conv2D(128, (3,3), activation='relu', padding='same'))
model.add(Conv2D(128, (3,3), activation='relu', padding='same'))
model.add(MaxPool2D((2,2)))
model.add(Conv2D(256, (3,3), activation='relu', padding='same'))
model.add(Conv2D(256, (3,3), activation='relu', padding='same'))
model.add(Conv2D(256, (3,3), activation='relu', padding='same'))
model.add(UpSampling2D((2,2)))
model.add(Conv2D(128, (3,3), activation='relu', padding='same'))
model.add(Conv2D(128, (3,3), activation='relu', padding='same'))
model.add(UpSampling2D((2,2)))
model.add(Conv2D(64, (3,3), activation='relu', padding='same'))
model.add(Conv2D(64, (3,3), activation='relu', padding='same'))
model.add(Conv2D(3, (3,3), activation='relu', padding='same'))

thank you so much for your comment! Just a quick question, if now i go on train my network using vgg16 by obviously training it again would it be a good choice for something to modify/increase performance in denoise autoencoders?? Because my main aim was to increase the accuracy and denoise autoencoders performance. Or there could be any different approach that is thereb which i can follow.

I will build a denoise autoencoder by today, and will share a link to the code.

thank you! please do let me know...although i have tried making a basic denoise autoencoder the results were good but i'm expecting a better result because of which i jumped to transfer learning using vgg16

@arizona_3 I've made a notebook on this, here is the link: https://www.kaggle.com/code/mushfirat/denoise-images-with-convolutional-autoencoders/
The explanations may not be so good, so I'll update it if you face any problem understanding it.

thanks a ton!! Really appreciate  this_/ _

just a doubt would mean_squared_error be a good choice above binary cross entropy??

@arizona_3 yes, MSE can be used, but binary crossentropy is like the standard loss function for calculating loss between images.
And It would be better if you could ask your doubts at kaggle, not here.

Alright!! sure will clear doubts there. Thanks again.


text:
Reconstructing 3D models from 2D images using autoencoders
I went through a research paper ("Voxel-Based 3D Object Reconstruction from Single 2D Image Using Variational Autoencoders") and tried to implement the approach following this diagram:
![link to image of reference network- https://ibb.co/4JgbQ9s
Here is my implementation for the same:
image = Input(shape=(None, None, 3))

# Encoder
l1 = Conv2D(64, (3,3), strides = (2), padding='same', activation='leaky_relu')(image)
l2 = MaxPooling2D(padding='same')(l1)
l3 = Conv2D(32, (5,5), strides = (2), padding='same', activation='leaky_relu')(l2)
l4 = MaxPooling2D(padding='same')(l3)
l5 = Conv2D(16, (7,7), strides = (2), padding='same', activation='leaky_relu')(l4)
l6 = MaxPooling2D(padding='same')(l5)
l7 = Conv2D(8, (5, 5), strides = (2), padding = 'same', activation = 'leaky_relu')(l6)
l8 = MaxPooling2D(padding='same')(l7)
l9 = Conv2D(4, (3, 3), strides = (2), padding = 'same', activation = 'leaky_relu')(l8)
l10 = MaxPooling2D(padding='same')(l9)
l11 = Conv2D(2, (4, 4), strides = (2), padding = 'same', activation = 'leaky_relu')(l10)
l12 = MaxPooling2D(padding='same')(l11)
l13 = Conv2D(1, (2, 2), strides = (2), padding = 'same', activation = 'leaky_relu')(l12)

#latent variable z
l14 = Reshape((60,512))(l13)
l15 = Dense((60*512), activation = 'leaky_relu')(l14)
l16 = Dense((128*4*4*4), activation = 'leaky_relu')(l15)
l17 = Reshape((60,4,4,4,128))(l16)

#Decoder
l18 = UpSampling3D()(l17)
l19 = Conv3DTranspose(60, (8, 8, 8), strides = (64), padding='same', activation = 'leaky_relu') (l17)
l20 = UpSampling3D()(l19)
l21 = Conv3DTranspose(60, (16,16,16), strides =(32), padding='same', activation = 'leaky_relu')(l20)
l22 = UpSampling3D()(l21)
l23 = Conv3DTranspose(60, (32, 32, 32), strides = (32), padding='same', activation = 'lealy_relu')(l22)
l24 = UpSampling3D()(l23)
l25 = Conv3DTranspose(60, (64, 64, 64), strides = (24), padding='same', activation = 'leaky_relu')(l24)
l26 = UpSampling3D()(l25)
l27 = Conv3DTranspose(60, (64, 64, 64), strides = (1), padding='same', activation = 'leaky_relu')(l26)

model3D = Model(image, l27)

This gives me error for l10 saying:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_33/351640059.py in <module>
     24 #Decoder
     25 l18 = UpSampling3D()(l17)
---> 26 l19 = Conv3DTranspose(60, (8, 8, 8), strides = (64), padding='same', activation = 'leaky_relu') (l17)
     27 l20 = UpSampling3D()(l19)
     28 l21 = Conv3DTranspose(60, (16,16,16), strides =(32), padding='same', activation = 'leaky_relu')(l20)

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    975     if _in_functional_construction_mode(self, inputs, args, kwargs, input_list):
    976       return self._functional_construction_call(inputs, args, kwargs,
--> 977                                                 input_list)
    978
    979     # Maintains info about the `Layer.call` stack.

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _functional_construction_call(self, inputs, args, kwargs, input_list)
   1113       # Check input assumptions set after layer building, e.g. input shape.
   1114       outputs = self._keras_tensor_symbolic_call(
-> 1115           inputs, input_masks, args, kwargs)
   1116
   1117       if outputs is None:

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _keras_tensor_symbolic_call(self, inputs, input_masks, args, kwargs)
    846       return tf.nest.map_structure(keras_tensor.KerasTensor, output_signature)
    847     else:
--> 848       return self._infer_output_signature(inputs, args, kwargs, input_masks)
    849
    850   def _infer_output_signature(self, inputs, args, kwargs, input_masks):

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _infer_output_signature(self, inputs, args, kwargs, input_masks)
    884           # overridden).
    885           # TODO(kaftan): do we maybe_build here, or have we already done it?
--> 886           self._maybe_build(inputs)
    887           inputs = self._maybe_cast_inputs(inputs)
    888           outputs = call_fn(inputs, *args, **kwargs)

/opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py in _maybe_build(self, inputs)
   2657         # operations.
   2658         with tf_utils.maybe_init_scope(self):
-> 2659           self.build(input_shapes)  # pylint:disable=not-callable
   2660       # We must set also ensure that the layer is marked as built, and the build
   2661       # shape is stored since user defined build functions may not be calling

/opt/conda/lib/python3.7/site-packages/keras/layers/convolutional.py in build(self, input_shape)
   1546     if len(input_shape) != 5:
   1547       raise ValueError('Inputs should have rank 5, received input shape:',
-> 1548                        str(input_shape))
   1549     channel_axis = self._get_channel_axis()
   1550     if input_shape.dims[channel_axis].value is None:

ValueError: ('Inputs should have rank 5, received input shape:', '(None, 60, 4, 4, 4, 128)')"```

Any help and guidance is appreciated.

You are missing a Reshape step between layers 9 and 10. In addition I suggest you to add an activation function to your dense layers.
Edit 1:
Actually in Keras we don't usually care about the tensor's first dimension, since it is the batch size: docs. I assume you set the batch_size at your input layer, so the tensors' dimensions first value isn't None?
Check the dimensions of your l7, according to the attached image it should be (60, 512). l8 should be Dense(512, activation = 'leaky_relu')(l7) and l9 would be Dense(128*4*4*4, activation = 'leaky_relu')(l8) which can be reshaped to (60, 4, 4, 4, 128) by calling Reshape((4, 4, 4, 128))(l9).
Now that I look at your implementation more carefully I noticed several issues:

It has six Conv2D layers but the reference architecture has seven
The number of filters and the kernel size doesn't match, except for the l1 layer. For example l2 should have 64 filters and a size of 5. And the number of filters increases as we go deeper, the last layer has 512 of them but yours has only one.
The reference architecture seems to use pooling layers, their resolution is halved at each step.

I suggest you try some simpler tutorials first, to get familiar with tensor sizes and how they relate to different network layers and their parameters.

l7 = Reshape((60,512))(l6)
l8 = Dense((60), activation = 'leaky_relu')(l7)
l9 = Dense((60), activation = 'leaky_relu')(l8)
l10 = Reshape((60, 4, 4, 4))(l9)```

i did this but its giving me error-> ```ValueError: total size of new array must be unchanged, input_shape = [60, 60], output_shape = [60, 4, 4, 4]```

Thanks for your explanation, i would also like to learn that whether upsampling layers needs to be inserted in latent variable layers as well? since between dense and reshape there seems to be increase in resolution?

I'm fairly sure they forgot to mention the batch size there, it is the only block which's size doesn't start with 60 x. And also it is a bit odd that a fully connected layer would have a such high output dimension, that is why I think it is actually 8192 (= 12844*4) before the reshape.

Niko i made some changes and have updated the code further some error still persists...unable to understand where exactly should i tweak...


text:
How is the training comlexity of NNLM word2vec calculated?
I was reading this paper on word2vec, and came around the following description of a feedforward NNLM:

It consists of input, projection, hidden and output layers. At the input layer, N previous words are encoded using 1-of-V coding, where V is size of the vocabulary. The input layer is then projected to a projection layer P that has dimensionality N × D, using a shared projection matrix. As only N inputs are active at any given time, composition of the projection layer is a relatively cheap operation.

The following expression is given for the computational complexity per training example:

Q = N×D + N×D×H + H×V.

The last two terms make sense to me: N×D×H is roughly the amount of parameters in a dense layer from the N×D-dimensional projection layer to the H hidden neurons, analogous for H×V. The first term, however, I expected to be V×D since the mapping from a one-hot encoded word to a D-dimensional vector is done via a V×D dimensional matrix. I came to that conclusion after reading this referenced paper and this SO post where the workings of the projection layer are explained in more detail.
Perhaps I have misunderstood what is meant by "training complexity".

Yes. Technically your understanding is correct; i.e. if all input neurons are active, the computational complexity would be like you said: Q = V$\times$D + N$\times$D$\times$H + H$\times$V.
So, if there are V words, the input, of course, will be a 1 of V one-hot vector.
And since there are N words in the input, the input matrix will be of size N$\times$V. If D is the dimensionality of the embedded vector, the projection matrix will be of size V$\times$D.
The product of the input matrix and the projection matrix will then have dimensions (N$\times$V)$\cdot$(V$\times$D) = N$\times$D.
But remember that this is a time series problem and not all input words are available at any given time. For a N-gram model, only N inputs are active at any given time. Hence, of the total V matrix elements in the projection matrix, only N elements corresponding to the input one-hot vector are referenced/updated at any given time. The remaining elements are referenced/updated at a different time.
As a result, at any given time, the input matrix will have dimension N$\times$N, and the projection matrix will only have to be N$\times$D. The product of these two matrices (N$\times$N)$\cdot$(N$\times$D) will still have dimensions N$\times$D. Hence the total complexity is Q = N$\times$D + N$\times$D$\times$H + H$\times$V.
But in practice, this is implemented as a look up table rather than a matrix multiplication, as mentioned here.


text:
Backprop to calculate mean and standard deviation in batch normalization?
On page 310 of the Deep Learning book by Ian Goodfellow (Page 310 can be viewed here for better context: https://www.deeplearningbook.org/contents/optimization.html ), it is mentioned that one crucial point about batch norm is that:

... Crucially, we back-propagate through
these operations for computing the mean and the standard deviation, and for
applying them to normalize
H
. This means that the gradient will never propose
an operation that acts simply to increase the standard deviation or mean of [the layer] h_i
; the normalization operations remove the eﬀect of such an action and zero
out its component in the gradient

The equations:

I do not understand why is it crucial that we backprop through these calculations and how that helps in setting the mathematical intuition to batch norm, as mentioned in the book.
Can someone help me connect the dots here?


text:
What is a 'degenerate run' in evaluating model performance?
I've recently come across a paper that uses the term "degenerate run", but I'm not sure if I understand what it means. The idea is that when they report the average performance of running fine-tuned models using multiple random seeds (e.g., a deep learning model where we need to initialize model parameters using multiple seeds to ensure results are robust,) they exclude the degenerate runs in some of their analyses.
As this paper mentions, a degenerate run is "where fine-tuned models fail to outperform the random baseline." But I wonder if this is a standard practice to eliminate such results when reporting the average performance? Or is the definition they give the correct meaning of a degenerate run?

Can you please cite the paper in the question?

Hello. Please, do as suggested in the previous comment. Include the link to and name of the paper you're mentioning.

@hanugm thanks for the suggestion. I just added the reference and more context.

The authors explain their use of the term in the paper:

Without the bias correction we observe many degenerate runs, where fine-tuned models fail to outperform the random baseline

Specifically, as it pertains to fine tuning, degenerate has this dictionary meaning in technical contexts: Lacking some usual or expected property or quality. In the paper's case the retrained classifier lacks the property of being able to classify inputs to any accuracy.
That is backed up by their use of the term to apply to failed training runs.
There is also a mathematics use of the term, which sometimes appears in computer science, and is about object classes e.g. a triangle with one angle 0 degrees is degenerate - it is just a line, not meaningfully a triangle. I don't think it applies here, although it could - the failed refined classifier has all the technical build qualities of a classifier, but does not classify.
I do not think degenerate runs is a commonly used phrase in machine learning, nor does it have any additional technical meaning when applied to supervised learning, beyond what they explain in the paper.
A more common term related to trained systems losing capabilities when trained further on new data is catastrophic forgetting. This may be subtly different to the effect that the authors are correcting for though, perhaps that is the reason for them using a different term.


text:
Structured policies in dynamic programming: solving a toy example
I am trying to solve a dynamic programming toy example. Here is the prompt: imagine you arrive in a new city for $N$ days and every night need to pick a restaurant to get dinner at. The qualities of the restaurants are iid according to distribution $F$ (assume [0,1]). The goal is to maximize the sum of the qualities of the restaurants that you get dinner at over the $N$ days. Every day you need to choose whether you go to a new restaurant and obtain a utility drawn at random from distribution $F$, or you go to the best restaurant you have attended so far, and obtain a known utility.
Here is my formulation:

Let $m_t$, the state at time $t$, denote the quality of the best restaurant visited so far.
Let $a_t \in \{Q, E\}$ be the action at time $t$; $E$ for EXPLORE, $Q$ for quit.
Given the action set, the rewards are ($W_t$ denotes the random variable of the quality of the restaurant drawn at random at time $t$):
$$r_t(m_t | E) = W_t \sim F $$
$$r_t(m_t | Q) = m_t  $$
The transition probabilities at time $t$ are:
$$ p_t\big (  m_t |  m_t, Q \big ) =1 $$
$$ p_t\big (  m_t |  m_t, E \big ) =F(m_t), \text{ with } W_{t} \in [0, m_t]$$
$$ p_t\big (  W_t |  m_t, E \big ) =1-F(m_t), \text{ with } W_{t} \in ( m_t, 1]  $$

First I write down the Bellman optimality equations:
For $t=N$, I get:
$$ u^*_N (m_N) = \max_{a_t \in \{Q, E\}} \big\{ m_N, \ \  E(W_N) \big\}  $$
For $t<N$, I get:
$$
 u^*_t (m_t) =
 \max_{a_t \in \{Q, E\}}  \bigg\{ m_t+ u^*_{t+1} (m_t) , \ \ \ E(W_t)+  u^*_{t+1} (m_t) F(m_t) + \int_{m_t}^1 u^*_{t+1} (x) dF(x) \bigg\}
$$
What I am trying to show is that, if it is optimal to quit exploration in period $\tau$, then it is optimal to not explore in every period $t=\tau+1, ..., N$. In agreement with the equations above, I assume:
$$m_\tau- E(W_\tau) \geq   \int_{m_\tau}^1 (u^*_{\tau+1} (x)-u^*_{\tau+1} (m_\tau))  dF(x) $$
and I want to show:
$$m_\tau - E(W_{\tau+1}) = m_\tau - E(W_{\tau})\geq   \int_{m_\tau}^1 (u^*_{\tau+2} (x)-u^*_{\tau+2} (m_t))  dF(x) $$
I have been reading the structured policies chapters on Puterman and Bertsekas (Vol II) and did not find anything that could get me rolling. Any help or pointer to literature is more than welcome!
After I show this I can argue that there will exist a sequence of thresholds $c_t$ so that is is optimum to stop exploration at time $t$ it $m_t \geq c_t$.
EDIT: Distribution $F$ is known to the tourist.

I think you need to declare whether the agent knows $F$. Knowing $F$ allows the agent to make decisions about optimality differently from if they don't know $F$.

Distribution $F$ is know. I added that in the question.

To clarify what your question really is, can you put it in the title of your post? "Structured policies in dynamic programming" is not really a question. I also don't see any question in the body of your post.


text:
How can I reduce the loss? Why do I have the high loss and why do I have the gradient?
I want to classify some images (there are about 200.000 images) with a CNN. But I get a very high loss, see figures:
Loss over the hole training run

Loss for each epoch

It's confused me, that there is a high gradient by the 15 batch of each epoch
I try some option in the history like editing the learning rate and add weights by initialization the network, but I dont unterstand what's wrong. In the first step I train my network with 2000 images, but it's possible to train it with more images.
The Question
How can I reduce the loss and why do I have the high loss? Rather, why do I have the high negativ gradient in the middle of each epoch (see in the second plot)?
Thank you for your help in foward :)
Some words to the dataset:
The dataset is a set of images and they looks very similar. There are two classes: the "good" images and the "error" images. For us - the humans - we will say, that's both images classes looks right, but a normal camera has not enough intelligence. That's the motivation for the projekt.
Now, you find here the code of my neural network:
The training method
model = net.Netz()
optimizer = optim.SGD(model.parameters(), lr= 0.0001, momentum = 0.8)

def trainM(epoch):
    model.train()
    for batch_id, (data, target) in enumerate(net.train_data):
        #data = data.cuda()
        #target = target.cuda()

        target = torch.LongTensor(target[64*batch_id:64*(batch_id+1)])
        #data   = torch.Tensor(data[64*batch_id:64*(batch_id+1)])
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()

        out = model(data)
        criterion = F.nll_loss

        loss = criterion(out,target)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1)
        optimizer.step()
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch,batch_id*len(data), len(net.train_data)*64, 100*batch_id/len(net.train_data), loss.item()))

for item in range(1,10):
    trainM(item)

The CNN
class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        torch.nn.init.xavier_uniform(self.conv1.weight)
        self.conv2 = nn.Conv2d(10,20, kernel_size = 5)
        torch.nn.init.xavier_uniform(self.conv2.weight)
        self.conv_dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(1050,60)
        self.fc2 = nn.Linear(60,2)
        self.fce = nn.Linear(20,1)

    def forward(self,x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        x = self.conv2(x)
        x = self.conv_dropout(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        #x = x.view(-1,320)
        x = x.reshape(x.shape[0], x.shape[1], -1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        x = self.fce(x.permute(0,2,1)).squeeze(-1)
        return F.log_softmax(x, -1)

The data prep method
def dataPrep(list_of_data, data_path, category, quantity):
    global train_data
    global target_list
    train_data_list = []
    mean = [0.0028]
    std = [1.0001]

    transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(torch.Tensor(mean), torch.Tensor(std))
        ])

    len_data = len(train_data)
    for item in list_of_data:
        f = random.choice(list_of_data)
        list_of_data.remove(f)
        try:
            img = Image.open(data_path +f)
        except:
            continue
        img_crop = img.crop((310,60,425,240))
        img_tensor = transform(img_crop)
        train_data_list.append(img_tensor)

        if category == True:
            target = 1
        else:
            target = 0
        target_list.append(target)

        if len(train_data_list) >=64:
            train_data.append((torch.stack(train_data_list), target_list))
            train_data_list = []

        if (len_data*64 + quantity) <= len(train_data)*64:
            break
    return list_of_data

Have you tried increasing the number of neurons or layers ?
And Try to use Adam Optimizer instead of SGD !

Yes I tried it, but the result is the same :(

There might be some bug in the code. From a first glimpse I don't really understand what's going on after your conv-layers. Specifically, this line preps the tensors for the classification head: x.reshape(x.shape[0], x.shape[1], -1) so you get a tensor of shape batchsize x img-height x features. By doing that you effectively run your classification head on the img rows separately, right? So they have very little info about the full input image. Is there a reason why you don't reshape the thing by x.reshape(x.shape[0], -1) and use a standard MLP as the classification head?

Your issue with high loss and a large negative gradient in the middle of each epoch could be due to a few factors. Let's explore potential causes and strategies to mitigate this:

Learning Rate & Gradient Clipping:
Problem: A very high learning rate or inadequate gradient clipping can cause large fluctuations in gradients, especially if your dataset is small or has imbalanced distributions between the two classes. A large negative gradient might indicate that the model is overshooting the optimal direction during backpropagation.
Solution: You can try lowering the learning rate further. You're already using a small learning rate (0.0001), but lowering it slightly more (e.g., 1e-5 or 5e-5) might stabilize the training. Also, try adjusting the max_norm value in torch.nn.utils.clip_grad_norm_. Lower it (e.g., from 1 to 0.5) to see if it helps reduce the gradient spikes.
Batch Size:
Problem: A batch size of 64 may cause unstable gradients if the data is noisy or highly similar. Batch size impacts the noise level in gradient updates.
Solution: Experiment with smaller batch sizes (e.g., 32 or 16). This may lead to more stable gradient updates. Smaller batches also allow the network to see more variability in each epoch.
Class Imbalance or Similarity:
Problem: If the two classes are extremely similar and there's imbalance in the dataset (e.g., more "good" images than "error" images), the model might struggle to distinguish between them, leading to loss spikes.
Solution: Ensure that your dataset is balanced. You can also try data augmentation to increase the variability between the classes, which might help the model generalize better.
Use Focal Loss: Instead of standard nll_loss, try using Focal Loss to penalize the model less for well-classified examples and more for hard-to-classify ones. This could improve learning for the similar images.
Normalization and Data Preprocessing:
Problem: The normalization you applied (mean=[0.0028] and std=[1.0001]) is quite unusual. If the images are not centered and normalized appropriately, this can destabilize training.
Solution: Recheck your normalization values. Typically, the mean and standard deviation should be calculated from the training dataset. Consider using mean=0.5, std=0.5 as a simple initial setup, or compute actual values from the dataset.
Weight Initialization:
Problem: If your weight initialization is too aggressive, it can cause sharp gradients early in the training process.
Solution: You’re using torch.nn.init.xavier_uniform, which is generally good. However, try switching to He initialization (torch.nn.init.kaiming_uniform_), especially since you are using ReLU activations, which tend to work better with He initialization.
Overfitting:
Problem: If you have too few training examples or the network is too complex, the model might overfit quickly, causing large gradient spikes as it tries to memorize the dataset.
Solution: You can introduce more regularization (increase Dropout2d rate) or add L2 weight decay in your optimizer (weight_decay parameter).
Data Shuffle:
Problem: If your dataset is not shuffled correctly, it may expose the model to batches of data that are very similar, causing spikes in loss.
Solution: Ensure that your DataLoader is shuffling the data at every epoch.
Plot Analysis
If you see a large negative gradient in the middle of each epoch:

Check if your data loader shuffles the data between epochs. This might explain why the gradients behave similarly at the same point each epoch.
Plot the learning rate over time to ensure it’s not oscillating or changing too abruptly mid-epoch.
for reference make changes in training code like this one
Consider lowering the learning rate further
optimizer = optim.SGD(model.parameters(), lr=1e-5, momentum=0.8, weight_decay=1e-4)

def trainM(epoch):
    model.train()
    for batch_id, (data, target) in enumerate(net.train_data):
        # Shuffle data if not already done
        data, target = data.cuda(), target.cuda()

        optimizer.zero_grad()

        out = model(data)
        criterion = F.nll_loss

        loss = criterion(out, target)
        loss.backward()

        # Try lowering gradient clipping norm value
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=0.5)

        optimizer.step()

        if batch_id % 15 == 0:  # Log every 15 batches to track potential issues
            print(f'Epoch: {epoch}, Batch: {batch_id}, Loss: {loss.item()}')

There is something called Dead Neuron Problem in Relu Activation function which means if your weights becomes too small there will a time when your model will stop learning and there will be no updates in parameters at all. Maybe that's problem in your case.


text:
How should I write the reward function to teach the agent the rules of this card game?
I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck.

In a 5 by 5 board, each position can contain a card with a color (0-4) and a value (0-9).
Some initial cards, all distinct from each other, may be on the board.
In each round, a card not on the board is chosen with uniform distribution.
The player has to place this card in an empty position before the next card is chosen.
When all positions are filled, the game ends and the final score is the sum of all points of each row, each column, and each of the two diagonals, where the points are calculated as follows (order does not matter, e.g. AAABB=ABABA, 12345=23415):

Combination
Points
Colors
Values

Straight flush
30
All identical
All consecutive

Straight
15
Not all identical
All consecutive

Flush
14
All identical
Not all consecutive

Five of a kind
28

AAAAA

Four of a kind
16

AAAAB

Full house
10

AAABB

Three of a kind
6

AAABC

Two pairs
3

AABBC

One pair
1

AABCD

Nothing
0

The goal is of course to get a score as high as possible.
Question:
How should I write the reward function to teach the agent the rules?
I used invalid action masking, so no invalid actions can be taken. However, it's still hard for high-score combinations to randomly show up. I tried setting reward to be the points possibly added due to the placed card (like if the placed card can form a pair with another one in the same row, add some points based on how much point a pair is rewarded in the final score), and reward 100*(final score) at the end, hoping the agent to learn that the final score is the most important, but the mean reward was not improving even after 2 million timesteps.
I use MaskablePPO with MlpPolicy (alias of MaskableActorCritic) from sb3-contrib. I represent each card as an int8. The upper 4 bits representing the color and the lower 4 bits represent the value.
The observation space I use consists of the card to be placed, the cards on the board, and all cards not on the board. I'm wondering if I should include cards not present in the observation space. I tried to exclude it but it does not change the results much.
Apart from direct answers, any pointers/reference would be much appreciated.

Is it a game that you can links the rules for? Your description is fine as-is, but the scoring rules may allow someone to try an experiment. One important detail: Is it possible to calculate a score for the incomplete board using the standard rules? As you have a large state space, and a policy network, I am assuming you are using a neural network. It may be worth explaining what you have tried there, and how you convert state to the representation used by the network. For instance, putting the raw 0-9 values in grid positions to feed into a CNN may not work well.

@NeilSlater I updated some info. I am not using a CNN tho, because stable-baselines 3 only allows CNN to work with images. I'm using an actor critic policy.

Thanks for the updates. Are the scores assigned according to strict sequence match ie. a straight has to have values in order in the horizontal (left to right) or vertical (top to bottom) row? Or is it enough to just have a set of consecutive values e.g. reading to to bottom the multicoloured array [6, 5, 8, 7, 4] would be a valid straight?

@NeilSlater Order does not matter. Updated. Thanks for pointing out.

I would recommend having the reward as the increase to score caused by adding the card in the chosen location. You could optionally include calculations based on partial rows for pairs, three-of-a-kind etc. I do not think you should grant a different final reward for the end score, and definitely not some large difference, as the gradient for a large difference may swamp the networks (unless you also drop the learning rate, but then the smaller early values will not get learned from).
I do not think that your focus should be on clever manipulation of reward signal. It is likely that looking at other things related to this problem will give you better impact.
First you should look at your state representation. If you are feeding the int8 values directly to a neural network, then this is very unlikely to work. Instead, you should use a short vector for each card you want to represent - of either 6 or 15 elements. You should be one-hot-encoding the colour for each position. You may find one-hot-encoding the value also helps, although that is trickier - if you don't one-hot encode, then you should scale the value e.g. $(v - 5)/3$ to keep it in range for best neural network learning.
The card in hand, about to be placed, should be part of the state. So far, before any feature engineering, that would give you a vector of at least 26 * 6 = 156 dimensions, but you could take that further.
If possible, use afterstates instead of enumerated actions. That may not be possible if you are using a library that expects enumerated actions. You probably won't find a version of Actor-Critic with afterstates because the Actor would need to express a probability distribution over the afterstates, which is awkward to do. You may be able to set the Actor to use enumerated actions and the Critic to score advantage based on afterstates though.
To help the agent, do some feature engineering to add summaries of each possible line to the state (so a summary for each of the 12 lines). One important feature will be whether that line is complete or not - if it is complete, no more reward is possible from that line (perhaps a scaled "how many spaces left" counter would work here). Other features that could be useful are a "bag of colours" and "bag of values" i.e. one-hot-encoded list of current variation in that row - such a summary makes it easier for the NN to tell whether an x-of-a-kind or flush or straight are possible.
In theory you could also pre-calculate the immediate reward from placing the card in each position, making that part of the state (another 25 features). That is so the agent is not trying to predict this value - it is hard for the NN, but easy for the game engine and not "cheating" at the game by e.g. peeking at the next card. That should allow the NN to use more of its capacity for planning ahead and statistically gambling on what might be available in future, which is perhaps the more interesting aspect of this game.


text:
Learning values in open ball: which final layers to employ?
I'm fairly new to deep learning and looking for some reference literature... Specifically, I want to train a neural network to predict vectors $v \in \mathbb{R}^3$ under the constraint $||v||\leq 1$.
I add that the case $v\simeq0$ is the most common in my dataset. Which final layer(s) would you suggest to accomplish this? These layers would follow a CNN + fully connected layers body.
Thanks in advance

Background: this vector would represent the $SO(3)$ element of a rotation matrix

Is the constraint only that you know the target dataset has nothing with $||v|| \gt 1$, or are there negative consequences for predicting magnitudes that are even slightly too large? How common is $||v|| = 1$ (lots of stuff on the boundary as well as near the origin?). What is your loss function? If you are not sure of the loss function, what are the consequences for inaccurate predictions and do they vary according to absolute error distance, ratio of prediction to true value or some other factor?

At $||v||=1$ a discontinuity in the loss function will happen. Basically, $cn$ and $(c+2k\pi)n$ are equivalent, with $||n||=1$, because they represent the same rotation vector (the constraint is actually $||v||=\pi$, but I put $1$ in the question for simplicity.

$||v||=1$ (actually, $||v||=2\pi$) will happen around 1-5 % of the times, whereas we're near the origin 60% of the times (meaning little rotation)

As you are asking about output representation you need to be clear how you want the raw output to be scaled. When considering unscaled rotation you have an interesting issue that $||v|| = 2\pi$ is equivalent to  $||v|| = 0$. As you are considering opposite rotations, then you could equally scale to $||v|| = pi$ as there is a discontinuity of sorts there anyway (when $||v|| = \pi$, then $v \equiv -v$) and you get full coverage of all possible new orientations under the constraint $||v|| \le \pi$

... Unless the rotations are defining the act of rotating as opposed to the final orientation? But in that case, it is not clear why you would stop at one complete revolution.

No, I really want to regress rotations. The problem is that I would like the rotation representation to be smooth, connected and easily invertible, but there is no such thing as this, apparently, unless I regress a full $9\times 9$ matrix, but then there is a problem of enforcing the orthogonality constraint, and I can't easily work with outputs on a non euclidean manifold (because of how I'll have to work with this NN).

The issue as I understand your problem is not the constraint. A predicted value that does not repsect the constraint, but that is close to the true value would still be good. The problem is that you have input data with similar inputs that should predict either $-v$ or $+v$ around 1-5% of the time, and the NN may have trouble learning this discontinuity. There is a good chance it will learn to predict close to zero rotation in cases where you want close to maximum rotation. You will be better off with a representation that does not have the discontinuity

@Leonardo "I can't easily work with outputs on a non euclidean manifold"  maybe this is a silly suggestion and you already know about it, you could check geotorch

@EdoardoGuerriero Didn't know about this, looks very interesting!

Can you please put your specific question in the title? It seems that your specific question is "Which final layer(s) can I use to produce 3d vectors such that their norm is less than one (an open ball)?". I would also recommend that you update your post with more details about your specific problem (e.g. what high-level task are you really trying to solve? in which context?) and with the info that you've gathered from the discussions in the comments. I would also describe more in detail your dataset. For example, what are the inputs? How many data points?


text:
MDP with a non-markovian reward function?
I have set up a RL environment and it converges to a decent solution when using a reward function:
$R(s_t,a_t) = fenv(s_t, a_t)$ , where $fenv$ is the environment dynamics.
Now, i want to change the reward function such that
$R(s_t,a_t) = fenv(s_t, a_t)*c(s_t:s_{t-10})$
where $c(s_t:s_{t-10})$ is a penalty term that depends on the average performance of previous 10 timesteps. So now the reward is dependent on agent's previous states and needs information from the past. I suspect this can change the underlying problem and the MDP assumption is not valid anymore.
I read about potential based reward shaping that guarantees invariance in terms of MDP but i am not sure if this transformation falls into that category.
So my question is, by including information from the past states into the reward definition, do we still maintain the underlying MDP or are we solving a different problem altogether? Assuming that we don't explicitly add previous action choices in the state representation, as suggested here https://ai.stackexchange.com/a/25991/54470

I believe this question is related and answers your concerns:
https://ai.stackexchange.com/questions/25990/reinforcement-learning-algorithm-with-rewards-dependent-both-on-previous-action?rq=1

Thanks @postnubilaphoebus! This confirms my suspicion about loosing the Markovian property in this case.

You're note asking any question explicitly. Could you please edit your post to actually ask a question? Thanks.

My bad @nbro! Hope the question is clear now. Thanks for your time.


text:
ML model to predict timeouts
I am new to ML and am trying to build a model to predict timeouts for a website.
The website is being monitored once a minute and the data consists of a timestamp and the response time in seconds. E.g.: 1650539220000 6.234041.In this particular case I do not have metrics from the server itself, just everything externally available.
When collecting the metric a timeout for the http request is set to 15 seconds. Meaning that every time the server does not respond in time I have no data and I want to predict when the next time out will happen. Currently, I have about two weeks' worth of data and want to predict the next likely timeouts/anomalies within some hours.
Since missing values are a huge issue with ML models I tried to handle the timeouts by setting the value to

20 to have large "peaks" which does not work too well with some models
integer-encode the data in 0 (no timeout) and 1 (timeout)

To have more features to work with I also tried splitting the timestamp into month, day, hour, minute.
So far I have tried multiple algorithms which did not perform too great

Prophet
LGBMRegressor (best so far)
AutoReg
ARIMA

I did not tune the algorithms, as I am quite new to ML.
What method, model, or approach could I try to make more accurate predictions?

What rationale do you have to expect the timestamp to be a predictor of timeouts? Typically timeouts are a function of the request type and overall server load. You don't appear to be using either of those as predictors, so your best result might be to roughly follow the server load on a daily/weekly basis, due to constructing the day and hour features.

While that is true there are patterns in the timeouts. LGBM has found one with the timeout (no data) being set to 20 and the timestamp split into multiple features. At least it predicts a response time of gt 10s. Could I improve my approch or try something else to make more accurate predictions?

Can you define what you mean by "timeout"? Do you mean how long it takes for the server to return a response after a request? Also, do you just have "timestamps" as inputs or do you have something else?

By time out I mean the cases where the server did not respond within 15 seconds to the request. I currently collect the timestamp and the response time. In this particular case I do not have metrics from the server itself, just everything externally available. Is there more I could collect or a better way to handle the time outs in the dataset?

@gwolter Please, edit your post to include this and the other information above directly there, as comments are temporary.

@nbro I edited the post to define timeout more clearly

For the time series i am using mostly deep learning(like LSTM).If you decide to try this be carefull  ,because this type of ML have issues with overfitting.
example of LSTM
https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/
This approach also sound promissing:
https://www.analyticsvidhya.com/blog/2021/12/time-series-forecasting-with-extreme-learning-machines/

Thank you very much for the answer! I will have a look at LSTM. Are there other deep learning models you are using? Some names would be great so I can dive down a bit deeper.


text:
How are 4D cost volumes constructed for DL based stereo matching?
I read a paper on Stereo Matching using Pyramid Cost Volumes (paper link: Semantic Stereo Matching with Pyramid Cost Volumes). At some point, in the proposed architecture, after:

Feature extraction from both left and right images
Performing adaptive average pooling at three scales (4, 8 and 16)
Reducing number of channels using 1x1 convolutions at each scale.

the authors form a cost volume by concatenating the corresponding unaries from the left and right image features and packing them into a 4D volume. The final dimensionality is:
$$C \times \alpha W \times \alpha H \times \alpha D, \alpha \in \{\frac{1}{4}, \frac{1}{8}, \frac{1}{16}\} $$
, where C - number of channels, W and H - width and height of the images, D - maximum disparity.
My question is, for all the possible disparities, how can this 4D volume be visualized? How is the volume, for each possible disparity, constructed in the first place?


text:
In the MuZero paper, how does backprop in the MCTS account for the immediate reward from each edge?
On page 12 of this paper: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model, it describes how MCTS works for the MuZero algorithm. It states in equation 4 that during the 'backup' after a simulation, the mean value (Q) for every edge in the simulation path is updated by:
$$Q\left(s^{k-1},\ a^k\right){\colon=}\frac{N\left(s^{k-1},\ a^k\right)Q\left(s^{k-1},\ a^k\right)+G^k}{N\left(s^{k-1},\ a^k\right)+1}$$
where $G^k$ is the return from depth $k$ and onwards in the simulation.
However, I don't see how this equation accounts for the immediate reward attained by transitioning from state $s^{k-1}$ to $s^{k}$.
Since $Q\left(s^{k-1},\ a\right)$ is used to determine what action to select from state $s^{k-1}$, shouldn't the backprop use the return from k-1 ($G^{k-1}$), to update the mean value instead of $G^k$?

Let's look at the first simulation where $k = 1$.
We have $G^0 = r_1 + \gamma v^0$ and $G^1 = v^1$, and we can pick one of them as the update target. The difference is that $G^1$ is fully conditioned on the action $a_1$ we just took, while $G^0$ is only partially so.
We choose $G^1$ for the same reason that we do the search: we assume the evaluations of child nodes conditioned on more actions is better than the evaluation of the root alone.
EDIT:
I believe the expectation should match so that $\mathbb{E}[G^n] = \mathbb{E}[r_{n+1} + \gamma G^{n+1}]$.
I looked deeper into it and I agree it's confusing.
I think what's happening is that in the paper they mixed two different definitions of $Q$: either $Q(s, a) = V(s')$ or $Q(s, a) = R(s, a) + V(s')$.
Their implementations (based on MuZeor's pseudo-code and MCTX) suggest that $Q(s, a) = V(s')$ because child nodes store values without considering the last received reward.
However, the reward plays a part in the child selection phase, so a better way to write their selection formula is
$$a^{k}=\arg \max_{a} R(s, a) + Q(s, a)+P(s, a) \frac{\sqrt{\sum_{b} N(s, b)}}{1+N(s, a)}\left[c_{1}+\log \left(\frac{\sum_{b} N(s, b)+c_{2}+1}{c_{2}}\right)\right]$$ and this reflects what they are doing in the code.
On the other hand, what you suggested make perfect sense too if we adopt the other definition of $Q$.

Shouldnt $G^{n}\ =r_{n+1}+\gamma G^{n+1}$? You still havent explained how the immediate reward is accounted for. In the first simulation if you go with $G^{1}$, you disregard $r_{1}$.

@Matrix001 Updated.

@Matrix001 does it make sense now?