text:
Do Quo et al (2013) perform backpropagation between layers?
Le et al. 2013's non-weight sharing CNN has inspired me to ask two questions on this site previously.
When training the three-layer autoencoder, do they compute dL/dW (where L is equation 1) independently in each layer or perform backpropagation from one layer to another?

I believe the gradients in the three layers are computed independently.
They mention minimizing each layer only with respect to the weights used in that layer.  If they performed some sort of backpropagation between layers, they would be minimizing the term for one layer with respect to the weights for another layer.  Since the explicitly state that equation 1 (the loss for each layer) is simply minimized for all three layers simultaneously, they clearly imply that only the weights for that layer are considered in the implementation.
But this only confuses me further as to why they train all three layers simultaneously, as asked in my second question.


text:
VAEs vs Autoencoders with BatchNorm and Dropout?
It struck me that regular auto-encoders with batch-norm and dropout have quite similar properties to VAEs which made me wonder whether VAEs where really much better than this simpler alternative. Let me explain:

BatchNorm: encourages activations to follow N(0,1), just like KL-divergence does to the output layer of the encoder
Dropout: during training time creates random (Gaussian due to CLT) encoder output distribution, mean and variance are learnable indirectly via network weights and mean/variance override params of batch norm layer

You might argue that KL-divergence adds more flexibility by allowing it to not exactly follow N(0,1) but batch norm allows learning to override this by default anyways.
All that considered are there really any practical benefits to VAEs which cannot be provided by this simpler setup?

Have you seen any work that uses autoencoders with batch norm and dropout and claimed that this would be (somehow) equivalent to VAEs? If yes, it might be a good idea to provide that reference. I know that, Monte Carlo Dropout with neural networks can be viewed as Bayesian neural networks. This is the closest thing to what you're describing, but for regular neural networks, not autoencoders.

No I'm just reasoning from first principles here, I could be wrong. But if I am I'm curious why.

@nbro
I take that back I did just find a reference trying to do something similar to what I mentioned: https://vitalab.github.io/article/2021/08/20/VariationalToDeterministicAE.html
They say they can use deterministic auto-encoders + injected noise for regularization as an alternative to VAEs. That is precisely what dropout is (though they don't use dropout explicitly).

De-noising auto-encoders also are essentially auto-encoders with dropout. But I don't know if anyone has directly compared them to VAEs. Still I sense they are similar...
https://proceedings.neurips.cc/paper/2013/file/559cb990c9dffd8675f6bc2186971dc2-Paper.pdf

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why do we use a linear interpolation of fake and real data to penalize the gradient of discriminator in WGAN-GP
I'm trying to better frame/summarize the formulations and motivations behind Wasserstein GAN with gradient penalty, based on my understanding.
For the basic GAN we are trying to optimize the following quantity:
$$\min_\theta \max_\phi \mathbb{E}_{x \sim p_{data}(x)}[D_\phi(x)] + \mathbb{E}_{z \sim p_G(z)}[1-D(G_\theta(z))]$$
The problem is that the dissimilarity measure between the two probabilities given by Jensen-Shannon divergence will not take into account any distance in a Euclidean sense. That's why we consider the Wasserstein distance defined as:
$$W(p_{data}, p_G) := \inf_\gamma \, \,\mathbb{E}_{(x,y) \sim \gamma(x,y)}\|x-y\|$$
that will account for a proper distance of our distributions. Computing it is very hard so we rely on Kantorovich-Rubinstein duality which states we can rewrite $W$ as:
$$W(p_{data},p_G) = \sup_{\|f\|_L \le 1}\mathbb{E}_{x \sim p_{data}(x)}[f_\phi(x)] - \mathbb{E}_{z \sim p_{G}(x)}[f_\phi(G_\theta(z))]$$
Now the crucial point, to enforce the constraint of $1$-Lipschitz continuity of the discriminator we add a penalty term to bound the norm of the gradient of $f$, so the final loss we consider is:
$$\mathcal{L} = \mathbb{E}_{x \sim p_{data}(x)}[f_\phi(x)] - \mathbb{E}_{z \sim p_{G}(x)}[f_\phi(G_\theta(z))] + \lambda \, \mathbb{E}_{\hat{x}}[(\|\nabla_{\hat{x}} f_\theta(\hat{x})\|-1)^2]$$
where
\begin{equation}
\hat{x} = tx + (1-t)z
\end{equation}
$t \in [0,1]$.
Now, I've understood that we bound the slope of discriminator because we want toavoid the vanishing gradient problem and keep gradient signal in order to make the generator learn, but why do we actually penalize the gradient of discriminator with respect to a linear interpolation of real and fake data?

Added bounty because it's a really interesting question. GANs made the waves they did because the discriminator they could quantify that the generator was getting "closer" in a more sophisticated way than just measuring pixelwise distance. It does seem quite strange that the gradient penalty would just be done with some pixelwise mixture.

In [1], section 4, the authors mention the following:

Sampling distribution We implicitly define $p_{\hat{x}}$ sampling uniformly along straight lines between pairs of points sampled from the data distribution $p_{data}$ and the generator distribution $p_{G}$. This is motivated by the fact that the optimal critic contains straight lines with gradient norm 1 connecting coupled points from $p_{data}$ and $p_{G}$ (see Proposition 1). Given that enforcing the unit gradient norm constraint everywhere is intractable (1), enforcing it only along these straight lines seems sufficient and experimentally results in good performance (2).

So, there are 2 reasons for enforcing $\nabla f_{\phi} = 1$ over $\hat{x} = tx + (1 - t) G(z)$. Starting with point (1),
(1) Enforcing $\nabla f_{\phi} = 1$ everywhere would amount to sampling $x$ from the whole space $\mathbb{R}^{d}$ uniformly. By the curse of dimensionality, this is unfeasible.
Now, let us consider why does it suffice to enforce on samples from $p_{\hat{x}}$. Note that, due to displacement interpolation [2, Theorem 7.21], the interpolation of two probability measures under the Wasserstein metric corresponds to following geodesics in the ambient space.
In the case of GANs, let $x \sim p_{data}$ and $\tilde{x} \sim p_{model}$, where $p_{model} = G_{\sharp}p_{G}$. The interpolation $p_{t}$ of these two distributions is supported on points $\hat{x}_{t}=tx + (1-t)\tilde{x}$, that is, the distribution $p_{\hat{x}}$, for fixed $t$, is actually the interpolation in probability space of $p_{data}$ and $p_{model}$. As consequence, the authors propose enforcing Lipschitzness on a much more reduced region of $\mathbb{R}^{d}$, that is, the support of $p_{data}$, $p_{model}$ and interpolations $p_{t}$. This may be justified (although the authors do not give theoretical reasons) via the Manifold Hypothesis, that is, the data is supposed to lie on a non-linear low-dimensional region of $\mathbb{R}^{d}$ (so that we do not need to regularize over the whole space).
Remark: the book by Villani [2] is rather technical, and for a reader not familiarized with optimal transport it may be challenging to understand it. A somewhat easier text is the book by Santambrogio [3], which discusses displacement interpolation under the Wasserstein metric in chapter 5. All in all I consider that the authors had an interesting theoretical intuition, and that they validated their approach empirically, but the question on why does this work is not very clear.
References
[1] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. C. (2017). Improved training of wasserstein gans. Advances in neural information processing systems, 30.
[2] Villani, C. (2009). Optimal transport: old and new (Vol. 338, p. 23). Berlin: springer.
[3] Santambrogio, F. (2015). Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63), 94.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How do I deal with a dataset of Images with variable sizes (width and height) when doing Image Classification?
I have a dataset in which the images which don't have the same width and height. How do I perform Image Classification with such images? I am trying as much as possible to steer away from image resizing because in my case it will lead to loss of important data/ give poor results.

check and if helpful upvote these answers on fully convolutional networks and pyramid pooling

@EdoardoGuerriero checked it out, but to no avail. Any ideas or help specific to my question?

could you explain more in details the issue? Both fully convolutional networks and pyramid pooling were precisely designed to tackle the variable input size issue in neural networks so it's not clear to me what else constitute the problem aside from feeding variable sized images to a model.

@EdoardoGuerriero I will need a fully connected layer in my network since my network has to output probabilities for a specified number of labels. So I will need resize. But what are alternatives to resize, because sometimes resize can cause you to lose data.

then again open and read the paper about pyramid pooling in the linked answer, it allows to map an image of generic size without resizing or reshaping to a dense layer of specified, fixed size.


text:
Can directly using expert policy in epsilon-greedy speed-up Q-learning?
In deep Q-learning we typically use epsilon-greedy policy during training. We choose a random action for a certain probability $\epsilon$, and choose the action that maximize the current Q-value estimate for probability $1-\epsilon$, and the value of $\epsilon$ decays slowly over the training process
I am wondering if directly swapping the random action with an expert policy can speed-up the Q-learning process?

For convergence to optimal value functions and optimal policies to be guaranteed in general, you need action coverage in a stochastic exploration policy. Typically that means implementing some kind of $\epsilon$-soft policy but not necessarily $\epsilon$-greedy. Without this, the agent will never look at the off-policy options that might be better. The agent also needs data about bad actions that it should not take, otherwise it will not learn to avoid them by itself.
In general, swapping the $\epsilon$-greedy policy out for a deterministic expert policy will not work. However, there are a few ways that it could be made to work:

The expert policy is stochastic and has coverage, but prefers "better" actions. You still need full coverage for the Q-learning agent to estimate returns from bad actions, unless the expert policy is also going to be applied in production to avoid those bad actions.

Instead of starting with an $\epsilon$-greedy policy and a high value e.g. $\epsilon = 1$, start with the expert policy for some fixed number of iterations, enough for the agent to approximately learn the value function for the expert policy. Then switch back to $\epsilon$-greedy with a relatively low value for exploration e.g. $\epsilon = 0.1$. This will initialise your agent with knowledge from the expert and help speed up leaning if that expert policy is any good.

If you have good knowledge of when the expert policy is truly optimal, and can detect that from the state, then use it instead of Q values in those states. You may still want an $\epsilon$ probability of taking random action instead applied to that choice, depending on whether you want the agent to properly learn about the worse alternatives. I have successfully used this approach for an agent learning a board game, where minimax searches that find forced end game results can easily be tracked and used in preference to Q tables when they apply.

If the environment is highly stochastic - e.g. in a board game using dice - it may not matter that you are using a deterministic fallback policy instead of $\epsilon$-greedy, because the agent will see enough variation in states despite not actively exploring itself.

If the expert policy is not necessarily optimal, but is perfect at avoiding terrible decisions, use it to filter available actions. This would need to be done in production as well as during learning, because the agent will never learn about those terrible decisions unless it experiences them. Although you could also fake that experience and feed it to the update step if the expert is also very good as emulating what would happen - this might be possible if the expert can reliably detect that the episode would end with a certain reward immediately for example.

Any hybrid variation of the above ideas, such as switching between three policies: greedy Q-table, expert and random choice based on evolving probabilities and knowledge of when the expert policy is best applied.


text:
Emulate program behavior using neural network
I have an exe file but no access to its source code. It takes as input a list of 8 parameters and prints text files containing the output. I was wondering if it is possible to write an AI-based program that (given that I am a newbie with IA) can "train" a neural network using the input and output of my exe file to kind of create a black box that emulate the given exe file.

If the program is a black box - that is, it actually computes its output from the values of the inputs within its own code (which means, it does not, for example, go out and read values off some device, or rely on some external service whose results are not predictable) - then, yes, you can write an AI that emulates the program.   Unless the program is fairly trivial, however, the emulation will be approximate - that is, for at least some inputs, the output will not be correct.   How close you can get to adequate emulation will depend on the details of how the inputs are processed, and how much training effort you are will and able to afford.
If you want to do this, you should probably start learning about reinforcement learning toolkits.   Rllib is one popular such tool set, if you're using (as I would assume you are) Python for your experiments.


text:
When using TD(λ), how do you calculate the eligibility trace per input & weight of a neural network neuron?
I have a Neural Network, each Neuron is made up of inputs, weights, and output. I have potentially multiple hidden layers. The activation function executed against the output is not known by the Neuron.
I would like to use TD(λ) to back-propagate errors through the network as it explores options. My understanding is that this is forward looking TD(λ) because I won't know the error until I reach a terminal state, and so an eligibility trace needs to be kept for each input+weight combination as I back-propagate the error between the NNs new output based on the state-change from the last prediction and the output from the last prediction.
To try and modularise my code as much as possible, the neuron won't know the loss function, but will instead be given the error as a derivative of whatever the activation function of its output was. It also won't know if it's in a hidden layer or not, it will just have inputs, weights, and an output
For example:

So when each Neuron is back-propagating the error signal from all its output connections (summed before it receives it), how do I calculate the eligibility trace?

What do you mean by "The activation function executed against the output is not known by the Neuron."? How is this possible? Also, you say "I would like to use TD(λ) to back-propagate errors through the network as it explores options.", but what do you mean by "as it explores options"? You're not talking about options in hierarchical RL, I suppose.

Q1: it's an implementation detail. The code for a "neuron" is unaware of the activation function used. As far as I understand it, this shouldn't matter because the "neuron" in my case will have available the output of the network as well as the derivative of it's output.
Q2: "explores" options I mean to say the NN will make predictions on what to do next, and the change in state that prediction results in is the next input the next epoch

Eligibility traces are not very common in deep-RL, but I suppose that if your network is not huge they might work. I think you are describing the backward view of eligibility traces since you are saying you want to keep a "trace" of each state visited before reaching the terminal state.
In this case, computing and updating the traces to later combine them with the TD-Error could be done as follows. You would need to initialize a $\textbf{Z}$ matrix with the same dimensions as the weights $(\textbf{W})$ for each layer of your network. Then, at each time step you update the trace by simply:
$$ \textbf{Z}_{ij} \leftarrow \gamma\lambda \textbf{Z}{ij} + \nabla_{\textbf{W}_{ij}}O(i) $$
where $O(i)$ is the output of the network. Assuming that $\textbf{O}$ is the last layer and either O1 is selected or O2 (but not both, as in a policy network) this is the rule. $\gamma$ is the usual discount factor and $\lambda$ the trace decay parameter. Once you have this then update the weights with the usual TD learning rule:
$$ \textbf{W}_{ij} \leftarrow \textbf{W}{ij} + \delta\textbf{Z}_{ij}$$
The TD-Error is whatever you prefer (i.e. Actor-Critic or SARSA). To understand better this combined rule, set $\lambda$ to be zero and see how you recover the usual TD(0) learning rule. I imagine you are familiar with Sutton&Barto 2018 fantastic book in RL (not deep RL). This topic is explained in chapter 12.
Recall that at each state (or time step) all the information to compute these rules is available thanks to the forward pass from the input to the output.
Hope this helped.


text:
Performance of augmented dataset with or without original images
I am training on yolo and I had a small dataset. I decided to increase it by augmenting it with rotation, shearing, etc to increase the size and increase accuracy.
Now I have seen augmented datasets labeled as with and without original images.
I was wondering if there is difference between training with and without original images besides there just being more images?

How many image for class in your dataset? Also are your image bird eye or satellite view?

My augmented has 10 k, original has 1k. My images are from monocular camera on a factory press

Thanks for the information. How many different class in you 1,000 original image? If your image collection at the factory pressing machine, is the photos of metal surface?

They are metal and plastic surfaces. So to clarify I have 25,000 images augmented total and 3,000k images original. There are 3 classes.

I think it depending on your input image for testing. If your testing image is similar to original 1,000 images, which is no rotation on the view, I don't think augmented images help increasing classification accuracy.
I don't know what metal and plastic surface condition you want to detect, but 1,000 images per class would be enough for to reach around 80% detection accuracy for simple surface feature, like nick, inclusion, and etc.

This is a good question.  The naïve view would be that there is no difference besides "there just being more images".  However, this is not necessarily the case.
The answer depends on how "true" your augmented dataset is to your original dataset.  In some cases, the augmentations are less representative of the original images.  In that situation, not including the original dataset in your training may be harming your training/model performance more than "there just being more images".  You would also influencing the quality of your training.
In some cases, you can tell a priori if the augmentation is representative of the original.  For example, a a car vs. truck dataset that is flipped horizontally is probably representative of the original.
In situations where you cannot tell a priori, you can train a model with the original dataset and another model with an equal number of augmented images.  In situations where the augmentation is "true", there should not be a huge difference in model performance. (With small datasets, the fluctuations in performance can be "huge", and this would not apply).


text:
How to use structural information in a Transformer?
I am performing a Neural Machine Translation (NMT) task. In my case, input data has relational information.
I know I can use a Graph Neural Network (GNN) and use a Graph2Seq model. But I can't find a good generational model for GNN.
So I want to use Transformer. But then the challenge is how can I embed structural information there? Is there any open source artefact for Relational Transformer that I can use out of the box?
Any pointers?

can you clarify the "relational information"?

Two nodes are connected by an edge with a relation. Think about a knowledge graph.

Would it make sense to define a positional embedding that is a function of the distance between nodes in your graph representation?

@JohnSt.John can you please explain in a bit more detail in an answer. Also can you please refer to an implementation if possible.

Done, hopefully that was helpful!

I am not totally sure that this solution would work for your problem, however I'll take a stab at it, and hopefully this helps!
One of my favorite resources so far for learning about transformers is this post http://nlp.seas.harvard.edu/2018/04/03/attention.html. I was able to follow along in a jupyter notebook and the only changes I remember I had to make to get it working were removing references to Variable. There is a section in there called Positional Encoding which may be relevant for what you want to do. Some people alternatively call this Positional Embedding. The choice of positional encoding is somewhat flexible. I have seen projects, such as Enformer from DeepMind (https://github.com/deepmind/deepmind-research/tree/master/enformer), which use custom position embeddings for the DNA sequence domain, which is centered in the middle of a sequence and extends out to the sides.
The above implementation/paper was very helpful for learning. Once I was ready to implement a transformer for a project I was working on, I found this implementation in the pytorch examples package a great place to start. In there you can see the class for the positional embedding and modify that as needed. https://github.com/pytorch/examples/tree/main/word_language_model
Now for your specific problem I did some googling for positional embedding/encoding and graph transformer, and I found a project that looks promising https://github.com/inria-thoth/GraphiT. Maybe that just does what you want, and their paper https://arxiv.org/abs/2106.05667 may have some useful background for you.
Hopefully these pointers help in your search!

could there be different kind of edges using "Positional Embedding?


text:
How to include additional numeric input in the Transformer architecture
I want to apply the Transformer architecture to my machine translation task, and provide the decoder with an additional parameter in the range of [0,1]. This parameter should provide the decoder with some additional information on how to translate the sequence, think, e.g., how pompous the resulting translation should be.
Although I have some idea on how to approach this task, I would like to ask for help and additional input. I have found a paper by Libovický et al., which studied multi-source sequence-to-sequence tasks. Here are some of the ideas that came to my mind:

Use an additional encoder, to encode this (single) numerical value, and apply one of the approaches mentioned by Libovický et al. This feels strange in the sense that we are encoding only a single element, and attending to it.
In a similar fashion, use an additional encoder, but this time instead of providing a single numerical value, create a sequence of length n, where all elements are the same input parameter. E.g., if the parameter was 0.5 and n = 3, the resulting sequence would be [0.5 0.5 0.5]. Then, use masked multi-head attention to attend to these output values. Note that this approach is based on my intuition only. I lack the needed experience to judge whether this approach could actually provide benefits over the previous one.
Add an additional input to the decoder, applying masked multi-head attention to it, but not necessarily positional encoding. Then, sum up the result of this operation and the "standard" input, resume with this interim result. I believe that for this approach I would also have to create a "fake" sequence, where each element is the same numerical parameter.

To sum it up, I am looking for a way to provide the decoder with an additional input, in my case a single numerical value.


text:
Is there a way use DQN to find the optimal combination of actions (control inputs) and environment parameters?
I am using DQN to find the optimal sequence of control inputs to a dynamic system. The setup is as follows:

At the beginning of each episode, the system is initialized to the SAME initial condition s0.
Each episode spans from 0 seconds to tf seconds. (=each episode takes the same time) A decision is made every h seconds. Thus there are tf/h iterations per episode.
In each iteration, the agent takes an action (He chooses 4 control inputs. For each input there are 3 seperate options to choose from). At the beginning of the next iteration, the agent observes the updated state and selects a new action (again this means that he chooses one of 3 possible values in each of the 4 control channels). Each iteration takes h seconds.
The reward is computed at the end of the episode (i.e. after t=tf).
The state observation includes the current time t. Thus, the agent knows how much time he has left within the episode.

The performance of the dynamic system depends on many parameters. (Considering the agent would control a car, exemplary parameters would be the weight or the aerodynamic drag coefficient of the car.) Changing these parameters will affect the dynamic behavior of the system, i.e. the environment, for the whole episode.
How can I include the optimization of these parameters in the problem? In essence I want to find the optimal combination of time-varying control inputs + parameters.
I though about including the parameters as actions. I then would have 5 action channels (4 control inputs a 3 options each and the parameter with x discrete values). However, considering that changing the parameter is only meaningful before starting an episode, taking this parameter action would only have an "effect" at t=0, i.e. when the first action is taken. (It makes no sense that the parameters are changed within an episode.) As a consequence, this "parameter action" would have zero effect for all t > 0. I do not know if this hinders the learning of the DQN agent, as this would increase the size of the action space.
Another idea of mine was pursuing a bi-level approach. Thus, I first train an agent that knows how to drive the car optimally (wrt. to my reward function). After training I could apply this pretrained agent to a range of cars, each with different parameter settings. Ideally I would then see where he performs best. However, this approach is not really "optimal", as I never optimize for both the control inputs and the parameters in one go...


text:
Normalisation of reward function
Problem
Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the environment becomes 1. The main reason why this is important is that the highest possible score can differ highly between each instance. For example, in one instance, the maximal score can be 50, and in another instance, it can be 150. However, the maximal score is never known.
The environment
The environment of my agent is a single graph of a few thousand nodes, where the goal is to maximise the coverage of a set of $k$ cliques (cliques are subgraphs in which all the nodes are connected to all other nodes in the clique). Each state consists out of the current candidate clique set $D=\left \{C_1,\dots,C_{k} \right \}$ and a newly found clique $C_\text{found}$, which means each state is a tuple of $s=\left (D, C_\text{found} \right)$. The agent then decides if the newly found clique will replace another clique in the set, and so yes, which one. For instance, if the action is $5$, then $C_\text{found}$ will replace the clique $C_5$ in the candidate clique set, and if the action is $k+1$, it will replace none of the cliques in the candidate set. The cliques themself are found through the Bron–Kerbosch algorithm implemented by NetworkX.
The Reward Function
The simplest version of the reward function should be the new coverage of $D \in {s}'$. The function below shows how this is calculated.
$$
R(s,a,{s}')=\left | \text{Cov}(D\in {s}') \right |, \\
\text{where } \text{Cov}(D) = \bigcup_{C \in D}C
$$
However, this reward function runs into the problem I described at the start of my question. Another reward function I experimented with is using the difference in coverage between the two states. This reward can be divided by the maximum clique in the graph, such that the score is normalised to a value between -1 and 1. The maximum clique can either be found through approximation or exactly by first going over all the cliques. Below is the formula of this reward function:
$$
R(s,a,{s}')=\frac{\left | \text{Cov}(D\in {s}') \right | - \left | \text{Cov}(D\in s) \right |}{\left | C_\text{maximum}\right |}
$$
This reward function also runs into some problems. First off, in my current training setup, I have never seen the reward be either -1 or 1, with the highest value seen being around 0.75. This difference is caused by most cliques, especially the largest ones, overlapping with other cliques in the graph. The second issue is that with this reward function, the agent will converge to a local optimum in which it rarely changes cliques in the candidate clique set. Lastly, I believe a proper normalised version of the first reward function will work better because this function is also used in the comparison.
Neural architecture and RL algorithm
Currently, I am using PPO with a highly custom neural implementation, in which a GNN is used to first embed all the nodes at the start of the episode, and another GNN embeds all the cliques at each step. The embeddings of all the cliques are then combined and flattened such that I can use them as input for both the actor MLP and critic MLP. The code is written in PyTorch and PyTorch Geometric. I do not believe the network or the algorithm itself is the issue, but let me know if you think so.
Conclusion
I have tried looking at graph and node attributes, which I could use to normalise the coverage score, but I could not find anything. Recently, I have been looking into adaptive normalisation; however, I am entirely new to this concept and would love to have some advice on it or get some links to implementations of it with PPO in PyTorch. Other advice about how to normalise this score function is also more than welcome. Also, if you need more information, just let me know.

I think your core problem is not reward normalisation, but lack of markov property in the state features. How are you representing the cliques in the state? The ids would be meaningless, and the number of nodes likewise. Potentially a binary vector of "in/not in set" might get close to markov trait, but it might not be enough. If the agent cannot assess the edge data or some proxy of it (perhaps from the algorithm you linked), then it will be very hard for it to track whether it could do better in future, and predict a final return. I don't think that goes away with normalisation.

Please, put your specific question in the title. "Normalisation of reward function" is not specific and it's also not a question.


text:
Neural networks with sparse inputs
I have a task I want to solve with neural networks. The task is predicting a certain vector of dimension K. The problem is that the inputs to the networks are sparse.
The input is a vector of size N, where N is huge (> 1M) and for most cases, the vast majority of the entries (> 99%) in the input vector are 0. Very rare examples however do have almost full inputs. It's clear that the model is hard to train, as there might be huge weight imbalance within examples, while the target vectors need not have such an imbalance.
I do have a working model, but could you point me to some papers / relevant literature about training a network whose inputs are so sparse? Perhaps there are some techniques that could be useful (maybe some preprocessing steps on the inputs, or something along those lines).
Any hint is appreciated!

Can you please put your specific question in the title? "Neural networks with sparse inputs" is not a question and it's also not specific enough.

What is the percentage of fully populated inputs? Can they be thrown away? Did you try usual dimensionallity reduction techniques? Did you try embeddings like in language processing? What else did you try? What kind of network is your working model?

I am not an expert in sparse computation, but this Pytorch tutorial might be a great start.

What I did when I encountered sparse inputs was to preprocess the data to only include vectors with more than n (threshold) nonzero values.
Not really an elegant solution (since you give up a huge part of your data), but you can try tweaking the threshold for best results.

The default solution to handle sparse or one-hot encoded input data is to use embeddings(PyTorch Embeddings).
The binary, n-dimensional input data is then transformed to a small dimensional(e.g., 4-128 dim) vector containing real values.

I'm not allowed to leave comments so I will put it here. I would use some sort of dimensionality reduction algo, PCA or something more sophisticated if need be. This way you can reduce your input to a smaller number of dimensions, which are more easy to handle


text:
In Graph Neural Network is Message Passing Step Agnostic of Output Values during Training?
So Graph Neural Networks is about representation learning where initially representation of graph is learned in the form of node embeddings. My question is: Are the output values back propagated and influence learned node embeddings? The paper I read seems to not account for output being backpropogated so that embeddings get learned. So does that mean learning node embeddings is agnostic of output values during training?


text:
Overfitting problem - poor performance on test data
I'm facing the problem of overfitting and I can't deal with it - I tried experimenting with optimizer, but nothing seems appropriate. My model has extremely poor performance on testing data and the loss even rises. Is there anything I missed during the model architecture planning or training?
I'm working on GTSRB.
n_epochs = 100
n_train = 4000
n_test = 1000

def load_split(basePath, subset_type, n_samples):
    csvPath = basePath + '\\' + subset_type
    #intialize the list of data and labels
    data = []
    labels = []

    # load the contents of the CSV file, remove the first line (since it contains the CSV header)
    rows = open(csvPath).read().strip().split("\n")[1:n_samples + 1]
    random.shuffle(rows)

    #loop over the rows of csv file
    for (i, row) in enumerate(rows):
        #check to see if we should show a status update
        if i > 0 and i % 1000 == 0:
            print("[INFO] processed {} total images".format(i))

        # split the row into components and then grab the class ID and image path
        (label, imagePath) = row.strip().split(",")[-2:]
        # derive the full path to the image file and load it
        imagePath = os.path.sep.join([basePath, imagePath])
        #print(imagePath)
        image = io.imread(imagePath)

        #resize the image to be 32x32 pixels, ignoring aspect ratio, and perform CLAHE.
        image = transform.resize(image, (32, 32))
        image = exposure.equalize_adapthist(image, clip_limit = 0.1)

        #update the list of data and labels, respectively
        data.append(image)
        labels.append(int(label))

    #convert the data and labels into numpy arrays
    data = numpy.array(data)
    labels = numpy.array(labels)

    #return a tuple of the data and labels
    return (data, labels)

print("[INFO] loading training and testing data...")
(train_images, train_labels) = load_split(DATASET_PATH, 'Train.csv', n_train)
(test_images, test_labels) = load_split(DATASET_PATH, 'Test.csv', n_test)

# Normalize pixel values within 0 and 1
train_images = train_images / 255
test_images = test_images / 255

train_labels = to_categorical(train_labels, 43)
test_labels = to_categorical(test_labels, 43)

model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dense(300, activation="relu"))
model.add(keras.layers.Dropout(0.4))
model.add(keras.layers.Dense(300, activation="relu"))
model.add(keras.layers.Dropout(0.4))
model.add(keras.layers.Dense(300, activation="relu"))
model.add(keras.layers.Dense(43, activation="softmax"))
model.summary()

#Compiling the model
model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])

#Fitting the model
history= model.fit(train_images,train_labels, epochs=100, batch_size=4,validation_data=(test_images,test_labels))

Have you tried increasing the batch size? - try 64, 128 or 256
Using a very smaller batch size (4 in your case) may not be optimal for the model to converge to the global optima.

I don't think this is an overfitting problem, it looks more like the model is not learning anything, but it would be useful to also see the performance on the training data to be sure. By not specifying the learning rate in the optimizer, you are using the default learning rate of 1e-2 which seems quite high. Try passing the optimizer with
keras.optimizers.SGD(learning_rate=1e-3)

instead for a learning rate of 1e-3 (or maybe try something even lower like 1e-4). And as Benjamin Cretois has mentioned, the batch size is pretty small. If you want to use smaller batch sizes you generally need a smaller learning rate as well. You can also try using other optimizers like Adam, but I think the learning rate is mainly responsible for the bad performance.


text:
Applications of one-to-one recurrent networks
I'm studying recurrent neural networks. Reading this page where it lists different types of recurrent network architectures, I think think of applications involving one-to-many (speech/sentence generation), many-to-one (a conventional time-series classification/prediction task) and many-to-many (machine translation). But I can't really think of an application where it's one-to-one. What is one-to-one used for?
[Edit] Note that on the page, it says ``Traditional neural network'' for one-to-one networks. But a traditional neural network would be a feedforward network rather than RNN? [\Edit]

I don't understand your last remark 'Note that on the page, it says ``Traditional neural network'' for one-to-one networks'. Why are you making this remark? What is your actual question? Are you asking what would be a 1-to-1 application of RNNs?

Yes to "Are you asking what would be a 1-to-1 application of RNNs". My remark refers to the link (https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks). In there, they've noted on the side of each architecture type the type of application that will use that architecture. For one-to-one, it just says "traditional neural network". I'm struggling to understand what that's actually referring to.


text:
How does SGD training error decrease in subsequent epochs when statistically, it requires that samples in subsequent epochs be i.i.d and they are not?
I have been reading the Deep Learning book by Ian Goodfellow and on pg. 277, they mention:

It is also crucial that the minibatches be selected randomly.
Computing an unbiased estimate of the expected gradient from a set of
samples requires that those samples be independent. We also wish for
two subsequent gradient estimates to be independent from each other,
so two subsequent minibatches of examples should also be independent
from each other

I understand that for any unbiased estimation we require the samples to be i.i.d. so that we ideally end up with a true representation of the underlying data and so the above statement makes sense. However in practice, the samples that SGD sees for the subsequent gradient update (the next epoch) are the same, and it still performs well in the sense that the training error decreases. The authors later mention :

...but of course, the additional epochs usually provide enough beneﬁt
due to decreased training error to oﬀset the harm they cause by
increasing the gap between training error and test error

I know this happens but if someone could explain to me why and how it happens perhaps from a statistical perspective, it would be great! Another way to put it: why does training error in SGD decrease in subsequent epochs even though the samples are not i.i.d anymore?

The samples might be the same in each epoch, but, before starting the new epoch, the training dataset might be randomly shuffled before splitting it into mini-batches (this is the default behaviour in Keras), which helps to de-correlate the samples.
For example, suppose that your training data consists of $D =\{4, 2, 3, 1, 5, 0 \}$. Now, you decide to perform mini-batch GD with batch size $m = 2$. You also shuffle the training data before each epoch, so before splitting the training data into mini-batches. In the first epoch, you could have batches $B_1 = \{3, 0 \}$, $B_2 = \{4, 5 \}$ and $B_3 = \{2, 1 \}$. In the second epoch, we could have $B_1 = \{4, 1 \}$, $B_2 = \{3, 0 \}$ and $B_3 = \{5, 2 \}$, and so on. So, if there was any spurious/noisy information associated with some specific order of the samples, this would unlikely bias the computation of the gradient.
This should give you some intuition about why shuffling might help to avoid the introduction of any bias due to some specific order of the samples.

Got it. I did not know that Keras randomly shuffles after each epoch. As far as I know in Pytorch, the shuffling only takes place once at the start of the training, which is why I assumed that the samples for subsequent epochs are not iid

Hm, it's strange that this is the default behaviour in PyTorch. I would like to know why.

my bad, Pytorch also shuffles every epoch.

Are you sure? The PyTorch documentation says that the DataLoader has a default parameter shuffle=False. It's been a long time since I used PyTorch.

Yes, by default it is false. But when set to true, it shuffles data at every epoch


text:
Convolutional network for multilabel classification in NLP
I am trying to label code snippets and I base on this article: https://arxiv.org/pdf/1906.01032.pdf
My dataset is just code snippets (tokenized as ascii characters) and 500 different labels from StackOverflow. Currently I have around 1,600,000 samples after filtering these with negative votes and less than 10 characters
This is my current implementation of network architecture in TensorFlow:
def build_cnn(config, hparams) -> tf.keras.Model:
    params = config["params"]
    inputs = tf.keras.layers.Input((1024, ))
    x = tf.keras.layers.Embedding(input_dim=1024, output_dim=16)(inputs)

    conv_outputs = []
    for filters, kernel in [
        (128, 2),
        (192, 3),
        (256, 4),
        (512, 5)
    ]:
        data = tf.keras.layers.Conv1D(filters=filters, kernel_size=kernel, padding="valid")(x)
        data = tf.keras.layers.BatchNormalization()(data)
        data = tf.keras.layers.Conv1D(filters=filters, kernel_size=kernel, padding="valid")(data)
        data = tf.keras.layers.Lambda(lambda a: tf.reduce_sum(a, axis=1))(data)
        conv_outputs.append(data)

    # Concatenation: output size is sum of all convolution filters
    concat_output = tf.keras.layers.Concatenate()(conv_outputs)
    concat_output = tf.keras.layers.BatchNormalization()(concat_output)

    concat_output = tf.keras.layers.Dense(480, activation="relu")(concat_output)
    concat_output = tf.keras.layers.BatchNormalization()(concat_output)

    concat_output = tf.keras.layers.Dense(480, activation="relu")(concat_output)
    concat_output = tf.keras.layers.BatchNormalization()(concat_output)

    outputs = tf.keras.layers.Dense(500, activation="sigmoid")(concat_output)

    return tf.keras.Model(inputs=inputs, outputs=outputs)

As my metric I use F1Score from tensorflow_addons:
model.compile(
    loss=tf.keras.losses.BinaryCrossentropy(),
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    metrics=[
        F1Score(num_classes=500, average="micro", threshold=0.5, name="f1_micro"),
        F1Score(num_classes=500, average="macro", threshold=0.5, name="f1_macro"),
    ]
)

After 5 epochs results look like this (later it wasn't really improving):

My first idea was to apply class weights but it didn't help. Next idea was an experiment with only 10 tags (still multilabel) and this is the result:
{
    "step": 12,
    "loss": 0.2092941254377365,
    "f1_micro": 0.6354409456253052,
    "f1_macro": 0.5389766097068787,
    "val_loss": 0.25258970260620117,
    "val_f1_micro": 0.547073483467102,
    "val_f1_macro": 0.4620901644229889
}

Last thing: after I saw some results with evaluation I saw that most of the outputs are empty or only 1 class, but using smaller threshold didn't increase the metric
Any idea how to improve the model for bigger number of tags? is it the metric?


text:
Is attention always better then an RNN/CNN?
We've all read the attention is all you need paper, but is it really all you need? Can you effectively replace any RNN/CNN with an attention transformer and see better results?

Note that some LSTM architectures (e.g. for machine translation) that were published before transformers (and its attention mechanism) already used some kind of attention mechanism. So, the idea of "attention" already existed before the transformers. So, I think you should edit your post to clarify that u're referring to the transformer rather than the general idea of attention, if that's really the case. Having said that, we have had a few similar questions in the past, like this, so you might want to look at them and then reviewing your question.


text:
How are CNN kernels trained when using FFT for convolutions?
CNNs (convolutional neural networks) are adept at processing images, as their construction is based on the biological neural networks found in the human eye. "Kernels", sometimes called "filters", are small feature detectors in the form of small matricies that are slid (called "convulving") across an image detecting features in a sample image. This process is computationally intensive, as each time we slide one or more units we have to multiply each kernel value by the section of feature map we are sliding over.
Recently I found here that you can use FFT to do the convolutions up to ~4.8 times faster than with all that multiplication.
Kernels are regularly trained by backpropagation, treating each kernel entry almost like its own axon weight. However with the FFT method I am at a loss as to how to train the kernels.
So, how do you train the CNN kernels when using the FFT method of CNN convolutions? Is backpropagation still relevant?

Have you checked the backpropagation algorithm (ref: https://brilliant.org/wiki/backpropagation/)? There is a Python code at the end. Please, check if you could apply FFT similar to that.

See my attempt at explaining this. Technical points to consider are the restriction of the convolution to a rectangular domain that can be considered as a separate linear operation, and that the forward mode is usually a "reversed filter" convolution that devolves to the scalar product in the "single output" case, as you also described. The reverse operation in the back-propagation of gradients is then the normal convolution.

I had that same exact question when trying to develop some kind of implementation of the Conv2D using the FFT.
To do this we need to consider that there are three steps that the layer needs to be able to do to actually be able to use it for inference and training:

1° calculate the convolution of the input samples with all of the filters:
$$y_{f'}=\sum_f x_f\ast w_{f'f}$$
2° calculate the gradients (derivatives) of the loss with respect to the parameters of the convolution, if you apply the chain rule carefully you will find that this can be done using a convolution, specifically:
$$\frac{\partial L}{\partial w_{f'f}} = \frac{\partial L}{\partial y_{f'}}\ast x_f$$
3° calculate the gradients of the inputs with respect to the loss (necessary for the layer before this convolution layer to do backprop as well) which can also be done using a convolution:
$$\frac{\partial L}{\partial x_f} = \frac{\partial L}{\partial y_{f'}}\ast w_{f'f}^{T}$$

This means that since you can apply the FFT in all of these steps in the same manner you can apply in the inference step.
If you want to know more you can try reading this 2014 article about doing convolution with the FFT by Yann LeCunn and some of his peers.


text:
1D Sequence Classification with self-supervised learning
I am working on a multi-class classification task on long one-dimensional sequences. The sequence length may vary in the range $[512, 30720]$, and there is one feature associated each time-step in the range. This means that the input to the model is of the shape $(N, 1, L)$ where $N$ and $L$ are the variables for the batch size and sequence length respectively. The singleton feature dimension contains values in the range $[0, 40]$. A small slice of a sequence might look like this: $[0,3,2,1,2,7,3]$.
The standard way of using deep learning (i.e. supervised learning) to solve this task is by choosing some neural network architecture (our model) with an appropriate inductive bias towards the data and encode the features, propagate it threw the neural network, optional pooling, decode the features and finally compute the loss w.r.t the ground truth in that order. In PyTorch this process would look something like this (the model is a basic ResNet block):
import torch

input_shape = (5, 1, 1024) # batch size, features, timesteps
X = torch.randint(0, 14, input_shape).type(torch.float) # sample input
y_true = torch.tensor([0, 0, 1, 2, 1]) # sample ground-truth (3 classes)

d_model = 64 # 64 is our hidden dimension
encoder = torch.nn.Conv1d(in_channels=1, out_channels=d_model, kernel_size=1)

class Residual(torch.nn.Module):
  def __init__(self, m):
    super().__init__()
    self.m = m

  def forward(self, x):
    return self.m(x) + x

model = torch.nn.Sequential(
  Residual(
    torch.nn.Sequential(
      torch.nn.Conv1d(d_model, d_model, 3, padding='same'),
      torch.nn.BatchNorm1d(d_model),
      torch.nn.ReLU(),
      torch.nn.Conv1d(d_model, d_model, 3, padding='same'),
      torch.nn.BatchNorm1d(d_model),
    ),
  ),
  torch.nn.ReLU(),
)

pooler = lambda x: x.mean(2) # average pooling over the length dimension
decoder = torch.nn.Linear(d_model, 3) # 3 output logits for each class

criterion = torch.nn.CrossEntropyLoss()

# training pipeline
X = encoder(X) # (N, 1, L) -> (N, H, L)
X = model(X) # (N, H, L) -> (N, H, L)
X = pooler(X) # (N, H, L) -> (N, H)
y_pred = decoder(X) # (N, H) -> (N, 3)
loss = criterion(y_pred, y_true)
loss.backward()

Assuming that the neural network architecture model is appropriate, I have considered using self-supervised learning (SSL) to boost the final classification accuracy on the withheld test set. I have seen the method gain attention in language modelling tasks with Transformers (i.e. BERT), but how does one apply SSL in the general case outside the language task domain? Is this approach successful without using Transformers?
As I understand, SSL is used on unlabeled data to learn the underlying structure of the data and is followed by a fine-tuning stage - i.e. supervised learning with the pretrained weights. Two SSL tasks that I can think of are 1) masking out certain timesteps and letting the model predict the values of the missing timesteps, and 2) training the model to predict the value of the next timestep given the previous timesteps. I'll use the following image to illustrate:

Intuitively, I think this should work but I am not sure how to implement it in a practical setting with PyTorch. For instance, when predicting the next timestep (red bar) can I treat it as a classification task with (40) classes? Would it work for varying sequence lengths? How big proportion of the training data should be used for pretraining?

Can you please put your specific main question in the title? "1D Sequence Classification with self-supervised learning" is not really a question.

In SSL (language modelling, for example), you do not have any explicit labels, just sequences of words that make sense together. SSL tries to model the language by next-word prediction, but the words it predicts are also parts of the sequence(s). The main goal of language modelling is to create (contextual) representations/embeddings for the words in the language, such that when put together, they preserve as much of the "meaning". Having modeled the representations, then you can change the head (last few layers) and fine-tune it on your classification/prediction task. This approach is important and useful when you want to learn the representations/features of the tokens in the sequence.

How does one apply SSL in the general case [...]?

The better question is, when does one apply SSL? I think it's usually when you want to learning some representations on unlabeled data, followed by the fine-tuning stage, on a downstream task.
I would say SSL is not appropriate in this context. Your data consists of the sequences (features) and the ground truths (classification labels). As far as I understand, there is no unlabeled data that you can use in the SSL way in this context. You don't need to know how to predict values in the sequence (missing/next), because you always know the representations of the features in your sequence (those singleton values).
The difference between this and language modeling is that in LM you do not have fixed, existing numeric values for each word, so you also want to learn these representations (word emebddings are not trivial; What is the best way to obtain them?).

Thanks, I believe that all of the weights in the model are adjusted during pretraining and not just the embedding layers in the case of LM? Word2vec has been a popular approach to learn the input representations and is entirely focused on the embedding layer (not the complete network). With the intuition that pretraining is not exclusive applied on the embedding layers, I thought that pretraining the model on a few input samples would be better than random weight initialization. But I agree with you that my task at hand is probably better suited from a pure supervised learning POV.

Yes, all the weights of the model are adjusted during pretraining. In the case of W2V it's the 2 linear layers (out of which the first is then used as the "embedding" layer for other tasks) , in the case of Transformer models it's the Embedding layer along with all the other attention layers and MLPs. What model are you thinking to use? Are you trying to learn different representations for your sequence elements?


text:
Can vanilla multi armed bandit problems be solved by RL algorithms like A2C and PPO?
Let's say we have N bandit machines with some distributions (assume some are gaussian, some are uniform, some are chi squared). We want to maximize rewards in X amount of time. I am aware that algorithms like Epsilon greedy, UCB can handle this. But is applying an RL algorithm like A2C the right choice to a plain vanilla Multi Armed Bandit problem (N independent machines that we have to maximize rewards from in X amount of time)? Is the notion even correct? Given that there is only always one state.


text:
Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
Sutton-Barto (Section  2.8-Gradient Bandit Algorithms, page 37):

Question: is $\bar{R}_t$  average of all rewards  or  average of  rewards corresponding to $A_t$?

Question: is $\bar{R}_t$  average of all rewards  or  average of  rewards corresponding to $A_t$?

It is average of all rewards seen so far. Usually a rolling recent average, so it slowly adapts to expected results from the current policy. But a mean value is good too.
It is important that the value does not depend on the action choice, otherwise the gradient calculation will not be correct. In fact you can use pretty much any baseline provided it follows this rule, including fixed value guesses. Some baselines work out better than others in practice.
If you have a contextual bandit (or an RL problem using a policy gradient method) then a good baseline choice might be average reward seen from the current state (note this depends on the policy, but not directly on the current action choice which might be an exploratory action). This leads to the various algorithms that use advantage - the difference between Q(s,a) and V(s) - as the scaling to $\nabla \text{log}(\pi(a|s))$.

@ Neil Slater, in the book it says "which can be computed incrementally as described
in Section 2.4 (or Section 2.5 if the problem is nonstationary)". However, in sections 2.4 and 2.5, we calculate average of rewards for a given action, not the average of all rewards seen so far. That was the reason why I am confused.

@user3489173 I think you are expected to extract the averaging technique from that reference, not what was being averaged.

@NeilSlater  Here in this problem statement it is said "On each step, after selection action $A_t$ and receiving the reward $A_t$, the action preferences are updated by...:".  I am confused because after you apply $A_t$, you obtain reward $R_{t+1}$. Hence. why in the update equation 2.12, do we not have $R_{t+1}$ instead of $R_t$?

@DSPinfinity The choice of labelling state, action, reward in a trajectory $s_t, a_t, r_t$ or $s_t, a_t, r_{t+1}$ is one of convention, with the majority of sources I have seen using the latter. Something to be aware of when using different learning sources, as it will change the notation in some of the equations, proofs etc

@NeilSlater I think Chapter 2 and Chapter 3 of Sutton-Barto are not consistent. Let me prove this. Chapter 2, page 26: "We denote the action selected on time step t as $A_t$ and the corresponding reward as $R_t$". Chapter 3:, page 48, Figure 3.1: "One time step later, in part as a consequence of its action, the agent receives a numerical reward of $R_{t+1}$". Do you agree on this consistency as well?

@DSPinfinity Well chapter 2 is about bandits, and time steps are entirely separate interactions with the bandit environment. It would make no sense at all to have action $a_t$ associated with reward $r_{t+1}$ when talking about bandits. I missed that in my comment above that the source was S&B

Perfect answer, Now clear! Thank you so much.


text:
Are there any guidelines on how to map the state space to integers in the case tabular RL algorithms?
Let's say that you want to solve a problem with a tabular reinforcement learning algorithm, for example, Q-learning. You can represent the value function $Q(s, a)$ as a $|\mathcal{S}|\times |\mathcal{A}|$ matrix. In practice, we could use an array to represent $Q$. To index this array, you need integers, unless you use e.g. hash maps or another data structure. Let's call the (NumPy) array Q, then Q[i, j] would be the value of action j at state s.
My question is: are there any guidelines on how to map the state space to integers, so that we can index our table? In the case of a grid environment, you can try to enumerate all possible states and use some convention to map a state to an integer. However, in more complex environments, this might not be straightforward. Do you know any other ways to try to solve this implementation problem?
Note that this is a design/implementation question rather than a programming issue. I am looking for guidelines or common practices (other than the one that I mentioned). Feel free to share with me examples of how this has been done in other projects/environments (links to the projects/repos are also appreciated).

It's hard to prove a negative, but I don't think there are any special considerations for reinforcement learning (RL) when enumerating and tabulating discrete states and actions. Instead, this is a computer science data structures issue. Depending on the problem at hand, it could be easy or hard to map states to something concise and useful. Unless there are hard constraints on the available memory or CPU, then it is common to pick a simple solution that fits into available RAM.
One important detail is that with tabular learning there is no meaning to records being adjacent or close. All Q values are completely independent. Which means that any map to a position in a table can be completely arbitrary. This is true even if your tabular states are a discretised approximation to a larger state space (although how you perform that discretisation is important - you should make some effort to group states with similar expected returns and next states). Which in turn means the primary concerns are:

Compactness. The state ids should ideally be short, and also the table of all states should not be wasteful.

Simplicity. Ideally the conversion between observable/usable state information for the environment to process actions, and the storage format for the table should not be taxing so that it takes lots of effort to code it.

Speed. Related to simplicity, but sometimes at odds with it, both converting the state to a table entry and accessing the table entry should be fast so that the machinery of RL is not a limiting factor when running the agent.

These three concerns interact, so you may need to compromise. Multi-dimensional arrays are faster and more compact in general than dictionaries/hashes, so it is common to use over-specified integer states, provided there are not too many wasted indices.
In examples in literature, you will commonly find:

Converting state representation into an integer, either using binary digits, decimal digits, or a "flattened index" (e.g. if state consists of 3 different traits A, B, C which can be enumerated as A $\in$ [0,5], B $\in$ [0,3], C $\in$ [0,7], then create a state_id = A + B * 6 + C * 24)

In the example, B is multiplied by the length of A, so that any allowed combination of A and B is a unique numer, without gaps. Then C is multiplied by the product of sizes of A and B for the same reason. Many implementations of multi-dimensional arrays do this internally (although they might start with C as opposed to A).
You could also implement the Q table itself with more dimensions, effectively having NumPy (or whatever) do the calculation for you. So you would use e.g. Q[A, B, C, action].

Concatenate state traits into a string or binary representation, and use that as a key to a hash/dictionary structure.

You can combine the two methods. If there would be lots of gaps of unreachable states from a numerical state id, then you may still prefer to use a dictionary for compactness. Integer keys to hashes may be faster to process and take less memory than string keys.
There are also other structures, such as tries, that may be more efficient for certain kinds of state representation. However, on a modern CPU, if you need extra efficiency to represent a tabular action value function, you may be hitting the size of problem where the speed improvement of generalisation from some kind of function approximation is more desirable than attempting to solve the MDP perfectly by visiting all state/action combinations exhaustively.

Can you explain why you do state_id = A + B * 6 + C * 24? If you know of any literature or implementations that illustrate different ways to implement e.g. the array or hashmap with strings, that could helpful for future users that want to have a reference. Your trie suggestion is interesting. I never thought of it, but it might be really possible to do it with a trie too.

@nbro: I added an explanation of the example.


text:
When training a seq2seq model is it better to train using the models outputs or expected outputs?
When training any seq2seq model you have a target and a source. The source may be a sentence such as:

I_walked_the_dog

And the target being

_walked_the_dogg

Where as you can see the expected output for the initial I is a space _. My question is, at training time, whether to use the models previous outputs for predicting the next output, or to run the training simultaneously using the expected outputs. To illustrate this more clearly, see below:

The incentive for training using the expected outputs is that all time steps can be trained simultaneously, so it speeds up training by a factor of the sequence lengths. However, it means that training is not representative of what the network will realistically be doing at testing time, as at testing time the network will not have perfect previous outputs, but rather only it's own.

I think you're referring to teacher forcing.


text:
Why my classification results are correlated with the proportionality of my data?
I'm facing a problem. I'm working on mixed data model with NN (MLP & Word Embedding). My results are not pretty good. And I observed that the proportionality of my data are corelated with my classification results. I explain:

As you can see, I have more LIVB than others data. The problem is that the predictions of my model are only LIVB
And I don't understand why ? Is it a high variance ? Is it a high bias ? What methods for classification problem should I use to detect the error ? Should I have more features ? Is my model is wrong ? Can someone has this problem before ?
Thanks for your help !

I see two main issues here:

you have really few data
you're using a generic MLP

What you observe if just overfitting. You multi layer perceptron is just learning to predict the majority class cause that's the class that lead to the lowest error possible when chosen all the time.
For sure more features will help, along with a different architecture (CNN would be a start). But considering how few training instances you have, i wouldn't expect great results anyway.
probably to maximize you're chances of training something that will learn to predict also the minority classes with only support 5 you should consider moving to finetuning pretrained models like BERT.

Despite how software might work, neural networks do not return labels. Neural networks return probabilities of class membership (typically fairly poor ones, which is a topic for a separate question). If you make probability predictions instead of having your software tell you the most probable category, I expect you to find that you have more diversity in those predictions than “LIVB every time”.
What’s happening is that LIVB is the most likely category going into the problem, and you need considerable evidence to shake your prior belief that LIVB is the most likely outcome. You are unable to produce enough evidence for another category to shake the mode away from giving LIVB the highest probability. Thus, this seems to be a matter of bias: your model lacks the ability to strongly discriminate between categories and tends to fall back on its prior probability that LIVB is most likely.
Annoyingly, it might be that this is just how your problem works: LIVB might always be the most likely outcome.
Finally, I agree with other comments that there are too few observations for a neural network to have much of a shot of being useful. Neural networks are a great way to get a lot of discriminative ability in order to get the model to scream, “This is not LIVB!” However, you probably lack the data needed for a large network not to overfit.

In my understanding , there could be multiple problems here:

Try to check the label on the training data manually -- you will be surprised to see that the training data could be mislabeled (i.e. all your training examples could be similar to LIVB). Also do the same thing for your test data.

You could varying the parameters of your MLP.

You could try context based embedding like a sentence encoder with a CNN as pointed out by @edoardo-guerriero.

Also instead of MLP, try with a simpler algorithm like multi-class logistic regression. If the logistic regression works good with your current word embedding, then you know the issue is with the MLP ( neural network) architecture.


text:
Why does triplet loss allow to learn a ranking whereas contrastive loss only allows to learn similarity?
I am looking at this lecture, which states (link to exact time):

What the triplet loss allows us in contrast to the contrastive loss is
that we can learn a ranking. So it's not only about similarity, being
closer together or being further apart, but now we want to learn how
much closer am I compared to another image.

The contrastive loss
$L(A, B) = y|f(A) - f(B)| + (1-y)max(0, m-|f(A) - F(B)|)$
would push similar samples together, and dissimilar samples apart.
The triplet loss
$L(A, P, N) = max(0, |f(A) - f(P)| - |f(A) - f(N)| + m) $
would push the positive close to the anchor, and the negative away from the anchor.

I fail to see why the quoted claim is or isn't true in either of these losses. To me, it looks like "same" samples are pushed together, and "different" samples are pushed apart by both.
Furthermore, with the contrastive loss, the distance in the embedding space would be, as I understand it, the ranking- which is claimed to only exist with the triplet loss.
Is there a clearer reference for this, or just a simple answer?

In the first you compare A and B. So if A is closer to B and y= 1 then it's better, that's it.
In the second, you compare A to P and A to N simultaneously.
So if A is closer to P it's good, but it's better if A is more distant to N. So with triplet you have a ranking.
I hope this was clear.

I think the presenter is referring to ranking in terms of the sample triplet.  With the contrastive loss, we take a pair $(A, B)$ and simply measure how similar $A$ is to $B$.  With triplet loss, we are measuring the similarity between an Anchor image $A$ and a positive sample $P$ and negative sample $N$, so intuitively we are comparing the similarities $s(A, P)$ with $s(A, N)$.  In the lecture, she presents this math.
$$||f(A)-f(P)||^2<||f(A)-f(N)||^2$$
which is how we derive the formula for triplet loss.  Triplet loss enforces this condition, i.e. that the distance between the embeddings for $A$ and $P$ are closer than the distance of the embeddings between $A$ and $N$.  Think of rank as an ordering of the distances $s(A, P)$ and $s(A, N)$.  We want to achieve the rank condition expressed in the above equation.  Therefore, the triplet loss measurement will be in terms of rank because it is dependent on $s(A, P)$ compared to $s(A, N)$.  To summarize, contrastive loss is learning similarity by pushing apart dissimilar embeddings and pushing towards each other similar embeddings based purely on the distance between the embeddings.  Triplet loss is pushing together similar embeddings and pushing apart dissimilar embeddings concurrently by enforcing a ranking condition where $s(A, P) < s(A, N)$.

Please don't upload math equations and text as images, it isn't searchable. There are tools available to convert PDFs to text and MathJax. Thanks.


text:
What architecture would be best to match images of torn pieces of tapes?
I am currently working on a project where the goal is to create a neural network that can determine if two pieces of torn tapes are a true fit or not. My current idea is a convolutional network that takes as inputs  2 pieces of tapes (256,256,2) and outputs a 1 or 0 if they are a true fit or not. The 2 channels are independent of each other, so I use grouped convolutions. Is there a better way to go about this? Any ideas would be appreciated!
Example of the data
Channel 1 (tape-1)

Channel 2 (tape-2)

These pieces of tapes are an example of a true fit

I would consider just trimming the black space on the left, flipping one image vertically and horizontally, concatenating the images left-to-right so the remaining black space is in the middle, and training a model based on those images

Why not try a Siamese network architecture?
This architecture is used to compute the similarity between two inputs by applying the same neural network $f$ to two pairs of inputs $(X_b, X_p)$ and $(X_b, X_n)$ and computing a loss function which is minimised when the distance between $f(X_b)$ and $f(X_p)$ is smaller than the distance between $f(X_b)$ and $f(X_n)$ plus some margin, where $X_b$ is any training sample, $X_p$ a matching training sample, and $X_n$ a non-matching training sample.
Your use case is not the typical one for Siamese networks (e.g. checking that two faces are the same), but it has the same structure, so it could work well.
This also requires some preprocessing of the data to make these pairs, but doesn't increase the size of your dataset given that you are already creating all the pairs of input images.


text:
Usability of power series in AI analysis
In mathematics, power series is given by
$$f(x) = \sum\limits_{n=0}^{\infty} c_n (x-a)^n$$
where $c_n , a \in \mathbb{R}$
Although most of the courses in academics cover moment generating functions in probability theory for AI, which is power series by itself, I didn't encounter yet an application or analysis that contains power series in either textbooks or research papers.
So, I want to know whether the power series has any usability in any branch of AI.

There is at least a TaylorSwiftNet (Arxiv). They model a (finite) Taylor series to predict the evolution of the hidden state, and a decoder to predict the observed state in a future time step.

In this work, we therefore propose to approximate the motion in a
video by a continuous function using the Taylor series. To this end,
we introduce TayloSwiftNet, a novel convolutional neural network that
learns to estimate the higher order terms of the Taylor series for a
given input video. TayloSwiftNet can swiftly predict any desired
future frame in just one forward pass and change the temporal
resolution on-the-fly. The experimental results on various datasets
demonstrate the superiority of our model.


text:
BERT2: How to use GPT2LMHeadModel to start a sentence, not complete it
I am using GPT2LMHeadModel to change the way GPT2 choose the next word in a sentence. At this point, I have to give the initial part of the sentence and GTP2 starts to predict the better next word.
I want GPT2 to read an entire sentence and then start a new one based on that (like it does with translation)
Is there any kind of parameter that I need to set up in order to make GPT2 start a sentence from zero, not complete an initial one?

You should be able to do this by feeding a period at the end of the sentence you're feeding to GPT2. It will understand that it needs to start a new sentence after that (because the most probably token after a period is the start of a new sentence)

It's still trying to "complete" the context, for example: I gave "A company director has a pecuniary duty."  and gtp2 come up with "A company director has a pecuniary duty. It's not to make money, but to serve customers". It didn't started a new one based on the first. I am using a paraphrase pretrained model.

What would you have wanted as a completion from the model after this first sentence? This will help understanding what you want to achieve.

A paraphrase, like:
"A company director has a pecuniary duty." .....
"A CEO has monetary responsabilities"
Something like that

Based on our conversation in the comment section, what Mucida wants is a reformulation of the input, e.g. if the input is:
"A company director has a pecuniary duty"
the output should be:
A company director has a pecuniary duty or "A CEO has monetary responsabilities".
By default, GPT2 returns what could be the next sentence in a longer paragraph, e.g.:
"It's not to make money, but to serve customers".
When you want large language models like GPT2 to give you a certain type of answers, what usually works well is to give it a few examples of what you want as an input.
For example, you'd give as input a few pairs of reformulations:
"What's a reformulation of "A company director has a pecuniary duty"? It's "A company director has a pecuniary duty". What's a reformulation of "Stackoverflow is a great place to ask questions"? It's "Stackoverflow is where you get the best answers to your questions". What's a reformulation of "Elon musk is the richest person on earth?" It's"
Then, the ouput of GPT2 should complete what comes after "It's" in the same style.
You should check what's the context length of the GPT2 model you're using is. Otherwise, if you feed an input that's longer than the context length, it's not going to be taken into account in its entirety by the model.

I think I will have to create my own decoding algorithm because it didn't work. I tried with a pretrained model for translation and for paraphrase. It stills try to complete a sentence

Can you show me the output you got from the model? Usually, getting what you want out of a big language model is a matter of crafting the right prompt.

If a start the sentece with "A company director has a pecuniary duty.", it completes with:   "A company director has a pecuniary duty."I'm not going to say that I'm going to be a good manager," he said" ... word by word each iteration. A print would be easier to show you hehe

Did you try what I suggested? Giving a few examples of the type of completion you want as an input to the model? It seems to me you just tried again with a single sentence as an input

I did, but didn't work. Sometimes it only repeats the first sentence, or completes with a random one. But I tried with only 6 pairs of sentences as examples, I will  try with more

I would try to figure out what's the context length of the model you're using

Is there any way we can get in touch?


text:
How can batch prediction make drift monitoring easier than online prediction?
In this video, I learned that drift monitoring would be easier in batch prediction than that in online prediction:

But I don't know why and I cannot find any information about it googling. In my opinion, in online prediction we only need to keep the prediction records and we have all the history data for drift analysis. What are the differences here between the two types of drift monitoring?

You're correct that, if you keep a record of all the predictions you've made, then you can monitor drift when doing online predictions. However, you probably need an extra database to log the predictions, and also a periodic job which calculates drift reading off that database. That can make for a complicated setup.
With batch predictions, if your batch is large enough, you can calculate summary statistics on that batch and detect drift at the same time as performing the prediction task. This is a lot easier to implement.


text:
What is the right way to find the alphas in this equation?
In the Grad-CAM++ paper the following equation (7) is posed (written here without the relu function):
$$
    Y^c =
    \sum_k \Bigl( \Bigl\{ \sum_{a,b} \alpha_{ab}^{kc}
    \cdot \frac{\partial Y^c}{\partial A_{ab}^k} \Bigr\}
    \Bigl[ \sum_{i,j} A_{ij}^k \Bigr] \Bigr)
    \qquad (7)
$$
$Y^c$ is supposed to be a function of the $A_{ij}^k$, and $\alpha_{ab}^{kc}$ are unknowns to be determined.
In order to isolate the alphas the paper computes partial derivatives w.r.t. $A_{ij}^k$ on both sides of the equation.  In the computation no partial derivatives of the alphas with respect to $A_{ij}^k$ occur, so it seems that the alphas are considered independent from the $A_{ij}^k$. However this does not look right.  To illustrate the problem assume that we want to find $\alpha$ in the equation $\alpha x = x^2$. If we follow the method used in the paper, i.e. differentiate both sides of the equation w.r.t. $x$ while assuming that $\alpha$ does not depend on $x$ (hence $d\alpha/dx = 0$), we will get $\alpha = 2x$, which is clearly incorrect. In fact differentiating both sides with respect to $x$ yields $\frac{d\alpha}{dx} x + \alpha = 2x$, which is not going to help isolating $\alpha$. The actual solution in this case is better obtained by dividing both sides by $x$, so $\alpha = x^2/x = x$, $x\neq 0$.
So, what it would be a right way to obtain alphas satisfying equation (7)?

Equation (7) in the Grad-CAM++ paper is linear.  In fact for a given class $c$ we have just one equation and many unknowns (the $\alpha_{ab}^{kc}$), hence the equation is underdetermined and will have infinitely many solutions. The general solution will consist of picking an $\alpha_{ab}^{kc}$ with non zero coefficient, and assigning arbitrary values to the other alphas. Then, the $\alpha_{ab}^{kc}$ picked can be written as a linear combination of the other alphas.


text:
Why exclude the first entropy bonus term in the soft Q-function in SAC?
Based on OpenAI Spinning Up description of Soft Actor Critic (SAC) the soft Q-function is defined as

and as they say

Q value is changed to include the entropy bonuses from every timestep except the first.

I feel like it should make sense somehow, but they do not give any further explanation, and I don't see why it is correct. Especially because in the soft value function the first bonus term is also used:

Could someone please explain this?


text:
How does learning the moves of chess show up in a neural network?
Is learning the moves a special case or just the same sort of thing that happens as the AI learns strategy? If you take two different neural networks and teach them each how the pieces move, what checkmate is, etc. will the two networks look identical or is there a random element that means that two networks with the exact same number of nodes, running on the same processors, etc. will not look identical even though they both know exactly the same things?

Perhaps part of your confusion: you generally don't teach the neural networks how the pieces move and what checkmate is. You just teach them directly how to suggest good moves, and an external program is used to ignore illegal move suggestions and pick the best legal move from the net's suggestions.

How does learning the moves of chess show up in a neural network?

If you have 2 neural networks that have been initialised in the same way, are trained with the exact same sequence of samples, and there is no other source of randomness, then they will be identical at the end of training.
So, the answer to your question is: it depends on the learning algorithm, how you initialise the neural networks, and how you sample the data and feed it to the neural network. There's no magic!
Finally, what they learn is encoded in their weights and how these weights are used to produce the output (i.e. the architecture). It's not easy to explain intuitively what the neural networks are exactly doing to produce the output (so they are sometimes called black box models), but there are explainable AI techniques that can be used for this purpose.

would it be possible to look at the network and identify, e.g., this is where the knight move is encoded? how long before the way the knight moves is stored accurately?

That sort of question is an open area of research in general, though as @nbro mentioned there are techniques for generating explanations (with varying degrees of accuracy), along with alternative interpretable models that may or may not have the same power as blackbox models.


text:
Can you make a Neural Network drunk or high?
We know that the human brain can become sozzled by various substances that are released into the brain, but can you make an artificial neural network drunk or high? For example on a RL Agent that still does the task somewhat but behaves silly?

In a way, the original Deep Dream was kinda like this - a network designed for a serious task modified to take it to absurdity

How do you define silliness?  For example, making a NN behave randomly (e.g., making random predictions independent of the input), is pretty trivial.

To achieve "abrupt" behavior we can try to multiply the network weights by some constant > 1 (10 or 100 for example, depends on wanted degree).
If we want sillier behavior - we can dropout/zero some weights portion.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
Implement 4D convolution as matrix-matrix multiplication - paper is confusing!
I am confused by this paper https://arxiv.org/pdf/1410.0759.pdf which displays on page 4 how to model a 3D convolution (input has more than 1 channel and filter has more than one output).
In this example, each filter kernel of one output dimension is reshaped into one single row.
The input channels are stacked above each other where a sliding window with the size of the filter defines the content of one row.
In this example, the number of columns of the filter matrix and the number of rows of the input matrix are equal, but this doesn't have to be.
Let's say the input data are 4x4 blocks - this means the input matrix still has 4 columns (because the filter is 2x2) but now has 9 x 3 = 27 rows which is more than the 12 columns of the filter matrix.
This doesn't make sense to me?

Can you please explicitly ask a specific question, which you should also put in the title? I don't see any question here. The last sentence is not really a question that we can answer.

It would be best if you could give a specific example and then ask a specific or multiple specific questions. I'm not sure what you mean.


text:
Using reinforcement learning for human-robot interaction
I have a scenario where a user is wanting to exercise and improve over time. They attend around 10 exercise sessions, doing 20 repititions of an exercise each session.
I want to develop a reinforcement learning agent to help the user get better at the exercises. The agent will observe the user exercising, and be rewarded when the user performs well (say, by completing a lot of repititions). The agent will then take actions to change the exercise routines to adapt to the skill level of the user.
Furthermore:

the RL agent will have no knowledge of the user's skill level before entering the first session
the user's skill level should increase from session to session, but at an unknown rate

I know that I must employ model-free RL methods, as the agent must be able to create unique policies for each user it works with. But I have three questions:

How do I determine what initial policy to follow during the first sessions?
I realise this problem description is vague, but generally speaking, would this be enough data for the agent to work with to create a policy quickly that performs well?
Does the fact the user gets better over time create a problem for the RL agent? As it would be like chasing a moving target?

Please, ask only one question per post. Edit your post to fix that and put your main specific question in the title. "Using reinforcement learning for human-robot interaction" is not a question.

Your question is difficult to answer because it is rather vague. Whether or not you can train an agent on your data depends on a few things. The two main things are: how much data do you have and how well do they represent the problem.
The first thing to do is to model the environment. What is your action space? What are potential states? What is the goal you want to optimise to? The answer to these questions determine what policy to use to reach that set goal. e-greedy is one of the simplest policies. I'd try that first and see how it performs. Even if it performs badly, you can find out why and then pick a better policy that will overcome these short comings.
As for your last question, it shouldn't create a problem if you model this aspect well. The goal itself, if well defined, would not shift. The performance of the user does and with every iteration the gap between performance of the user and goal should decrease.


text:
I have a 3 class classification problem. Detection of one of classes is very important. How to design the problem? one class classification or ...?
I have a 3 class classification problem. Correct detection of one of the classes is very important. How to design the problem:

one class classification?
a normal 3 class classification?
two distinct 2 class classification?(I mean, for example I have 3 lables A,B,C , and detection of class B is very important. Design a binary classification A,B and another binary classification B,C ?
or ....?
Please recommend me!

What have you found and tried so far? Share with us that info. Also, what do you mean by "design a problem"? Maybe you mean "design an algorithm or model to solve this problem" or "describe the problem mathematically".

When one class is more important than the others, you can simply use a higher sample weight for those and use normal three-class predictor.
Alternatively you can bias the network to a desired direction. Let's say the detection of class B is more important, and the model predicts class probabilities of 50%, 40% and 20% for classes A, B and C. You could "override" the prediction to B when ever its probability is above a certain threshold, for example 35%. Naturally this will increase the odds for false positives.
Maybe this threshold tuning is easier if you use "B vs. the rest" classifier first, you'll get clear false-positive vs. false-negative plots for given thresholds. The secondary classifier would be trained for an "A vs C" problem.

May be worth linking some examples of sample weight in use. In some libraries it may be called class weight (it can be viewed as "weighting a sample due to its class" or "weighting all samples of given class") so the name varies

Maybe, but I don't know which library the OP is using. I expect it is trivial to set the weights, and if not he may ask an other question.


text:
Why is my convolutional neural network failing to classify user inputted images after having high accuracy in testing?
My CNN was trained on the Kaggle A-Z Dataset and consists of:
      model.add(Conv2D(32, (5, 5),input_shape=(28, 28, 1), activation='relu'))
      model.add(Dropout(rate=0.15))
      model.add(Conv2D(64, kernel_size=(3, 3), activation='relu', input_shape=(28, 28, 1)))
      model.add(MaxPooling2D(pool_size=(2, 2)))  # choose the best features via pooling
      model.add(Dropout(rate=0.15))
      # flatten since too many dimensions
      model.add(Flatten())
      # fully connected to get all relevant data
      model.add(Dense(64, activation='relu'))
      # fully connected to get all relevant data
      model.add(Dense(128, activation='relu'))
      # one more dropout
      model.add(Dropout(0.4))
      # output a softmax to squash the matrix into output probabilities
      model.add(Dense(26, activation='softmax'))

Data used was split into training sets and test sets. Oversampling was used to account of the unbalanced dataset. The final accuracy was over 95%. Therefore, I expected my program to be able to read alphabetical letters with ease so I loaded one in:
def predictImageFromDrive(model):
  imageLocation = filedialog.askopenfilename()
  #image_numpy = cv2.imread(imageLocation, 0)
  image_numpy = cv2.cvtColor(imread(imageLocation), cv2.COLOR_BGR2GRAY)
  #print(image_numpy.shape)
  #print(imageir.shape)
  imageir = image_numpy.reshape(28,28,1)
  #print(imageir.shape)
  test_image = np.copy(imageir.reshape((1, imageir.shape[0], imageir.shape[1], imageir.shape[2])))

  print(test_image.shape)

  prediction = model.predict(test_image)
  classification = np.argmax(prediction, axis=-1)[0]
  print(prediction)
  plt.imshow(image_numpy)
  plt.title("Classification: " + chr(ord('@')+classification+1))
  plt.show()

The model now seems to predict every letter that I put in as part of the same category. For example, if I input an image of letter V, it shall predict it as an A, if I input an image of letter B, it shall classify it as an A and so forth. Why is this happening?

Check the numerical representation in your prediction code with the prepared dat ain training. Is the range of values for each pixel similar?


text:
ML algorithm suggestion for databases that change a lot with time after model training
I have a classification problem and I'm using a logistic regression (I tested it among other models and this one was the best). I look for information from game sites and test if a user has the potential to be a buyer of certain games.
The problem is that lately some sites from which I get this information (and also from where I got the information to train the model) change weekly and, with that, part of the database I use for prediction is "partially" different from the one used for training (with different information for each user, in this case). Since when these sites started to change, the model's predictive ability has dropped considerably.
To solve this, an alternative would be, of course, to retrain the model. It's something we're considering, although we'll have to do it with some frequency given the fact that the sites are changing every couple of weeks, considerably.
other solutions considered was the use of algorithms that could adapt to these changes and, with that, we could retrain the model less frequently.
Two options raised were neural networks to classify or try to adapt some genetic algorithm. However, I have read that genetic algorithms would be very expensive and are not a good option for classification problems, given the fact that they may not converge.
Does anyone have any suggestions for a modeling approach that we can test?

It sounds to me that while the data is changing every week, it is still in the same domain. That should make things easier. You need a neural network that generalises well. Faster RCNN with ResNet as a backbone can be used for classification. It is a state-of-the-art network that generalises very well. You could also try Yolov4 with Darknet as a backbone. It is also state-of-the-art. On benchmark sets, Faster RCNN has a higher mAP while Yolov4 is faster.


text:
How to compare images for similarity?
I'm working on a system that reads 3 images per second and stores them in a collection.
For each image I have all its keypoints and descriptor vectors, using ORB detector. On average there are 250 features in an image
To add a new image to the collection, I need to check if a similar image already exists or not.
What is the most efficient way to compare the images using the descriptors/keypoints?
what I tried?
Brute force
compare each descriptor vector in the new image with each one of all the images in the collection. This takes a lot of time that increases gradually with the collection size.
Reference vector comparison
Decide on a reference vector, for example (1,1,1,....,1), and calculate the distance between each descriptor vector and the ref vector, then use the distance as a metric to determine descriptor vectors similarity: if 2 descriptors have the same distance from the ref vector, then they're similar.
While this is very efficient computationally, it doesn't yield the correct result in most cases. especially in ORB where the distance is hamming distance.
Reducing to smaller set of vectors
This is a half cooked idea: From each group of vectors with a certain distance to the reference vector, take the K nearest vectors to the ref vector.
The idea is to reduce the number of descriptor vectors to compare with.

I would like to share some ideas on how to make the "brute force" approach (more) feasible. The theme throughout all of these ideas is to have the vector you are using to represent an image to be as small as possible.
Assuming you want to use descriptors/keypoints as described in your question:
In this case, you want to transform your vector with high dimensionality into one with low dimensionality. Essentially, you want to throw away all unnecessary information. If this transformation is a linear map, you would be looking at PCA (Principal Component Analysis). For an example, see Principal Component Analysis for Dimensionality Reduction in Python.
Alternatively, you could use an autoencoder (neural network), which would be capable of expressing non-linear transforms as well (assuming you use non-linear activation functions).
An auto encoder usually has a part that encodes the input into a smaller vector, and another part that decodes that smaller vector back to the input. After training both, you'd take the output of the encoder, which receives an input (e.g. an image) and outputs a vector. Note that training the auto encoder is done once on a powerful machine.
For an example, see Intro to Autoencoders.
If you don't actually care too much about the descriptors/keypoints:
You could consider training a neural network to find a vector representation suitable for image similarity for you. For an example, see Image similarity estimation using a Siamese Network with a triplet loss. Since you mentioned in a comment that you are building a mobile app, note that there are neural nets for mobile vision applications, like MobileNet.
Regarding the time complexity
Keep in mind that what we are doing is applying a map from an input vector (e.g. image, keypoints, descriptors, etc.) and thus reducing it to a smaller vector. This transformation only needs to be done once; afterwards, you can cache the result. The time complexity of this transformation depends on the method and model you are using; an encoder with more parameters is more "expensive". You can find more information about PCA and its time complexity in the scikit-learn user guide.

Thank you Daniel!
The auto encoder will receive and image and outputs one vector? i'm concerned though about the overhead and time complexity of the model, if it's less efficient that ORB detector.
same concern about time complexity for PCA.
My use case here is that im building a mobile app, so this calculation is being run on less powerful hardware, also adding more external libraries will also impact the performance of the app, hence im considering mainly "old school" methods

I edited the answer to hopefully answer your questions. Keep in mind that these ideas aren't that new; for example, take a look at this paper from 1991: Eigenfaces for Recognition (in particular step 5 under "Summary of Eigenface Recognition Procedure" on page 6 of the PDF).