text:
How to calculate the total number of inputs in CNN?
I search this kind of question for a while and I find many discussions involve on counting the number of parameters of a Convolutional Neural Network, but not on the inputs. Using the Fashion MNIST dataset as an example, each black and white image has $28 \times 28 \times 1$ pixels and there are 60,000 images in the training dataset. Does that mean we have total number of $28 \times 28 \times 60,000=47,040,000$ inputs for the input layer of CNN?
My partner critics my baseline/simplest CNN model (for demo purpose) with just one convolutional layer with 10 filters/kernels (the kernel size is $3 \times 3$
), padding has been used and strides = 1. The Keras model information is listed below. He says the training set only have a sample size of 60,000, but you have 78,510 parameters. He concerns over-fitting issues because I have more parameters than the inputs.

I really don't know how to explain the concepts to him clearly that the inputs of CNN are pixels. Could anyone help? A more detailed explanation will be very helpful and I am also happy to learn!

Let's say you are training a CNN to classify whether an image has a dog or a cat. Your dataset has only one (grayscale) photo of each, but they have a high resolution of 6000 x 4000 pixels. Wouldn't you expect the network to overfit, even though you have 2 x 6000 x 4000 = 48 million input pixels?
It will overfit, because after all it has only seen one dog and one cat. It might learn to make the distinction based on the background, or other irrelevant information. The network needs sufficiently many examples of each class to learn what is relevant and what isn't. Increasing the number of pixels / image doesn't change this.

Thank you. Are there any rules apply to CNN to compare the number of inputs and the number of parameters? And also for the sample size, for sample size, I am referring to the data points here. For traditional models, we use rule of thumb.


text:
How does Supervised learning models handle time-varying data
I need to train a supervised learning model which would take some input which differs in its output relating to time. to better understand my question I would give a simple binary classification, the model would receive an object that changes according to time so on time period t0 the output of the model related to that input differs from the output of the time period t1. I might be describing something either very easy and I missed it or something that doesn't yet exist.
EDIT: I am preparing a model that classifies fruit based on its appearance to either be consumable or not. The only different thing in my work  is that in specific time periods what was considered not consumable should be considered consumable so a picture of a fruit in a certain state could be classified both ways in general but in a specific time period it has only one classification.

I would advise going over recurrent neural networks as a starting point.

Your question is quite abstract, could you give a more concrete example? For example, are you trying to predict a temperature, product sales, stock market price or what? You mention binary classification, what would the two classes be?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

Supervised learning models can handle time-varying data by using a variety of techniques. One common approach is to use a Sliding Window, where the model is trained on data from a fixed period and then applied to data from a different period. Another approach is to use a Recurrent Neural Network, which can learn temporal dependencies in data. If the input data changes over time, the model can learn to adapt and update its predictions accordingly.
It sounds like you are looking for a model that can learn to classify the fruit based on its appearance, and then can also learn to update its classification based on the specific period. This type of model does not currently exist, but it could be developed with enough data and training.You will need to design your own AI algorithm to accomplish this task.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to predict time signal based on multi-input signals?
I would like to approximate the following relation by a neural network
$y = \mathcal{f}(x_1(t),x_2(t))$
Here, I have only one output variable that is a function of 2 other variables which vary in time. Now, I want to be able to predict $y$, given any shape of the 2 independent variables in time. For this reason, I have a training set corresponding to different input and output signals. However, I don't know how to make the neural network understand the concept of time which is very important since I expect the solution at time $t_k$ to be influenced by the previous instants in time. For this reason, I added as input variable the time derivative as
$y = \mathcal{f}\left(x_1(t),x_2(t), \dfrac{\partial x_1 (t)}{\partial t}\right)$
This solution seems to work quite well for the fully connected neural network that I'm using. However, I would like to know if there are other ways to treat such problems where the time history is important.

Please, put your specific question in the title. "Time series prediction" is not a question and it's also not specific.

Instead of feeding the network the samples at just a single timestamp $t$, you should feed it the last $n$ observations. Since you have two "channels" $x_1$ and $x_2$, your input to the network would have dimensions $\texttt{batch_size} \times n \times 2$ (or $\texttt{batch_size} \times 2 \times n$ depending on your framework's conventions). These can then be processed by 1D convolutions, flattened and then the final output is produced by fully connected dense layers.
Alternatively you could flatten the input to $\texttt{batch_size} \times 2n$ and use only dense layers, but this is more prone to overfitting especially as $n$ grows larger.
Since $n$ is a hyperparameter, you'll need to run experiments to determine how large it should be. It is best to test it with exponential steps, like $1, 2, 4, 8, 16, ...$

As for using a Neural Network for a multivariate problem I suggest that you investigate an LSTM architecture, that takes into account the temporality of events.
E.g.
model = Sequential()
model.add(LSTM(200, activation='relu', input_shape=(n_steps_in, n_features)))
model.add(RepeatVector(n_steps_out))
model.add(LSTM(200, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(n_features)))
model.compile(optimizer='adam', loss='mse')

I also suggest that you investigate traditional statistics models such as SARIMAX, and test for Autocorrelation and Partial Autocorrelation (the incidence of the last event, and the incidence of the events chain on your actual sample).

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How does GPT use the same embedding matrix for both input and output?
My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $V$ be the vocab size, $D$ the number of embedding dimensions, and $E$ be a $V \times D$ embedding matrix:

On input, if $x$ is a one-hot $V$-dimensional vector, GPT uses $Ei$.
On output, if $\hat y$ is a $D$-dimensional prediction vector, GPT uses softmax($E^\top{\hat y}$) as its predictions.

Q1. Is the above correct?
I cannot find this stated clearly in the paper, but it is stated explicitly here.  It's also clearly implied by the parameter count listed here, and argued for as best practice here.  Yet, for example, Karpathy's mini-GPT implementation seems to use two different matrices:
self.tok_emb = nn.Embedding(config.vocab_size, config.n_embd) # <--- This would be E
self.pos_emb = nn.Parameter(torch.zeros(1, config.block_size, config.n_embd))
self.drop = nn.Dropout(config.embd_pdrop)
# transformer
self.blocks = nn.Sequential(*[Block(config) for _ in range(config.n_layer)])
# decoder head
self.ln_f = nn.LayerNorm(config.n_embd)
self.head = nn.Linear(config.n_embd, config.vocab_size, bias=False) # <--- This has the same dimensions as Etranspose but is clearly a different matrix

Q2. If it is correct, how does can it work?
This seems to be tasking $E$ with two very different, even opposing, functions:

Map vocab to their meaning on the input side; higher magnitude indicates "more meaning"
Map meaning to the most likely vocab on the output side; higher magnitude indicates greater likelihood

When outputting, we want the softmax to be highest when the word is most likely; magnitude of the output matrix should be roughly proportional to how likely the word is two appear.
Yet, when inputting, magnitude has nothing to do with likelihood.  Magnitude on the input side captures some element of meaning: perhaps how extreme or intense the meaning is, perhaps another aspect (not necessarily easily interpreted).

A GPT produces output based on its own previous output, so it must be able to understand its output.
The learning input is provided as a stream of tokens, and these tokens are defined before learning starts. So it has to use the same set of tokens to understand its own output. The set of possible output tokens is fixed, it learns only to assign probabilities of the next token.
Looking on it in a per token perspective, when it gets a token as input, it learns that it has a non-zero probability to be following the previous input. If the same happens again, it learned that the probability is higher than previously assumed. The probability is expressed as a number of occurrences in the input, and a number per token in the output, but both are about the relative probability of tokens following the previous text.

Yes, GPT uses the same embedding matrix. See here.

Regarding your second question - on the input side, a given token selects one row from the token embedding matrix; it is not clear that higher magnitude signifies anything. On the output side, the magnitude of the output vector $h_n$ also doesn't signify anything. However, once you take the dot product of the output vector with each of the previous "rows" of the token embedding matrix, this gives you a dot-product similarity of the computed output with the pre-stored embeddings. A large value for a given token element means the model is more likely to predict that token.

Please don't upload text as images, it isn't searchable. Use MathJax whenever possible. Thanks.


text:
Can objective function and gradient be unlimited in reinforcement learning?
I'm looking at an example where they define a policy $\pi_\theta(a_t|s_t)\sim \mathcal{N}(ks_t, \sigma)$, where $a_t$ and $s_t$ are action and state, while $\theta=(k,\sigma)$ are the parameters of the policy. Now I take the sampled approximation of the objective gradient for a policy gradient algorithm (like REINFORCE)
$\bigtriangledown_\theta J(\theta) \simeq \frac{1}{N}\sum_{i=1}^{N} \left[\sum_{t=1}^T \bigtriangledown_\theta log\pi_\theta(a_t^i|s_t^i) \sum_{t=1}^T r(s_t^i,a_t^i) \right] $,
where $log\pi_\theta(a_t^i|s_t^i) =-\frac{1}{2\sigma^2}(ks_t^i-a_t^i)^2+const$
and $\bigtriangledown_\theta log\pi_\theta(a_t^i|s_t^i) =\begin{bmatrix}
\frac{\partial J(\theta)}{\partial k} \\ \frac{\partial J(\theta)}{\partial \sigma}
\end{bmatrix} =
 \begin{bmatrix}
  -\frac{s_t^i}{\sigma^2}(ks_t^i-a_t^i)\\ \frac{1}{\sigma^3}(ks_t^i-a_t^i)^2
\end{bmatrix}$
in the example they say that the optimal solution is $k=-1,\sigma=0$, but as i get closer to the solution, the objective and the gradient get bigger and goes to infinity. I believe this is a problem, isn't it? Ok, maybe it's a problem only in practices but not in theory... however, I wonder if we can always fall in this situation, since we don't decide the objective function


text:
Text rewriting with learning of writing style
I do a lot of scientific writing in my job. Unfortunately, I often also have to rewrite amateurish text that some students write for manuscripts. They use tools like grammarly to fix grammar issues, but even after that, I find myself doing a lot of rewriting as grammarly does not catch all grammar problems, even introducing some in many cases. My postdocs are of limited use in this respect as their writing styles are not that great either. Importantly, I like the introduction and discussion/conclusions written in a particular way that maximizes their impact in terms of the arguments made. Can I train an AI (maybe GPT3 based) tool using my published articles (over 200 at this stage), figure out how I like to structure my papers, and then rewrite student submitted manuscripts to match that style, without changing the scientific content?

Don't write "A non-specialist's question...". If you're here, we know that you have a question. Please, just write your specific question in the title.


text:
Is there a literature on the time complexity of Neural Networks?
There exist various blog posts describing the time complexity of Fully Connected Neural Networks (1, 2, 3, 4); Convolutional Neural Networks (CNN) (5) and of Long Short-Term Memory (LSTM) networks (6). However, this question seems to be absent in the academic literature or at least I couldn't find it. Furthermore, papers that go towards this question seem to consider mostly experimental results (7, 8).
Therefore, my question is:

Is there a literature on the time complexity of Neural Networks that gives analytic expressions in the fashion of $\mathcal O(\cdot)$?
If this is not the case this is because...

...(1-6) is common sense?
...it is not that easy to answer and the blog posts only present a simplified form of the problem?
...something different?

Although I consider the posts mentioned above very useful, there are various reasons why one would prefer to cite a paper rather than a website (10)!

We have other posts other than the only one from our AISE site that you're linking to. See this, this, this, this, etc., some of them answered by myself. So, I recommend that you go through all these posts and update yours with links to them, and revise your question, if necessary.


text:
Dummy variable trap in neural networks and class visualization
Let's say I have data records looking like that: (x1, x2, x3, x4, ..., x100), where each x can be either alpha, gamma or omega.
An example of record could be ('gamma', 'alpha', 'omega', 'alpha', ..., 'gamma') .
So the shape of my dataset is (N, 100) (with N the number of records).
I want to train a neural network to predict some binary label. As there is no underlying ordering in my input categories, I use dummy variables to feed my network. Therefore, I end up with a dataset of the following shape: (N, 100, 3).
My problem is that I don't really know how to deal with the dummy variable trap. According to this answer, I should drop one category when I use a network without weight decay. However, I thought that even without weight decay, the non-linearity of neural networks (assuming I'm using non-linear activation functions like relu), would be enough to avoid the issue without actually needing to drop one category. Am I wrong?
Would a neural network without weight decay behave badly if I do not drop one column?
Some context
Ideally, I would like to avoid dropping one column as my next step is to create inputs that maximize a class prediction (starting from noise and using gradient ascent to "improve" the input).
If I do that with a model trained with a dropped category, I can end up with values close to 0 for my (n-1) categories, probably meaning that the category that would maximize the output would be the dropped one. This interpretation looks correct to me, but it leads to a generated input that has very high-frequency components.
However, I know that consecutive features in a record are likely to be the same (like ('alpha', 'alpha', 'alpha', 'gamma', 'gamma'...)), so input with such high frequencies is quite unrealistic. Generally, this kind of method imposes some L1 or L2 regularization in order to get more global coherency in the generated input. But here is my issue, if I impose some regularisation, the generated input will very likely be biased toward the dropped category right?
As I'm writing these lines, I'm wondering if these considerations about frequencies are not relevant only for numerical inputs and not for categorical inputs. I'm open to any clarification as I'm a bit lost.
Inspiration: The interpretation part of this work is greatly inspired by the first part of this repo. It deals with images as input so the visualization is easier than with my data, but the problem is the same: visualize classes predicted by my network by generating regularized inputs that maximize the class prediction.
EDIT
I believe that the part about using L1 or L2 regularization to improve global coherency is not true. I'm still trying to work out how to solve this problem, but I'm struggling quite a lot.

Assume that you have a (3, 3) dataset (3 records, 3 variables), and a, b, and c are alpha, beta, and gamma, respectively.  Then, an example dataframe would be as follows:
   A B C
1  a b b
2  a a b
3  g g a

This dataframe could be one-hot encoded as a (3, 9) dataset.  You don't need a (3, 3, 3) dataset as you suggest in the post.
One-hot encoded
   A_a A_b A_g B_a B_b B_g C_a C_b C_g
1  1   0   0   0   1   0   0   1   0
2  1   0   0   1   0   0   0   1   0
3  0   0   1   0   0   1   1   0   0

After you know the value of 2 columns of a feature (e.g., A_a and A_b), you can predict the value of the 3rd column( e.g., A_g).  You can, therefore, drop the 3rd (redundant) column for each feature without loss of information, as follows:
One-hot encoded
   A_a A_b B_a B_b C_a C_b
1  1   0   0   1   0   1
2  1   0   1   0   0   1
3  0   0   0   0   1   0

However, the article you refer to suggests to not drop the third column for certain models because "If you remove a category from input of a neural network that employs weight decay, it will get biased in favor of the omitted category instead."  Therefore, you should retain the third columns if you employ those algorithms, is what that answer seems to suggest.
For these other models, the answer suggests to drop the third category to avoid multicollinearity:

Linear/multilinear regression
Logistic regression
Discriminant analysis
Neural networks that don't employ weight decay

I'm wondering if these considerations about frequencies are not relevant only for numerical inputs and not for categorical inputs.
This issue specifically arises with categorical variables, not numerical variables.


text:
Concise and mathematically-oriented book on AI and neural networks suitable as a gift
I would like to buy a book about AI and neural networks written on accessible level for a 17 years old mathematically very gifted student interested in these topics. The book should contain some sections about perceptrons and optical character recognition. I am aware of https://www.deeplearningbook.org/ but it does not fully satisfy me. Mostly, because it goes too slow and too long. For instance, in order to grasp the back-propagation algorithm one needs to read 60 pages! Eqs.(6.49-6.52) are particular shocking, I thought every student should know rules of chain differentiation! We do not write such trivial things in, e.g., theoretical physics.
Now, Internet is full of all possible blogs and tutorials, but it is impossible to filter some nice and concise exposition for people with mathematical background. I notice some popular extremes such as i) prolonged discussion of a single neuron, ii) XOR example, iii) very technical tutorials, which require a lot of python, a lot of packages, web-servers etc.
I am seeking some nice text to create a neural network completely from scratch, and with some impressive performance for e.g. written digits recognition. No reliance on external packages, no object-oriented features, as it may deter young students. But functional programming paradigm is welcome.
Therefore, I was thinking about more refined and concise books, preferably with good typography and illustrations, hard cover, suitable as a gift for a mathematically-inclined student. What would be your recommendation?

Please, first rewrite the title in the form of a specific question. A specific question is something like "Do you know of a book that satisfies requirements X?". Second, please, go through this https://ai.stackexchange.com/q/3647/2444, and let us know if it satisfies your requirements.

I didn't downvote. Second, you should try to be more polite when people try to help. Third, your question should be specific but not lead to opinions. Currently, the title says "a beautiful...", beautiful is subjective. So, again, try to ask a question that will to faccts.

While this is a moderately interesting question, I don't believe it fits here in this stack.

@DavidHoelzer Why do you think so? Mathoverflow with >130k questions is full of questions about sources of theorems and books. Stackoverflow with >22m questions contains alone 288 posts about "Book  Guide" recommendations. The The Definitive C++ Book Guide and List was upvoted 4234 times! In cs.stackexchange  with 44k questions the question "books for algorithms" was upvoted 23 times. Yet, on ai.stackexchange with just 10k posts, questions about book are not tolerated and frown upon. I do not see the logic. Is it that this field is so rapidly developing that no one has time to write books?

After a lot of searching the following seems to be a good choice, but sometimes with repetitions of material. The math is very accessible.
Neural Networks and Deep Learning: A Textbook by Charu C. Aggarwal

My vote would go to Artificial Intelligence: A Modern Approach. It is not concise, but that's a feature, just select the chapters you are interested into.


text:
What is the best approach/model for classifying document images with over 70+ classes of documents?
What is the best approach/model for classifying document images with over 70+ classes of documents? I have tried LayoutLM which is a model that incorporated both NLP and Computer vision to classify but it's accuracy is not upto the mark. Are there any other models/methods I could try to come up with a better solution?

It is difficult to answer your question like this:

you speak of image classification but you mention documents and tagged the question with natural language processing, so are you using images or text?
how much data do you have? Do you some computational power as well? The default answer would probably to use a deep learning model, but even that is quite vague. You could use a pre-trained model and fine-tune for example.

What is a 'document image'?

A document image is a scanned version of any document. The task is to classify these scanned images into their respective doc types. We use images and we need NLP because we need to extract the text out of these images and make sense out of it.

The model LayoutLM has been used but it's accuracy is not upto the mark.

@darthvader40 Edit your post to include all this info in the comments directly in the post, because comments are temporary. Also, please, share with us your findings so far, which shows that you've tried at least a little bit to answer your question first.


text:
What is the basic difference between NAS, Hyperparameter Optimization/Tuning and Pruning?
As far as I understand:

NAS = an algorithm that searches for the best NN architecture, i.e., how many layers, what activation function to use, how many neurons, etc.
Hyperparameter Optim = finding the best set of values for an already designed NN model.
Pruning = optimizing the connections between layers in a trained model.

My question is,

When should I use which in the above?
Can all three be used at a time?

Regarding question 1:

Use NAS when you don't know which is the best NN architecture for your problem. Basically, you start from a set of pre-defined blocks and NAS finds the best way to connect them and also their hyper-parameters like number of blocks, filters, etc.
As you said, hyper-opt is useful for an already designed NN (i.e. you know that such an architecture should work well) but also for the optimizer (e.g. learning rate, weight decay), and other hyper-parameters related to some custom layers, operation, loss. For example, in auto-encoders the size of the latent space is another hyper-parameter.
Finally, pruning is a model compression technique. So you use that to remove a given % of connections in your trained model: basically it sets some weights to zero. Pruning is done to get a lightweight version of your initial model, with the aim of deploying it on mobile, low-power devices and even FPGAs (field programmable gate arrays) boards. The pruned model has to be compiled by specific tools in order to get a practical advantage of the weights set to zero, to reduce the number of operations and also memory usage.

About question 2:

Well, I would use NAS first to determine the best NN architecture, then hyper-opt to find the best hyper-params of, e.g., the optimizer (and so not the ones related to the NN since NAS should have already found them), and finally, pruning as a post-training step but only if you want to deploy the model on some computing device with limited resources.


text:
Feature Extraction for timeseries temperature signal
i have two temperature signals from which one is sensitive toward a specific event. I would like to know what other features can be useful to extract apart from:

Angles (between the two).
Slopes ( between the two).

P.S. Language is Python

Cross correlation
A high cross-correlation value indicates a strong correlation between the two signals, which can be useful for identifying the specific event that one of the signals is sensitive to.
TimeLag
Which represents the time delay between the occurrence of the event in one signal and its effect on the other signal.
Frequency Analysis
This can be useful for identifying any periodicities or cycles in the signals that may be related to the specific event.
Statistical Analysis
You can perform various statistical analyses of the two signals, such as mean, median, variance, and standard deviation, to compare their overall characteristics.


text:
Image classifier model which predicts objects and it's relevant areas with a combination of words
I have experience with image classification models such as CNN and Vision Transformers but this time I want to try a new thing (For me).
First please check the below image to understand what I want

In this picture I want to feed an image along with it's label (which is a string) and then I want to predict same string from my model.
In some cases there can be multiple strings with one image as we call multi class problem.
First I want to know if it's possible? Second which state of art CNN/Transformer model will be good for this model, Third has anyone worked on this before?

Can you please put your main specific question in the title? "Image classifier model which predicts string type labels" is not formulated as a question, so I would recommend you do that. Having said that, it seems that you want to predict one or more locations in the image and then you want to describe what is in those locations with more than one word, possibly. Is that correct?

Yes Sir, You truly understood my question. I want to predict information from specific locations which is the disease type and the surrounding information in an image.

And as You said sometimes there can be multiple locations (2 or 3) in an image and the model should describe these locations with more than one word.

@nbro Sir is it possible?  Is there any model developed before for this type of problem?

The task name is "Image captioning". That is the keyword for paper searching. Dataset, evaluation methods, and the state-of-the-art are all aggregated on the forum: paperwithcode.

Can I use these image caption generator models to generate multi captions from my image?

I'm sorry, that's not my field. However, I think it's possible since we have multi-label classifying, the captioning could also return multiple outputs. Or you can just use two models (in some cases, it's work).


text:
What papers can I read that explore model performance vs dataset size?
I am trying to estimate how many images I need to label for an object detection task. I understand a lot of variables are at play, but I'd like to find some papers that have already explored this further. Specifically it would be helpful to find charts for specific model architectures on specific classes that plot:
Mean Average Precision vs Training Set Size (num instances)
What data exists to help me estimate how much data I need to train an object detection model?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

I found a paper that might be helpful. The paper is titled “Sensitivity Analysis of Dataset Size vs. Model Performance”. It discusses the relationship between training dataset size and model performance, especially for nonlinear models. It also provides an approach to quantifying the relationship between model performance and dataset size for a given model and prediction problem.
I also found another paper. It discusses the relationship between batch size and training time for object detection models. The paper also provides an approach to quantifying the relationship between model performance and dataset size for a given model and prediction problem.
I hope this helps!
https://machinelearningmastery.com/sensitivity-analysis-of-dataset-size-vs-model-performance/
https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123660477.pdf


text:
Can a concept/feature be represented using more than one layer of a Neural Network?
I was reading Goodfellow. At the start of the text it was mentioned that there are two ways to represent depth of a deep neural network. One is using the depth of the computation graph and the other is using the depth of the chain of features/concepts that is learned by NN.
Don't every layer in NN represent a concept so the depth would be the number of the layers, but then each layer also does calculation on input from previous layer so the length of computation graph will be same as number of layers. So both representation of depth will be same as number of layers in NN.
Or is it that features can also be represented using more that one layers and not individual layers.
Maybe I have misunderstood the concept of NN.

You cannot reason in a mathematical way over features in my opinion, as they are not defined. However, you can think of deep neurons as a hierarchy of always more high level concepts, as observed in the human brain.
For example consider the task of identifying animals: the first layer, can at most identify simple stuff like edges, then the second layer combines edges and finds corner, the third layer might combine edges to find paws/ears/mouths and so on
This is what he ideally describes as features, but there is nothing stopping the net to learn features that makes no sense for us humans, or not to learn anything on a layer and just propagate the identity


text:
Remove already reached targets from the system to enable reaching other targets?
This may be a very fundamental question, but somehow I can't decide.
I have a graph and the user can take several actions while traversing it and there are multiple points with rewards. When I execute the MDP process, it ends up finding the first (the one it reaches first) target repeatedly and cannot find the others. So, now I am removing the reward value of that node once its reward is reached. Is it a correct approach? If not, what should I do instead of that?
Thanks in advance, Kind Regards, Ferda.

So, now I am removing the reward value of that node once its reward is reached. Is it a correct approach? If not, what should I do instead of that?

Partially correct. For the environment to continue making sense to the agent (and be a valid MDP), you need to set a state variable to represent this change. A list of booleans "goals reached so far" would be one simple way to do it. Note that each such goal state that toggles between giving a reward or not depending on history doubles the state space.

Dear Neil, thanks for the response. I already had the impact_reached booleans for the nodes, so it is ok.  I also have other related questions, but I will ask them separetely.. Thanks again

Dear Neil, I have another question related to this on this link https://ai.stackexchange.com/questions/36381/markov-decision-process-how-to-get-the-correct-policy-if-targets-are-reached-onc  Can you please check that one if you can?


text:
What type of neural network has an unorganized structure?
I am looking for a network that has an unorganized structure like this, is feed-forward, does not have back-propagation functionality, and is trained with a genetic algorithm.
What would I be looking for? I also want to be able implement it in python.
This library would help me make a network like what I described, but its in Javascript.
Are there any others like it?

What do you mean by unorganized? And do you just not want to use backpropagation, or it must be impossible in the network?

I do not want to use backpropagation and I mean unorganized by https://i.sstatic.net/DV2Dq.png.

A few thoughts: 1. why do you want to do this? 2. with Torch you can implement any network you like, you do need to have an algorithm that determines that "unorganised" structure; 3. training networks with genetic algorithms is extremely tricky AFAIK

well... for nothing in particular, 2. I will have a look at it, 3. when I say genetic algorithm, I mean it will have to do with natural selection and I will be doing things my own way.

What would I be looking for?

This seems a lot like neuroevolution of augmenting topologies, or NEAT for short.
There is a Python library neat-python.
Even if your own idea wil have some differences to NEAT, it is worth checking out either as something similar to benchmark against, or as a starting framework to modify.

I have had a look at it, I don't want the algorithm, I just want the network. Have a look at my stack overflow question.

@coder Maybe look at FANN - http://leenissen.dk/fann/wp/ - it's an older NN framework that allows for more or less arbitrary connections. Failing that you may be better off writing one yourself, although you'll want the NN processing part to be in C, not Python, for perfromance.

Someone answered my stack overflow question. I am now using NEAT's FeedForwardNetwork by itself so its only fair that I accept your answer. What do you think?

@coder: That is up to you and how you interpret Stack Exchange's "accept". I clearly didn't give you enough detail to understand precisely what to do, but then the question here is simpler and broader in scope. In your situation I probably would accept, but there's no pressure :-)

Well... If you can answer this question I will: Exactly what is this sort of neural network called? Just plain "Feed Forward Neural Network"?

@coder: I think so, you can even re-arrange that diagram into strict layers with "skip" connections, and importantly there are no loops. It's still different from what you would get out of the box with keras and PyTorch though.


text:
Effective fast mobile ocr model
I plan to develop OCR application which is mobile oriented and fast. (like 10~30 fps) The images that will be detected is not wild images. They are refined data such as cell phone capture images. In this situation what will be the most effective OCR model? I'm thinking yolo for text detection and bilstm for text recognition. However I don't have idea to make the model to end to end like this paper. Any idea or paper will be appreciated. Thanks for reading my question.


text:
LSTM exploding? - multiple parallel time series with multiple variables
I have the following situation:

Stock
Time_Stamps
Feature_1
Feature_2
Feature_n
Price

Stock_1
2019
0.5
1.0
1.0
100

Stock_1
2020
0.7
1.3
0.9
90

Stock_2
2019
0.3
0.9
1.1
110

Stock_2
2020
0.2
0.8
1.1
120

Stock_n
year_n
value_n
value_n
value_n
price_n

So this is how my data table is structured. My original df has 100+ features and 70000k observations resp. 2000+ stocks - so this is only a simplification.
I want to train a LSTM on this data table and look for features correlation with the price.
Common idea, nothing new, so pls save your time giving me "this will not work" bla bla.
I am generally interested in how you would approach this problem. We have multiple inputs (features) for our time series forecast with 8 time stamps (8 years) per stock. However, in my understanding, I'd have to train my model for every stock seperately which is inconvenient.
How would you pre-process my data, so that I can train a decent model?

Maybe you can try to encode "stock" column via one hot encoding https://stackoverflow.com/questions/37292872/how-can-i-one-hot-encode-in-python
so for example, if you have "AAPL" and "MSFT" stocks, than AAPL would be encoded as [0, 1] and MSFT would be encoded  as [1, 0].
And maybe it worth to do the same for "Year", like 2022 -> [0, 0, 1], 2021 -> [0, 1, 0], 2020 -> [1, 0, 0]
And also, I think price should also be normalized, so instead of absolute values you can use minmax normalization, to convert price to number from 0-1 range https://stackoverflow.com/questions/48178884/min-max-normalisation-of-a-numpy-array

thanks for the ideas. In a prior approach, I have categorised the price (-1,0,1), however, if I would encode my 2000+ stocks, wouldn't that create a horribly huge dataframe for my model?

No idea if this would result in a "decent" model, but can you not take the disjoint union of all features?
Roughly that means you concatenate the $n$ features for all 8 stocks into a single "megastock" that has $8n$ features and $8$ outputs ($1$ price per stock), as opposed to 8 models with $n$ features and $1$ output each.
In other words, for each stock you have $n$ input features and 1 output (price), so taking the disjoint union would result in $8n$ input features and 8 outputs (price for stock $1$, price for stock $2$, etc.).


text:
Play against your own RL-trained AI from gym retro
so far I have seen people implementing reinforcement learning to build an AI to play and complete games on gym retro, such as street fighter, racing games and so on. However, I was wondering if it is possible to play against your own trained AI to get some sort of human evaluation? Anyone has any idea for this? Thank you

Most of the games on Gym are one player. If this is specifically about gym retro, then identifying a game you want to do with this, and any progress/ideas you have so far would help narrow things down. Clearly in general it's possible to play a game against an AI, so it would help to know a little more about your setup and your plans

I dont understand why the post is downvoted.

I understand that you can use gym-retro to build a reinforcement learning agent to play some particular game, lets say street fighter.
What I want to know is, is it possible to fight against that agent that you have trained, rather than watching a mere trailer (render) of that agent fighting against the CPU bot in that game. If it is possible, then how should I go about doing it, assuming I have a trained rl agent for street fighter. Any resources out there that can guide this, since the python gym-retro does not have too much details on deployment of trained agents vs humans

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.

The new details are helpful. However, you have posted them as an answer to your question. Please edit the question instead and put the new details in there.


text:
Non-Convex loss-surface although quadratic loss function
there is one problem which bugs me quite a long time, it is the non-convex loss shape (multiple minima, e.g. shown here) of neural networks which use a quadratic loss function.
Question: Why is a “common” AI problem usually non-convex with multiple minima, although we are using e.g. quadratic loss functions (which is in lectures usually drawn as a simple,convex,quadratic function such as x^2)?
My guess:
Is it because we are feeding the loss function with our highly non-linear model-output and therefore the resulting total loss surface is highly non-linear/ non-convex? Specifically, the quadratic loss is just approximating the infinitesimal small neighbourhood around a specific point (minibatch) as quadratic. Is that guess correct? This would imply, that highly-non linear / very deep and complex models have a highly-non linear resulting loss-surface, while shallower models have less minima and a one-layer network has a convex shape ?

If I understood your question correctly - the quadratic loss function is not always convex. Its' convexity depends on the input it takes. For example, consider a very basic NN that takes some input $x$ and returns
\begin{equation}
f(x)=relu(w_1relu(w_0x+b_0)+b_1).
\end{equation}
Using quadratic loss, or $\ell_2$ is minimizing the objective $\sum_i(x_i-f(x_i))^2$, but even when examining only one of the terms (say $i=0$), which is $(x_0-f(x_0))^2$, we see that it is not convex at all, as the following plot shows:

Yes, you did. So the convexity of the loss depends on both inputs (prediction_function and groundtruth_function) ? As soon as one is non-convex, the loss will also be non-convex right? What confuses me, I guess most of the real-world problems will be non-convex and thus would always lead to a non-convex loss, and I thought simple linear classifiers (1/1+e(-(wx+b))) are convex ? Or does the convexity just depends on (in your example) f(x_i) and not as well x_i ?

Generally speaking, you are correct as the composition of a convex function with a non-convex one may very well be non-convex. Notice that a sigmoid function is not convex (think of it as an "S" shape)

So the convexity of the loss is also depending on my underlying groundtruth function, which I try to approximate, right ? So it is very likely that a very complex task, which I try to solve with a linear classifier would yield to a non-convex loss-function (although I am using the BinaryCrossEntropy loss) ? Or does the loss-convexity just depend on the "prediction_function" ?


text:
How can I learn to transform one input signal (time series) into another?
I'm posting this question here because I've been trying in vain to solve a problem for weeks and I hope some of you might have some useful suggestions.
Basically, the problem is as follows. I have 7 quantities that vary simultaneously as time changes. These quantities are somehow related to each other, but for simplicity's sake we could also consider them independent.
To represent each of these quantities, I have a continuous signal that takes on values between 0 and 1 as time varies (x-axis). In essence, this is a time series as can be seen from the image below.

What I would like to be able to do is to transform the continuous signal into another signal that can only accept values 0 or 1. Practically, from the signal shown in the figure above, I would like to be able to generate the signal shown in the figure below.

I have a lot of data at my disposal, so I could create a set of training, validation and testing. The solution I thought of was to build a deep network that could learn the transformation between the input signal and the output signal.
Ideally, the network should learn through a tailored loss to indicate the difference between ground truth and input signal. In addition, some consideration should probably be given to the context and thus the relationships between the various time instants. For this reason (i.e., temporal correlation) I had thought of recurrent networks or Transformers.

However, I don't how to model this network and whether such a solution would actually make sense. I have also searched the literature for work of this kind, but to the best of my knowledge the problem has not been addressed.
The problem seems well defined to me, but unfortunately I cannot find the right solution. Do you have any suggestions for me? Thank you in advance

is not clear to me how the ground truth is defined, i.e. why in the second pictures are there two spikes mapper to 1 but not the others? I could think it's just a matter of thresholding, in which case would it be possible to learn to predict a threshold rather than the entire time series? (asking cause that would change the architecture completely).

This problem looks like a matched filter, which takes into account time correlation. Would that help?
https://en.wikipedia.org/wiki/Matched_filter

@EdoardoGuerriero The first signal is a kind of confidence measure and, ideally, should only result in a 1 in the output if its value is above 0.5 in the original signal. However, the signal is noisy and very far from perfect, so a mere thresholding does not seem to work.

Thank you @JaumeOliverLafont. I did not know about matched filters, I will look into them and let you know if they were useful.


text:
How can evaluate the success of my algoritm?
A little bit of context. I have a classification algorithm based on mathematical discriminator and I am not applying any machine learning or AI technique, just moving window and several relative comparisons. This is a signal segmentation problem, where I need to detect three states based on their mathematical properties or morphology. In addition, I need to perform this with minimum delay and almost real-time.
The output of my algorithm is a labelled signal with the different sections of it labelleded depending on those mathematical descriptors. The reference model was labelled by professionals that tagged the data for me and I am in charge of creating an automatic labelling tool for it.
Here you can see on the top the reference model and on the bottom the output of my algorithm.

My question is, I have defined a measure of success function that compares sample by sample the two signals but it is not very accurate and I want to know what is the state of the art in this type of classification problem.
My "measure of success" function:
def measure_of_success(signal_size, result_labels, model_labels):
""" Measure of success function """
incorrect_matching = 0
for index in range(signal_size):
    expected_section = identify_section_from_sample(model_labels, index)
    result_section = identify_section_from_sample(result_labels, index)
    if expected_section != result_section:
        incorrect_matching += 1
matching = (1 - (incorrect_matching / signal_size)) * 100
return matching

This function performs a sample-by-sample comparison. My problem is: the labels from the upper picture were set by hand but, I does not matter how accurate the section is determined but just to know that the event has been detected for a considerable time. This is a bit confusing but basically, if I detect the same "b" event and it coincides in time with the majority of the reference model, it should be 100% ok. On the other hand, I feel this measure of success is a bit simplistic and there might be better ways like a confusion matrix or something like that.
What would you suggest me to do in order to evaluate the success of the classification?
Thanks!

Welcome to AI stack exchange. I suggest you share your "measure of success function" since this is key to understanding how you are evaluating. If you want to compare to state-of-the-art, you will need to give more information about the problem domain and what you already know about state-of-the-art algorithms (this looks like a signal sequence, so I would guess some kind of RNN or CNN might do well - if it is a sequence please make clear whether prediction needs to happen at the moment of sampling for each record or can be done retrospectively). Use [edit] to add details to the question

I have added some information in the post! Thanks for pointing that out


text:
Why is multilayer perceptron a nonparametric model?
E. Alpaydin, Introduction to ML, 4-th edition, page 46:

Over time, it has been realized
in the neural network community that most neural network
learning algorithms have their basis in statistics—for example,
the multilayer perceptron is another class of nonparametric
estimator.

Why is multilayer perceptron a nonparametric model?

The terms "parametric" and "non-parametric" have been used ambiguously off-late, see discussion - https://stats.stackexchange.com/a/322051/219259. A comment from that answer says " As the wiki-page on nonparametric statistics says, there is no general consensus on what non-parametric means".

@GaneshTata Towards the end of 2020, there was a joke in the Cross Validated Meta that our new year’s resolution would be to stop using the term “nonparametric” for exactly that reason.

“Nonparametric” is a controversial term in statistics whose definition lacks a consensus.
One definition is that the parameter count increases as the data size increases. This is how kernel density estimation works: for each new point, we add a new kernel. Perhaps a neural network could be seen in this light by allowing the number of neurons to increase as data are added.
One definition is that we allow the data to describe themselves rather than shoehorning them into a particular form that is governed by parameters. A neural network could be seen this way, as the idea is to give a large amount of flexibility to allow the data to extract their own features that involve curvature and variable interactions; we don’t have to specify these like we do in a linear regression or generalized linear model if the model figures them out.
If the author had referred to a neural network as a particular type of nonlinear regression, then the statement would not be controversial.
Because of the lack of consensus on what “nonparametric” means, towards the end of 2020, there was a movement/joke on Cross Validated (Statistics Stack Exchange) that our new year’s resolution would be to stop using the term “nonparametric” for exactly that reason.


text:
How do you evaluate a k-medoids cluster model?
So I'm planning on clustering a bunch of observation data using k-medoids. There are seven attributes for each instance and the data is numerical and discrete. I'm a little uncertain of how to evaluate the model to find the correct number of clusters. I was thinking I could run the cluster technique for an increasing number of clusters (say start at 1 and increase by one each time), measure the silhouette coefficient for each cluster model and then select the number of clusters with the highest value?
Would anybody be able to tell me if this is a good idea for evaluating the model and if not what else I could do?

In Finding Groups in Data, Kaufman and Rousseeuw describe ways to evaluate the quality of clusterings. If I remember correctly (it's been some time that I worked with this), for k-means algorithms you add up the average distances of each element to each cluster medoid. This gives you a score for k clusters. Then you repeat the clustering with fewer or more clusters, and do the same calculation. You can then compare the scores and choose the best configuration.
If you have few clusters, the differences will generally be large, but there won't be many of them. If you have too many clusters, there will me more distances to add up, so the score will also be larger. The optimum number of clusters will minimise that score.
PS: I can only highly recommend that book.


text:
Can RL still learn if part of my actions are only used once, at the beginning of the episode?
I am working in an environment with 3-dimensional action space. The first two actions are only used at the first timestep and never again. The third action is used at every timestep.
Say, the action is $a = (a_1, a_2, a_3)$. At the start of an episode $i$, the agent uses actions $a_1, a_2$ only at timestep 1. Action $a_3$ is used at every timestep in the episode starting from 1 till the horizon H. The agent receives rewards $r_i$ at each timestep $i$ till the end of the episode.
I am using SAC. Since the actions $a_1, a_2$ only affect the agent's behavior at timestep 1 and are not used at any of the later timesteps, I am not sure if the RL policy will get better at choosing "good" values for $a_1, a_2$.
Will the RL be able to learn a good policy even if it doesn't quite see the effects of the first two actions in the episode data except at the first timestep?

You should be able to learn a good policy even if you use the first two actions only at the first timestep.
Using this OpenAI reference, the loss for the state action value function (from which the policy loss is later derived) is:
$$L(\phi) = \mathbb{E}_{(s, a, r, s') \sim D}\left[\left(Q(s,a|\phi) - (r + \gamma Q(s', a'|\phi_{target})\right)^2\right]$$
where $D$ is a set of transitions, $\phi_{target}$ are "old" parameters for the action state value function which are left unchanged in the parameter update, and $a' \sim \pi(.|s,\theta)$.
Note that I've simplified the equation for clarity.
The expectation in the loss is replaced in the actual algorithm with an average on a batch of transitions.
At timestep 0, the target $r + \gamma Q(s', a'|\phi_{target})$ for $Q(s_{t_0},a_{t_0}|\phi)$ (with $a' \sim \pi(.|a_{t_0},
\theta)$) in the loss will be non-zero, because $Q(s', a'|\phi_{target})$ will be non-zero and will reflect the value of $(s',a')$ accurately (e.g. thanks to transitions which happen at later timesteps).

Thank you for your answer. But, I believe I did not make it clear in my question (I have edited it now). I am getting rewards at each timestep but, only use the third action at every timestep and not the first two, which are used only on the first timestep of every episode. My concern is that since the first two actions do not affect the agent's behavior at each timestep, the RL policy might not be good at choosing good values for $a_1$ and $a_2$.

I had understood that the first two actions are only used in the first timestep. When I say "even if you don't see a reward", that wasn't to say that I thought your issue was with having a reward or not, but that even if you didn't see a reward in the first timestep, the policy would be accurate for the action $(a_1, a_2)$ because the target will be non-zero in either case (reward or not). I've amended the answer to be clearer


text:
Categorical loss function for variable number of labels
I have a model for binary classification. The target variable has the different number of labels (instances) in each sample. For example, a batch of size 2 with 2 and 3 instances and correspondingly with 2 and 3 labels (0 and 1):
y_true = np.array([[0., 1., -1., -1.],
                   [1., 0., 1., -1.]])

The maximal number of labels (instances) in a sample is equal to 4. -1 values are used as a mask. I created a function (in TensorFlow) that masks all -1 values and calculates the loss for each unmasked value and then the average of all losses:
def my_loss_fn(y_true, y_pred):
    mask = tf.cast(tf.math.not_equal(y_true, tf.constant(-1.)), tf.float32)
    y_true, y_pred = tf.expand_dims(y_true, axis=-1), tf.expand_dims(y_pred, axis=-1)
    bce = tf.keras.losses.BinaryCrossentropy(reduction='none')
    return tf.reduce_sum(tf.cast(bce(y_true, y_pred), tf.float32) * mask) / tf.reduce_sum(mask)

Is it a correct way mathematically? Should I add some weights depending on number of labels in a sample? Or calculate loss per sample (per row in my example) and then get the average? Should I mask unused labels at all? It looks like my model doesn't learn to classify correctly and predicts 1 for all labels.

Correct for what model? The ordinary categorical cross-entropy loss arises from a specific probability model of the data. The categorical cross-entropy loss is the "correct" loss for that probability model, but it's "incorrect" if the data are assumed to arise from a different probability model. This reads a bit like an
XY Problem. What problem are you trying to solve, and how does having a variable number of classes for each instance fit into solving it?

Here's one possible solution. For a single instance, the categorical cross-entropy loss for $K$ classes is
$$
L=-\sum_{k=1}^K y_k \log p_k=0 + \dots + 0 - \log p_t + 0 + \dots + 0
$$
where $y_k$ is the one-hot class label and $p_k$ is the estimated probability that the instance belongs to class $k$. Because $y_k$ is one-hot, there's only one nonzero term in the sum. The only part of this that involves the number of classes $K$ is the fact that there are $K-1$ zero entries in the sum. If this is the model you want, then no weighting or other refinement is needed!


text:
Why "Good Model" that performs great on holdout validation data fails on production data
I have this binary regression model that has ~500 futures with an unbalanced dataset with the following results.
Loss is 0.06152007728815079,
Accuracy is 97.71724343299866

How I split the data:
X_train, X_val_and_test, Y_train, Y_val_and_test = train_test_split(X, Y, test_size=0.20, random_state=42)
X_validation, X_holdout_final_validation, Y_validation, Y_holdout_final_validation = train_test_split(
    X_val_and_test,
    Y_val_and_test,
    test_size=0.5
)

The model :
model = tf.keras.Sequential([
    tf.keras.layers.Dense(X.shape[1], activation='relu', input_dim=X.shape[1]),
    tf.keras.layers.Dense(X.shape[1] * .66 + X.shape[1] + 1, activation='relu'),
    tf.keras.layers.Dropout(.10),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(
    loss='binary_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

lr_sched = step_decay_schedule(initial_lr=1e-4, decay_factor=0.75, step_size=2)
callbacks = [
    lr_sched,
    PrintLR(),
    monitor
]

hist = model.fit(
    X_train,
    Y_train,
    epochs=50,
    batch_size=128,
    shuffle=True,
    validation_data=(X_validation, Y_validation),
    callbacks=callbacks
)

And here are some confusion matrix results
50.0 %
['6634', '77', '104', '1114']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['83.67%', '0.97%', '1.31%', '14.05%']

80.0 %
['6691', '20', '283', '935']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.39%', '0.25%', '3.57%', '11.79%']

85.0 %
['6693', '18', '328', '890']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.41%', '0.23%', '4.14%', '11.22%']

90.0 %
['6699', '12', '388', '830']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.49%', '0.15%', '4.89%', '10.47%']

95.0 %
['6705', '6', '507', '711']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.56%', '0.08%', '6.39%', '8.97%']

97.0 %
['6709', '2', '585', '633']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.61%', '0.03%', '7.38%', '7.98%']

99.0 %
['6711', '0', '757', '461']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.64%', '0.00%', '9.55%', '5.81%']

100 %
['6711', '0', '1218', '0']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.64%', '0.00%', '15.36%', '0.00%']

The function for the confusion matrix:
def my_rounder(num, threshold):
    if num >= threshold:
        return 1
    else:
        return 0

def print_conf_matrix_assessment(data, y_expected, threshold, prediction_model):
    y_actual = [[int(my_rounder(x, threshold)) for x in y] for y in prediction_model.predict(data)]
    conf_matrix = confusion_matrix(y_expected, y_actual)
    label_names = ['True Neg', 'False Pos', 'False Neg', 'True Pos']
    result_percentages = ["{0:.2%}".format(value) for value in conf_matrix.flatten() / np.sum(conf_matrix)]
    print()
    print(threshold * 100, "%")
    sample_counts = ["{0:0.0f}".format(value) for value in conf_matrix.flatten()]
    print(sample_counts)
    print(label_names)
    print(result_percentages)

So in general things look pretty good. But as we know if it looks too good there is something wrong and I am trying to figure that out since when I get some production data and put it through the model I get results like that instead:
50.0 %
['2585', '1681', '23', '767']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['51.13%', '33.25%', '0.45%', '15.17%']

60.0 %
['2683', '1583', '30', '760']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['53.07%', '31.31%', '0.59%', '15.03%']

70.0 %
['2782', '1484', '38', '752']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['55.02%', '29.35%', '0.75%', '14.87%']

80.0 %
['2918', '1348', '51', '739']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['57.71%', '26.66%', '1.01%', '14.62%']

85.0 %
['3007', '1259', '59', '731']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['59.47%', '24.90%', '1.17%', '14.46%']

90.0 %
['3129', '1137', '81', '709']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['61.89%', '22.49%', '1.60%', '14.02%']

95.0 %
['3305', '961', '113', '677']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['65.37%', '19.01%', '2.23%', '13.39%']

97.0 %
['3446', '820', '149', '641']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['68.16%', '16.22%', '2.95%', '12.68%']

99.0 %
['3704', '562', '254', '536']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['73.26%', '11.12%', '5.02%', '10.60%']

100 %
['4266', '0', '790', '0']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.38%', '0.00%', '15.62%', '0.00%']

And here we are talking of data that is 2 hours apart. So concept shift is doubtful.
The data is on the same scale - using the min max values from the training dataset and using the mean if some value is out of bounds.
Does anyone have any idea what could be the reason for this to happen?


text:
How to handle out-of-bound values in Production data?
So I have this model but the data may vary. And it is virtually impossible to always have the values in bounds. If I do I`d have to use larger period leading to concept shift which is worse.
The question is what is the best way to deal with the values of futures that are out of the model bounds? I see 3 options

If the value is greater than max set it to the max value the model has seen
If the value is less than min set it to the min value the model has seen
If the value is greater or less set it to the mean that kind of eliminates the future weight for the prediction.

So what would be the best approach here any thoughts?
Note: I am retraining the model daily and the model has a lot of futures ~500 so it is highly likely even right after retraining some to be out of bounds, excluding futures is not the best option since it's never the same future showing this behaviour.
I am using this function for scaling.
def min_max(value, min_max_map):
    result = 0
    if min_max_map['max'] - min_max_map['min']:
        result = (value - min_max_map['min']) / (min_max_map['max'] - min_max_map['min'])
        if result > min_max_map['max'] or result < min_max_map['min']:
            result = (min_max_map['mean'] - min_max_map['min']) / (min_max_map['max'] - min_max_map['min'])

    return result

is not clear to me why data out of training range represent a problem. Is it cause of feature preprocessing? Are you using min max normalization or something similar?

Yes I am using normalization. As far as I know, models tend to give wrong predictions if the values they are analyzing are never seen before. I am using min max scaler to cramp the values on 1 scale

Steps 1 and 2 are basically the same operation, clipping, which is a possibility but not the best since your loosing information. Imagine having two instances with same features except for feature $n$, which assume value 101 and 201 respectively. Let's say that 100 was your observed maximum in the training data for feature $n$, after clipping both instances will look the same and lead to same predictions, good for instance 1 cause its value of $n$ is close to the training maximum, but nonsense for instance 2.
The best way would be defining a theoretical maximum, so independent from the training data, it could be also a value that we set as maximum and after which we do indeed clip cause occurrences of higher values are rare.
An alternative is also to use z score standardization instead of min max normalization. The output range is [$-\infty$, $\infty$] but in practice you'll end up with values almost all within the range [-1, 1], and rare
occurrences below and above this range are totally fine.

If I put let's say a theoretical maximum of 100 for features that usually vary between 0 and 35 and we get a value of 71. In the training, the model will see a max value of .35 but in the prediction, it will get .71. Won`t that mislead the model and create some kind of bias? It's binary classification if that matters.

no, the purpose of feature scaling is just to set all features in the same order of magnitude, but is not required that every feature includes instances with values 0 as min and 1 as max after being processed. It's actually extremely rare for that to happen, basically only in computer vision where images are always between 0 and 255.

And also if the variability between training and test is huge, than there is a problem in the quality of the training data that are not representative enough, and that can't be solved with preprocessing.

I see thank you. I will actually post a question exactly about the data quality topic soon :).


text:
What temperature would you recommend for the chatgpt api?
I believe that it is recommended to have a tiny bit of temperature with GPT 3 even for noncreative tasks like 0.2 or something (I am not entirely sure why).
Last I checked, and if I remember correctly, the examples from openai on their GitHub page use 0 temperature.
Is there any benefit in choosing a non-zero temperature for the chatgpt API when the query does not request a creative task? If so, are there some categories or examples?
[EDIT: to make the answer less subjective, perhaps I could ask what are the benefits of increasing the temperature in the chatGPT API]

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

@AnshumanKumar The question is fuzzy I agree, it's asking for a list of benefits of increasing the temperature

Like with many things, it depends on what you're trying to accomplish.
The two major parameters you can tweak are:

temperature - Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.
top_p - An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered

In terms of some reference values, the forum post on Mastering Temperature and Top_p in ChatGPT API provides the following examples:

Use Case
Temp
Top_p
Description

Code Generation
0.2
0.1
Generates code that adheres to established patterns and conventions. Output is more deterministic and focused. Useful for generating syntactically correct code.

Creative Writing
0.7
0.8
Generates creative and diverse text for storytelling. Output is more exploratory and less constrained by patterns.

Chatbot Responses
0.5
0.5
Generates conversational responses that balance coherence and diversity. Output is more natural and engaging.

Code Comment Generation
0.3
0.2
Generates code comments that are more likely to be concise and relevant. Output is more deterministic and adheres to conventions.

Data Analysis Scripting
0.2
0.1
Generates data analysis scripts that are more likely to be correct and efficient. Output is more deterministic and focused.

Exploratory Code Writing
0.6
0.7
Generates code that explores alternative solutions and creative approaches. Output is less constrained by established patterns.

Note: It is generally recommended that you set either temp or top_p, but not both.  So for the table above, the values can be seen as roughly equivalent ways to achieve the desired outcome.
Another reason you might include a higher temperature is if the user indicates the first response is wrong and you want to attempt again, but get a different response.  Although re-prompting with the new information can also work as well.

Thanks. I suppose the temperature=0.2 rather than 0 for coding is for alternative solutions when trying again (like the regenerate answer button in the web interface). I am not sure however if the table means T=0.2 OR top_p=0.1 or it means T=0.2 AND top_p=0.1. In the API documentation for top_p they mention We generally recommend altering this or temperature but not both., it's mentioned here : https://platform.openai.com/docs/api-reference/chat/create

Yeah, should have clarified - I'll update the answer.  The table means you should set one OR the other, but not both

Temperature = 0 means deterministic. The same input gives the same output every time.
If you are fine with fixed outputs, go for temperature = 0, since it's easier to debug.

Hi, thanks. I know that T=0 means deterministic. I am more interested in the advantages of adding temperature with the chatgpt 3.5 model. I believe it is recommended to add a little bit of temperature to the gpt 3 model to avoid it repeating itself in some cases. ChatGPT is a less raw model than GPT 3 and is more knowledgeable of what people like and dislike (such as answers with a lot of repetitions). Hence, I am wondering if there is still a reason to add temperature to the ChatGPT model.

The only reason I see is that it's weird to have the same exact response every time, in normal conversations. Imagine replying always "Hello there" to my "Hello". Usually some days you would say "Hi" or "Hello" to reply instead. ChatGPT repeating itself seems more a problem which could be handled with repeat penalty.

That's a good point for a conversation indeed. I suppose if these gpt models are ever trained in chess and are equipped with some tree search algorithms or something it could also be good to not have a predictable game also. Those are some benefits of nondeterministic answers. I am still interested in how the type of answers change when the temperature is increased. I might run some statistics to see how answers change. I am not yet sure what to measure other than correctness. Maybe I could also message word frequency and text length or tendency to digress.

temperature = 0 does not necessarily imply that the output will be deterministic. See the OpenAI docs.

Can you share a link to the relevant documentation page?

@nbro I think Rexcirus was refering to your comment but did not mention your username

A temperature set to 0 doesn't guarantee a deterministic result.

@FranckDernoncourt Besides the fact that the link is for GPT4, I believe that the spirit of temperature=0 is to ensure deterministic output. The fact that this may not happen in practice (e.g. due to floating point roundings, etc.) should be viewed more as a bug in the implementation.

But it would be nice to have a link to the docs proving or disproving this.

@Rexcirus the link also mentions gpt 3.5 https://community.openai.com/t/observing-discrepancy-in-completions-with-temperature-0/73380/7 AFAIK the official docs are not mentioning that, but openai employees confirmed that behavior. If you want to reproduce it yourself: Prompts that reliably result in unreliable completions


text:
What is the "attack success rate" of an Adversarial Attack?
For a typical adversarial attack, a sample $x_{0}$ is chosen from a training set belonging to class $C_{i}$ and a transformation $A$ is applied such that the adversarial example $x=A(x_{0})$ would be misclassified as $x \in C_{target}$. The adversarial example $x$ could be found iteratively through ${x}={x}_{0}+\eta \cdot \operatorname{sign}\left(\nabla_{{x}} J\left({x}_{0} ; {w}\right)\right)$ for instance.
In this documentation of TextAttack I read

Out of those 157, 29 attacks failed, leading to a success rate of 128/157 or 81.5%.

My question is: how do we chose the starting samples $x_{0}$ (here the 157) to evaluate the success rate of a type of adversarial attack? Do we chose several target classes? Several initial classes $C_{i}$? How many samples do we chose? From which set (training? validation?)
In the DeepFool paper, they'd use the entire test set- but is it standard practice to claim the success rate of a type of attack this way?

See: https://ai.stackexchange.com/a/14264/17742 and  https://ai.stackexchange.com/a/9767/17742 - the success rate is supposed to be 100%; otherwise it's not an attack, only an attempt.

The adversarial example is supposed to have a 1/1 success rate, but I can't find a consistent way to chose the samples to evaluate the type of attack throughout the literature


text:
Fine-tune GPT-Neo with prompt and completion?
I'm new to AI and machine learning.
To fine-tune GPT-3, I understand that we need a set of training examples that each consist of a single input ("prompt") and its associated output ("completion").
I have prepared a dataset with "prompt" and "completion". And I expect that a fine-tuned model would return the corresponding completion after receiving a prompt in my dataset. But due to some reason, I cannot fine-tune GPT-3 at the moment.
So I plan to fine-tune GPT-Neo (or GPT-J or GPT-NeoX). From this video and this video, it seems that they only accept a dataset containing only "prompt".
Does anyone know how I could modify my dataset with "prompt" and "completion" such that it could be used to fine-tune GPT-Neo?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

as far as I can tell training gpt neo with only prompt (if you use the completion as the last part of the prompt) will let you achieve the same results.
You probably notice that, on HuggingFace, there is no way to make a supervised Learning with a text generation model using X as the prompt and y as the completion and the reason is that is not needed. Just add the completion to the prompt and you’ll be fine.
Everything I said is well written and explained inside this HuggingFace notebook: https://github.com/huggingface/notebooks/blob/main/examples/language_modeling.ipynb


text:
Last linear layer of the decoder of a transformer
I am learning the transformers architecture from these two sources:
https://arxiv.org/pdf/1706.03762.pdf
https://jalammar.github.io/illustrated-transformer/
I just wanted to ask about the final step in the decoder. Let's fix testing time. As I understand, the decoder starts with an input of dimension $(N_{words},d_{emb})$, where $N_{words}$ is the number of words already predicted and $d_{emb}$ is the embedding dimension.
Now if we "follow" the following decoder steps, at each step (after e.g. the attention layers) we should have a vector of dimension $(N_{words},d_{model})$ where $d_{model}$ is the model dimension. In other words, up to the final linear layer we have $N_{words}$ vectors which are $d_{model}$-dimensional.
Are all these $N_{words}$ vectors fed into the last linear layer (before the softmax) or, as I suspect, only the last of these vectors is used ? In the latter case the last linear layer would be a matrix of dimension $d_{model}\times N_{vocab}$, where $N_{vocab}$ is the vocabulary dimension.
Is this correct ? Are there any issues in what I wrote ? Unluckily from the online sources I was not able to clarify this point...
PS: I conjectured that the last linear layer is using just the last vector, because than I would understand what happens in training time, one would just use in that case all the output vectors from the decoder, instead of just the last one, to have a parallelized prediction.

Anyway I am starting to be quite convinced that I am right. If the final linear layer were processing all the vectors output of a decoder than the number of parameters of this dense layer would be dependent on the sequence length, which would give problems in handling inputs with different lengths (which we need to handle for language translation)

I agree with this:

PS: I conjectured that the last linear layer is using just the last
vector, because than I would understand what happens in training time,
one would just use in that case all the output vectors from the
decoder, instead of just the last one, to have a parallelized
prediction.

Indeed when you look at the code of Tensorflow shared previusly you see this line
# select the last token from the seq_len dimension
  predictions = predictions[:, -1:, :]  # (batch_size, 1, vocab_size)

where they take the last token of the sequence as a prediction for the next token. And this is consistent with the fact that the input sequence to the decoder is shifted to the right by adding a <START> token at the beginning (see transformers original paper section 3.1-Decoder).
So to correct the previous answer, the input and output sequence of the decoder have the same length and the last vector is used for the next token prediction which is then added at the end of the input sequence.

Well well, this text in the link posted in one of the answers above (https://www.tensorflow.org/text/tutorials/transformer) gives the answer to the question.

Note: The model is optimized for efficient training and makes a
next-token prediction for each token in the output simultaneously.
This is redundant during inference, and only the last prediction is
used. This model can be made more efficient for inference if you only
calculate the last prediction when running in inference mode
(training=False).

Edit
Based on the comments to the original version of this answer, OP indicated that the use case was translation between two languages.
Answer:
At sampling time, the last linear layer of the decoder is going to output a sequence whose length is incremented by one each time you apply the encoder-decoder transformer to the input sentence.
Let's take a practical example, with $w$ denoting the words in the original sentence and $w^{i}$ those in the target language after iteration $i$ of applying the transformer model.
If you have already sampled a sequence $(w^1_1, w^2_2, ..., w^{l}_l)$, the inputs to the encoder and the decoder the next time you apply the model (to get the next token $w^{l+1}_{l+1}$) are going to be respectively $(w_1, w_2, ..., w_T)$ (the original sentence) and $(w^1_1, w^2_2, ..., w^l_l)$. The output of the decoder (after sampling) will be $(w^{l+1}_1, w^{l+1}_2, ..., w^{l+1}_l, w^{l+1}_{l+1})$, but we're only going to keep $w^{l+1}_{l+1}$.
Then, applying the model again to get $w^{l+2}_{l+2}$, the new inputs to the encoder and the decoder are going to be $(w_1, w_2, ..., w_T)$ (again the original sentence) and $(w^1_1, w^2_2, ..., w^l_l, w^{l+1}_{l+1})$. The output of the decoder will be $(w^{l+2}_1, w^{l+2}_2, ..., w^{l+2}_l, w^{l+2}_{l+1}, w^{l+2}_{l+2})$, and we'll keep only $w^{l+2}_{l+2}$.
This continues until the sentence is fully translated.
At each step, we give the whole original sentence to the encoder so that the model can build its translation for the next word by looking at the whole sentence. And we give the sentence translated so far to the decoder so that the translation of the next word can attend to what has already been translated.
To confirm, you can have a look at what they do in the TensorFlow transformer tutorial.

Thanks a lot for your answer. Give me please some time to look at the implementation... In the meanwhile there are some clarifications I would need to better understand.

(1) You say  "the inputs to the encoder and the decoder the next time you apply the model to the sequence (to get the next token wT+1) is going to be (w1,w2,...,wT)". --> what setting do you have in mind ? I am thinking in this thread about sentence translation. For example I would like to translate in german "The dog is on the table". For my understanding the encoder is used (forward pass) just once with the whole english sentence "The dog is on the table". Instead the decoder, as you say, receives as input the translated sentence. "Der" , than "Der Hund", than "Der Hund ist", and so on...

(2) "The output of the decoder (after sampling) will be (w2,w3,...,wT+1)."--> Since we already have in your example "(w1,w2,...,wT)". Let's call "(w'2,w'3,...,w'T+1)" the output of the decoder, since there is no guarantee that w2=w2',... and so on... The only word we will use in the following steps is w'T+1, right ? (see also next question)

(3) Then, applying the model again to get wT+2, the new input to the encoder and the decoder is going to be (w2,w3,...,wT,wT+1) --> According to the last question, I thought that the new input would have been (w1,w2,w3,...,wT,w'T+1) (see also video in section "The Decoder Side" of the link "The illustrated transformer)

Let me update my question for the translation use case.

Thanks Raphael. It is very useful your update, so that we can communicate on the same lines. Before asking for some clarifications ( I may update my question ), I have an other preparatory point on what you wrote.

(4) If you have already sampled a sequence (w11,w22,...,wll), the inputs to the encoder and the decoder --> why do you say that at each step we pass the sentence throught the encoder at testing/sampling time ? For my understanding, the sentence is passed throught the encoder just once. It is the output of the encoder that is given as an additional input to the decoder at each translation/sampling step. Performing the encoding of the sentence at each step would mean repeating the same operation at each step of the translation, since the output of the encoder does not depend on tranlsation steps

Indeed, as an optimisation you can reuse the output of the encoder rather than recalculating it. Depending on your implementation, it might be easy or hard to do. Also, it doesn't change the overall time complexity of the algorithm (although in practice speed might be important!)

Is there anything which is still unclear? Otherwise, could you accept my answer?

Hello Raphael. Actually, if I have to be honest, I did not accept your answer because we spent quite some time to clarify what you wrote ( a useful exercise for me though )... and at the end I could not focus on clarifying my question... I did not get actually the answer to what I was really asking and therefore I was planning to make my question more explicit. But this requires me to find some time to modify my question, and I ended up being quite busy lately...


text:
Reinforcement learning with $\gamma=0$ versus supervised framework
I am on a learning phase (still to enter the details but I would like first to get a bird's eye overview). I would like to understand the difference between:
(1) a reinforcement learning framework, with $\gamma=0$, policy gradient optimization, and where the environment presents a new state at every iteration sampled from an input distribution. These conditions define a setup that we call RL0
(2) a standard neural network supervised classification framework. Call this second setup ANN.
To me the setups seem equal if we make the correspondence:

ANN weights <-> RL0 policy parameters
ANN inputs <-> RL0 states
ANN predicted values <-> RL0 deterministic action given the state
ANN loss function for one sample <-> RL0 reward given the state and action

My questions:

Is it really so? In other words can an ANN used in a supervised framework can be seen as a special case of reinforcement learning with a parametrized policy and discounted rewards with $\gamma=0$ ? Or am I missing some fundamental difference?

Does this mean that every RL technique applied in this setting should fall in some supervised learning category ? How would any RL technique applied to solve a supervised problem compare with a standard supervised framework?

NB: question Can supervised learning be recast as reinforcement learning problem? is very similar and answers point 1, but still left point 2 not very clear to me.

Hi. Yes the question is very similar indeed. I started also reading the article you linked. Indeed it looks that one can rewrite a supervised learning problem in a RL framework.

So question 1 is solved. I have some follow up on question 2. You write in your answer to the other question "However, note that the learned policy might not be able to generalise to observations not present in the training dataset. Moreover, although it is possible to solve an SL problem as an RL problem, in practice, this may not be the most appropriate approach (i.e. it may be inefficient)."

This seems to suggest that the answer to question 2 is "no", because you (and also Barto in his chapter) suggest that using RL to answer a SL learning problem would be suboptimal. The follow up question would be than "what would be the differences in tackling a SL problem as a RL one and why this would be not an equivalent choice but a suboptimal one"?

An effort to answer more quantitatively the second question is in the answer I wrote. Feedbacks welcome.

I add some thoughts of mine following Sutton and Barto, chapter 13, also to keep track of my work. This effort should go in the direction of answering point 2 of the question.
Policy gradient theorem
We consider the setting RL0 as in the OP, with the additional notation:

Each episod goes to the terminal state immediately after the first action.
Each state $s$ can be denoted as $x$, a member of the input state $X$, more suggestively remembering the notation in a supervised setting.
The action is the (deterministic) prediction of the parametrized model (e.g. a neural network), so that $\pi(a,s)$ can be rewritten as $\pi(y,x)=\delta(y-Y^{pred}(x,\theta))$, where $a$ has been substituted with $y$ to match notation with a supervised setting.
The reward is the squared error: $R=G=|Y^{pred}(x)-Y^{ex}(x)|^2$

Such a setting looks very similar to a supervised one. Indeed the policy gradient theorem reads:
$$\partial_{\theta} J=\int dy |Y^{pred}(x,\theta)-Y^{ex}(x)|^2\partial_{\theta}(\delta(Y^{pred}(x,\theta)-y)) [1]$$
After some maninpulations and integration by parts one can rewrite the policy gradient theorem as above in the form:
$$\partial_{\theta} J=\partial_{\theta} |Y^{pred}(x,\theta)-Y^{ex}(x)|^2 [2]$$
which is the form one would use in a supervised form for the update. So up to the policy gradient theorem RL0 and ANN would be equal in this setting.
REINFORCE update
If we look at the REINFORCE update rule in this setting it reads:
$\delta \theta=|Y^{pred}(x,\theta)-Y^{ex}(x)|^2 \partial_{\theta}(\delta(Y^{pred}(x,\theta)-y))/\delta(Y^{pred}(x,\theta)-y)$
To get a better understanding of the ratio of $\delta$ functions we rewrite:
$\partial_{\theta}(\delta(Y^{pred}(x,\theta)-y))/\delta(Y^{pred}(x,\theta)-y)=\delta'(Y^{pred}(x,\theta)-y)/\delta(Y^{pred}(x,\theta)-y)\partial_{\theta}Y^{pred}(x,\theta)$
So we would like to understand if the expression $\delta'(x)/\delta(x)$ can be given a finite meaning. Unluckily if we use a regularization of the delta funcion $\delta(x)=\frac{1}{\epsilon}e^{-\pi x^2/\epsilon^2}$ we see that this expression diverges when x is not null.
INTERPRETATION (for the moment):
It looks that the manipulations starting from the policy gradient theorem and leading to the reinforce update is where RL0 and ANN start to diverge substantially. I think the main issue there is that when in the derivation of the REINFORCE algorithm one multiplies and divides by $\pi(a|s)$, this operation provides larger estimators when $\pi(a|s)$ is low (because the $\pi$ at the denominator is the one surviving, the other gets replaces by sampling). As a consequence the REINFORCE update breaks when the densities are deterministic, i.e. when no sampling is necessary.
Comments welcome.


text:
Which neural network/ML framework to use for a partial amount of a fixed-time amount of sensor data?
Assume I've run a set of initial experiments. For each experiment, I have a set of input signals $I_e(i, t)$, and output signals $O_e(j, t)$ for $i \approx 10$, $j \approx 10$, and $0\leq t\leq t_{max}=7200$ for half-second timesteps. This gets me a set of experiments $1 \leq e \leq N$. So $O_e = F(I_e)$ for some function $F$.
I have a target signal to match the desired output signals for a given $T(i, t)$ by controlling the $j$ input signals on the fly for a new experiment but only have the data for the run for the start to the present time, e.g. $O_{e+1}(i, t')$ with $ 0 \leq t' < t_{max}$. I'd like to do this iteratively, so hopefully each succeeding experiment would be more accurate. This would also need to be run in real-time.
Which ML approach should I use? It seems like reinforcement learning/q-learning would be the right approach? I'm not sure how that would work with partial time data as a state.


text:
Linear Actor Critic for continuing task and 1 continuous action => Any comment?
I wish to implement an Actor Critic agent using linear functions for a continuing task with one continuous action. Below the resulting pseudo-code I have reached by my own (the initialization part is removed).
$$
\begin{aligned}
& Observe \hspace{0.1cm} X \\
& Loop \hspace{0.1cm} until \hspace{0.1cm} STOP:\\
&  \hspace{0.5cm}\mu = \theta_{\mu }^{T} * X \\
&  \hspace{0.5cm}\sigma = ln\left (\theta_{\sigma }^{T} * X  \right ) \\
&  \hspace{0.5cm}a \leftarrow N(\mu , \sigma ) \\
&  \hspace{0.5cm}Take \hspace{0.1cm}action\hspace{0.1cm} a, \hspace{0.1cm}observe\hspace{0.1cm} X' \\
&  \hspace{0.5cm}\delta \leftarrow R - \overline{R} + w^{T} * X' - w^{T} * X \\
&  \hspace{0.5cm}\overline{R} \leftarrow \overline{R} + \alpha _{\overline{R}} * \delta \\
&  \hspace{0.5cm}w \leftarrow w + \alpha_{w} * \delta * (R-\overline{R}+w^{T}*X'-w^{T}*X)*X \\
&  \hspace{0.5cm}\theta_{\mu} \leftarrow \theta _{\mu}  + \alpha_{\mu}*\delta * \frac{a-\mu}{\sigma^{2}}*X \\
&  \hspace{0.5cm}\theta_{\sigma} \leftarrow \theta _{\sigma}  + \alpha_{\sigma}*\delta * \left (  \frac{\left ( a-\mu \right )^{2} }{\sigma^{2}}- 1 \right )*X\\
&  \hspace{0.5cm}X \leftarrow X'
\end{aligned}
$$
My mathematics level is not high. I would be grateful if somebody could check. My questions:
 1) Can somebody help me to complete the update of theta_mu and theta_sigma (lines with question marks) ?
I finally found something I missed in Sutton & Barto book, so I could complete the updates and remove question 1.
2) Does somebody see any mistake or have any remark ?
Many thanks for the attention.


text:
Why does my neural network perform different on the same images during training and testing?
I use tensorflow keras to build a neural network that classifies images of covid-19 rapid tests into three classes (Negative, Positive, Empty).
During training the tensorboard logs denote a validation accuracy of around 90%. But when I test the network after being trained with the same images it was trained on, the classification performance is way worse (~60%). I observed the same behavior when I trained the network with different images (see section What I have tried).
During training the images are preprocessed to grayscale and resized before being fed into the model. The batch size is 16.
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
image = cv2.resize(image, (height, width))

To augment the sparse data that I have (~450 images) I am using the keras.preprocessing.image.ImageDataGenerator and its parameters are: width_shift_range=0.1, brightness_range=[0.9, 1.1], height_shift_range=0.1, zoom_range=0.2, horizontal_flip=True, rotation_range=10, shear_range=0.2, fill_mode="nearest", samplewise_center=True, samplewise_std_normalization=True
Edit: The images are split into 90/10 training/validation sets.
(trainX, testX, trainY, testY) = train_test_split(data, labels, test_size=0.1, random_state=24)
I am converting the model to tflite because we need it for mobile platforms. I am using this code snippet:
model = tf.keras.models.load_model(model_path)

converter = tf.lite.TFLiteConverter.from_keras_model(model) # path to the SavedModel directory
# converter.optimizations = [tf.lite.Optimize.DEFAULT] # optimizations
tflite_model = converter.convert()

# Save the model.
with open('rapid_test_strip_cleaned_model.tflite', 'wb') as f:
  f.write(tflite_model)

What I have tried:

crop the images to the strip of the casette, train and test the network again
check in the testing (inference) script if the labels are correct
check if the images are converted to grayscale and resized correctly before being fed into the network during testing
test the model before converting it to tflite, using tensorflow.keras.models

Model:
img_width, img_height = (256, 256)

model = Sequential()
inputShape = (img_width, img_height, 1)
model.add(Conv2D(32, (3, 3), activation="relu", input_shape=inputShape))
# to prevent overfitting
model.add(Dropout(0.25))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

model.add(Conv2D(64, (3, 3), activation="relu"))
# to prevent overfitting
model.add(Dropout(0.25))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

model.add(Conv2D(128, (3, 3), activation="relu"))
# to prevent overfitting
model.add(Dropout(0.25))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

model.add(Flatten())
model.add(Dense(512, activation="relu"))
# to prevent overfitting
model.add(Dropout(0.5))
model.add(Dense(3, activation="softmax"))

opt = Adam(learning_rate=INIT_LR, decay=INIT_LR / EPOCHS)

model.compile(loss="categorical_crossentropy", optimizer=opt,
            metrics=["accuracy"])

This is the Tensorboard graph of the training. The straight line is from another training run.
Testing/Inference script:
interpreter = tf.lite.Interpreter(model_path=model_path)

# Load TFLite model and allocate tensors.
interpreter.allocate_tensors()

# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

height = input_details[0]['shape'][1]
width = input_details[0]['shape'][2]

labels = ["positive", "negative", "initial"]

# load image into numpy array
image = cv2.imread(image_path)
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
image = cv2.resize(image, (height, width))
input_arr = img_to_array(image)

input_arr = np.array([input_arr])
# normalize values
input_arr = input_arr / 255.0

interpreter.set_tensor(input_details[0]['index'], input_arr)

interpreter.invoke()

# The function `get_tensor()` returns a copy of the tensor data.
output_data = interpreter.get_tensor(output_details[0]['index'])

results = np.squeeze(output_data)
top_k = results.argsort()[-5:][::-1]

print(labels[top_k[0]])

Ideas about where the problem may be are very appreciated. I am stuck with this problem for two months now.
Thank you!

there is a reason why we use train/validation/test split, and this is one of the reasons... if you use a validation set for hyperparam tuning, you should consider that a "unbiased (possibly overshooting) estimation of your network performance"

I am using a train/validation split. I edited the post with the LOC. I did not really get how that influences the performance of the network. Shouldn't the network reach a performance around the average validation accuracy?

Your network seems to include modules, such as Dropout, that operate differently during training and inference/testing. For example, dropout is usually active during training, and may drop some connection, whereas all connections are preserved in inference mode. This may be the reason.

I found the issue with a tip from stackoverflow! It was the normalization in the inference script. It is supposed to be input_arr = (input_arr - np.mean(input_arr)) / np.std(input_arr). Thanks a lot!

@SohrabTawana if you've found the answer, please write it as an answer and accept it, so that future users can make use of it.

I found the issue with a tip from stackoverflow!
The problem is the normalization in the inference script. It is supposed to be input_arr = (input_arr - np.mean(input_arr)) / np.std(input_arr).


text:
Does sequence matter in LSTM?
In general, as long as the items are in order as per time sequence, does it matter if it's in ascending time sequence or descending time sequence for LSTM within the input vector?
eg:
ABCDEF -> G
BCDEFG -> I
vs
FEDCBA -> G
GFEDCB -> I

It can depend on what you are learning to predict and what is your dataset.
If I correctly understand your question I will assume that the training dataset is: ABCDEF -> G
And then you ask about prediction on: FEDCBA -> G
I would say yes. I does matter.
Reference: colah.github.io tutorial (or PapersWithCode), and their paper: "Order Matters: Sequence to sequence for sets", by Vinyals, Bengio, and Kudlur, (revised 23 Feb 2016)
Where you can see that there is memory to remember the most important information. But the network is really sensitive to this type of data as it transforms it step by step. Not as a chunk in input
After winter there is spring not the autumn.
Noticed that the similar question was answered before at Cross Validated: "Do RNNs such as LSTM only provide value for sequences where order matters?"