text:
How to re-train an AI model to have smaller input image size
I need a PyTorch Model which can do road segmentation on OAK-D camera.
The model provided requires Input Image Size: 896x512, which is too big for running on OAK-D camera. Thus I need to re-train it with a smaller input size(224x224) and just need the BG(background) and road classes, or if any other options available which can easily make it running on the OAK-D camera.
Does anyone know how to do this?

The model you're using has a couple of 2 strided convolutions. You can manipulate those in your liking, omitting some for example and finetuning the other layers..

hi @FarhoodET I am still very new to this area, could you please share a piece of code to explain what you mean?? thanks

@Franva unfortunately I think there is a lot of experimentation involved in AI. It is not simply a matter of taking some code and running it.

@user253751 definitely understood it. there are huge amount of things to twist. I'm not asking for listing all of them which is impossible. But just one viable code. One can write hundreds of words to explain things, but a piece of working code is more straightforward and appreciated.

@Franva What you're trying to do is somewhat similar to Transfer Learning. You can see an example here: https://www.pyimagesearch.com/2019/06/24/change-input-shape-dimensions-for-fine-tuning-with-keras/. This only changes the input shape tho, you must change the other model's dimensions to your liking and load the weights of remaining identical layers. See this for loading some but not all weights: https://stackoverflow.com/questions/43702323/how-to-load-only-specific-weights-on-keras.

What you need to search for is a Fully Convolutional Network, i.e. a network that use global pooling to overcome the issue of a fixed input size. Unfortunately the model you found is not fully convolutional, and every workaround to make those pre-trained weights usable implies retraining.
At this point it is more convenient to find something else, or train something yourself. You can take a look at this repo which contains also a fully convolutional network for segmentation among other models (they don't seem to link to any pre-trained weights though).


text:
Are the "artifacts" in select Keras MNIST training images really there or is my download corrupt?
I'm having fun with a ludicrously well known and used dataset: mnist.
I am doing it with a huge and well known tool: keras.
Please excuse the red dots, something else I was doing.  I have otherwise not modified the image at all except via the process of plotting it to a file.  All images below are plotted in R.

Image #48, a "5" digit:    There is a perfect vertical slice through the upper segment.  For all the letters, they look blurred.  This slice is perfectly vertical and does not have blurring.

Image #54, a "4" digit:    This has artifacts to the left and above the image.  It looks like part of a border around the number.

Image #68, a "1" digit:    This has artifacts to the left and above the image.  It looks like part of a border around the number.

Maybe something is odd with my download or version, so perhaps someone else can verify this.
Are there the artifacts as shown in the images above, in the canonical dataset as presented by the well known tool?

There are many versions of MNIST digits now, and some of them are corrupted, binarized, or otherwise altered (see TensorFlow datasets) intentionally; but I don't think the keras MNIST digits has these "artifacts" intentionally added.  These are real handwritten digits, so some of the "artifacts" are probably "people" mistakes.  Some may be due to the fact that we are "zooming in" on low resolution images (consistent with your observation of images being "blurry").  I think the "stuff" around the digits that looks like a border may be adjacent text that was cropped out, but I really don't know.
Yes, I also get the same images as you do, so your download is not corrupt.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Find maxima and minima of feed forward neural network given interval inputs
I have a feed forward neural network defined in pytorch as follows:
    model = torch.nn.Sequential(
        torch.nn.Linear(2, 50),
        torch.nn.GELU(approximate='tanh'),
        torch.nn.Linear(50, 50),
        torch.nn.GELU(approximate='tanh'),
        torch.nn.Linear(50, 1),
    )

The model has been trained.  Now, I have specific intervals of input values (e.g. [0, 0.1), [0.1, 02.),...[0.9, 1]) within which I want to find the maximum and minimum values of the model's output.  I am not looking for specific code or software to do this.  Rather, what are some general strategies for doing this?

this is a question more suited for StackOverflow since it's about programming. Anyway, you can check the pytorch documentation to find the answer to these type of questions, start from https://pytorch.org/docs/stable/generated/torch.argmax.html

@EdoardoGuerriero I added the pytorch model definition for context, but am not expecting a code answer. I'm looking for a theoretical answer on how the solution could be computed. I'm happy to figure out how to do the implementation myself.

ok I see, maybe the phrasing is a bit unclear, are you asking if it's possible to know beforehand, i.e. without running a model,  the range of a network predictions based on the range of its inputs?

@EdoardoGuerriero Apologies if it was unclear. I have the trained model, and am willing to run it. But it's unclear to me how you would run it when the input would be intervals rather than fixed values. I.E rather than doing inference on input (0.3, 0.4), I want to do inference on ([0.3, 0.35], [0.4, 0.49]).

You can only input into your model the type of data that you used to train it.  Your model, therefore, will not accept intervals.  As suggest in the answer by @NikoNyrh, you can approximate the minimum and maximum values in the interval by sampling within your interval.  For example, the $[0, 0.1]$ interval can be sampled as $0, 0.01, 0.02, ...0.09, 0.1$.  Then, run each number in the sampling individually as an input into the model (perhaps in a for loop) and collect outputs in a list.  Find the min and max of the list.  Repeat for each interval.

This is kind of a black-box optimization problem, but luckily you can calculate gradients between its input and output. Since the number of inputs is so small, you could just brute-force it by evaluating the network for a grid of input samples. Naturally you'll need to choose how many samples to do along each dimension. Since the network has quite many parameters, I expect the "surface" to be quite intricate.
A more efficient approach is to run a gradient descent algorithm on it, starting from various random points. Then again the brute-force approach will take better advantage of a GPU.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why MSE and MAE yield poor results when used with gradient-based optimization for classification?
Deep learning book chapter 6: In 6.2.1.2 last paragraph:
Unfortunately, mean squared error and mean absolute error often lead to poor results when used with gradient-based optimization. Some output units that saturate produce very small gradients when combined with these cost functions. This is one reason that the cross-entropy cost function is more popular than mean squared error or mean absolute error, even when it is not necessary to estimate an entire distribution p(y | x).
explain the above sentence
Doubt: But we use mean squared error (MSE) and mean absolute error (MAE) for regression.

is there more context? i.e. are they talking about classification? I agree that in general the statement isn't true, MSE is used in regression as you point out, but for e.g. a classification task the statement would be correct.

Yes, I went back through the sections and checked, it refers to classification, though not mentioned explicitly.

We use Mean Squared Error (MSE) and Mean Absolute Error (MAE) for regression, so why is it a problem for classification?
First, note that while this passage from the book does not explicitly specify whether it is talking about classification or regression, the mention of crossentropy loss necessitates that the discussion revolves around classification.
In linear regression, the outputs range is (-inf, inf), and in this case, the MSE and MAE functions are both convex and can be minimized.  In classification, the output (assuming softmax activation of the final dense layer) is a vector of probabilities (with a probability for each class), each of which ranges from [0, 1] and together add to 1.  In this situation, MSE and MAE are non-convex and may have a broad plateau.  When you have "output units that saturate", you are in the plateau portion of the curve.  In the plateau regions, gradients (slopes/derivatives) are very small and, therefore, training is very slow (see Fig 4 of article).  Therefore, MSE and MAE "often lead to poor results when used with gradient-based optimization."


text:
How can an ensemble be more accurate than the best base classifier in that ensemble?
BACKGROUND:  Ensemble classifiers are said to reduce bias by taking an "average" of predictions of several base classifiers that comprise the ensemble.  However, I am uncertain if this necessarily means that they can increase accuracy.  My intuition tells me that the ensemble classifier should perform no better and possibly even worse than the best base classifier in the ensemble.  This seems especially true for bagging approaches which use strong classifiers anyway.  When you have a "star performer", it just doesn't seem to make intuitive sense to "dilute" its performance with subpar performers.
Nonetheless, from my novice-level reading, it seems that ensembles can be as good or possibly even better than all of the individual base classifiers, but I'm still not clear why.
QUESTION:  How can an ensemble be more accurate than the best base classifier in that ensemble?

Lets consider binary classification.
Imagine you have an ensemble made up of $K$ models. Assume each model has exactly $51\%$ accuracy. Further assume each model's error is uncorrelated with each other model. The ensemble simply takes a majority vote.
For $K = 11$, what is the probability we get a single question right?
Note: This problem can be simply modeled as flipping an unfair coin $K$ times and calculating the probability that one side will be the majority of flips.

$P(correct\ \geq\ 6)\ =\ \sum_{k=6}^{11}{\binom{11}{k}({0.51}^k)({0.49}^{11-k})= 0.53}$

What about when $K = 101$?

$P(correct\ \geq\ 51)\ =\ \sum_{k=51}^{101}{\binom{101}{k}({0.51}^k)({0.49}^{101-k})= 0.58}$

If you do the math you will find that as $K$ approaches $\infty$, the odds of the ensemble being correct asymptotically approaches $100\%$.

$\lim_{K\rightarrow\infty}{P\left(correct\ \geq\frac{K+1}{2}\right)}={lim}_{K\rightarrow\infty}\left(\sum_{k=\frac{K+1}{2}}^{K}{\binom{K}{k}({0.51}^k)({0.49}^{K-k}})\right)=1$

Now, we made 2 key assumptions:

Errors are uncorreleated between models
Each model has more than $50\%$ accuracy

In practice these assumptions often may not hold, especially the first one. But if they do, this simple probability analysis shows ensembles will be better

Note that your models having more than 50% accuracy is a very weak assumption because any model that does worse than random can be used to construct a model that does better than random.

@OscarSmith, can you explain how that is possible, maybe along with a concrete example?

you take the model and make a new model that predicts the opposite. (for the n way version it's a bit harder, but you can make a model that chooses a random model other than the one your original model produced)

Sorry, I may have missed something here.  Is it implied somewhere in the first comment (Note that your models having more than 50% accuracy is a very weak assumption because any model that does worse than random can be used to construct a model that does better than random.) that it is referring to a multiclass situation?

The generalization of the 50% accuracy condition (which is better than random) is the weak assumption.


text:
Combining GANs and NLP for AI-Based Programming: Generating Input-Output Templates for Computer Functions
I would like to combine GANs and NLP to create a system that can take an input and generate an appropriate output. For example, given the input 9 to the power of 2, the system would output pow(9,2).
I am not entirely sure how to research this, but I can provide some examples of what I am trying to accomplish.
Example:
Input: 2 * 5
Output: mul(2,5)
Input: (5*3) + 9 / 7
Output: sum(mul(5,3),div(9,7))
My ultimate goal is to combine AI with programming to give artificial intelligence the ability to use many computer functions. For example, it could retrieve the current date with Date.now() or use ParalelAICapture.getSubject() to identify what is currently in front of it.
Actually, code suggestion algorithms are giving me somewhat similar results to what I am looking for, but they are a bit blurry for my goal. Therefore, I will use input and output templates for more precise results.
Can someone please explain how to achieve this and provide some examples?


text:
How do we define greedy action for policy improvement for a given stochastic policy?
If we have a deterministic policy $\pi$ with action-value function $q(s,a)$, then a greedy action for policy improvement is defined as
$\pi^\prime(s)=\arg\max_{a}q^{\pi}(s,a)$.
How do we define a greedy action for policy improvement for a given stochastic policy $\pi(a|s)$
with action-value function $q^{\pi}(s,a)$.

Probably related: https://ai.stackexchange.com/questions/40149/mean-or-mode-of-action-distribution-when-evaluating-policy-gradient-agents. In my opinion, the action that can be considered "greedy" for a stochastic policy, should be one of the distribution mode(s).

Could you please write your guess mathematically?

Sure, have a look at my answer

I think the question is missing some context. The greedy action is by definition the action that the agent predicts will result in the highest return. That's it, there's no edge cases or special rules beyond that in different circumstances. I'd like to know what you are trying to do with this information. For instance, it would not be appropriate to use the greedy action to train a policy gradient method - you should use the action taken by the policy even if it was "wrong".

Other than that, if you are talking about policy improvement in Q-learning, or SARSA or other action-value based method, then the definition for greedy action/greedy policy does not change. If you are interested there is an extension to the policy improvment theorem that the resulting $\epsilon$-greedy policy (in e.g. SARSA) is a strict improvement over the previous one.

Deterministic case: the greedy action $a_g$ (discrete) yield by the greedy policy $\pi_g$ is:
$$\pi_g(s) = a_g = \arg\max_{a\in\mathcal A} q^\pi(s,a)$$
For continuous actions, the $\arg\max_a$ is infeasible to compute. One can thus learn a deterministic policy $\pi_d$, according to the deterministic policy gradient theorem, such that $a_d=\pi_d(s)$ achieves the maximum action-value, i.e. $q^\pi(s,a_d)\ge q^\pi(s,a),\ \forall a\in\mathcal A$.
Now, for both discrete ($a_g$) and continuous ($a_d$) actions these are greedy, in the sense that their respective policies should assign a probability of one to those (if viewed as if being stochastic), and zero to all sub-optimal actions:
$$
\pi_g(a\mid s) = \begin{cases}1 & \text{if $a=a_g$} \\ 0 & \text{otherwise}  \end{cases},
$$
similarly for $\pi_d$ and $a_d$. This means that $\pi_g$ (viewed as stochastic) samples the greedy action all the times, so it always exploits.
Stochastic case: to simplify let's assume the policy is Gaussian, i.e. $\pi(a\mid s) = \mathcal{N}(\mu,\sigma^2\mid s)$. I assume $\mu$ and $\sigma$ to be meaningful, like being learned to maximize the action-value function, $q^\pi$. If so, in my opinion, the most likely action (i.e. the distribution's mode) should have the largest action-value. In the case of Gaussian distributions, the mean $\mu$ is the most likely action (being also its mode) and so it should be the greedy action, i.e. $a_g=\mu$ such that $q^\pi(s,\mu)\ge q^\pi(s,a),\forall a \in\mathcal A$.
Therefore $\pi$ assigns the following probabilities:
$$
\pi(a\mid s) = \begin{cases}\alpha e^{-\frac12} & \text{if $a=\mu$} \\
                            \alpha e^{-\frac12\big(\frac{a-\mu}{\sigma}\big)^2} & \text{otherwise}  \end{cases},
$$
where $\alpha = (\sigma\sqrt{2\pi})^{-1}$. Now, from this we can see that the considered stochastic policy is greedy $\alpha e^{-\frac12}$ of the times (since is the probability of sampling the mean $\mu$ as action.) Since there always exist a deterministic policy (see slide 41) we can, in principle, turn the stochastic $\pi$ into a deterministic policy $\pi_d$ such that:
$$
\pi_d(a\mid s) = \begin{cases}1 & \text{if $a=\mu$} \\ 0 & \text{otherwise}  \end{cases},
$$
in order to be greedy all the times: with probability one, $\mu=\pi_d(s)$ always outputs the mean $\mu$ (action that maximizes $q^\pi$) in state $s$ - indeed, the value of $\mu$ is state-dependent, meaning that a different state could have a different mean.
(Disclaimer: I don't know if the above is mathematically sound and correct but that's my intuition. So, please correct me eventually.)

In the Gaussian example, those are probability denisties, not probabilities. I'm not sure this is answering what the OP asked though, see my comments on the question

The stochastic policy trained in the standard policy gradient method are inherently incompatible with any 'greedy' exploitative action during policy improvement which is a deterministic concept. Policy improvement in policy gradient method simply means its parameters update rule follows the gradient ascent of its specified performance metric $J(\theta)$.
Having said that, the closest counterpart of a greedy action perhaps is the actor component in the bootstapping actor-critic methods as detailed in Sutton & Barto's Reinforcement Learning book section 13.5.

As we have seen in the TD learning of value functions throughout this book, the one-step return is often superior to the actual return in terms of its variance and computational congeniality, even though it introduces bias... When the state-value function is used to assess actions in this way it is called a critic, and the overall policy-gradient method is termed an actor–critic method.

Also take a look at its detailed algo without eligibility traces on page 332, and you can treat its actor's parameters $\theta$ update as some form of 'greedy' action based on the learned critic's feedback $\delta$ if you will, but it's not the same sense as taking the action with maximum action value in all action based RL methods.

See my comment on the question. This answer assumes the OP does not want to calculate the greedy action, and is searching for something else. However, the question is asking for the definition of the greedy action given the action value function, which is straightforward. Also, if you are using $\epsilon$-greedy behaviour policy, then the greedy action choice is also used in the policy improvement step. The OP has not returned to the question to clarify, although they are still active on the site.


text:
Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents
Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for discrete ones it can be a Categorical (or a relaxation).
The question is: during training we sample from such distribution and then compute the log-probability, but during inference (also agent evaluation/test) what is the best to do? Usually, during evaluation we want to exploit so we should't sample the action (unless is recommended for a given environment setup, to avoid an opponent to exploit a deterministic policy - but I'd like to exclude this case), and so we want to take the action that exploits the more.
I'm puzzled about whether to take the mean or mode of the action distribution during inference and evaluation (I assume the agent training has ended): also, is the most likely action the action that exploits?
In pseudo-code you can view the problem as follows:
function act(state):
   action_distribution = policy(state)

   if evaluation:
      # here is the problem: mean or mode?
      actions = action_distribution.mean()  # or mode()
   else:
      # training phase
      actions = action_distribution.sample()

  return actions

Clarifications: I assume a generic action distribution in which its mean can be different from its mode, like a Beta distribution for example. What I also ask is:

Considering a given state, does taking the mean of the distribution corresponds to an action that is associated to the highest expected value?
Similarly, considering the mode should correspond to the most likely action (to simplify I assume the mode to be unique) which should be associated to the highest state-action (or Q) value. Is that so?

Additionally, I'm aware of this answer: it's related but does not answer my question.
What I'd like to know is: at evaluation/inference time, assuming for a given environment is fine to have a deterministic policy, how to best implement/obtain that from a learned action distribution?

How could you choose the mean action from a Bernoulli distn? The mean is $p$, whereas the mode is 0 or 1. for a discrete distn, you would usually want to choose the modal action. In a continuous distribution, the mode might be complex to calculate so I’d recommend using the mean.

Sure, the Bernoulli is a bad example (I'll edit that) so assume you have a Beta instead (indeed, the action space is continuous), what you would do in that case? still choosing the modal action? If so, can you motivate further

I think my last comment is still valid. Finding the mode could be non-trivial (though I suppose in a Beta it is will known). So, in this case, the advice would simply to be to use whichever gives the best performance.

(This does not fully answer your question. Actually, it just attempts to make sure you understood certain things).
The distribution that you learn should put more weight/mass/density on the best actions. So, if you sample during inference, you should get the best actions more frequently than other actions.
You would use the same action every time if you are certain that the optional policy is deterministic, which is the case of finite MDPs.
There are cases where the best policy is really stochastic (e.g. rock-paper-scissors, which is naturally represented as Markov Game, so not a finite MDP), so choosing always the same action is likely not a good idea, so the best approach during inference would actually be to sample.
So, you would need to pick the same action every time if you have a finite MDP (finite action and state spaces).
See also this answer

Maybe I should convert this to 1 or more comments, if this doesn't even partially answer your questions.

Keep as an answer it's interesting. I'm aware of cases like rock-paper-scissor in which the best policy is always stochastic to avoid the opponent to exploit the determinism. But excluding that (I may clarify this in the question), I'd like to understand what is the best to do at inference time with the policy. Actually, could you elaborate a bit more about the difference of finite MDPs and continuous MDPs, in particular why in the continuous case one don't want to have deterministic actions?

@LucaAnzalone That's actually a good question. I shouldn't have used "only if" but "if" in my last sentence. In other words, I just wanted to point out that, if you have a finite MDP, you're sure that the optimal policy is deterministic. In other cases, it might not be. So, I did not want to say that in continuous actions MDPs the optimal policy is always stochastic, because I don't know if that's the case (I think not - one can probably think of a very simple environment with 2 continuous actions where the optimal policy is deterministic, but I didn't think much about this).

Well, I can think of an env in which there are three paths that lead to a terminal state, the paths on the left and right are highly rewarding while the middle path is poor. Suppose the actions are steer left/right (from -1 to 1), the "mode" policy will always pick something close to either -1 or 1, while the "mean" policy just average to 0 (or so) leading to the non-rewarding path. In this case the best to do would either to be deterministic or taking one or the modes randomly at each trial.


text:
How can I not only classify an intent, but also identify slots and values in it?
I've been working on text -> intent -> command execution for a particular application and while I've found many papers and code that work well for intent classification (1, 2, etc.), they stop there. For example, given a standard music intent "Play some music by U2" such classifiers return me an intent class play_music. But the information about the artist, track requested aren't part of it. Everytime I search for papers on text -> intent, I end up with classifiers instead of any deep learning models that can not only classify, but maybe give me an importance vector to extract the key parameters from my intents to pass onto the intent class.
Does anyone know of any papers or implementations like this?
Note: I'm not looking to make a voice assistant but a text->command execution engine for a very specific purpose.

Can you please put your specific question in the title?

@nbro changed it to better reflect the intent.

Found it! After a lot of searching and reading through different papers, this was the exact one I needed.
"BERT for Joint Intent Classification and Slot Filling"
There's also an example dataset based on Snips that shows how to create custom intents, entities, slots and train different kinds of transformers to classify text and extract words to fill the appropriate slots.


text:
Role of $f$ Target Network in DDPG
I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and also target networks $V^{+}$ and $f^{+}$ (but no $r^{+}$ or $Q^{+}$). Here, $V$ is the value function $V^{\mu} = Q^{\mu}(s, \mu(s))$, i.e., it is simply $Q^{\mu}(s, a)$ when $a$ is chosen to fit the policy $\mu$, $f$ is the state dynamics function and $r$ is the reward. Altogether, $Q^{\mu}(s_t, a_t) = r(s_t, a_t) \Delta t + V^{\mu}(s_{t+\Delta t})$, setting the discount factor $\gamma = 1$.  In the task given, I am instructed to train

$\langle r\rangle$ based on minibatches drawn from a replay buffer, using a loss function that is half the square error $e^{r} = \langle r \rangle (s\_, a\_) - r\_$, where $s\_, a\_, r\_$ are from the buffer.
$\langle f \rangle$ using a loss function that is half the square of $e^{f} = \langle V \rangle(\langle f \rangle (s\_, a\_)) - \langle V \rangle(s_{next}\_)$
$\langle V \rangle$ by using an analogue to the Bellman error for $\langle Q \rangle$

What I do not see in all of this is the need for a target network $f^{+}$. The use of the target network $V^{+}$ arises because $e^f$ and $e^{V}$ admit two instances of the same network $V$–– thus, to simplify things, one can assign the first instance of $V$ to $\langle V \rangle$ and the other to the target $V^{+}$, before ultimately nudging the parameters of $V^{+}$ towards those of $\langle V \rangle.$ But role does $f^{+}$ play?


text:
Clustering by number of spacial nodes in an area
I have a huge a amount of coordinates (hundred of millions) and I want to group them in areas, where every area should have approximately the same number of nodes and no more than a limit.
I was thinking about implementing a RTree so every Leaf will be an area, but because the amount of data, maybe there is a better way using some clustering algorithm.

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.


text:
Recognize patterns within random sequences
I am familiar with ANNs  as I studied them back in the days for regression and currently I'm working with CNN's for image recognition. But recently I was reading more about pattern recognition in sequences and also sequence prediction, so I end up reading about RNN's and more especifically about LSTM.
However, I realized all the examples I read always use a more or less easy predictable dataset like a normal numerical sequence (1,2,3,4..) or a kind of wave form that repeats some specific shape every X samples. In these cases the RNNs can not only fit a good regression on training dataset but also make a reasonably good prediction for the future. On the other side, with more random dataset it mostly can only fit the existing dataset with a good regression but not really predict anything very well due to the random nature of the data, which is understandable.
The question I am then making myself is: what is the best approach to tackle a problem in between those extremes? Say we have a limited weather dataset (limited in the sense of not extensive enough to make good predictions) as an example. Even if the dataset is not complete enough so we can make general good predictions, it is still possible that one can find some specific patterns within the dataset that may happen kind of randomly, namely, that almost everytime we see a sequence of 2-3 days raising the temperature together with humidity and wind speed, it rains (just as a simple example). So even though our model might not learn to make a good day-to-day prediction, it could understand that this kind of pattern happens every once in a while, so whenever it recognizes this small sequence, it can say with some good probability that it will rain. Another example would be analyzing traffic data on a network. You might not be able to predict what is being transmitted but you might be able to say, for example, a constant stream of data (video) is about to be transmitted when you see someone has requested data from youtube.com.
In those scenarios, I'm not looking for a model that generally predicts the next step of sequence, but rather to identify within an "apparent random dataset" when some small patterns take place. For these cases, is the RNN's (like LSTM) still the best recommendation of model? If yes, is there something one might need to adjust or tune in order to achieve that? I also read a bit about transformers and apparently you can give focus to some specific events in comparison with others. Is this maybe a use case for that?
I hope the question is not broad enough and I'm looking forward for some tips and advices with this regards. Thanks
UPDATE:

when predicting a probability like suggested by maxy, how should I represent the output vector exactly? Having a practical example:

I have 3 features (humidity, temperature and hours of sunshine in a day).
I want to predict the probability of rain on the next day.

So in this case, which I do not want to predict all features, but rather just one of them, how should I design it? I can imagine, from what I understood on the suggestion, I would have something like a LSTM layer, followed probably by a Dense layer and then a Softmax as output layer. To train, I would provide a normalized version of my input (physical values in percentage, degrees and hours) and use a float value between 0.0 and 1.0 as label, where 1.0 is used for the past days that rained and 0.0 for the ones it did not. However this is probably not what was suggested, since I would not be dealing with mean/variance of the temperature, neither with any kind of proability distribution, since my model would then just predict a percentage between 0.0 and 1.0.
Also I am not sure if I follow how to fit step 3 into it. Do you mean using the predicted values to try predicting more steps into the future or as another way of training the model mixing past with predicted data?
Sounds like many questions from my side but I find the approach quite interesting and would like to understand it better to give it a try. But for that, I need to understand how exactly is the data format. If you could use my example above to illustrate what you mean, would be very helpful! Thanks

Elaborate follow-up questions don't work so well on this site. Or asking multiple things at once. It's better to make a new question at this point, I think. (You can link them for context, but ideally they are self-contained.)

Can you please put your specific question in the title?

Instead of predicting a single output value (like next temperature), you predict a probability distribution (a range of plausible next temperature values).
Then you can do this:

Feed the known past into your model.
LSTM is a good model choice for this. You could also do something much simpler, e.g. a linear model or a look-up table, just taking the last 5 time-steps as inputs.

Predict a distribution over the next inputs.
For language models, this is a categorical distribution: a probability for every possible next token. (A softmax layer transforms a bunch of numbers into a bunch of probabilities that sum up to one, exactly for this purpose.) For weather data, you could predict the mean and variance of the expected temperature in one hour, same for humidity, etc.
(Predicting a normal distribution may be too simple because it's unimodal, but you can always predict a mixture of multiple normal distributions instead.)

Take a random sample from the distribution that you predicted, add it to your "inputs", and go back to step 1.

You can obviously train your model on historical data (step by step), and the loss function of choice is the cross-entropy loss (which measures the difference between two probability distributions: your output, and a usually very sharp "what really happened" kind-of-distribution).
If your model is flexible enough (e.g. a large LSTM), it will be able to learn (reproduce) the kind of random trends you had in mind. The next question is: how do you make use of the fact that it learned them?
For some known input history, you can repeat the above process to 100 times, and you get 100 possible futures. (Or 100 possible text continuations, in case of a language model.) All of them are somewhat plausible (according to your model).
Often, you want a "representative" future, instead of a random future. You could always pick the most likely prediction (instead of a randomly sampled one). Maybe have a look at "beam search" (for language models). It depends on what you want to do with your model. But the point is, the way you build the model is the same either way.

Suggested reading: (all of them "solid theory", but in different directions)

Kalman Filters
MacKay's book on Information Theory (free PDF download)
(Partially Observable) Markov Decision Processes

Thanks for the answer. The way I imagine using the model is something like (as example): if I predict that the probability of raining is greater than X %, I would cancel an outside event. Having the event with raing causes losses due to damages from water. Cancelling the event as well. But finding a correct X might give me an optimal balance. For that reason I updated my post with some question about your answer, in case you have time to take a look at it. Thanks

The way you put it fits into decision theory. You have a cost function ("loss") that you want to minimize. (In expectation, meaning: on average over multiple instances.) But this takes quite a different spin from your other follow-up questsions. I would suggest to study some probabilistic ML theory, maybe logistic regression (which is equivalent to a simple NN with yes/no output).


text:
Is manual binding output to input also an AI?
I know AI is primarly training a machine by samples of input-output in order it would learn itself about relations between the input and the output.
What if I manually add the relations? Is that still can be called a type of AI? If so, what type is that within the context of AI classifying?
In other words, does manually bound inputs and outputs belong to some "official" sort of modern AI due to specific case(s) when we need such a manual binding instead of training a machine? Or AI is exclusively creating relations by machine learning only?
For example, if I save some text notes (output) and add multiple tags manually to each of the notes in order to find a note in the future by a few tags inputted (input), is such a search system a type of AI? Or I can call it just "intelligent", but not AI-like? Where is the "official" difference?

@user253751 Thanks for the comment! I added an update to make it clearer

AI is not exclusive to machine learning. Back in the early days of the AI, in the 1960's or thereabouts, they were programming rules-based systems. A lot of work was done on manipulating facts and using combinations of facts to derive new facts. In 1972 the Prolog language was created which does exactly this - the programmer defines a bunch of rules and then the interpreter finds ways to answer queries using the rules. (E.g. "Tom is a cat. All cats are animals. What animals exist?" -> "Tom" [but written in Prolog and not in English])
There was also work on natural language processing - you can see e.g. SHRDLU. It was like ChatGPT but it only knew about stacking blocks and it was written in 1968 using Lisp. PUT THE BLUE CYLINDER ON THE RED CUBE. WHERE IS THE BLUE CYLINDER?. Today we're quite impressed to see ChatGPT answer this, but it's also easy to see how it could be done simply by adding enough rules to the system. SHRDLU would not only answer it, but also do it with a robot arm. Oh look, they also made DALL-E in 1985.
Nowadays, AI and Machine Learning are near-synonymous, because the other AI stuff - such as natural language processing - is no longer called "AI".
Your tag search system is not AI in any sense of the word - it's just a tag search system. Such rigid mechanical systems were never called AI.

Do I understand you correctly: a system built by ONLY manually added rules can not be classified as a modern AI? (could be in the past, but not nowadays). The rules rather are being manually added to a modern AI on two cases only: 1) as algorithms before the training a model and 2) later when a programmer can add some rules directly to the model to improve it, correct? Hence, a modern AI is mostly built by machine learning instead of manually added rules. You may add it to your answer as a summary or takeaway

Okay, thanks again! What one more a little other question: you wrote natural language processing is just an AI stuff. Why? I read it's a "branch of artificial intelligence or AI". I mean, is this a type of AI or just stuff, branch, subfield, component of AI etc?

It seems to be a reasonable idea combining machine learning with predefined knowledge.
A recent example of this symbiosis is Wolfram plugin to provide hard-coded mathematical and data information into the conversational capabilities of ChatGPT.
This addition improves truthfulness in the enhanced hybrid (ML+handcrafted) AI system.

I think Wolfram itself used the label "computational intelligence".


text:
using bitmaps for storing and analyzing numeric data (use case: flight sim game AI)
I had idea of storing flight mission data in bitmaps for analysis.
Data stored could be velocity vector of our aircraft, velocity vector of target, location relation vector between the two, aim lead angle as a vector... and maybe a few more such as altitudes, etc. we could end up in 30 or so parameters that we want to store. At least for most the 0...255 resolution is enough.
With each RGB pixel storing 3 parameters we could have snapshot of the situation stored in an 12 pixel image with room to spare.
A small 64x64 image could store 340 samples (seconds) of gameplay. This would be a long enough to represent an encounter in the game. So say I then fly (with human as a pilot) gazillion missions and store these pictures as samples of each encounter.
Goal would be to have a system that would look at current "12 pixel sample" and based on that tell us what the best next "12 pixels" options are.
Are there any benefits in storing data in image format in terms of analyzing the data and suggesting next successful action. Naturally same data can be just tables or similar but would the image format give allow leveraging existing image tools or similar benefits?
cheers.
And by the way I am a total amateur when it comes to algorithms etc. But would still appreciate any input.

The image processing and related models like CNNs are using the fact that pixels close together in an image are more related and small translations in objects are not important so using images do not necessarily improve results as there are also few pixels. Another problem may be when data is stored, it may be compressed such as by jpeg if not careful and lose data. Otherwise, should have little overhead (from image headers) compared to pure binary files. Text files on the other hand would likely be a bit larger but would be much more readable (and flexible).


text:
Would converting embeddings updated by a transformer into tokens (eg: by searching for the nearest embeddings) produce results that make sense?
In the transformer architecture, one of the step is to update the embeddings, allowing words to pass information to whichever words they are relevant to. For example in the sentence "a small cat is walking in the garden" we want the embedding of "small" to cause a change to "cat" and to moves it to a different part of the embedding space that more specifically encodes a "small cat". "small" is an adjective but the same thing can be done with verbs, pronouns, ....
If we would take the updated embeddings and start looking for the closest tokens (the ones that minimize distance), what would we got ? Are the updated embeddings a vector that only make sense inside the transformer layers and for prediction (the last step), or do they somehow still have a relation with the initial embeddings used for tokens ?
Is this plausible that the updated embedding of "small cat" to be closer to "kitty" than "cat" ?

The typical example used to explain word embeddings to beginners is to conceptualize each word embedding as a vector. When a word embedder is successfully trained, it should enable operations such as:
$$
\text{king} - \text{man} + \text{woman} = \text{queen}
$$
Let each embedding be denoted as $\vec{x}_{t}$, where $t$ represents a word. . In practice, the following quantity is computed:
$$
\vec{y} = \vec{x}_{\text{king}} - \vec{x}_{\text{man}} + \vec{x}_{\text{woman}}
$$
Next, one searches for the closest embedding $\vec{x}_{t}$ to $\vec{y}$. In this case, $\vec{x}_{\text{queen}}$ is expected to be the closest embedding to $\vec{y}$.
Thus, to answer your question: yes, this has been implemented in studies such as "Linguistic Regularities in Continuous Space Word Representations" by Mikolov et al.


text:
How to extract product codes within unstructured data
I am working on a project where I have a dataset consisting of unstructured data from multiple ERP systems. Each dataset (extracted from an ERP) has different columns, and unfortunately, there is no standard format for the data. Among the columns, there is a product code, along with other product-related information. The product code can be in various columns, or even within a larger column description.
My goal is to extract the product code from each row from this unstructured data. I am looking for advice on which type of model or strategy I can apply to extract the product codes automatically.
Here are some key points for consideration:

The data is unstructured and comes from various ERP systems;
There is no standard format for the columns or the product code placement;
The product code can be within a larger column description;
The extract data isn't natural language (it doesn't have a syntax). I don't have actual sequences. The columns are extracted from an ERP system and they basically contain a bunch of keywords, like: '3/4" SHOE RED BLUE NIKE", stuff like that;
The are millions of possible product codes.

Any suggestions or recommendations on models, strategies, or tools that would help me achieve this goal would be greatly appreciated. If you have any experience with a similar problem, please share your insights or any relevant resources.
Thank you in advance for your help!

It would be very useful to see an example of the data. Like an image? Also, please explain what an ERP is as not everyone will know and how that information is needed for understanding the case.

There are two class of methods that could potentially solve this: hardcoded rules or learning. As with any other problem, go for the first one (rules) first and only if it's not enough go for the second one (learning).
Here's how I would proceed:

try to think about what makes something a product code or not. For instance, if you get a column and split it in strings separated by spaces or column separators, you could find things such as

minimum length,
combinations of letter, numbers and characters that required, allowed or forbidden,
maximum length,

Even if these criteria aren't enough to get exactly the codes you want they will help you on the next step. The important thing is that as much as possible you shouldn't miss any valid code. You should refine the rules as much as possible and if they aren't enough you can start learning.

With the first method you have a way of generating a dataset of potential codes. Now you should look at the dataset and find those datapoints that aren't codes. If you can create a rule to exclude them, do it. If not, annotate them by hand (this is, add a "wrong" flag to those). Then you can train a model to classify the potential codes as correct or incorrect.

Let me know if this didn't answer your question and if you have questions :)

Thanks for your detailed response! I have a couple of questions:

Unfortunately we can't specify what makes a "product code". The brand of the project has it's own rules, and sometimes the brand doesn't follow that rules either. So, one product can have a product code of "ABC" and another one "13123KDD-D/3". The only thing that is certain is that the product code is usually a random string combination (with some arbitrary logic)
(continue on the next message)

One thing that I know by exploring the data is that the product code is usually composed by 2 parts. The first string is usually the "real" code (that is shared across multiple products), while the rest of the code is usually a "description" (think about a "product type" + "product specification"

Based on the above, I was able to code a NER model using Spacy that identifies both parts (the "product code" and "product description"). I got ~80% accuracy, but it miss most of the times the "description", while it gets right most of the code itself.
I tried using Spacy Spancat, but I was not able to do this. Do you have any suggestion how to improve that? Specially the part of getting the actual description.


text:
How to extract the high-level features of YOLOv5?
I am faimilar with extracting the high-level features from any pretrained model for classification problem such as ResNet version, VGG, etc. It is easy to extract the features because there is a fully connected layer after the convolutional layers and the high-level features are usually extracted from the penultimate layer (the output of neurons of the layer before softmax layer).
Today, I spent a lot of time to understand how I can extract the high-level features from YOLOv5.
Firstly, I want to ask please, how do the high-level features look like? I mean what is the shape of the features, because I do not see any fully connected layer in the repository here.
Second question, from which layer should I extract the features of final predicted objects?
Thank you so much for any help! I am really stuck for a long time. :(

I recommend you read this StackExchange post.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

Set export = True at the yolo head. If the net is the AutoShape instance, you can achieve it by:
m = net.model.model.model[-1] if net.dmb else net.model.model[-1]
m.export = False  # Allow obtaining features after detection/segmentation head

Also, you might need to fix non maxima suppression to pass the correct features.


text:
Summarizing articles with ChatGPT
You cannot upload files such as articles on ChatGPT. As an alternative I provided links to ChatGPT of the corresponding webpages but it is clear from its summaries that it is not able to open the link and is not actually summarizing that article. Similarly, telling it the title of the article and author doesn't work because it does not access the article. For example, if you give it the prompt "Write the 4th paragraph of this article: https://www.msn.com/en-ca/news/canada/this-ontario-teacher-wants-schools-to-be-more-open-to-muslim-student-needs-starting-with-prayer/ar-AA19wzbe?ocid=hpmsn&cvid=b735bc2290e14ee385647759d7de753f&ei=11 " it will not write the correct paragraph so I am not sure how it summarizes the article. On the other hand, you need to pay to access some articles so I find it unclear how it summarizes those articles. Also, the articles can be long so if you paste sections in separate chunks ChatGPT will summarize each section, which is not ideal because it should summarize the whole article at the same time.
How do you summarize articles with ChatGPT?

I usually copy paste the whole article and ask him to summarize . it works for me .

Although, you can't trust it to not make something up in the process.

This prompt worked for me with ChatGPT.
Can you summarise this
https://www.msn.com/en-ca/news/canada/this-ontario-teacher-wants-schools-to-be-more-open-to-muslim-student-needs-starting-with-prayer/ar-AA19wzbe?ocid=hpmsn&cvid=b735bc2290e14ee385647759d7de753f&ei=11

If you want a more elaborate summary. Use the prompt
Can you elaborate?

Note: I have not verified the correctness of the summary.


text:
Do you know of a website/resource that gives the latest status of "deep fake" use cases/tools?
I'm looking for a good summary of the state of tools for making AI generated videos of people talking. This can be no-code, low-code, or running/tweaking actual code.
Here are some use-cases. I would like to know, for each of these, what tools exist, what tools might exist soon, etc. I'm using Donald Trump as the example, but it could be any person.

Start with a video of me talking and making facial expressions, and create a video with Donald Trump's face making the same expressions and mouth movements, and still with my voice. (It looks like https://github.com/deepfakes/faceswap does this well.)

Like #1, but with Donald Trump's voice instead of mine. (I looked at voice.ai for this, but it didn't have great quality, and also required you to do a fairly good impression of the person to start with. It also has what seems like a flaky/buggy UI. Plus I'd like an opensource tool. I also looked at fakeyou.com and wasn't able to get an example without paying.)

Write a script, perhaps including some direction for facial or hand expressions, and create a video where Donald Trump speaks the script in his voice, with his cadence/speaking style, and ideally with his corresponding facial and hand mannerisms / body language.

I did do a bunch of googling for this stuff. I got buried in an avalanche of marketing speak and hucksterism. I'm looking for a reliable index of tools that actually work. Naively, based on how well visual training works, I would think that a model could be trained on lots of Donald Trump talking videos, and then you could give it a script and it would do the voice, cadence, mannerisms, expressions. I did read that voice is just harder than video, because of the many overlayed frequencies. Does that make #3 impossible currently or in the near future?


text:
Why does the accuracy of my neural network stay constant?
I'm testing my own implementation of a neural network on recognising the type of a function. I generate sine, linear and quadratic functions with random parameters, compute their values for a linspace of size 100 and pass the y-values as input into the network, expecting a vector size 3 as the output.
I've already checked my gradient with the one returned by the numdiff library and it's spot on. The input is normalized.
The structure of the neural network is 100 nodes in the input layer, 10 nodes in the hidden layer and 3 nodes in the output layer. These are the results I got for 200 epochs with batch size 20, learning rate of 0.001 and 400 training samples:

The cost (E_mean) is decreasing but the accuracy on training data isn't. What happened too was that the accuracy has sky-rocketed but then dropped immediately.
I'd be grateful for any help!

If the accuracy is sky-rocketing unexpectedly during training (and then dropping immediately), my first hunch would be to experiment with the learning rate - it seems that the learning rate may be too high. Please remember that the learning rate is just a parameter that magnifies your step in the direction of the gradient calculated at point (t). It means that in (the case of first-order optimization) there is nothing that prohibits it from overshooting the local or global minimum.
The second thing that I would like to inspect is the architecture of the network. Given that you have 100 inputs and then only 10 neurons, it is very likely that the architecture is just too simple to approximate the target function. However, this seems strange, as you are trying to approximate the sine, linear and quadratic functions. This should be feasible by using a rather simple architecture. I omit here a detailed explanation of the approximation of common functions. If you are interested, jump straight into an approximation of common polynomials and the minimum number of parameters required to do so.
Finally, I wonder about your samples. You say that you generate some common functions and then sample points that serve as training data. Please keep in mind, that your sample should be reflective of the data-generating function. If it not (e.g. you sample only around some small interval) - you will not approximate the function correctly, even if your acc is 100% ;)

Your network is always guessing the same output. I would recommend using cross-entropy loss rather than mean-squared error.

One cause I can think of is that your network is too simple for the task. That would explain why the  cost stops decreasing at around 0.33. Your network might be outputting the same prediction independent of the input, by using the bias values and setting the weights to low values. That would also be the reason why the accuracy is 0.33, because one third of the time the network is lucky and its guess matches the input.
Another problem could be your that you are using a bad model architecture, for example using the wrong activation function. I suggest using the relu or the sigmoid function.
I hope you are able to solve your problem.


text:
Blurring of image in generative model using diffusion probabilistic method
In the 2015 paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" by Sohl-Dickstein et al. on diffusion for generative models, Figure 1 shows the forward trajectory for a 2-d swiss-roll image using Gaussian diffusion. The thin lines are gradually blurred into wider and fuzzier lines, and eventually into an identity-covariance Gaussian. Table App.1 gives the diffusion kernel as:
$$
q(\mathbf{x}^{(t)} \mid \mathbf{x}^{(t-1)}) = \mathcal{N}(\mathbf{x}^{(t)} ; \mathbf{x}^{(t-1)} \sqrt{1 - \beta_t}, \mathbf{I} \beta_t )
$$
The covariance of the diffusion kernel is diagonal, so each component $x_i^{(t)}$ (i.e., each pixel in the image at time step $t$) is independently sampled from a 1-d Gaussian based on the prior time step's pixel value at the same x-y location in the image. So a given pixel should NOT diffuse into neighboring pixels; instead, the action of the diffusion step is a linear Gaussian 1-d transformation of the number held in the pixel, with the mean slightly reduced and some noise added.
Question: This seems inconsistent with Figure 1? Instead of the blurred line (wider and fuzzier line), we should have a line that has the same width, but exhibits more noise? In order to have a pixel diffuse into neighboring pixels, we would need a diffusion kernel with a non-diagonal covariance, so that there is nonzero covariance between components?

Figure 1. The proposed modeling framework trained on 2-d swiss roll data

The data is actually 2d coordinates, plotted on a scatter-plot. It doesn't represent an $n \times n$ image with individual pixels. And the added noise is 2d, well 1d to the x-coordinate and 1d to the y-coordinate, if you consider them separately.
But when the process is applied to images, as seen in Figure 3, it behaves as you described and expected the result to look like.

I did not realize that "2-d swiss roll data" is actually represented as a (x,y) scatter plot, thanks very much!!


text:
Can we use Low rank approximation in Markov decision process problem?
I am newbie in MDP.I have started reading Ronald Howard Dynamic Programming and MDP book as well as Sutton and Barto An Introduction to Reinforcement Learning. To my understanding MDP is a model based (not free) process unlike SARSA and TD. The toy examples presented in both books (specially Ronald Howard's that I have run in my computer) for the value iteration have the element by element multiplication of matrices (Hadamard product) $\Pi$ (t.p.m) and $R$ (reward matrix).
$$q = \Pi \odot R$$
$q$ matrix might have large dimensions (columns).Is it proper to factorize this $q$ matrix (decompose it) ? I was thinking low rank approximations. But I have a concern: If I lose some elements of this $q$ matrix I will lose some paths, possible actions and policies that will not lead me to my target state. Am I correct ?
In addition I reading approximations of policy iteration, approximation of value iteration and so on.What do they approximate ? Do they lose some potential paths as fas I have understand ?
I would like some help here in order to figure out what is going on with MDP and RL in general.

Can you please put your specific question in the title? "Low rank approximation of MDP" is not really a question.

Regarding "If I lose some elements of this q matrix I will lose some paths", I think your system won't reach the target, so you have to set different scenarios. By the way, what case will you loose elements?


text:
Query regarding the minimax value function of GANs
In the book Generative AI with Python and TensorFlow 2 from Babcock and Bali (page 172), it is stated that the value function of a GAN is the following:

where D(x) is the output of the discriminator and G(z) is the output of the generator. However I don't understand why there is a product of two logarithms. The value D(x) is supposed to be a probability, meaning that D(x) lies in the number interval between 0 and 1. Having that into account the log D(x) would be a negative number, so log log D(x) shouldn't exist because the log of a negative number doesn't exist.
Can anyone shed some light into this? Is the function wrong or is there anything I am missing?

Instead of writing "Query regarding...", can you please just put your specific question in the title?

Assuming that there is no other trickery done in the book regarding notation, I'd say this value function is wrong. Minimizing log(log(x)) is similar to minimizing log(x). However, log(log(x)) indeed cannot take values lower than 1, which is what the discriminator would output.
Maybe someone else who has access to the book can clarify exactly how they come to the value function they use. Until then, I'd recommend simply thinking of the value function that is used in the original GAN paper (Goodfellow et al. 2014)
$$min_Gmax_D(D,G)=E[log(D(x))] + E[log(1-D(G(x)))]$$


text:
Playing around with Transformer - accuracy not improving
I am playing around with a decoder only transformer model,
The Colab is here if you find that easier
https://colab.research.google.com/drive/1SHyJ9Oa3E4j1x8YFlXQbd1mjUjWhHGOV#scrollTo=60e13119
or see the code below (should run with minimal deps in a notebook)
Goal:

I am playing around to understand the transformer architecture. I wanted to build a minimal version of a decoder only model and see if i can train the model to predict the last digit in this sequence (think of it as a recall function from a key value store).
Want to get an intuitive understanding on the linear transformations of the weight matrices to the sequence input + positional encodings.

n9 v8 a5 p7 k0 j1 i3 e2 g6 c4 c - (should be 4)
d5 t3 q8 r7 y1 i0 c2 n9 s4 u6 i - (should be 0) ...
l8 u9 p5 y3 f7 k0 g6 v4 r1 x2 l
a7 x5 b6 v0 i1 f3 z9 d4 y2 k8 x
m9 h4 g5 t2 l3 f1 w7 b6 a8 j0 g
x0 g7 q9 u2 j8 v4 h3 o1 f5 r6 r
r6 c4 d0 p3 j2 g9 a7 n1 e8 l5 d
r2 z7 y6 x5 v4 u1 s3 a8 l9 p0 z
k0 u3 t1 r4 g8 p2 j5 x9 s7 v6 t
o7 a1 u3 r2 k6 j0 m8 y9 e4 c5 j

Questions:

The accuracy is not improving, so i guess there is some fundamental issue with the model. It looks like it is learning that it is a digit, but not what digit.

Code:
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import torch.optim as optim
import random
import string
import time
import math

if torch.cuda.is_available():
    dev = "cuda:0"
else:
    dev = "cpu"
device = torch.device(dev)

class PositionalEncoding(nn.Module):
    """
    compute sinusoid encoding.
    """
    def __init__(self, d_model, max_len, device):
        """
        constructor of sinusoid encoding class

        :param d_model: dimension of model
        :param max_len: max sequence length
        :param device: hardware device setting
        """
        super(PositionalEncoding, self).__init__()

        # same size with input matrix (for adding with input matrix)
        self.encoding = torch.zeros(max_len, d_model, device=device)
        self.encoding.requires_grad = False  # we don't need to compute gradient

        pos = torch.arange(0, max_len, device=device)
        pos = pos.float().unsqueeze(dim=1)
        # 1D => 2D unsqueeze to represent word's position

        _2i = torch.arange(0, d_model, step=2, device=device).float()
        # 'i' means index of d_model (e.g. embedding size = 50, 'i' = [0,50])
        # "step=2" means 'i' multiplied with two (same with 2 * i)

        self.encoding[:, 0::2] = torch.sin(pos / (10000 ** (_2i / d_model)))
        self.encoding[:, 1::2] = torch.cos(pos / (10000 ** (_2i / d_model)))
        # compute positional encoding to consider positional information of words

    def forward(self, x):
        # self.encoding
        # [max_len = 512, d_model = 512]

        batch_size, seq_len, d_model = x.size()
        #print("seq_len: ", seq_len, " batch_size: ", batch_size, "d_model: ", d_model)
        # [batch_size = 128, seq_len = 31]
        #print("self.encoding: ", self.encoding[:seq_len, :].shape)
        return self.encoding[:seq_len, :d_model]
        # [seq_len = 30, d_model = 512]
        # it will add with tok_emb : [128, 30, 512]

# Generate random strings
def random_string(length):
    #return "1 2 3 4 5 6 7 8 9"
    a = random.sample(string.ascii_lowercase, length)
    d = random.sample(string.digits, length)
    r = ' '.join([a[i] + d[i] for i in range(length)])
    n = random.randint(0, length)-1
    return r + " " + a[n] + d[n]

# Synthetic Dataset
class RandomStringDataset(Dataset):
    def __init__(self, num_samples, seq_length):
        self.num_samples = num_samples
        self.seq_length = seq_length
        self.data = [random_string(seq_length) for _ in range(num_samples)]

    def __len__(self):
        return self.num_samples

    def __getitem__(self, idx):
        input_seq = self.data[idx][:-1]
        target_seq = self.data[idx][1:]
        return input_seq, target_seq

# Tokenizer and detokenizer functions
def tokenize(text):
    return [char for char in text]

def detokenize(tokens):
    return ''.join(tokens)

# Map characters to indices and vice versa
vocab = string.ascii_lowercase + string.digits + " "
char_to_idx = {char: idx for idx, char in enumerate(vocab)}
idx_to_char = {idx: char for idx, char in enumerate(vocab)}

# Convert tokens to tensor
def tokens_to_tensor(tokens):
    indices = [char_to_idx[token] for token in tokens]
    return torch.tensor(indices, device=device)

# Convert tensor to tokens
def tensor_to_tokens(tensor):
    return [idx_to_char[idx.item()] for idx in tensor]

def collate_fn(batch):
    inputs, targets = zip(*batch)
    input_tensors = [tokens_to_tensor(seq) for seq in inputs]
    target_tensors = [tokens_to_tensor(seq) for seq in targets]
    return torch.stack(input_tensors), torch.stack(target_tensors)

class DecoderOnlyTransformer(nn.Module):
    def __init__(self, vocab_size, d_model, nhead, num_layers):
        super(DecoderOnlyTransformer, self).__init__()
        self.pe = PositionalEncoding(d_model, 128, device)
        self.embedding = nn.Embedding(vocab_size, d_model)
        self.transformer_decoder = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(d_model, nhead),
            num_layers
        )
        self.fc = nn.Linear(d_model, vocab_size)

    def forward(self, x):
        x = self.embedding(x)
        pe = self.pe(x)
        #print("x: ", x.shape)
        #print("pe: ", pe.shape)
        x = x + pe
        tgt = torch.zeros_like(x)
        output = self.transformer_decoder(x, x)
        output = self.fc(output)
        return output

# Hyperparameters

seq_length = 10
batch_size = 16
num_samples = batch_size*1000
learning_rate = 0.001
num_epochs = 100
d_model = 4
nhead = 4
num_layers = 2
vocab_size = len(vocab)

# Create dataset
dataset = RandomStringDataset(num_samples, seq_length)
train_loader = DataLoader(dataset, batch_size=batch_size, shuffle=True, collate_fn=collate_fn)

# Initialize model, loss, and optimizer
model = DecoderOnlyTransformer(vocab_size, d_model, nhead, num_layers)
model.to(device)

criterion = nn.CrossEntropyLoss()
lr = 5  # learning rate
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.95)

for epoch in range(num_epochs):
    correct = 0
    count = 0
    total_loss = 0
    start_time = 0
    log_interval = 100
    num_batches = int(num_samples/batch_size)
    for inputs, targets in train_loader:

        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs.reshape(-1, len(vocab)), targets.flatten())

        probabilities = torch.softmax(outputs[0, -1], dim=-1)
        next_token_idx = torch.argmax(probabilities).item()
        token = idx_to_char[next_token_idx]

        if idx_to_char[targets[0, -1].item()] == token:
            correct+=1
        count+=1

        optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
        optimizer.step()

        total_loss += loss.item()
        if count % log_interval == 0 and count > 0:
            lr = scheduler.get_last_lr()[0]
            ms_per_batch = (time.time() - start_time) * 1000 / log_interval
            cur_loss = total_loss / log_interval
            ppl = math.exp(cur_loss)
            print(f'| epoch {epoch:3d} | {count:5d}/{num_batches:5d} batches | '
                  f'lr {lr:02.2f} | ms/batch {ms_per_batch:5.2f} | '
                  f'loss {cur_loss:5.2f} | ppl {ppl:8.2f} | '
                  f'Acc: {correct/count}'
                 )
            total_loss = 0
            start_time = time.time()
    print(f'Epoch: {epoch + 1}/{num_epochs}, Loss: {loss.item()}  Count: {count}')
```

Ok, so the solution to this was that I was randomizing both the key (a-z) and the value (0-9). That was too hard for a simple model to learn, so only randomizing the value worked. I also changed the positional embedding to be learned, which improved learning speed.
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import torch.optim as optim
import random
import string
import time
import math

if torch.cuda.is_available():
    dev = "cuda:0"
else:
    dev = "cpu"
device = torch.device(dev)

def flatten(l):
    return [item for sublist in l for item in sublist]

def random_string(length):
    a = string.ascii_lowercase[0:length]
    d = random.sample(string.digits, length)
    r = ' '.join([a[i] + d[i] for i in range(length)])
    n = random.randint(0, length)-1
    return r + " " + a[n] + d[n]

# Synthetic Dataset
class RandomStringDataset(Dataset):
    def __init__(self, num_samples, seq_length):
        self.num_samples = num_samples
        self.seq_length = seq_length
        self.data = [random_string(seq_length) for _ in range(num_samples)]

    def __len__(self):
        return self.num_samples

    def __getitem__(self, idx):
        input_seq = self.data[idx][:-1]
        target_seq = self.data[idx][1:]
        return input_seq, target_seq

# Tokenizer and detokenizer functions
def tokenize(text):
    return [char for char in text]

def detokenize(tokens):
    return ''.join(tokens)

# Map characters to indices and vice versa
vocab = string.ascii_lowercase + string.digits + " "
char_to_idx = {char: idx for idx, char in enumerate(vocab)}
idx_to_char = {idx: char for idx, char in enumerate(vocab)}

# Convert tokens to tensor
def tokens_to_tensor(tokens):
    indices = [char_to_idx[token] for token in tokens]
    return torch.tensor(indices, device=device)

# Convert tensor to tokens
def tensor_to_tokens(tensor):
    return [idx_to_char[idx.item()] for idx in tensor]

def collate_fn(batch):
    inputs, targets = zip(*batch)
    input_tensors = [tokens_to_tensor(seq) for seq in inputs]
    target_tensors = [tokens_to_tensor(seq) for seq in targets]
    return torch.stack(input_tensors), torch.stack(target_tensors)

class DecoderOnlyTransformer(nn.Module):
    def __init__(self, vocab_size, d_model, nhead, num_layers):
        super(DecoderOnlyTransformer, self).__init__()
        self.pe = nn.Embedding(32, d_model)
        self.embedding = nn.Embedding(vocab_size, d_model)
        self.transformer_decoder = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(d_model, nhead, batch_first=True),
            num_layers
        )
        self.ln = nn.LayerNorm(d_model)
        self.fc = nn.Linear(d_model, vocab_size)

    def forward(self, x):
        B, T = x.shape
        x = self.embedding(x) + self.pe(torch.arange(T, device=device))
        output = self.transformer_decoder(x, x)
        output = self.ln(output)
        output = self.fc(output)
        return output

# Hyperparameters

seq_length = 10
batch_size = 16
num_samples = batch_size*1000
num_epochs = 10
d_model = 32
nhead = 4
num_layers = 2
vocab_size = len(vocab)

# Create dataset
dataset = RandomStringDataset(num_samples, seq_length)
train_loader = DataLoader(dataset, batch_size=batch_size, shuffle=True, collate_fn=collate_fn)

# Initialize model, loss, and optimizer
model = DecoderOnlyTransformer(vocab_size, d_model, nhead, num_layers)
model.to(device)

criterion = nn.CrossEntropyLoss()
lr = 5  # learning rate
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.95)

for epoch in range(num_epochs):
    correct = 0
    count = 0
    total_loss = 0
    start_time = 0
    log_interval = 100
    num_batches = int(num_samples/batch_size)
    for inputs, targets in train_loader:

        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs.reshape(-1, len(vocab)), targets.flatten())

        probabilities = torch.softmax(outputs[0, -1], dim=-1)
        next_token_idx = torch.argmax(probabilities).item()
        token = idx_to_char[next_token_idx]

        if idx_to_char[targets[0, -1].item()] == token:
            correct+=1
        count+=1

        optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
        optimizer.step()

        total_loss += loss.item()
        if count % log_interval == 0 and count > 0:
            lr = scheduler.get_last_lr()[0]
            ms_per_batch = (time.time() - start_time) * 1000 / log_interval
            cur_loss = total_loss / log_interval
            ppl = math.exp(cur_loss)
            print(f'| epoch {epoch:3d} | {count:5d}/{num_batches:5d} batches | '
                  f'lr {lr:02.2f} | ms/batch {ms_per_batch:5.2f} | '
                  f'loss {cur_loss:5.2f} | ppl {ppl:8.2f} | '
                  f'Acc: {correct/count}'
                 )
            total_loss = 0
            start_time = time.time()
    print(f'Epoch: {epoch + 1}/{num_epochs}, Loss: {loss.item()}  Count: {count}')

# Some evaluation
rs = random_string(10)
print(rs)
t = torch.argmax(torch.softmax(model(tokens_to_tensor(tokenize(rs))[:-1].reshape(1, -1)), dim=-1), dim=-1)
"".join(tensor_to_tokens(t[0]))


text:
Translation Algorithm for CILP model (Garcez, Lamb, Gabbay) - why difference in answer between my implementation and the answer from the book?
I have implemented the Translation algorithm for CILP (Connectionist Inductive Learning and Logic Programming model by Garcez, Lamb and Gabbay) from their book "Neural-Symbolic Cognitive Reasoning"
The paper which describes the example Neural Network can be found here:
https://www.staff.city.ac.uk/~aag/papers/cilp.pdf
My implementation can be found here:
https://github.com/nravindranath10/CILP_Translation_Algorithm
While the book claims that the 2 iterations are required for the Neural Network to converge to the final answer which is {B}, in my code the Neural Network seems to take only 1 iteration to converge to the same answer {B}. I have highlighted the "m = 2" line in my attached image from the book.
I would like to understand if I am going wrong somewhere in my implementation.
Thanks,
Ravi

I don't see any question being asked here. Can you please edit your post to ask a specific question explicitly and put it in the title too?

my question was that my code was taking only 1 iteration to reach the answer while the book says that 2 iterations are required to reach the answer. So I wanted to know there was anything wrong with my code logic. However, now I have figured out the fallacy in my code and have made a small modification to the logic of the code. So now my code too requires 2 iterations to reach the answer. So now my implementation is in agreement with author. So, closing this ticket now.


text:
How do Multimodal LLMs of 2023 score on the ARC benchmark (in 2020: 20% Accuracy)
Q: I wonder if anyone has tried to solve the ARC tasks with one of the state-of-the-art Multimodal LLMs?
Can LLMs that can process graphics input do the following?

(This question is not about the The AI2’s Reasoning Challenge (ARC) dataset, a multiple-choice question-answering dataset, containing questions from science exams from grade 3 to grade 9.)

Background:
I came across a paper from 2019 "On the Measure of Intelligence" by Francois Chollet who is a Researcher at Google, Creator of the Keras Deep Learning Library.
In that paper he proposed the Abstraction and Reasoning Corpus, ARC. Actually you can view this as  a Benchmark that consists of solving 400 graphical reasoning puzzles,  to test AI systems' abstraction and analogy abilities.
To illustrate, here is an example from the paper:

(A machine must infer what to do (transforming a test picture without seeing a "result" picture on the right) from ~3 similar training imagepairs per task)-
2020 state of the art:
In 2020 there was a Kaggle Competition and the 1st place solution got 20% of the 400 task right. (Leaderboard Score in competition was "Percentage still wrong" or something - so lowest score of 79.4% won)
The  C++ code contains special-purpose image manipulation and inference routines, carefully crafted and combined as described in a write-up by the author "icecuber".
Now it's 2023 and OpenAI is making GPT-4V multimodal generally available.
In April 2023 someone asked a similar question on the OpenAI discussion forum; no answers. That post was written independently from this one,  and I just discovered that post right before submitting this post.
Yesterday I have  given ChatGPT-4 Advanced Data Analysis a single example, and the LLM quickly solved the puzzle shown below; sort of. Admittedly, my prompt was  in sloppy natural language, and I uploaded 3 badly cropped training images as input.
Task f8c80d96:

With human intelligence I was not able to infer the solution to task f8c80d96 easily. Made 3 wrong guesses.
(Solution: Add 1 vertical yellow line but also replace black background with grey background)
Clearly my attempt is not a systematic study, just practice.
To conclude, I suspect if GPT-4 can solve 1 task, it can solve many more than 20% of those 400 examples, e.g. when using the API with carefully designed prompts and instructions.

I have gathered a few notes about ARC, with links to projects that use LLMs. Too many links to post them all here.
As of 7-may-2024, Jack Cole+Mohamed Osman solves 34 of the 100 hidden tasks. This is SOTA with LLMs and the hidden dataset.


text:
Are transformer decoder predictions computed in parallel during training?
I've been studying the transformer from the original "Attention is all you need" paper and from various other sources. I have a question about the behaviour of the decoder during training that I cannot find the answer to anywhere.
During inference I understand that the decoder input is its own previously generated token from the prior time step. Tokens are fed into the decoder one-by-one and predictions made one-by-one.
However, during training the target sequence is known and I have read from several sources that the entire sequence is used as the decoder input to allow parallel processing and to improve training efficiency. To keep the decoder autoregressive, a masked attention sub-layer is introduced where a masking matrix is added to the scaled dot product attention mechanism.
So my question is, since during training the decoder input is the entire sequence, is the entire output sequence predicted in parallel (simultaneously), or are tokens predicted one-by-one, as in inference?
To me it makes sense that if the entire target sequence is used as the decoder input, then an entire sequence is output. If it wasn't, the decoder would be using the same input at every timestep whilst being expected to produce different tokens.

Can you please put your specific question in the title? "Transformer decoder during training" is not a question and it's not specific.

In short, masking is just sequence padding in the decoder!
I'll try to slightly rephrase your question first to ensure I properly understood it. You are confused about:

The transformer is fed the whole sequence (during training) for efficiency,
But it needs to be autoregressive so it cannot look at the whole sequence,
So, how can it look at the whole sequence and not look at the whole sequence at the same time?

I was also confused about this, so I'll try to explain.
Say we want to train on a sequence of tokens, i.e. s = [1, 13, 64, 40]. The next token to predict after 40 is 50. Now, instead of having one training sample, s, we can have several sequences from s, such as s1 = [1] (with target 13), s2 = [1, 13] (with target 64) etc., giving a total of 4 sequences.
In total we have the following training data
sample = [
 [1],
 [1, 13],
 [1, 13, 64],
 [1, 13, 64, 40]]

targets = [13, 64, 40, 50]

As usual, Tensors do not like to have weird shapes. Therefore, the training data will look something like this
sample = [
 [1, -inf, -inf, -inf],
 [1,  13,  -inf, -inf],
 [1,  13,   64,  -inf],
 [1,  13,   64,   40]]

In this new tensor, the -inf is the masking! Now you might say, "that's just sequence padding", and it is! The only reason that they call it masking instead of padding is because the architecture in the paper has an encoder (next to the decoder), in which the complete sequences are used without masking. However, when only using the decoder (as in LLMs), masking is essentially just padding.

During training in a transformer model, the entire target sequence is indeed fed into the decoder as input. However, the output sequence is still generated autoregressively, one token at a time. This means that the decoder generates each token based on the previously generated tokens, rather than generating all the tokens in parallel.
To achieve this, the decoder uses a masked self-attention mechanism, where the attention weights are masked to prevent attending to future positions in the sequence. This ensures that the decoder only generates tokens based on the previously generated tokens, as it would during inference.
The reason for feeding the entire target sequence as input during training is to enable parallel computation of the decoder layers. By doing so, the model can process multiple tokens in parallel, which speeds up training compared to generating the tokens one at a time. However, the autoregressive nature of the decoder output generation is still maintained, as this is crucial for ensuring that the model can generalise to unseen sequences during inference.

Attention is all you need, p. 3:
"We also modify the self-attention
sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This
masking, combined with fact that the output embeddings are offset by one position, ensures that the
predictions for position i can depend only on the known outputs at positions less than $i$."


text:
Why clamp std for reparameterization trick between -20 and 2?
In the Soft Actor Critic Paper (found here https://arxiv.org/pdf/1801.01290.pdf), they use a neural network to approximate a diagonal gaussian distribution. In the sample function you can see that it has a function called reparameterize. As you can see in the reparameterize function, we use a tanh function to squash the action bound between -1 and 1. So why is it that we clamp the standard deviation between -20 and 2. I have read both Soft Actor Critic papers and can't find why we do this. Is there something about the normal distribution that would make that range of clamping of the std more desirable? Does bounding the range of the std help with convergence. Is there a paper you my recommend that would help with this answer? This is my first post here so if you need more information please let me know.

Link to code Repo

Can you please provide the code instead of a screenshot? Although note that programming issues are off-topic here.

This is a common trick done in practice: this helps to stabilize training, and prevent large values that can blow up in NaNs.
The reason is that the std of the Gaussian is learned in log-space (because it's easier to learn with neural-nets) that is unbounded ($-\infty, +\infty$), and so you want to bound to something like $[-20, 2]$ because such log-std will be exponentiated later: you have $\exp(-20)\approx 0$, and $\exp(2) = 7.389$ that is reasonably small; a similar trick is also employed for log-probabilities.
Practical tricks like this one are often not documented in the paper (unfortunately), but can be found in code implementations.

Thank you, I tried to look at other methods papers to see if it was documented and could not find any. I really appreciate your answer. One final question if you have a second, so the bound are truly arbitrary and I could make them anything as long as it it reasonably small?

Yes, the bounds are arbitrary but just consider that you're are in log-space and that the $\exp$ can easily underflow or overflow going to infinite. If you want to change the bounds, think about the range of values you want and take the log of that, finally clamp: say you want the std to be in $[1,3]$, you take the log to determine the bounds i.e. $[\log(1), \log(3)] = [0, 1.0986]$ for example.

I finally found a reference to your answer, the stable baselines, https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/distributions.py, say that they clamp to avoid NaN values in the comments of the code. You can find it in the TanhBisector function.


text:
Classifier-Free-Guidance with Transformers

I'm working on music generation using transformers.
Using the decoder part for the audio tokens with text conditioning by the T5 encoder

In Classifier-Free-Guidance, the text conditioning randomly dropped with a probability of $p_{uncond}$. So when it's dropped, what should the cross-attention input be, or how is the transformer architecture modified?
Is it like when the text condition is not present the middle sub-block is skipped with an if statement?

https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569

Is it like when the text condition is not present the middle sub-block is skipped with an if statement?

Answer is Yes. See the link

Cross-attention reduces to self-attention

Hi @Sahil Goyal, and welcome to AI Stack Exchange! It may help the OP if you would add some more details to this answer. Thank you for posting, and we hope to see more of your posts here in the near future!

Can you provide references from previous works?

A recent reference that I remember is, https://arxiv.org/pdf/2301.00704 (see section 2.7).

Yes, Thanks! Also In below link (music gen using Transformers), there's a if statement as I asked in the question.
https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569


text:
How can an MLP be implemented with convolutional layers?
I am studying the architecture of the network pointnet, specifically the MLPs stages of the pipeline highlighted in red in the following image (taken from the author page here):

It is strange to find out that this part was implemented with 2d convolutional layers by the authors, here is the snippet where they create the MLP(64,64) part (here you'll find the complete code base):
 net = tf_util.conv2d(input_image, 64, [1,3],
                     padding='VALID', stride=[1,1],
                     bn=True, is_training=is_training,
                     scope='conv1', bn_decay=bn_decay)
net = tf_util.conv2d(net, 64, [1,1],
                     padding='VALID', stride=[1,1],
                     bn=True, is_training=is_training,
                     scope='conv2', bn_decay=bn_decay)

Same thing happened when I looked into the pytorch implementation, but now with 1d convolutional layers! (code base here):
 self.conv1 = torch.nn.Conv1d(k, 64, 1)
    self.conv2 = torch.nn.Conv1d(64, 128, 1)
    self.conv3 = torch.nn.Conv1d(128, 1024, 1)

The extent of my knowledge is that MLPs are densely connected networks (MLP definition), even more so, in this application of pointnet the authors depict the MLP stages as fully connected networks:

I am lost here, for what I can gather from the images there should be a dedicated MLP to transform every point independently, how does convolution help in this situation? Your insight will be greatly appreciated.
ERRDATA
I have just figured that the kernel size in the original tensorflow implementation is [1,3], meaning, that is also a 1D convolution, are they feeding the convolution of the three channels to the MLP?
Regards

When I try to understand convolutional layer, I always look at the shape of filters. I try to stack matrices in my mind and visualize the convolution.
If we consider that each point has 3 features (x, y, z), than k will be equal 3 in this particular layer  self.conv1 = torch.nn.Conv1d(k, 64, 1). The weight matrix of this layer will have the shape (64, 3, 1). Meaning we have 64 filters in total, each with shape (3,1).
You can think of 1 filter as the weight in a neuron. The number of neurons will be equal to the number of filters. The input to your neuron would be the point, that has shape(1,3). This point will then get multiplied by the weight matrix ( aka. X * W + b, bias is optional). Each point will be go through each filter, that is why I think they call it "shared".


text:
Is reinforcement learning the correct approach for power source location?
I am trying to attack a problem for my thesis and I feel RL is the correct framework for this problem, however I am completely new to this topic and I am not sure about it.
The idea of the problem is to locate the position of a power source, in order to obtain the position I can perform several questions asking if the source is located in that interval. If the source is inside the interval I will obtain higher power, in other case the received power will be lower (stochastically). Additionally, if the interval is greater or smaller the received power change, the most narrow the interval the greater the power.
In my personal opinion the problem fits quite well the idea of reinforcement learning of discovering an environment by an agent. However, I do know how to model the rewards (maybe with the received power). Finally, the biggest problem is related to the states, in the RL literature I read about "gridworld" or similar problems but in my case I do not feel comfortable defining the state space. I thought about compute the posterior distribution in a Bayesian sense and using that distribution as state.
Thank you in advance and sorry for my English. If you have some recommendation about RL literature to strong my knowledge I would appreciate it.
EDIT:
Suppose we have a noisy function $f(x)$ where $x\in[-10,10]$. The idea is try to find the value $x^*$ which maximimize the function. For that, we can try $K$ different intervals $A_t=[a_t,b_t]$ with $t=1,\dots,K$. For example $A_1 = [-1,0]$, $A_2=[-10,0]$, etc.
For each tested interval we obtain a sample $y_t$ which depends on two factors:

If the optimum $x^*$ is inside the interval, the average power of $y_t$ will be higher.
The width of the interval, in other words, if $x^*=0.5$ and the tested interval si $[0,1]$, the average power of $y_t$ will be higher than using $[-10,10]$.

The action space would be the intervals $A_t$ and the state space is my doubt. I thought about computing the posterior distribution of the parameter $x*$ and update after each new sample $y_t$, therefore each state $S_t$ would be the posterior probability distribution computed with the samples $\{y_1,y_2,...,y_{t-1}\}$. I mean $p_t(x^*|y_1,\dots,y_{t-1})$.

What is the "interval" you mention?

What is your state space and action space?

Thank you for your responses. I have updated the explanation, i hope it clarifies the problem.

Given an interval $A_t$, how do you evaluate $f(x)$? I mean you sample a value from the interval, i.e. $x\sim A_t$, then compute the function? I'm wondering whether the agent should also predict $x$..

Sorry. I deleted my previous comment because it was incorrect. The idea is evaluate the function piecewise in this manner:
\begin{align} f(x) &= \frac{P}{\Delta_t} + n, \text{if}\ \ x^\in A_t\ f(x) &= n, \text{if}\ \ x^{} \notin A_t \end{align}
with $n$ gaussian noise $\Delta_t$ is the width of $A_t$ and $P$ the power of the source.

Alright, so how do you know if $x^*\in A_t$? You can compute that?

No, we only get a sample $y_t$ which can be $\frac{P}{\Delta_t}+n$ if $x^*\in A_t$ and only noise if It is not inside. (Sorry if I havent explained propertly)

RL approach
I guess this problem has not a clear definition of a state space, unless you can describe the power source by some features that change in time.
Anyway, if so, it seems more like bandit problem rather than RL. As I understand, the hidden location of the power source remains the same during time, so it's like having only an initial state.

Your reward function $r(s_t,a_t)=r(A_t) = y_t$, where $y_t = f(x; A_t)$.
As you said you have two actions that define an interval $A_t=[a_t, b_t]$; probably is better to predict an offset: say your range is $[-10, 10]$, the agent predicts two offsets $\Delta_a,\Delta_b$ (both $>0$) such that $A_t=[\Delta_a-10,10-\Delta_b]$.

Since you obtain a sequence of $y_1,y_2,\ldots, y_t$ you may try to recover $P$ by averaging out the (Gaussian) noise $n$ if possible, and multiplying by $\Delta_t$ (computed from $A_t$). If you're able to do this you can consider it as your current state, even it's not a proper state.

If you frame the problem as a bandit task, if I recall correctly, you don't need a state at all because you only care about the actions.

Lastly, you may want to consider genetic algorithms (GA) and simulated annealing too.

Algorithmic approach
Are you sure that an algorithm based on dynamic programming (by working on a discretized range of intervals) or divide-et-impera won't work?
I mean if you consider a grid $N\times N$ that discretizes the initial interval $[-10,10]$, you can design an (inefficient) algorithm that loops over each $N^2$ combinations of intervals, evaluating $f(x)$. Then, you keep track of the interval that achieves the higher $y_t$, and that will be your (approximate) solution.
Maybe you can star from such simple description to design something more sophisticated, or even implement both approaches and compare them to see which is the winning method.

Thank you for your time and for your response. I am goint to try to attack it from bandit perspective. About the algorithmic approach it is true that is a way to solve, however as you said it is inneficient. I am trying to solve this problem efficiently and accuratly that is the reason I tried to find a broader framework like RL to implement classical methods like Q-learning to my problem.