text:
How can a convnet learn with a 3x3 output layer?
I was studying the "Deep Learning with Python" book, I came across this MNIST example and this is how the last conv2d layer looks like:
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 3, 3, 64)          36928
=================================================================

I have a hard time understanding how can this neural network figure out any feature from such a small image like 3x3 pixels. It would make sense to me if it were something like 10x10 but a 3x3 image makes no sense when I look at one, yet the network can achieve %99+ validation accuracy. How is this possible?

Do you know what (None, 3, 3, 64) means? Because it seems that you're assuming that the input image to the convolutional network is 3x3. 3x3 is probably the size of the kernel. So, please, edit your post to clarify that.

@nbro I think I have a vague understanding of what these parameters means, I don't think it is the kernel. It is the output shape from a conv2d layer.

Well, which software library is the author of the book using? You should edit your post to include this detail. The answer below assumed that you're using TensorFlow/Keras, and that's probably a good guess.

I'm going to assume that what you posted is the output of something like model.summary() from TensorFlow/Keras. With that assumption, (None, 3, 3, 64) is the output shape. We can ignore the None, as it usually encodes the batch size.
Thus, the output of the last layer can be seen as a 3x3 image with 64 channels. Alternatively, you can think of it as 64 3x3 images. For more information about shapes, see this question: Keras input explanation: input_shape, units, batch_size, dim, etc
So what is (probably) happening, is that given a large(r) input image, the neural network extracts relevant features which are very good at describing the validation data set.
Answering the question of how this output is enough to describe the validation data set so well is probably a bit harder. If you'd like, you can chalk it up to neural network "magic". But keep in mind that the MNIST dataset isn't really that "hard" - at the end of the day, it is a database of handwritten digits from 0 to 9. So it isn't that suprising that 3 * 3 * 64 = 576 numbers would be capable of describing the data set to a degree that allows high accuracy. In fact, outputting 9 numbers would be enough, if each number encoded the probability of high likely it is that the digit in the image is a 0, 1, 2, etc..
Nonetheless, you might want to take a look at what those 3x3 images look like. In that case, How to Visualize Filters and Feature Maps in Convolutional Neural Networks might be of interest to you.


text:
Why do we use the same parameters for the joint, marginal and conditional distributions in VAEs?
I've noticed in several resources on variational autoencoders (for example the wikipedia article), we use the same parameters theta for the prior, likelihood, posterior, etc distributions. For example the equation $p_\theta(x) = \int_z p_\theta(x|z)p_\theta(z)dz$. Aren't $p_\theta(x)$ and  $p_\theta(z)$ two different distributions, so how can we parameterize them with the same $\theta$ params. I might be misunderstanding something about what it means to parameterize a distribution...

I think this is very confusing to many people. I had to deal with VAEs (and Bayesian neural networks) multiple times in the past, and I've seen so many inconsistent notations and unclear explanations. Here's what I know about the issue that you're raising, but I will not give you a definitive answer.
In the appendix of the VAE paper (sections C.1 and C.2), they consider the cases of

a multivariate Bernoulli decoder, and
Gaussian encoder and decoder

They write

in the case of the multivariate Bernoulli decoder, $\theta$ are the weights and biases of the neural network that outputs the Bernoulli parameters (each Bernoulli distribution has only one parameter that basically defines the probability of getting one sample rather than the other - this is not written in that section - I am just telling you this, in case you didn't know it).

in the case of the Gaussian encoder, $\phi$ (which parametrizes the variational distribution $q_\phi$) are the weights and biases of the neural network that represents the encoder (which outputs the $\mu$ and $\sigma$ of the Gaussian, which are the parameters that define the specific Gaussian - you can see this from the equations in appendix C.2)

in the case of the Gaussian decoder (appendix C.2), the weights and biases of the neural network (the decoder) are part of $\theta$ (I don't know what they mean by "part of" here, honestly).

So, in the case of the VAE (with a multivariate Bernoulli decoder, or a Gaussian encoder and decoder), which is really a neural network (or 2 neural networks - encoder and decoder) that attempts to model the random process which is represented by the graphical model in figure 1 of the VAE paper, $\phi$ and $\theta$ are the weights and biases of the neural networks that represent the encoder and decoder (partially, in the case of the Gaussian decoder, they say), respectively. $\phi$ and $\theta$ are really the learnable parameters. In fact, if you look at algorithm 1 or any implementation of the VAE (I like this simple PyTorch one), they are what we're trying to optimize.
Now, if you look at the definition of a parametric model, e.g. here, you will see that the parameters $\theta$ in $p_\theta$ actually refer to e.g. the mean and variance of the Gaussian, i.e. $\theta = \{ \mu, \sigma \}$. This is one thing that causes confusion - the subscript $\theta$ (or $\phi$) refers to different things in different articles.
HOWEVER, ultimately, what we're trying to learn are the parameters of a probability distribution, let's say it's a Gaussian, so we are trying to learn a mean and variance $\{\mu, \sigma \}$. In the VAE, we use a neural network to output the mean and variance, i.e. $\mu, \sigma = f_\phi(x)$, where $f_\phi$ is the neural network with weights $\theta$ (see the equations in appendix C) and $x$ is an input. So, $\{\mu, \sigma \}$ are defined by $\phi$.
Now, back to your actual question. Here are some possible interpretations of that notation.

If $\theta$ represented the parameters of a specific probability distribution (let's say a Gaussian), so $\theta = \{ \mu, \sigma \}$, then, as you noticed, you would have all your distributions with the same mean and variance, and I don't think this is generally the case (but could happen?) - for example, $p_\theta(x)$ and $p_\theta(z)$ would exactly be the same distribution, so that means that $x$ and $z$ follow exactly the same distribution (e.g. $\mathcal{N}(0, 1)$).

However, if you look at figure 1 of the VAE paper, both $x$ and $z$ depend on $\theta$, but is it the same $\theta$? So, even though the letter $\theta$ is used as a subscript in the likelihood, prior, marginal and posterior, do these distributions actually have the same specific values of $\theta$? If you look at section 2.1, they also use that notation you found on Wikipedia.

If $\theta$ represented the weights of neural networks, then all neural networks would be equal (this is not the case in practice - see the PyTorch implementation).

If $\theta$ were hyper-parameters, that notation could make sense (because the mean and variance of the Gaussians could still be different), but this does not seem to be consistent with the actual explanation in appendix C and the PyTorch implementation.

In most cases θ is used to indicate weights of a neural network, and with sloppy notation that could mean any NN so not necessarily the same one every time.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to properly combine multiple readings/measurements?
In an AI application (for example, self-driving), there are usually many different reading devices/sensors to ensure the outcome is correct. More specifically, a self-driving car can use object tracking with cameras, road-integrated optic fiber, sound analysis, and so on.
In many cases, these readings can be very difficult to integrate into a single model and get a single output. Assume each device provides an independent output. What are some ways that I can do to combine them and find out the most likely readings?
For instance, device A says there are 10 people standing 5 meters away, B says there are 5 people standing 10 meters away, and C says there are no people at all. The easiest way is to do a weighted average/voting, but it can make more correct readings less effective. If A is totally precise 95 percent of the time, no matter how much weight I assign to A, it will be affected by the less accurate ones.

First of all, I would exclude correlated inputs. If input data has a completely different format, you can feed it using different heads. I may also want to normalize it if possible

@ArayKarjauv thanks for helping. I have tried to just feed all these data into the same Neural network, but it has no improvements (actually, only getting worse performance at most of the time). There are better algorithms than deep learning for certain input types. In addition, all these data measurements are unavoidably correlated (for example, a optical fiber will receive measurement when a car passes by. If a car passes by, the car will also make sound, and the camera will receive video).

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Embedding Quality of Transfer Learning model vs Contrastive learning model
I am working on Contrastive learning which is a technique to learn features based on the concept of learning from comparing two or more instances.
The downstream task is a classification problem.
Transfer Learning
Due to limited data, I tried to use Transfer learning model trained on "Imagenet"(ResNet50 V2 "Deep Residual Learning for Image Recognition Kaiming He, et.al").
I used the embedding from the pretrained model and trained Linear SVM and achieved a F1 score of 0.84.
Contrastive Learning
I also trained a model for contrastive learning using Facenet technique("FaceNet: A Unified Embedding for Face Recognition and Clustering Florian Schroff, et.al") and further used the embedding for training a Linear SVM for classification problem. The achieved F1 score is 0.83.
Problem
Though the scores of both the concept are closeby, I tried to evaluate the quality of both the embeddings using Silhouette Coefficient.
Overall Silhouette Coefficient:

Transfer Learning Embedding = 0.05
Contrastive Learning Embedding = 0.49

I do not understand this behaviour of the system that even with lower Silhouette Coefficient, the transfer learning model is able to perform well.
Kindly provide me with your views on it

Can you please be more specific and put your specific question in the title (too)? "Kindly provide me with your views on it" is not a question and it's also not specific and "A vs B" is also not a question.

Thanks a lot @nbro. I was bit unsure how to frame such a long question in the title.
But I will try to explain the part which was unclear to you. Thanks once again.

what are the dimensions of the embeddings? The first thing that comes to mind is that you probably should be wary of using euclidean distance in high dimensions (Distribution of Distances between Elements in a Compact Set), as done when computing the Silhouette Coefficient.
However, that's probably not an issue for typical dimensions of 10 to 100. It would be good to get more details on your training.

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.


text:
How to create a loss function that penalizes duplicate indices in the output tensor?
We're working on a sequence-to-sequence problem using pytorch, and are using cross-entropy to calculate the loss when comparing the output sequence to the target sequence. This works fine and penalizes the model correctly. However, we also have the added constraint that the prediction can't contain repeated indices, e.g.
good: [1, 2, 3, 4, 5]
bad: [1, 2, 2, 4, 5]
We would like to add an additional penalty term that punishes the model further for producing duplicate indices, which would be added to the cross-entropy loss.
How would I construct this additional loss function in pytorch?
PS: Yes it's true that we could just hack the code-generation piece to not generate duplicate indices and then incorporate this into our beam-search, but I would like to first see whether this additional constraint produces a better model!

This is a bit confusing - do you want a loss that acts on each output individually or all at once? Or just adjacent ones? I've created a diagram here of my understanding of what you want

The indices from the sequence correspond to token positions in the embedding, so we need a loss function that is 0 when all indices are unique, and then (say) 1 when there is 1 duplicate index generated by the model (as in the "bad" case above).


text:
Value iteration algorithm converge to max reward with discount of 1?
I'm running the value iteration algorithm of a gridworld of 4*3, with two terminal nodes with -50 reward and one with +20 reward like so:
+--+--+-----+-----+
|  |  | -50 | +20 |
+--+--+-----+-----+
|  |  | -50 |     |
+--+--+-----+-----+
|  |  |     |     |
+--+--+-----+-----+
|  |  |     |     |
+--+--+-----+-----+

There are 4 possible actions, up, down, left and right.
There is some noise in the system and for any action, there is a 0.8 probability of following it, or 0.1 probability of going either left from the action or 0.1 of going right from the action (never back).
When going to a wall, the agent just stays in place (with no negative reward)
I want to compute the best policy and value function for this scenario using value iteration.
If I chosse a discount factor of 1 (gamma=1), then my values all converge to 20 and my policy ends up like so:
       value table                 policy*
+----+----+-----+-----+  +-------+-------+------+-------+
| 20 | 20 | -50 | +20 |  | down  | left  | -50  | +20   |
+----+----+-----+-----+  +-------+-------+------+-------+
| 20 | 20 | -50 | 20  |  | down  | left  | -50  | right |
+----+----+-----+-----+  +-------+-------+------+-------+
| 20 | 20 | 20  | 20  |  | right | right | down | up    |
+----+----+-----+-----+  +-------+-------+------+-------+

Am i doing something wrong here ? is it normal to converge to the max reward ?
Will a MDP in a grid world always converge to max reward if the discount factor is 1 ?

I linked to a question that effectively includes a duplicate of yours. In brief, yes this is normal, and there are some considerations, such as how to convert the values to a policy.


text:
Is it mandatory to multiply every activation of a layer by droupout factor during testing?
Dropout is a regularization technique used in neural networks. It is useful in preventing overfitting by making a neural network as good as an ensemble system.
In dropout, we switch off $p$ percent of neurons of the input or hidden layer during the training phase.  But, we do not switch off them during the testing phase.
We generally multiply each activation by $p$, the dropout ratio used during the training phase, in the testing phase. Is it mandatory to do so? Or is it okay to use the whole network as it is?

Isn't it 1-p? Also, it is mandatory to do so from what I know, except there are other ways to ensure the scale of the weights are the same (such as inverted dropout).

There are two types of dropout, depending on whether a scaling correction is applied during:

testing - without dropout applied, to decrease logits by a factor $1-p$ to match the expected magnitude that layers were trained with.

training - with dropout applied to increase logits by a factor $\frac{1}{1-p}$ to match expected magnitude without dropout.

(Both these assume $p$ is your probability that a node value is dropped out. Some will use $p$ as the probability that a node value is kept).
The first version above was the original invention of dropout, but it has fallen out of favour nowadays, and the second one - sometimes called inverted dropout - is more commonly used. The advantage of inverted dropout is that it is purely a training adjustment and the whole dropout layer can be removed for production use.
If you do not apply the correct numerical adjustment, then networks with dropout layers will perform poorly.
To answer your question title:

Is it mandatory to multiply every activation of a layer by droupout factor during testing?

Only if you are using the original formulation of dropout. Many neural network libraries use inverted dropout, so do not do this. Those libraries have related adjustment during training instead.

So, it is mandatory to do.

@hanugm Yes, if by "mandatory" you mean "the algorithm doesn't work without it". However, you should check which version you are using. Inverted dropout does not apply any correction in testing phase, by design


text:
PPO: how to scale rewards
I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried various normalization methods. The best one so far is to simply take the log of the cost, as normalizing with a running mean/std (using Welford's algorithm) seems to collapse the value loss to almost zero and learning becomes very unstable. Are there any guidelines for dealing with reward normalization in these cases?

have you tried tuning the discount factor? Because that might be more important than normalization. From what I've seen/experienced, value function normalization is typically not too important or necessary

What if you have values on the scale of billions and trillions?

I could take cartpole, and make the value function large by making each timestep +1e6 instead of +1. If I then scaled the learning rate by 1e-6, this would be equivalent to solving the regular cartpole with the regular learning rate


text:
PPO: multiple discrete actions per step, one depends on the other
I have a custom PPO implementation, and it works fine, but I need to add to it the ability to select 2 actions per turn, one different in nature from the other, one dependent on the other.
Imagine that a turn a had 20 possible actions. Now, for each of these action of type A, I need to choose one from 3 possible actions B. They are not the same, and thus they can not be on different turns.
What I tried was to flatten the 20x3 space to a 60. Then, action A is 60//3 and action B is 60%3. But this does not train well. Are there any good methods for this issue?

Hi @Antonis Karvelas! Could you explain what you mean by the algorithm not training well? Thanks!

Nothing converges. I imagine that it can pick good action A and bad action B or vice versa but it can't figure out which is which...

What you describe will be possible, but you need to ensure that your policy returns the probability of the action(s) taken.
In the two step process you describe, you would have something like
\begin{align*}
\mathbb{P}(a_t) = \mathbb{P}(a_{(t, 1)}=A)\mathbb{P}(a_{(t, 2)}=B \, | \, a_{(t, 1)}=A)
\end{align*}
where $a_t$ is the action taken, which consists of the two steps $a_{(t,1)}$ and $a_{(t,2)}$.
If you aren't strong in probability, here is another way to think about it. You have 60 possible actions (20 possibilities for A, and for each A, 3 possibilities for B). For each of the 60 actions, the policy must return the probability of selecting that (A, B) action tuple. If you sum the probability over all 60 of the action pairs, it must sum to 1. If it does not sum to 1, the gradient will not be correct.


text:
DDPG agent with convolutional layers for feature extraction
I'm trying to come up with a definition of the critic for a DDPG agent in PyTorch using a CNN as a feature extractor. It is pretty straight forward for the actor model. However, for the critic model I am not sure. Below, I've given my code for the actor model.
class CNN(nn.Module):
    def __init__(self, obs_dim, act_dim, act_limit):
        super(CNN, self).__init__()

        channels, height, width = obs_dim
        self.act_limit = act_limit

         self.actor = nn.Sequential(OrderedDict([
            ('conv1', nn.Conv2d(in_channels=channels, out_channels=32, kernel_size=8, stride=4)),
            ('relu1', nn.ReLU()),
            ('conv2', nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2)),
            ('relu2', nn.ReLU()),
            ('conv3', nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1)),
            ('relu3', nn.ReLU()),
            ('flatten', nn.Flatten()),
            ('linear1', nn.Linear(3136, 512)),
            ('relu4', nn.ReLU()),
            ('linear2', nn.Linear(512, act_dim))
        ]))
    def forward(self, obs):
        # Return output from network scaled to action space limits.
        return self.act_limit * self.actor(obs)

My confusion lies at the part with the concatenation of the action and the observation vectors and how to actually integrate that in my CNN model. A typical critic model is a MLP that looks like this:
class critic(nn.Module):
    def __init__(self, obs_dim, act_dim, act_limit):
        super(critic, self).__init__()
        self.act_limit = act_limit

        self.fc1 = nn.Linear(obs_dim + act_dim, 256)
        self.fc2 = nn.Linear(256, 256)
        self.fc3 = nn.Linear(256, 256)
        self.out = nn.Linear(256, 1)

    def forward(self, x, actions):
        x = torch.cat([x, actions / self.act_limit], dim=1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        q_value = self.out(x)

        return q_value

Any ideas on how to define a critic with the same (convolutional) feature extractor?


text:
RL-based trading bot: how to deal with overfitting
I've been playing around building a reinforcement learned-based trading bot using the stable-baselines3 library. I've come up with an environment that seems to be able to learn how to make profitable trades on the dataset that I feed it. If I keep training long enough it is seemingly making good money.
Now, I have a background in supervised machine learning, and I was already quite sure that my agent was "overfitting" to the dataset that I was feeding it. This was confirmed by my experiment: when feeding my trained model a new validation dataset the performance was on-par with a random agent, and it was no longer making any money. In other words, my agent manages to exploit relations in the training data very well, but when evaluating the out-of-sample performance on unseen test dataset it appears that my agent just learned noise. In the context of my trading bot it seems that the agent memorises the financial time-series, and it not actually useful for the purpose of trading as the patterns it learned are not generalisable.
Now my question: how do we overcome this issue?
In supervised learning, there are various ways to help prevent that our model will overfit. We can use a validation set during training so that we can apply early stopping (stop training when validation accuracy no longer improves). We can use regularization techniques such as drop-out layers. But how do we deal with the concept of overfitting in reinforcement learning? Some things I had in mind that could be useful:

Does it make sense to apply regularization to my neural network that learns the value function and policy?
Is there something equivalent to early stopping?
Will adding more data or simply decreasing the number of timesteps help?
Could it possibly help to generate training batches by randomly selecting certain periods of my full financial time-series, rather than going through the same time-series over and over again? My assumption is that in that way the network no longer relies on the same sequence, and the patterns it learns could be more robust...

Looking forward to hearing your view on this!

Can you comment on what you're doing a little further? I'm guessing you're representing the state space as price volume data/financial data/economic data/news surrounding a given security X. Your choice of action (buy/sell in x amount) supposing that you are not large enough to move the market does not affect the next state s' - in fact, modelling s' is the main task it seems. Furthermore, the reward function r(s,a) does depend on your action, but in a non-deterministic way - does your algorithm handle this? A little more information is needed to answer your question.

But to point you in a general direction, you're dealing with batch RL (as opposed to online RL). The difference is that in the batch RL setting, your dataset is fixed, and you do not get to interact with the environment to observe rewards and next states. For any bootstrapped target, in a continuous batch RL setting, you may encounter overestimation bias, which can lead your agent to behave poorly. This is somewhat analogous to overfitting in the supervised learning setting. I suggest reading this: https://arxiv.org/pdf/2005.01643.pdf

Finally, just to make a comment about the U.S. stock market: for most liquid stocks, in the short run, the price movements will be incredibly noisy (near-random), mostly governed by news (which, by definition is unknown information prior to it coming to light). The noise to signal ratio is incredibly large, so this problem is probably harder than any commonly available RL benchmark you can find. This may be why it seems like your agent is behaving randomly. Furthermore, it doesn't seem to me transition dynamics are stationary in financial markets - you may want to also consider this obstacle.

Thanks for your comments! Actually I am using data from cryptocurrencies, but the idea is indeed like you mention. I feed historical data related to price/volume to the agent as the state, and the action space is [hold, buy, sell]. The reward function is triggered when the agent buys or sells. The agent gets rewarded if it sold on a profitable trade (i.e., sell price is higher than buy price) or if it has bought low/avoided an unprofitable trade (i.e., if a buy is on a lower price than previous sell). I do am aware of the high signal-to-noise ratio, so have realistic expectations.

Additionally, the train and test dataset should be reasonable equal, as they both describe the price of the same asset (bitcoin) and have similar dynamics, in the sense that there are "bull markets", drawdowns, volatile periods, etc. So my hope is that is can extrapolate some the things learned from the train data to actually outperform a random agent on the test data.

Does the state space also contain your current portfolio position?

Prior to throwing it real data, I would try to generate some autocorrelated data with obvious momentum trends just to see if your setup is working on clearly inefficient markets.

Yes, the state space does include the current position [cash, bitcoin] and if it is holding bitcoin, the difference between the entry and current price. Moreover, I am using maskable PPO: https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html
This way, I am preventing that the bot tries to buy bitcoin while it is already holding a position, or selling while it is holding cash.

Running a simulated timeseries through the bot surely sounds like a good idea to start with. Also the term "overestimation bias" seems to perfectly fit my case. I was curious to what extent general techniques in reinforcement learning exist to overcome this issue of "overestimation bias". My intuition says that stopping training at some point could help (i.e., using "early stopping"). But not sure if that's a thing in RL...

It's not exactly like overfitting in the supervised learning sense. What happens in a continuous action space is that you are using function approximation to estimate Q(s,a). For actions a_hat that are out of distribution (i.e. not drawn from your dataset), you have no idea what Q(s,a_hat) should be, and are forming an estimate based on your dataset. These estimates turn out to be very important, because TD learning relies on next state value estimates. If Q(s, a_hat) is erroneously high, due to overestimation, then you're going to overestimate the value of actions which you have not observed.

Essentially you mistakenly exploit areas of the Q-function which are erroneously high, and base your actions on this.


text:
How do we estimate the value of a stochastic policy?
I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select the actions from a probability distribution).
I don't understand how the value function for stochastic policies is estimated accurately. The value function $V^{\pi}(s)$ is the expected return when starting at state $s$ and following policy $\pi$, but if we are choosing actions randomly then we are not really following any particular policy.
How are we getting an accurate estimate of the value of policy $\pi$ if we are not following a deterministic policy $\pi$ during training?

The value function is the expected (discounted) future rewards from the current state. The expectation is taking with respect to the policy, so if the policy is stochastic then it is measuring the returns under the policy in expectation, I.e. what the policy does on average.

"I've noticed that many algortihms use stochastic policies during training", I know that you're talking about policy gradients and actor-critic algorithms, but I recommend that you provide 1-2 examples of a specific algorithm that does this so that there's no ambiguity and people can answer your question more concretely.

We are choosing actions randomly with probabilities given by the policy $\pi$. For example, one policy might make two actions A and B equally likely, another might choose A 90% of the time.
What we're actually calculating is not the value of a particular path, but the expected value of following the policy (sort of the average value).
To calculate the expected value, we sum (over all possible actions) the probability of each action (as given by the policy) times the return from that action.
$V^\pi(s) = \sum_a \pi(a | s) [R(s,a) + \gamma  V^\pi(s')]$
If the transitions are also stochastic, we do a similar sum over next states:
$V^\pi(s) = \sum_a \pi(a | s) \sum_{s'} P(s' | a,s) [R(s,a,s') + \gamma  V^\pi(s')]$

Deterministic policies are just a special case of stochastic policies. The probability distribution of a deterministic policy has a single spike of value 1 for the deterministic action.
You are right that the value function keeps changing based on the random action chosen but all of these changes cause the value function to converge to a fixed point which is the value of that policy. This follows directly from the law of large numbers.


text:
Visualizing the loss landscape in deep NN to compare optimization methods
I'm comparing 2 optimization algorithms for deep neural nets through visualizing the loss landscape. The visualization method is described here.
Besides the qualitative observation that how trajectory moves w.r.t. the loss level-sets, are there any quantitive measures to compare the two methods?

How about:

speed of convergence
stability/variance w.r.t to the initial random seed (or other sources of variance like learning rate)
presence/number of saddle points in your loss landscape


text:
Why are embeddings added, not concatenated?
Let's consider the following example from BERT

I cannot understand why "the input embeddings are the sum of the token embeddings, the segmentation embeddings, and the position embeddings". The thing is, these embeddings carry different types of information, so intuitively adding them together doesn't really make sense. I mean, you cannot add 2 meters to 3 kilograms, but you can make a tuple (2 meters, 3 kilograms), so I think it's more natural to concatenate these embedding together. By adding them together, we are assuming the information about token, segmentation, and position can be simultaneously represented in the same embedding space, but that sounds like a bold claim.
Other transformers, like ViTMAE, seem to follow the trend of adding position embeddings to other "semantic" embeddings. What's the rationale behind the practice?

Can I know why you are comparing with physical units?

@hanugm Sorry for the confusion! I was analogizing 2 meters to token embedding, and 3 kilograms to position embedding. The idea is that token/position embeddings respectively describe the semantics/position of the tokens, so they should have different units, just like length and weight have meter and kilogram as their units. In dimensional analysis, it is established that you cannot add quantities with different units together (i.e. dimensional homogeneity), which is why I think it doesn't make sense to add embeddings together.

In high-dimensional spaces, the token embeddings and positional encodings can be thought of as forming two separate subspaces that are approximately orthogonal to each other. This is based on the mathematical property that

Randomly chosen vectors in high dimensions are almost approximately
orthogonal [1].

This means that these subspaces can be manipulated essentially independently of each other by the same learned transformation, despite the addition operation.
So, by adding the embeddings, we’re not losing any information. Instead, we’re allowing the model to decide how best to use the information from the token embeddings and the positional encodings. This is more efficient than concatenation, which would increase the dimensionality and the number of parameters the model needs to learn.
In essence, the addition of positional encodings to token embeddings in Transformer models might seem counterintuitive, but it’s a clever trick that leverages the properties of high-dimensional spaces to efficiently combine different types of information.
P.S. - There isn’t a definitive answer to this question. However, after extensive research, I found this intuitive discussion here [2] that beautifully conveys the idea explained above.

First of all, I think it is very hard to properly reason about these things, but there are a few points that might justify using sum instead of concatenation.
For example, concatenation would have the drawback of increasing the dimensionality. So for subsequent residual connections to work, you would either have to use the increased dimensionality throughout the model, or add yet another layer to transform it back to the original dimensionality.

The thing is, these embeddings carry different types of information, so intuitively adding them together doesn't really make sense. I mean, you cannot add 2 meters to 3 kilograms, but you can make a tuple

I would say that because the token-embedding is learned, you cannot really compare it to a fix unit like kilogram. Instead the embedding space of the token can be optimized to work with the positional encoding under summation.

By adding them together, we are assuming the information about token, segmentation, and position can be simultaneously represented in the same embedding space, but that sounds like a bold claim.

The same applies here, the problem is not to embed them into the same space, but rather that subsequent layers can separate the position information from the token information.
And I think this is possible for two reasons. Firstly, if you look at the visual representation of the positional embedding, the highest distortion by summation would happen in the first dimensions:
 (Image taken from here)
Therefore the token embedding could learn to encode high-frequency information only in the last dimensions to be less affected by the positional embedding.
I think another interesting statement in the Transformer paper is that the positional encoding behaves linearly w.r.t. relative position:

We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset $k$, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$.
[Source: Transformer Paper]

So this property shouldn't add additional non linearity to the token embedding, but instead acts more like a linear transformation, since any change in position changes the the embedding linearly. In my intuition this should also enable easy separation of positional vs token information.
This is my intuition so far, I am happy to hear your thoughts and additions

The confusion here is that we believe positional embedding is a more complicated version of adding positional information to the work embedding; however, it is not actually. Adding new dimensions to each embedding increases the dimensionality of the problem. On the other hand, please note that the added positional embedding is static, as shown in this image for a 2D positional embedding:

The added positional embeddings are the same for all the inputs, and the transformer can separate the positional information from the actual work embedding through the training process. Therefore, the positional embedding doesn't mess with the work embedding information, and adding them is a more efficient way of adding the positional information that concatenates them.

Could you elaborate how does the training process can learn to separate the positional and word embedding information?

@Glue The positional embedding is like a bias added to all word embeddings located at each specific position. It is like adding a constant to your inputs. Not sure if this actually happens or not, but separating them is like learning a bias. The value of bias is your positional embedding, and what remains after removing the bias is your word embedding.


text:
Why can't I reproduce my results in keras using random seed?
I was doing a task using RNN to predict a time series movement.
I want to make my results reproducible. So I strictly followed this post:
https://stackoverflow.com/questions/32419510/how-to-get-reproducible-results-in-keras
My code are as follows:
# Seed value
# Apparently you may use different seed values at each stage
seed_value= 0

# 1. Set the `PYTHONHASHSEED` environment variable at a fixed value
import os
os.environ['PYTHONHASHSEED']=str(seed_value)

# 2. Set the `python` built-in pseudo-random generator at a fixed value
import random
random.seed(seed_value)

# 3. Set the `numpy` pseudo-random generator at a fixed value
import numpy as np
np.random.seed(seed_value)

tf.compat.v1.set_random_seed(seed_value)

tf.random.set_seed(seed_value)

# 5. Configure a new global `tensorflow` session

# for later versions:
session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
tf.compat.v1.keras.backend.set_session(sess)

However, every time I ran my codes, I still got a different result, what could the reasons be?

This question seems to be off-topic (please read the rules). Consider asking programming questions on StackOverflow.

@ArayKarjauv I don't think this is a programming question, but OK.

FYI If using GPU, non-random reproducibility is more complicated than that because of random results ordering.


text:
How to properly name given type of classification problem?
What is the proper technical name of the classification problem where each data sample can be classified according to two different criteria and each of them can have two or more classes?
For example age/gender estimation problem where age is one criterion, gender is second.
Age can be divided into 4 age groups and gender into 2. And model should generated estimate of both age and gender for every sample.

I think what you are looking for is multi-label classification. Multi-label classification can take care of both of your constraints -

each data sample can be classified according to two different criteria
and each of them can have two or more classes

If you just had the latter constraint, I'd have suggested multi-class classification, but multi-label classifiers are a subset of multi-class classifiers.
See What is the difference between Multiclass and Multilabel Problem.

This is called multi-task learning, as you have two independent classification tasks. Note that this is not the same as multi-label learning.


text:
In-batch negative training Improves the results
I have read Dense passage retrieval for Open Domain Question Answering, and in page 6 it talks about in-batch negative training, it states the following:

We find that using a similar
configuration (7 gold negative passages), in-batch
negative training improves the results substantially.
The key difference between the two is whether the
gold negative passages come from the same batch
or from the whole training set. Effectively, in-batch
negative training is an easy and memory-efficient
way to reuse the negative examples already in the
batch rather than creating new ones. It produces
more pairs and thus increases the number of train-
ing examples, which might contribute to the good
model performance. As a result, accuracy consis-
tently improves as the batch size grows.

I know that it is more efficient and hence we can increase the number of negatives by increasing the batch size (or we can use other techniques if we cannot do that), but in the paper they have the following table:

How having the same number of negatives (3rd & 4th rows) gave different results?
Is that something realted to the gradient or something, since we sample from the same examples that contribute to calculating the gradient in one step?

Yes, the In-batch negative training improves the results but not objectively, it is more helpful in reducing the confusion of semantics of various pattern which I guess tends to happen in deep-layers.
In the context of Blip-2, By including these hard negative samples in the training batch, the model is forced to learn to distinguish between positive and negative samples, which was helpful in Visual Knowledge Reasoning, but it didn't improve accuracy significantly.
For your last question regarding the link to gradient, yes it has to do with gradient, I guess.
        # sample way to generate hard negative samples
        with torch.no_grad():
            outputs = model(inputs)
            _, predicted = torch.max(outputs.data, 1)
            negatives = inputs[predicted != labels]

        #  hard negative samples with the positive samples
        inputs = torch.cat((inputs, negatives), dim=0)
        labels = torch.cat((labels, torch.zeros(negatives.shape[0], dtype=torch.long)), dim=0)

        # Zero the parameter gradients
        optimizer.zero_grad()


text:
How do I preload a conversational AI assistant like GPT-3 with complex relational data to draw on?
I'm exploring options to build a virtual assistant type of product.
Creating good dialog is mostly solved with GPT-3 or even DialoGPT.
My main question is how do I add larger amounts of relational data, for the AI to draw on.
e.g.
I have a list of properties, with address, details like number of rooms, bedrooms etc.
Each properties as a historic list of tenancies, as well as owners, each tenancies has a list of tenants and rent agreements. Each Owner & Tenant has personal details. And we can go all the way down to financial transactions, invoices against properties.
etc.
In a prototype I got GPT-3 to write SQL code for me, which works reasonably well for Q&A. Not so much for a continuous conversation, that allow you to dig further into details.
I could also write out all details on a per record bases. e.g. "Property 15125 has address '20 W 34th St., New York, NY 10001' with 3 Bedrooms and 2 Bathrooms". I haven't tried it, but I'm relatively confident that if a dataset becomes sufficiently large (hundreds of records) the system would fail to make the correct connections.
An ideal conversation would look something like this:
Me: "Who is the owner of 20 W 34th St., New York, NY 10001?"
AI: "Peter Pan and Wendy Darling share the property."
Me: "Who is the most recent tenant?"
AI: "Dr. John Darling is the current tenant."
Me: "How much is the rent?"
AI: "Then rent is $1902 a week."
Me: "What's John's phone number?"
AI: "His mobile number is (311) 555-2368."

can you show the prompt snippet for "GPT-3 to write SQL code for me"?


text:
Neural network: Initial weights for layer with non-negative constraint
I wonder how to initialize the weights of a layer with non-negative weight constraints and sigmoid activation afterwards. I did not find any guidelines. I thought about taking inspiration of glorotUniform initialization and using $\text{Uniform}[0,b]$ with $b=2 \sqrt{6 / (n_{in} + n_{out})}$ where $n_{in}$ and $n_{out}$ are the number of input and output units respectively. Another idea I had was choosing $a=\frac{n-12}{2n}$, $b=1-\frac{n-12}{2n}$ where $n$ is the number of input units and sampling out of $\text{Uniform}[a,b]$ such that the variance of each output variable is equal to $1$.
My question: Are there any guidelines how to initialize non-negative weights? If no, do you think my suggestions run in problems?

Can you provide a bit more detail on why your weights have a non-negative constraint? Is that constraint enforced during optimization? Or is it only during initialization? What is your reason for wanting this?

The input and output lives in a cone, i.e. multiplication with positive values stays in the cone as well as addition of two cone elements. In some layers I use specific cone-operations which require that the input is a cone element. Non-negative weight constraints can be easily implemented using e.g. constraints in tensorflow. Problem is that I don't know about initialization guidelines.

I'm also curious: Do you have a reason to believe that it will benefit training to keep the weights $\gt 0$? To me it would make more sense to only constrain the network output like this. Regarding the question, I don't know about any guidelines, but I think your idea to augment the glorot uniform like this makes sense. Except that I'd choose $b = 0.5 \sqrt{6 / (n_{in} + n_{out})}$ to keep the layer output at a similar magnitude. Multiplying by two would yield outputs that are four times larger compared to the standard glorot-uniform-initialized layer, which might harm training.

@Chillston Thanks for the input, yes, positivity was more or less theoretically motivated because I'm working in the cone of positive definite symmetrical matrices and I thought about the values staying in it all the time but I also might try with negative values

I, unfortunately, cannot provide you with a scientifically based answer, so I'll try to logic my way to an answer.
I know that there are things called 'nonnegativity-constrained autoencoders'. I do not know the details of their initialization but you could look into those for possible guidelines. There are possibly also other networks which have papers published talking about a 'nonnegativity-constrained' variant which I am unaware of.
Furthermore, I would have some concerns/thoughts about nonnegative neural nets:

I would assume that having only positive network weights could lead to an exploding gradient. If anything, I would scale the input in a similar range as your weight initialization and have the upper bound of your weight initialization not exceed 1.
Logically, but also building further on point 1, I would not use (variants of) ReLu activations. With non-negative constrained weights and non-negative inputs, ReLu becomes a simple linear activation. I would advise against any activation function without an upper limit, as it might again result in exploding weights/gradients.
I would start testing the model with sigmoid activations in all layers (except possibly the last, of course).
Xavier/glorot initialization is usually best for layers that also use negative weights and a tanh activation function. I do not know how a nonnegative weight constraint changes its effectiveness. However, if you centre your input around (i.e.) 0.5, taking a uniform distribution around 0.5 and vary its range depending on the layer size might work.

Again these are just my insights, not necessarily based on scientific research.


text:
Data Imbalance in Contextual Bandit with Thompson Sampling
I'm working with the Online Logistic Regression Algorithm (Algorithm 3) of Chapelle and Li in their paper, "An Empirical Evaluation of Thompson Sampling" (https://papers.nips.cc/paper/2011/file/e53a0a2978c28872a4505bdb51db06dc-Paper.pdf).  It's a contextual bandit model using a Laplace approximation for regularized logistic regression with Thompson Sampling.
My question is about the effect of data imbalance.  Say the log files of the online advertiser showed that after implementing the algorithm, Ad#1 accounted for 40% of the impressions and achieved 60% of the rewards (clicks).  Ad#1 has the 3rd highest conversion rate compared to 30 other ads.  (The two ads with higher conversion rate accounted for <1% of weekly impressions/rewards.  The number of impressions is still high though, N>300K).
Question: What would be the effect of downsampling in the training set, the examples for which Ad#1 has positive reward?  The paper doesn't mention anything about data imbalance.  Is data balancing needed/advisable?
Option A
The model will impress Ad#1 less and result in fewer weekly aggregate rewards (less exploitation of Ad#1 resulting in less rewards).  The concern is that the model has chosen to exploit Ad#1 and if we downsample its rewards in the training set, we will exploit less and allocate more impressions to ads with lower conversion rates.
Option B
The model will impress Ad#1 less, however will learn better when to serve the impressions in a targeted way so that the weekly aggregate rewards will not change and/or will increase.
The idea is that the imbalanced dataset is causing Ad#1 to be impressed in situations in which it shouldn't.  By downsampling, and having a balanced dataset, we will better model when Ad#1 should actually be impressed and this will lead to higher rewards for Ad#1
Option C
Something Else


text:
What would be the reason for having a different network architecture for the actor vs. value function networks in PPO?
I was reading this link , and saw some creative architectures for PPO.
I know the "No Free Lunch Theorem" and all, but what would be the logic/reasoning for why you would choose to have a different size/shape/depth for the actor vs. value func. networks in PPO?
I just came across this question, which is related but not the same. It asks about the utility of sharing parameters across the two networks, to which the answer is essentially "that might accelerate training by having less parameters". I think my question is distinct and more general.

I would usually go with two separate networks for several reasons:

Having a single network with two heads (one for policy and one for value) might be more challenging to train. I believe that training two separate networks is more stable. Especially in the case of PPO, you want to have small policy updates and you clip if the update would be too large. But now updating the value network also updates your policy, so you also have to clip the value loss as well, but what should be the clip size for the value net ? (see here for more)

In the case of a single network you have a single objective where you add the policy loss and the value loss: $L = L^{CLIP} + cL^{VF}$, and now you need to adjust the hyperparameter $c$. See Equation. 9 from the official PPO paper. Of course, if you have two networks you also have other hyper parameters, like different learning rates and so on, but these are not so much interdependent and can in general be adjusted separately, i.e. if your value net is training well but the policy is not, then don't touch the hyperparams of the value net and only adjust the hyperparams for the policy net.

The architecture of the two networks doesn't have to be the same. I usually use a much simpler network for the value network and a more complex model for the policy network. The task that the policy network is learning is much harder and at the end when all training is done you are actually using only this network. On the other hand your value network is trained on a regression task, i.e. predicting a single number, which is much easier and also better understood theoretically.

You might also want to regularize the models differently. For example I would regularize the value network using L2-regularization, but I would not use this for the policy network. For the policy network I would use entropy regularization.


text:
What makes a 'good' dataset
for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own.
While datasets from external sources are of course marketed as beeing 'good' or 'high-quality' (and in most cases not explained on how the authors come to this conclusion), creating a dataset yourself doesn't come with these labels.
This brings me to my questions: How can one (objectively) quantify the quality of a dataset for a given problem?
This, of course, includes some points which are more or less 'accepted' within the community, e.g. the dataset has to contain enough datapoints for every modeled state (which leads to the question of what is 'enough' ...), or that the datapoints for each modeled state should be roughly equally split (e.g. a dataset for cat/dog image discrimination would not work well with one dog image and 10k cat images) and so on.
I recognize that this is a rather open ended and maybe even philosophical question, but I believe that, given the importance of data for ML (and other disciplines), I am in need of an objective way to evaluate my datasets in relation to the task at hand and determine their quality in an objective way. Also my goal here is clearly ML oriented, but since this topic is not only valid in the ML context (and ML is in its core more or less complex statistics), I don't want to restrict this only to the ML, but to datasets overall

This is not a simple answer, and I think it really depends the goal, quality is not clearly defined and can vary. However there are some points that are commonly seen as positive.
The quantity
As you said, having enough data is important to train models, and it is even better if the dataset is balanced for classification. This means that it i spossible to build more complex models, and then learn more from the dataset.
Learn more does not mean to learn from more data, but from more diverse data. If you take the example of dog/cat classification, a larger dataset will probably have more angles (from above, behind or with an obscured view), and more cat/dog races which means that the algorithm can learn all these differences that needs a larger amount of data.
The main point of quantity is not how much data, but how much of the input space is covered.
A clean dataset
The first step of every project is often cleaning the data and preprocessing it, which can be avoided if the dataset is of better "quality". This means that all instances are correctly labeled, there is no missing value, no duplicates, the data types are the same...
There the data is synthetic and can be used nearly as is, without need of further work to process these.
The ambiguous points
If a dataset present the attributes above I'd say it's quality is already great, but there are some additional points that can be considered. Some attributes of the dataset can vary (for example image size/resolution, background noise for audios, typo in language corpus...).
The quality of the instances with such difference can be said to be low, if example if for example only the tail of the dog can be seen. However the issue is that real world data is not perfect.
In some cases you can pre-process these differences or perturbations and correct them, but sometimes you can't. There some people will say it's part of the challenge to deal with these, some will say it is useless.
There it depends on your goal and subjective point of view to decide if these inconsistency is needed or more harmful.
To add a more personal thought, I'd say that when building a production model it is important to keep these as they reflect the real use and the perturbations brought by real data.

For the dataset to be useful for the community it should be publicly available in a stable domain and establish what the test set is (if you can have a hidden one and make a webpage to evaluate the submissions that's that's better). It's also useful to have baseline performance numbers.
The dataset doesn't have to be balanced. High-quality datasets are regarded as such if the label noise is small (note that annotators do fail), the data are not corrupted and care has been taken regarding potentially harmful biases. Of course, having the dataset checked and reviewed by experts in the field is desirable. Creating a good dataset involves lots of work.


text:
How to use information on a function to design a neural network learning that function?
I have a function $g$ that takes a vector $x$ of size $n$ and an integer $k$ in $1, \ldots, n$. I know this function is of the form
$$g(x,k) = G\left(\sum_{i=1}^k f(x_{i})\right),$$ where $f$ and $G$ are some unknown functions.
I want to train a neural network that estimates the function $g$. What is the best way to proceed given the structural information I have about the function?
Of course, I can train the neural network without taking the specific structure into account, but I believe this is not the best way. I can also estimate $n$ neural networks for $g(x,1), \ldots, g(x,n)$, but I guess this is not the best way either. Is there a good / standard way to proceed?

Do you have any information about f and/or G?

A first observation is that $f(x)$ is shared across all the $n$ dimensions of $x$ so you can approximate it using a fully connected neural network with a 1-d output. You can think of it as a block in your architecture. To get a variable number of $f(x_i)$ you just run the whole network on each $x_i$ up to $k$. I expect this won't be hard to parallelize.
The other function $G$ doesn't really need to know anything about $k$, it just requires the sum of $f(x_i)$ which is nothing but a scalar. Hence, to approximate $G$ you just need another fully connected neural network with 1-d input and 1-d output layers.
As for how big the middle layers should be, this depends on many factors and foremost on your data and how complex the functions $f$ and $G$ are. You'd better experiment a bit on both cases.


text:
Why do we use gradient descent to minimize the loss function?
The purpose of training neural networks is to minimize a loss function, in this process we usually use gradient descent method.
But in Calculus, if we want to find the global minimum of a multivariable function, we usually first calculate the partial derivatives of this function with respect to its variables, and then set these partial derivatives to zeros, and then find the solutions of these equations. Usually we get a bunch of points. We can use the second derivative method(involving Hesse Matrix) to determine whether these points give local minimum values, or we can directly evaluate the values of the function at these points and compare them to find the minimum.
So, I'm curious why don't we use this classical method in Calculus to find the global minimum value of the loss function and instead using gradient descent? Is it hard to for computer to find zeros of multivariable equations?

and then set these partial derivatives to zeros

but how do you do that? This is analytically possible in very simple cases, such as least squares on linear regression. In that case, it's possible to invert the function and explicitly build the weight matrix and the biases.
But in maths, the vast majority of equations is not analytically solvable, and needs iterative approximation methods that lead you to a numerical solution. This is what gradient descent does.
To convince yourself of this, I suggest you try: write a simple function representing a neural network with 2 layers, then see if you can do as you say. This will give you a better understanding of why it doesn't work.


text:
What is the correct interpretation of the discount factor in MDPs?
In infinite-horizon MDPs one can consider the expected discounted return from a distribution of start states as the objective[^1]. i.e. $\mathbb{E}[V^{\pi}(S_0)] = \mathbb{E}[G_0] = \mathbb{E}[\sum_{t=0}^\infty \gamma^t R_{t+1}]$ where the expectation is over start states $S_0$, following states taken from the transition dynamics of the MDP, and actions from a policy $\pi$.
In this setting, discounting is interpreted as (1) giving less importance to future rewards, or (2) having a $1-\gamma$ probability of being truncated forever at every step as mentioned in this answer. The answer claims that the or is exclusive and that "choosing one or the other means tackling a different problem". That (1) is implemented by adding $\gamma$ to the return, while (2) by simulating the truncation.
I believe this not to be true and that it does not make sense to include $\gamma$ in the  return (thus the state and action value functions implicitly as well) without interpreting it as a probability of being truncated at the same time (i.e. part of the MDP. Interpretation 2 in the answer).
In the context/question where the answer is taken from, if one derives the policy gradient with respect to the discounted return as the objective, then the discounted state distribution arises (the one corresponding to interpretation 2), not the original one. Proof here.
Hence, using $\hat{\nabla J_1} = G_t \nabla\log\pi(A_t | S_t)$ where $G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1}$ is a biased estimator for the policy gradient when "giving less importance to future rewards". $\hat{\nabla J_2} = \gamma^tG_t \nabla\log\pi(A_t | S_t)$ would be the correct one.
It's as if one implicitly changes the MDP by discounting (or equivalently that $\gamma$ is part of the definition of the MDP not only the objective).
Does this make sense?
[^1]: Another objective is the average reward.


text:
Is regularization in machine learning and deep learning same
As there are L1 , L2 , etc out and other technique are those all same for machine learning and deep learning while using Ml algorithm and DL algorithm

What are your findings so far? What is your understanding of ML/DL? Do you know what L1 and L2 are?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

ML and DL are the same thing,

Yes L1 and L2 in DL and ML refers to the same kind of regularization. DL can also benefit from dropout which is an additional method of regularization. See this article.


text:
How to encode time embeddings in a diffusion model for 1D vectors?
For a project I'm working on, I'd like to try a diffusion model like illustrated in the paper by Ho et Al to generate 1D vectors. What I'm trying to figure out at the moment, is what kind of architecture should I use for the 1D case, given that those people use a U-net-based model including different layers like attention, residual, group norm, etc. The only requirement that such a model should satisfy is that the input and output shapes should be the same, as we're interested in predicting the noise associated with a particular timestep.
$$\mathcal{L}(\theta) := \mathbb{E}_{\textbf{x}_0, \, t, \, \varepsilon}\left[\|\varepsilon - \varepsilon_\theta(\sqrt{\bar{\alpha}_t}\textbf{x}_0 +\sqrt{1-\bar{\alpha}_t}\varepsilon,t)\|^2\right]$$
So for now, I'm considering a simple autoencoder architecture but I'm not sure how and where I should inject knowledge of timesteps using some sort of positional encoder as proposed in the paper by Ho et Al.
Additionally, I'm not even sure about they inject such knowledge into the model? Do they sum the obtained time embeddings over channels? Or over image dimensions?

1D diffusion is already supported. See https://github.com/lucidrains/denoising-diffusion-pytorch/issues/116


text:
Is data augmentation inducing bias?
I am using Keras to build a CNN model to classify spectograms and using the following layers:
Conv2D 32 filters, size (5,5), relu
MaxPooling2D (2,2)

Conv2D 32 filters, size (5,5), relu
MaxPooling2D (2,2)

Conv2D 64 filters, size (5,5), relu
MaxPooling2D (2,2)

Flatten
Dense 64 , relu
Dense 32 , relu
Dense 5 (softmax)

With this model I was able to comfortably achieve more than 97% training accuracy and around 84% validation accuracy. Unfortunately, I have very few data samples (slightly more than 1200), which is my main suspicion on why the model is overfitting.
As I am not able to get more data samples, I opted to use SpecAugmentation. I used to get all my classes (5) to 1000 samples each, meaning I generated around 3800 samples.
After I fit the model to this augmented data I got more than 97% validation data, which I found very strange. I am thinking that I am introducing bias because I am performing the data augmentation before splitting into training and validation sets, which may lead to having the same sample in both sets (obviously they are not 100% equal but one is derived from the other).
Should I only be performing data augmentation after splitting? If I use data augmentation only on the training set will I not be overfitting to the training set as well, as it will see very similar samples several times?
What other techniques would you recommend for data augmentation besides SpecAugmentation?
Thanks

You are right you should apply the augmentation after splitting. The goal of the validation data is to assess the results on data that has not been seen during training. When doing so you way have different versions of the same sample in the train and test splits.
The goal of SpecAugment is to make the model learn from different part of the audio. Time and frequency masking hide some information so the model will try to build its decision based on the other parts of the spectrogram.

Thank you for your validation. Any idea on how I could prevent the overfitting of the CNN with the small dataset I am using? I am still exploring BatchNormalization or even changing the batch size, for example, but I am not sure if this will fix the overfitting problem. Any suggestions?

SpecAugment and batch normalization should help. Maybe a smaller batch size and lower learning rate can help, what optimizer algorithm are you using? I recently tried flipping the audios along the time axis (you can consider it as playing it in reverse) and it helped me a bit. It is also possible to randomly crop a part of the audios, although it never helped me.


text:
What is the distribution of autoencoder embeddings?
Is there any result on the distribution of autoencoder embeddings?
For example, the following image (taken from this article) visualizes the latent space with t-SNE. As you can see, images from the same class (in this case MNIST digits) roughly form a cluster. Is there any theory explaining why that is the case?

Note that I am talking about autoencoders, not variational autoencoders, whose embedding by definition has a multi-variate Gaussian distribution.

If you don't make assumptions about your input distribution, and the form of your network, it's very difficult to express the embedding distribution in closed-form. Here's an idea on how to formalize the change in distribution under the application of an autoencoder.
Let $g:\mathcal{X}\rightarrow\mathcal{Z}$ be the encoder, and $f:\mathcal{Z}\rightarrow\mathcal{X}$ be the decoder. $\mathcal{X}$ corresponds to the input space, whereas $\mathcal{Z}$  corresponds to the latent space. Let $P$ be the true distribution of your data, from which you have an i.i.d. sample $S = \{\mathbf{x}_{i}\}_{i=1}^{n}$. You can approximate $P$ using your sample,
$$\hat{P}(\mathbf{x}) = \dfrac{1}{n}\sum_{i=1}^{n}\delta(\mathbf{x}-\mathbf{x}_{i})$$
this is called an empirical approximation of $P$. Now, if you want to study the distribution of $\mathbf{z}_{i} = g(\mathbf{x}_{i})$, you can write a new distribution,
$$\hat{Q}(\mathbf{z}) = \dfrac{1}{n}\sum_{i=1}^{n}\delta(\mathbf{z}-\mathbf{z}_{i})$$
which is an approximation of $Q = g_{\sharp}P$, where $g_{\sharp}$ is the pushforward of $P$ by $g$. Note that this does not tell us much about the properties of $Q$, since we did not make enough assumptions. In some frameworks (e.g. Optimal Transport), this formulation is very useful. If you want to read more on an probabilistic/transport analysis of the distributions in an autoencoder you can take a look on [1].
References
[1] Sonoda, S., & Murata, N. (2019). Transport analysis of infinitely deep neural network. The Journal of Machine Learning Research, 20(1), 31-82.


text:
What Kind of Reinforcement Learning Algorithms Can Be Used when the Action Space is Unfeasibly Large?
I know Deep Q network as a $S\times A$ DNN which maps the $S$ dimensional statespace to q-values of $A$ distinct actions.
In my problem, the action space is still discrete, and finite, but depending on some parameters (e.g. number of users in a system) can grow exponentially large. More concretely, if I have $n$ users, then there are about $3^n$ actions. Clearly, the vanilla DQN is infeasible for this problem.
So I wonder if there are any reference algorithms (best if they come with mature library supports for production code) catering to the scenarios where it is infeasible to calculate individual Q values for each action.


text:
Parametric vs Non-parametric generative models
I have a little perplexity trying to distinguish parametric vs non-parametric generative model.
In my understanding, a parametric generative model would try to learn the probability density function by estimating the parameters of an underlying distribution we are assuming. So just doing for example,
$$\theta^* = arg\max_\theta  \,\prod_{i=1}^N p_\theta(\textbf{x}_i)$$
I realize that in practice, we need to figure out what is the basic distribution that we are going to modify by adjusting the parameters $\theta$. So in the case of VAEs we use latent variables assumption to make training feasible, we jointly train $q_\phi(\textbf{z}|\textbf{x})$ and $p_\theta(\textbf{x}|\textbf{z})$ prametrizing both distributions with neural networks (i.e. encoder & decoder). In such case, we end up with the situation that all our distributions are Gaussians (assuming that prior and conditional are gaussians). So, having said that, can we conclude that VAEs are parametric? Also, what could be an example of non-parametric generative model?
I would say that, for example, GANs maybe an example of non-parametric model, as we start with a latent normal distribution but then applying a stack of non-linear transformations ending up with something potentially very complicated.

I call this difference prescribed vs implicit models.
To my knowledge, a parametric model has a fixed,finite number of parameters with respect to the sample size (e.g a linear model), while nonparametric model have an increasing number of parameters with respect to data. See this blog post and this answer for more precise info. Since most models use a Neural Network to generate an output, I prefer the terms prescribed vs implicit.
Viewing this way,

Implicit models directly generate the output by passing a noise vector through a deterministic function (Generally a NN). They do not have a global likelihood, and are less limited in the architecture.
A great example of an implicit model, as you stated, are GANs.

Prescribed models provides an explicit parametric specification of the output distribution $p(x)$. They can estimate the likelihood of data, but are more constrained in the architecture.

An Autoregressive model fits a Bernoulli distribution , learning a neural network to model $p(x_i|x_{<i})$
Normalizing flows learn a change of variable to reshape a gaussian source noise. They must learn an invertible function, so the architecture is heavily constrained.
VAEs assume a Gaussian prior $p(z) \sim \mathcal{N}(0,\mathbb{I})$ that is  decoded by a NN which models $p(x|z)$. They have less limits in the architecture, but they lose the closed form likelihood, and they are trained by maximizing an ELBO. Since they still have (even if approximated) likelihood, they are prescribed models.


text:
Is this simple game solvable with reinforcment learning?
Let's imagine this simple environment :
Each episode has a length of 1 step. Each action leads to a reward for this action.
The action space is of 3 : 'UP', 'DOWN', 'UNKNOWN'
Most of the time, the observation is a random vector. But sometimes, it is a sinusoid. The player has to predict if the next value is greater or smaller than the last one. If the vector is a sinusoid, it is very easy to predict what the next value would be. If it is random, there is no way to know.
If the answer is correct, the player gets a reward of 1. If it is incorrect, -1. If the action taken is 'UNKNOWN', 0.
A good model would predict 'UP' or 'DOWN' when the observation is a sinusoid, and 'UNKNOW' when it is random.
I thought it would be possible to solve this game with a Reinforcment Learning algorithm, what are your thougths ?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

This looks like a job for a multiclass classifier. You can build a classifier model to predict one of your three classes and train it on test data with known correct outputs very easily.
Although you can fit reinforcement learning (RL) to the problem, there is no real benefit to doing so, and learning will be less efficient, since RL works on trial-and error. There is no need for trial-and-error here, or conversion into a reward score. Instead you can directly use an error gradient from the prediction.
If your purpose is to find a simple game where RL could be applied, so you can practise RL, then you need to make a more complex game, which has time steps and actions that have consequences involving changes to state. A common toy game for teaching yourself RL would be Blackjack (aka 21), where the actions are "twist" or "stick" - see Sutton & Barto: Reinforcement Learning, an Introduction chapter 5, section 5.3 which presents a simplified version of the game and what a solution looks like.

I'd suggest multi armed bandit as an even easier entry point


text:
Mathematics books for reinforcement learning
This question is not about the math prerequisites of reinforcement learning, but about the textbooks of mathematics that are enough to understand the literature on reinforcement learning.
What are the mathematics books that are recommended to study in order to understand the majority of the reinforcement learning literature?

Two books that really helped me to understand RL/deep RL are:
Reinforcement Learning: An Introduction by Andrew Barto and Richard S. Sutton
Deep Reinforcement Learning: Fundamentals, Research and Applications by Hao Dong, Zihan Ding, Shanghang Zhang
Before, I started getting familiar with RL through online courses and papers, but after even coding and research, I read Sutton's book again and found it really great. It deserves to be the reference for RL. Start with it, and you can see how it improves your vision of RL.

to piggyback onto this answer, I'd also recommend doing some probability/statistics. Enough so you're familiar with (conditional) probability distributions, expectations, etc.

The both are RL books but I am asking for math books.

This doesn't answer the question, as mentioned by the OP above. The question is not: which RL books do you recommend? Of course, some RL books may contain 1-2 sections on math prerequisites, but that wasn't the question.

in my opinion both books are enough for getting involve into RL world.


text:
Can an AI generated image (such as pic of human face) be detected that it's AI generated?
AIs are getting better and better at creating images and art. Some of the stuff is almost impossible to be detected by the naked eye. But what about programs and algorithms? Instead of creating an image, can anything detect that this image was created by an AI?
Take this one for example:
This picture of a woman's face was generated by AI

Once we work with bit arrays or matrices after discretization, things become difficult. The artificial bitstream, either coming from a natural or artificial source at the origin, is identical. Take a simple "hello world" as an example. 11 characters, at say 8bits/char, 88 bits. But all that information encodes the text content only, no bit encodes the natural/artificial origin. Therefore, how can we tell if this was produced by a human typing on a keyboard or a code program? That information is by definition lost in discretization.

Images such as this one are produced using generative adversial network, which is build from two models:

one to generate images given a random vector as input
another trying to detect the generated image from two images, with one of them being real

Then the weights of the first model are updated if the second one detected which image is artificial, and the second model is updated if its prediction is wrong.
Of course you might build a model that can sometime detect AI generated images, but it is probably not possible to differentiate them all the time. Then, if you build such model that is better than any other model to detect generated images, it is possible to create another model trained to fool it.

I am not an expert, but it feels like these GANs are not paying attention to the clothes and the background and make them "fluid".
Like, what is this hat the woman in your example is wearing? Why is the right side of the background looks like it is a mix of liquid paint?
Or here:

What is she wearing? Did she kill a rat to make these clothes? And similar fluid background.

TL;DR: Yes, but it's becoming more and more difficult, even for humans, as generative models get better and better. It's a quite hot research topic.
Disclaimer: I am not affiliated with any of the authors, I'm just studying this research topic.
Humans usually look for some visual artifacts (as all the other answers point out), for example

Colour or texture artifacts (colour blobs, unrealistic texture)
Asymmetries or inconsistencies in the image (this is easy to spot in faces or hair, for example)
Anomalies in color, lighting, image parts

But as models become more and more advanced, these artifacts are becoming harder to spot, if not completely disappeared. As in 2023, images coming from diffusion models like Stable Diffusion or Midjourney API have a photorealistic quality, and often are already indistinguishable from real images (see some examples here).
For these reasons, we want to find automatic and more robust detection approaches to prevent malicious uses of these AI generation models.
Detection approach
A simple approach is to train a detector on AI generated images, which classifies the image as real or AI generated either by looking at the whole images or at single patches [Chai et al., 2020].
A more performing approach is to exploit some invisible artifacts created by the convolutional upsampling, which is commonly used in GANs and in some Diffusion models (Stable Diffusion, for example) to create high-resolution images. While invisible in the image domain, this trace can be easily extracted and identified in the frequency domain [Marra et al., 2019, Yu et al., 2019].

The detector needs to be robust to common image modifications (contrast/luminosity, colour jittering, jpeg compression,etc.) and also to adversarial attacks on the detector.
Watermarking/Fingerprinting
To proactively improve the detection performance, the developers of generative models could include a watermark in their images, to mark the images produced by their models as AI-generated. This watermark is typically invisible, and can be generated in several ways:

Traditional approaches are based on frequency decompositions of the image, constructed through DCT, DWT, Fourier-Mellin, or complex wavelet transformations Cox et al., 1996, O’ Ruanaidh et al., 1996,
O’Ruanaidh and Pun, 1997]. These frequency transformations all share the beneficial property that simple image manipulations, such as translations, rotations, and resizing are easily understandable and watermarks can be constructed with robustness to these transformations in mind.
Model-based approaches use a different learned model to embed a watermark in the image. Hayes and Danezis [2017] and Zhu
et al. [2018] propose strategies to learn watermarking end-to-end, where both the watermark encoder and the watermark decoder are learned models, optimized via adversarial objectives to maximize transmission and robustness [Zhang et al., 2019]. Zeng et al. [2023] present a related approach, in which a neural network watermarked encoder and its associate detector are jointly learned using an image dataset. Notably these approaches still work like a traditional watermark in that the encoder imprints a post-hoc signal onto a given image - however the type of imprint is now learned.
More recent approaches use another model or the generative model itself to either embed the watermark after the generation process or during the generation process. Some approaches consist in embedding a watermark in training data [Yu et al., 2022], in some components of the model (e.g the convolutional decoder) [Fernandez et al., 2023], or by slightly modifying the distribution from where sampling is performed [Wen et al., 2023].

As detectors, watermarks need to be robust to common image transformations (such as cropping, contrast/luminance editing, jpeg compression, etc.) and adversarial attacks that actively try to remove the watermark.
References

Gragnaniello, Diego et al. “Are GAN Generated Images Easy to Detect? A Critical Analysis of the State-Of-The-Art.” 2021 IEEE International Conference on Multimedia and Expo (ICME) (2021): 1-6.
enbo Wan, Jun Wang, Yunming Zhang, Jing Li, Hui Yu, and Jiande Sun. A comprehensive survey on
robust image watermarking. Neurocomputing, 488:226–247, June 2022. ISSN 0925-2312. doi: 10.
1016/j.neucom.2022.02.083. URL https://www.sciencedirect.com/science/article/
pii/S0925231222002533.
[Cox et al., 1996] Cox, I., Kilian, J., Leighton, T., and Shamoon, T. (1996). Secure spread spectrum watermarking for images, audio and video. In Proceedings of 3rd IEEE International Conference on Image Processing, volume 3, pages 243–246 vol.3.
[O’ Ruanaidh et al., 1996] O’ Ruanaidh, J., Dowling, W., and Boland, F. (1996). Watermarking digital
images for copyright protection. IEE PROCEEDINGS VISION IMAGE AND SIGNAL PROCESSING, 143:250–256.
O’Ruanaidh and Pun, 1997] O’Ruanaidh, J. J. and Pun, T. (1997). Rotation, scale and translation invariant digital image watermarking. In Proceedings of International Conference on Image Processing, volume 1, pages 536–539. IEEE.
[Zhu et al., 2018] Zhu, J., Kaplan, R., Johnson, J., and Fei-Fei, L. (2018). Hidden: Hiding data with deep networks. In Proceedings of the European conference on computer vision (ECCV), pages 657–672. 3
[Marra et al., 2019] Marra, F., Gragnaniello, D., Verdoliva, L., and Poggi, G. (2019). Do gans leave artificial fingerprints? In 2019 IEEE conference on multimedia information processing and retrieval
(MIPR), pages 506–511. IEEE.
[Yu et al., 2019] Yu, N., Davis, L. S., and Fritz, M. (2019). Attributing fake images to gans: Learning and analyzing gan fingerprints. In Proceedings of the IEEE/CVF international conference on
computer vision, pages 7556–7566.
2
[Chai et al., 2020] Chai, L., Bau, D., Lim, S.-N., and Isola, P. (2020). What makes fake images detectable? understanding properties that generalize
[Yu et al., 2022] Yu, N., Skripniuk, V., Abdelnabi, S., and Fritz, M. (2022). Artificial fingerprinting for generative models: Rooting deepfake attribution in training data.
[Corvi et al., 2023] Corvi, R., Cozzolino, D., Zingarini, G., Poggi, G., Nagano, K., and Verdoliva, L.
(2023). On the detection of synthetic images generated by diffusion models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
[Wen et al., 2023] Wen, Y., Kirchenbauer, J., Geiping, J., and Goldstein, T. (2023). Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust.
[Fernandez et al., 2023] Fernandez, P., Couairon, G., J ́egou, H., Douze, M., and Furon, T. (2023). The stable signature: Rooting watermarks in latent diffusion models.

I have not worked practically with GANs and just know their theory, but I do not agree 100% with this comment that AI chooses stupid things for clothes or backgrounds. I remember it could be detected when a video was generated with deep learning methods from Obama.

You are supposed to use comments for this.

I wanted to do that but I can not comment. it is restricted

It is better to wait a little then. Maybe a little frustrating, but you risk getting downvotes, which will make your rep go down, if you try and work around the way the tools are set up. Sadly it has to be set up like this, because spambots would overwhelm things (costing extra volunteer moderator time) if there wasn't some significant barrier to commenting from a new account.

There's a paper that claims to detect AI generated images with a 95% accuracy.
https://www.researchgate.net/publication/326053461_Detection_of_GAN-Generated_Fake_Images_over_Social_Networks
A search with the right keywords can reveal more such research.


text:
Tensorflow-gpu and multiprocessing
I have finished implementing an Asynchronous Advantage Actor-Critic (A3C) agent for TensorFlow (gpu). By using a single RMSprop optimizer with shared statistics. To do so, a central controller holds both the Global Network (ActorCriticModel) and the Optimizer. Workers, to update the global network, need to communicate gradients to the Controller, while receiving back the updated model weights. It works decently: despite there might be some errors in the code, it still achieves the results pretty fast.
Each Worker is a thread on its own.
class Worker(Thread):
Problem is Workers need to extend the Thread class while I would like to use heavy Processes (from multiprocessing) instead (Queues used in the following snippet of code are imported from the multiprocessing class).
class Controller:

    def __init__(self, **kwargs):

        self.terminated_workers = 0
        self.env_name = kwargs["env_name"]
        self.available_actions = kwargs["available_actions"]
        self.action_size = len(self.available_actions)
        self.env = make_env(kwargs, env_name=self.env_name)
        self.state_shape = self.env.observation_space.shape

        self.number_of_workers = kwargs["number_of_workers"]
        self.workers = []
        self.controller_queue = Queue()
        self.worker_queues = []

        # Initialization of threads
        for i in range(self.number_of_workers):
            worker_queue = Queue()
            self.worker_queues.append(worker_queue)

            updater = GlobalUpdater(self.controller_queue, worker_queue)
            worker = Worker(i, updater, self.state_shape, kwargs)
            self.workers.append(worker)

        # Only import tensorflow after heavy Process initialization
        import tensorflow as tf

        self.global_model = ActorCriticModel(self.state_shape, self.action_size)
        self.opt = tf.keras.optimizers.RMSprop(
            learning_rate=kwargs["lr_schedule"],
            rho=kwargs["rho"],
            centered=kwargs["centered"],
            clipvalue=kwargs["clipvalue"]
        )

As soon as i change the Worker to extend Processes instead of threads, I get the following message for every worker instantiated:
F tensorflow/stream_executor/cuda/cuda_driver.cc:152] Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error
Technically, since multiple methods of the Worker class need to use tensorflow, the library is imported statically, before declaring the Worker class, as follows:
import tensorflow as tf

class Worker(Process):
   ...

I'm using Linux and the libraries' versions are:

TensorFlow: 2.9.1
CUDA: 11.2.2-1
cuDNN: 8.1.1.33-1+cuda11.2

I tried every other solution I found online, including setting mp.set_start_method('spawn') and trying to postpone as much as i can the import of the tensorflow library, so to postpone as much as possible the heavy process initialization but nothing seems to work. How would you approach this problem? Do you have any idea on how to solve it? Thanks in advance.

Firstly, this question is well suited for StackOverflow, but this forum is less about implementation issues - so this will probably get closed.
However, this sounds to me like an issue with memory being not shared petween Processes. Have you tried to import tensorflow in the constructor and reference it as self.tf or use global tf?
Also you should use gpu memory growth, otherwise the first process will occupy the whole space on the GPU.

Thank you for your kind answer. That's pretty smart, but unfortunately the self.tf is not compatible with the code (as I'm using the @tf.function).

So, I gave it a try by commenting tf.function, setting multiprocessing.set_start_method('spawn', force=True), importing TensorFlow within the spawned process upon creation and putting global tf right after the import.
And it actually works. You're a beast! I've spent months trying to figure this out, it's a pity this post will be removed as I think a lot of people would actually need to see this.

Oh Really? :D That's amazing! I'm glad I was helpful
Maybe the post stays I don't know, usually these post get removed but who reads the manual anyways^^


text:
Is the loss calculation step in Logistic Regression even needed?
I was reading about Logistic Regression and trying to implement the model from scratch. Maybe I am wrong, but I have noticed that the loss calculation step is meaningless in training a Logistic Regression model.
For example:
loss = -(y * np.log(y_hat) + (1 - y) * np.log(1 - y_hat))

Almost every article or video I have watched it was said that "We need calculate the loss based n output and the ground truth values". But then when we do backprop through the model we calculate gradient for dz like this:
dz = y_hat - y

And parameters are calculated like this:
dw = np.dot(x, dz.T) / m
db = np.sum(dz) / m

Gradient descent:
w = w - learning_rate * dw
b = b - learning_rate * db

Through all of these steps there is not a single place where the variable loss was used. From what I can conclude is that we are calculating loss just to print it on the screen for the programmer to see how is the training process going (to see if the model is learning something or not).
Does that mean that the loss calculation step is not needed at all or am I missing some key points?
If answer t the previous question is NO, does that mean that loss calculation step is trivial in Neural Networks as well?


text:
Cannot understand/ reproduce reuslt in figure 1 from MobileNetV2?
Hi I am recently reading the MobileNetV2 paper and I found I cannot understand the following figure in the paper.

I made a simple code trying to reproduce the results, but I got opposite results.
These are visualizations of (1)my input, (2)after ReLU in 3 dimensional space, (3)transform from 3D back to 2D:

And here is the result for transforming the same input to 15 dimensional space, applying ReLU and transform back to 2D.

I don't see an obvious difference as expected in the paper. Since this paper is cited many times, I bet I made mistakes somewhere. Could anybody help point out where I am wrong? I used the following code to generate my results:
import numpy as np
import matplotlib.pyplot as plt
n=256
angle = np.linspace(0, 12*2*np.pi, n)
radius = np.linspace(.5, 1., n)
x = radius * np.cos(angle)
y = radius * np.sin(angle)
plt.plot(x,y)
plt.show()

# ==== embed into 3 dimensional space and transform back after applying ReLU ====
T = np.random.random((3, 2))
T_inv = np.linalg.pinv(T)
after_T = T @ np.row_stack((x, y))
after_relu = after_T
after_relu[after_relu < 0] = 0

# ---- visualize transformed 3d shape ----
ax = plt.axes(projection='3d')
ax.plot3D(after_relu[0],after_relu[1],after_relu[2])
plt.show()
# ---- ----

new_x_y = T_inv @ after_relu
plt.plot(new_x_y[0], new_x_y[1])
plt.show()

# ==== embed into 15 dimensional space and transform back after applying ReLU ====
T = np.random.random((15, 2))
T_inv = np.linalg.pinv(T)
after_T = T @ np.row_stack((x, y))
after_relu = after_T
after_relu[after_relu < 0] = 0
new_x_y = T_inv @ after_relu
plt.plot(new_x_y[0], new_x_y[1])
plt.show()


text:
Help with model architecture for a racing game
I’m working on a model for a racing game using pytorch. The model gets frame from the game as input and produces a controller state as output. The dataset consists of frames from the game and corresponding controller state (buttons state + joystick state)
Until now I’ve been using a cnn connected to linear layers to predict the output. The problem is that this model cannot establish connection between buttons state output and the joystick state output so it can’t learn how to drift or aim items. (when the player drifts using b, the joystick output need to be updated accordingly)
I thought about the following architecture:

I’m new to deep learning and I would like to know if someting like that even makes any sense because I haven’t seen anything like it before. If it does not make sense , can you suggest another model architectre?
Thanks in advance.
dht


text:
is it possible to train the same neural network with different numbers of inputs and outputs?
is it possible to create an adaptative neural network that can change the number of its inputs and outputs without having to train it each time it changes?
the neural netwrok has to take purchases and affect them to the delivery man that will minimize the global traveled distance.

Yes, it is. For example recurrent neural networks may process sequences with different lengths, that is, with a different number of inputs.

you say the input data are purchases, you should specify more in details how these data look like to have a precise answer. mikkola might be right but maybe recurrent neural nets are not suitable for your use case.

the customer can do purchase only two products( A and B) in specefic quantities. for exemple a customer can purchase only 1, 2, 3, 4, 5, 6, 7, 8, 11, 14 or 18 items of a product (A or B) or any linear combination of them.

the number of delivery men can vary from day to day but  the quantities they can deliver per day is already fixed


text:
how to manage the impact of Covid on building a machine learning model
I need your suggestions for using historical data to build a machine learning model for analyzing the market and build an AI model(tree based model/random forest or regression analysis) for setting the price.
I have data from these years:

< 2020
2020
2021
2022

I am not confident about predictive power of pre covid data, my question form data scientist guys is how they manage the probable effect of covid. I divided my historical data as above because I think we have

pre-covid, normal data,
covid-time and its shock
after-covid data, data with the impact of covid
next-normal, I am not even sure that the industry meets new normal or not

do you have any guide line that says how to start, I need a framework to manage these data.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.


text:
Neural network and logical gates
I have a network witch consist of two fully connected layers (without bias) and a ReLu function in between. The network input is two binary numbers, and the output should be the a logical gate result:

I need to find the weights values for each one of the following logical gates: AND, OR, XOR.
In addition, I need to figure out witch one of the three gates cannot be achieved without using an activation function.
I am trying to think what the weights values should be, but I can't figure it out. I thought that $h_1$ can be $x_1+x_2$ and $h_2$ can be $x_1-x_2$ but it's not working for any of the gates.