text:
What is ergodicity in a Markov Decision Process (MDP)?
I have read about the concept of ergodicity on the safe RL paper by Moldovan (section 3.2) and the RL book by Sutton (chapter 10.3, 2nd paragraph).
The first one says that "a belief over MDPs is ergodic if and only if any state is reachable from any other state via some policy or, equivalently, if and only if":
$$\forall s, s', \exists \pi_r \text{ such that } E_\beta E_{s, \pi_r}^P [B_{s'}] = 1$$
where:

$B_{s'}$ is an indicator random variable of the event that the system reaches state $s'$ at least once, i.e., $B_{s'} = 1 \{ \exists t < \infty \text{ such that } s_t = s'\}$
$E_\beta E_{s, \pi_r}^P[B_{s'}]$ is the expected value for $B_{s'}$, under the belief over the MDP dynamics $\beta$, policy $\pi$ and transition measure $P$.

The second one says "$\mu_\pi$ is the steady-state distribution, which is assumed to exist for any $\pi$ and to be independent of $s_0$. This assumption about the MDP is known as ergodicity.". They define $\mu_\pi$ as:
$$\mu_\pi(s) \doteq \lim_{t \to \infty} \Pr\{s_t=s \vert a_{0:t-1} \sim \pi\}$$

i.e., there is a chance of landing on state $s$ by executing actions according to policy $\pi$.

I noticed that the first definition requires that at least one policy should exist for each $(s, s')$ pair for the MDP to be ergodic The second definition, however, requires that all policies eventually visit all the states in an MDP, which seems to be a more strict definition.
Then, I came accross the ergodicity definition for Markov chains:

A state $i$ is said to be ergodic if it is aperiodic and positive recurrent. In other words, a state $i$ is ergodic if it is recurrent, has a period of $1$, and has finite mean recurrence time. If all states in an irreducible Markov chain are ergodic, then the chain is said to be ergodic.

This leads me to believe that the second definition (the stricter one) is the most appropriate one, considering the ergodicity definition in an MDP derives from the definition in a Markov chain. As an MDP is basically a Markov chain with choice (actions), ergodicity should mean that independently of the action taken, all states are visited, i.e., all policies ensure ergodicity.
Am I correct in assuming these are different definitions? Can both still be called "ergodicity"? If not, which one is the most correct?

In short, the relevant class of a MDPs that guarantees the existence of a unique stationary state distribution for every deterministic stationary policy are unichain MDPs (Puterman 1994, Sect. 8.3). However, the unichain assumption does not mean that every policy will eventually visit every state. I believe your confusion arises from the difference between unichain and more constrained ergodic MDPs.
Puterman defines that a MDP is (emphasis in the following mine):

recurrent or ergodic if the state transition matrix corresponding to every deterministic stationary policy consists of a single recurrent
class, and
unichain, if the state transition matrix corresponding to every deterministic stationary policy is unichain, that is, it consists of a
single recurrent class and a possibly empty set of transient states.

Before unpacking this further, let's first recap what recurrent and transient states in a Markov chain are. The following definitions can be found in Puterman, Appendix A.2. For a state $s$, associate two random variables $\nu_s$ and $\tau_s$ that represent the number of visits and the time of the first visit (or first return if the chain starts in $s$) to state $s$. A state $s$ is

recurrent (sometimes also called ergodic), if $P_s(\tau_s < \infty) = 1$, that is, the state is eventually visited (or returned to), and
transient, if $\mathbb{E}[\tau_s] = \infty$ and $P_s(\tau_s < \infty) < 1$.

It is also true that $s$ is recurrent if and only if $\mathbb{E}[\nu_s] = \infty$, i.e., it is visited infinitely often, and $s$ is transient if and only if $\mathbb{E}[\nu_s] < \infty$, i.e., it is visited only finitely often (and thereby never again after some finite time).
So let's now return to the two types of MDPs above. Consider an arbitrary deterministic stationary policy $\pi$ which maps any state $s$ to an action $a = \pi(s)$. If the MDP is ergodic, then the stationary distribution $\mu_\pi(s)$ exists and is unique, because the Markov chain over states induced by any policy has a single recurrent class (it does not matter in which state $s_0$ the chain starts, the same stationary distribution is reached). There is a single class of recurrent states, i.e., all states are recurrent, therefore any $s$ is visited infinitely often and $\mu_\pi(s) > 0$.
Now, if the MDP is unichain, then once again the stationary distribution $\mu_\pi(s)$ exists and is unique, because the Markov chain over states induced by any policy has a single recurrent class. But, importantly, there may exist a policy $\pi$ for which the set of transient states in the induced Markov chain is not empty. Because any transient state $s$ will only be visited a finite number of times, in the (infinite horizon) stationary distribution $\mu_\pi(s)=0$!
So indeed, if the MDP is of the stricter ergodic type, every policy will eventually visit every state. This is not true for unichain MDPs however.
A final remark: some authors define a policy as ergodic (e.g., Kearns & Singh, 2002), if the resulting Markov chain over states is ergodic (i.e., has a unique stationary distribution). The unichain MDP is a type of MDP where every policy is ergodic.
References:
Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming.
Kearns & Singh. Near-Optimal Reinforcement Learning in Polynomial Time. Machine Learning, 49, 209–232, 2002

From what I gathered, I have misunderstood what the second definition of ergodicity in my post (the one from Sutton) implied. It does not imply that "there is a chance of landing on state $s$ by executing actions according to policy $\pi$". Only that when following a policy $\pi$, the probability of visiting a state $\mu_\pi$ is "fixed", i.e., the only thing that changes the probability of visiting a state is the policy $\pi$. I wrongly assumed that ergodicity meant $\mu_\pi(s) > 0, \forall s \in S$, as some states may have $\mu_\pi(s) = 0$ (like unichain MDPs).

But then, I still don't understand if there's a definitive definition for ergodicity. In this thread there are 3 different definitions. (1): there exists a policy $\pi$ such that $\mu_\pi(s) > 0, \forall s \in S$ - the one from Moldovan; (2): for every policy $\pi$, and for all states $\mu_\pi(s)$ exists (and may be equal to 0) - the one from Sutton; (3): for every policy $\pi$, $\mu_\pi(s) > 0, \forall s \in S$ -  the one from Puterman. Should I just clarify what I mean by "ergodicity" when I refer to it, given this ambiguity?

Certainly makes sense to clarify which definition you use. The Moldovan definition concerns something other than ergodicity of MDPs, namely "beliefs over MDPs" in their language. For that I have no comments to add. Sutton (2) is a bit vague, and seems like they could mean something that either equals Puterman's ergodic OR unichain case. All this shows that ergodicity in a MDP is context dependent, could be a property of the MDP as a whole, or a policy, or something even different. Puterman is the authoritative source on MDP theory, and I find he gives very clear defintions of the fundamentals.

I was unaware of the Puterman book. I'll certainly look into reading it. I have considered your answer as correct, as you provided a formal definition of ergodicity in an MDP. Given that I have always associated ergodicity with being able to eventually visit any state $s'$ from any other state $s$, I will probably be using Puterman's definition in the context of MDPs and Moldovan's in the context of policies. Thank you very much!


text:
Why does my neural network to solve the XOR problem always output 0.5?
I'm trying to create a neural network to simulate an XOR gate.
Here's my dataset:
╔════════╦════════╗
║ x1, x2 ║ y1, y2 ║
╠════════╬════════╣
║  0, 0  ║  0, 1  ║
║  0, 1  ║  1, 0  ║
║  1, 0  ║  1, 0  ║
║  1, 1  ║  0, 1  ║
╚════════╩════════╝

And my neural network:

I use logistic loss to get the error between target $y_{k}$ and output $\hat{y}_{k}$:
$$ E(y_{k}, \hat{y}_{k}) = - y_{k} \cdot log(\hat{y}_{k}) + (1 - y_{k}) \cdot log(1 - \hat{y}_{k}) $$
And then use the chain rule to update the weights. For example weight $w_{3}$'s contribution to the error is:
$$ \sum_{k=1}^{2} \frac{\partial E(y_{k}, \hat{y}_{k})}{\partial w_{3}} = \sum_{k=1}^{2} \left(\frac{\partial E(y_{k}, \hat{y}_{k})}{\partial \hat{y}_{k}} \cdot \frac{\partial s_{k}}{\partial c_{1}}\right) \cdot \frac{\partial c_{1}}{\partial w_{3}} $$
Which in developed form is:
$$ \sum_{k=1}^{2} \frac{\partial E(y_{k}, \hat{y}_{k})}{\partial w_{3}} = \left( \left( - \frac{y_{1}}{\hat{y}_{1}} + \frac{1 - y_{1}}{1 - \hat{y}_{1}} \right) \cdot \hat{y}_{1} \cdot (1 - \hat{y}_{1}) + \left( - \frac{y_{2}}{\hat{y}_{2}} + \frac{1 - y_{2}}{1 - \hat{y}_{2}} \right) \cdot (- \hat{y}_{2}) \cdot\frac{c_{1}}{c_{1} + c_{2}} \right) \cdot s_{0} $$
My issue is that, after a couple epochs of training on the entire dataset, the network always outputs:
$$ \hat{y}_{1} = \hat{y}_{2} = 0.5 $$
What am I doing wrong?

One neuron on its own can only solve linearly separable problems. You need a combination of neurons to solve non-linearly separable problems.
For the XOR case, you need at least 2 neurons at the first layer, and 1 neuron at the output layer to properly classify it.
Keep in mind that, sometimes, the 3-neurons network might get stuck in a local minimum as well. You will need some luck in the random initialization of weights. Using the right seed during the random initialization of weights can help converge, and some other seed will only result in a stuck network.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Viola-Jones algorithm: Haar-like features, how are the features extracted?
If I have an image like this
1 2 3 4 5 6 7 8
a b c d e f g h
...

And I apply a Haar-like feature with a template
1 1 1 1
-1 -1 -1 -1

Then in the first position we get X1 = 1+2+3+4+a+b+c+d. If we slide one side to the right, we again get X2 = 2+3+4+5+b+c+d+e.
This way we will get X1 and X2 and X3 and so on. Now, how are these values combined to get the feature? Because when we say a feature we are not just running that template in one place, rather we will run it over multiple places in the image. It gives lots of values like X1,X2 and X3 and so on. Now, how are those combined to get the final feature which will be passed to Adaboost?

I would look at table 1 of the original paper. While you're reading the alogorithm, try to really focus on Step 2 when you get to it.

In summary, each feature is used to train it's own classifier. So in your example, the calculated features X1, X2, ... Xn you describe coorespond to apply some set of feature transforms f_1, f_2, ... f_n to a single image. This is a bit backwards from what actually happens. What the method really does is train a classifier for each feature. So if you had n features, you would have n classifiers. Then in an adaboost fashion, you upweight the classifier that performed the best. I.e, you are upweighting the classifier based soley on th best performing feature. You then repeat and re-weight all the classifiers until you reach convergence.

Sorry if this is stupid thing to ask - but in your answer when you say there are X1,...Xn the n features - they are coming from the same Haar featuretemplate shown in example in the question right? So when are we using the all of the haar feature templates? (Btw +1 for your effort in answering my bad question maybe)

Nah, no stupid questions! Only stupid answers on my part. More concretely, each haar feature template would be used to create a single classifier. So if you have n haar feature templates you have n classifiers. For each of those n classifiers, you train the entire image set and use adaboost to come to a final solution.

I have n haar feature templates. So for example, one template is as shown in my question. Now if I slide this feature template over an image I will get many scalar values due to different location on which it is slid over. All I am asking this, what would I do with all these values? Do I just put them a vector and tell that it is my feature corresponding to the feature template? Also just notice, in my question I said 'X1is the output of running this feature templated in one location.X2is the output of running the *same* feature template over different position.You defineX1` otherway.

In regards to you saying,
"Do I just put them into a vector and tell that it my feature cooresponding to the feature template"
The answer is yes. The point is that you're going to have a ton of features in the begining (on the order of thousands) where most of them are actually useless and will get filtered out (or upweighted) by adaboost.
I found a link to a python implementation that seems much easier to read than staring at the opencv implementation if that helps https://github.com/ZihengZZH/viola-jones/blob/master/src/adaboost.py#L44.

I will select this as answer and I will put another answer drawing whatever you said. Thanks.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
In ensemble learning, does accuracy increase depending on the number of models you want to combine?
I want to predict using the same model as multivariate time series data in a time series prediction problem.
Example:
pa = model predict result(a)
pb = model predict result(b)
pc = model predict result(c)
...
model ensemble([pa, pb, pc,...]) -> predict(y)

Can I expect a better performance of our model by using a model ensemble with more kinds of time series data here?

Do you mean that would be a single model, but you have multiple timeseries with same target and you plan to ensemble prediction for these timeseries?

Yes. I want to make predictions with multiple time series data of different kinds with the same model.

Yes, you can. Let's say you have 5 classes named a,b,c,d,e. You fit your data into a SVM Classifier and a Random Forest Classifier. Assume that, SVM classified "a" and "b" class well and RFC classified "c","d","e" well. So, ensembling these two models is going to increase accuracy  dramatically. Ensemble learning is really good when a model generating many "False-Positive" and "False-Negative". You can also use weighted ensemble learning methods. You can set greater weights for reliable models & lower weights for untrustable models.


text:
Are there RL algorithms that also try to predict the next state?
So far I've developed simple RL algorithms, like Deep Q-Learning and Double Deep Q-Learning. Also, I read a bit about A3C and policy gradient but superficially.
If I remember correctly, all these algorithms focus on the value of the action and try to get the maximum one. Is there an RL algorithm that also tries to predict what the next state will be, given a possible action that the agent would take?
Then, in parallel to the constant training for getting the best reward, there will also be constant training to predict the next state as accurately as possible? And then have that prediction of the next state always be passed as an input into the NN that decides on the action to take. Seems like a useful piece of information.

Comments are not for extended discussion; this conversation has been moved to chat.

Yes, there are algorithms that try to predict the next state. Usually this will be a model based algorithm -- this is where the agent tries to make use of a model of the environment to help it learn. I'm not sure on the best resource to learn about this but my go-to recommendation is always the Sutton and Barto book.
This paper introduces PlanGAN; the idea of this model is to use a GAN to generate a trajectory. This will include not only predicting the next state but all future states in a trajectory.
This paper introduces a novelty function to incentivise the agent to visit unexplored states. The idea is that for unexplored states, a model that predicts the next state from the state-action tuple will have high error (measured by Euclidean distance from true next state) and they add this error to the original reward to make a modified reward.
This paper introduces Dreamer. This is where all learning is done in a latent space and so the transition dynamics of this latent space must be learned, another example of needing to learn the next state.
These are just some examples of papers that try to predict the next state, there are many more out there that I would recommend you look for.

Check out Imagination-Augmented Agents paper - seems like it does what you are talking about. The agent itself is the standard A3C that you are familiar with. The novelty is the "imagination" environment model which is trained to predict the  behavior of the environment.


text:
Can someone explain R1 regularization function in simple terms?
I'm trying to understand the R1 regularization function, both the abstract concept and every symbol in the formula.
According to the article, the definition of R1 is:

It penalizes the discriminator from deviating from the Nash Equilibrium via penalizing the gradient on real data alone: when the generator distribution produces the true data distribution and the discriminator is equal to 0 on the data manifold, the gradient penalty ensures that the discriminator cannot create a non-zero gradient orthogonal to the data manifold without suffering a loss in the GAN game.
$R_1(\psi  ) = \frac{\gamma}{2}E_{pD(x)}\left [ \left \| \bigtriangledown D_{\psi}(x) \right \|^2 \right ]$

I have basic understanding of how GAN's and back-propagation works. I understand the idea of punishing the discriminator when he deviates from the Nash equilibrium. The rest of it gets murky, even if it might be basic math. For example, I'm not sure why it matters if the gradient is orthogonal to the data.
On the equation part, it's even more unclear. The discriminator input is always an image, so I assume $x$ is an image. Then what is $\psi$ and $\gamma$?
(I understand this is somewhat of a basic question, but seems there are no blogs about it for us simple non-researchers, math challenged people who fail to understand the original article )

Rather than providing a link to an external resource, can you just type in latex (yes, you can use latex on this site) of the specific formula that you're referring to? Moreover, please, provide some context, and do not just say "explain X".

@nbro I added latex. I initially avoided adding context because I wanted to keep this question more generic as this really seems to be mostly uncovered by popular blogs. I tried to add some directions but I'm afraid I don't have any more concrete context other than I encountered that StyleGAN uses for one dataset and not for another, and I want to "get a feel" as to how it works.

Thank you! Even if you're looking for a general answer, providing some context is always a good idea.

Here is how I understand this regularization (updated).
$R_1$ is simply the norm of the gradients w.r.t. to the real data (gradients indicate the direction and magnitude by which the weight should be updated during training). So, $R_1$ prevents large weights update for the discriminator if it has high gradients on real data, effectively smoothing its decision boundary around the real data points.
$$
R_{1}\left(\psi\right) = \frac{\gamma}{2}E_{p_{D}\left(x\right)}\left[||\nabla{D_{\psi}\left(x\right)}||^{2}\right]\text{,}
$$
where $\psi$ is discriminator weights, $E_{p_{D}\left(x\right)}$ means that we sample data only from the real distribution (i.e. only real images) and $\gamma$ is a hyperparameter.
The intuition is that we want the discriminator's output to be minimally sensitive to small changes in the real data distribution. This helps prevent overfitting to particular noise in the training data. It encourages the discriminator to be less confident overall, thus indirectly helping the generator to improve by preventing the discriminator from becoming too perfect too quickly (a very confident discriminator is a known problem for KL-loss).
The authors also investigate which value is best for $\gamma$ by analyzing the eigenvalues of the Jacobian of the associated gradient vector field, but in my opinion, this value is highly dependent on the dataset and architecture.
When we say "gradient orthogonal to the data manifold," we are referring to changes in the discriminator's parameters that would affect its decision-making process in ways that are not along the data manifold. Since the data manifold represents the set of points in the high-dimensional space that corresponds to real data, gradients in directions orthogonal to this manifold don't improve the discriminator's ability to classify the real data correctly; such gradients might signify overfitting or instability in training.

The current accepted answer is not exactly correct, we are not actually concerned with properties of the model parameters here. The $R_1$ regularization is dealing with the gradient with respect to the data $x$, not the model parameters. The gradient being orthogonal to the data manifold doesn't make sense if we are talking about the gradient with respect to the parameters.
With that out of the way, the essential idea behind $R_1$ normalization is that you want to "flatten" the response of the discriminator. That is, you don't want the value of the discriminator to change too quickly between similar data points. This is a problem because when your generator starts to get good and produce realistic points, your discriminator starts panicking and getting sensitive and paranoid about realistic data points, since the generator is such a good imposter. So the discriminator might classify one datapoint as real, but an almost identical datapoint as fake. You don't want this as it leads to unhelpful extreme feedback from the discriminator so we employ the $R_1$ regulizer, which is a form of gradient penalty to smooth out the discriminator output.
Now let's talk about the gradient being orthogonal to the data. The key point here is the context. The generator is producing "good" samples, and the discriminator is 0 on the real data, meaning the generator is doing so well that the discriminator has essentially no skill yet and claims everything is fake. A non-zero gradient orthogonal to the data means that the discriminator thinks that as you go in the direction towards unrealistic data (orthogonal), then I will mistakenly think that data is more realistic. The gradient penalty helps stop this by squashing the gradients on the data manifold.

Good points, thanks. I've updated my answer


text:
Can residual connections be beneficial when we have a small training dataset?
I have a classification problem, for which an inadequate amount of training data is available. Also, there is no known practical data augmentation approach for this problem (as no unlabelled data is available either), but I am working on it.
As we know, deep neural networks require a large amount of data for training, especially when a deep architecture with many layers is used. Using these complex architectures with less data can easily lead to over-fitting. Residual connections can shortcut some blocks or layers, which can result in simpler models, while we have the benefit of complex structures.
Can residual connections be beneficial when we have a small training dataset?

Can residual connections be beneficial when we have a small training dataset?

The usual rule of data science investigations applies here: Try it, measure the results, then you will know.
It is very hard to tell, a priori, whether a specific architectural or hyperparameter choice will impact the performance of a neural network on a given problem.
In this case, you are wondering whether residual networks using skip connections might help when you have a relatively low amount of training data.

On the pro side, effects of skip connections that help correct vanishing gradients, and treat each block as learning the difference from an identity function, will still work for your problem. That means you will have some freedom to explore adding layers without worrying about the negative impacts of doing so.

On the con side, it is unlikley that you will benefit from very deep networks as there will not be enough examples to learn truly complex functions from.

You may find that adding depth, but reducing "width", i.e. the number of artificial neurons in each layer, will work.
If you have a low amount of training data and a difficult problem to solve, then residual networks are not a magic fix. The best you could hope for is a relatively stable statistical model that works on the simpler differences between your training examples. However, it may be possible that tuning a neural network by searching through different architectures will be a worthwhile exercise.
I would also suggest that for one of the good networks that you vary the number of examples used to train the network in multiple training runs, and plot a learning graph showing number of examples versus accuracy (or other metric that you may be interested in). This graph will help you decide whether collecting more training data would be worthwhile because you will have a rough estimate of the gradient for how much new training examples could improve your results.


text:
What are possible ways to combat overfitting or improve the test accuracy in my case?
I have asked a question here, and one of the comments suggested that this is a case of severe overfitting. I made a neural network, which uses residual boosting (which is done via a KNN), and I am still just able to get < 50% accuracy on the test set.
What should I do?
I tried everything from reducing the number of epochs to replacing some layers with dropout.
Here is the source code.

There are a few issues you need to address first.

Normalise your data. You should try and keep your values for each input in a good range, otherwise you're never going to train anything useful. A simple way of doing this could be to divide each value by the maximum value for that input. This will ensure they are between 0 and 1, or you could divide by the average so you get approximately 1. Whatever the case, you need to do something about decreasing the magnitude of your inputs (also make sure your are consistently scaling your data, especially during validation and training, otherwise your data will likely be misinterpreted by your model)
Optional: Augment your data. You have a very small dataset to work with here, if there's any way you can augment this to make it larger, you will likely get much better performance on validation and testing sets. Augmenting is just creating new data that is still representative of the task (as a visual example, you might augment your image dataset by flipping some images and including those in training as well as the unflipped versions)
Split your data into Training, Validation and Testing sets. Training is used for what you expect - training. Validation is used to validate your model on unseen data. The reason for the testing set is obviously you are going to try your hardest to tweak and modify the model so it performs best on your validation set, but this incurs a kind of bias as you are only tweaking for this specific set of data, which is not completely representative of the real world. So try your hardest to perform best on the validation set by changing model hyper-parameters, but confirm it's real use on the testing set. As a recommended split for your entire data set, I have had success with 10% testing, 20% validation and 70% training, but you might be better off with a different split.
Test different models. Train your network for a small amount of time (say only 10 or so epochs) and see how it performs based on the validation set. Keep in mind because your dataset is particularly small (only ~300 samples in your training set) you don't need to train for many epochs at all to get a good representation of how your model will perform (an epoch being a complete loop over all training data). Make sure when you're doing this you're training in batches.
Try different regularization methods. One of the primary ones being good initialization. Make sure you're initializing your parameters with something like Xavier initialization of similar. Other suggestions include: dropout (though you said you already tried this, maybe try different values for dropout), weight decay and batch normalization layers. To be honest though, I think one of the 4 steps above will solve this issue, so you likely won't need to do any of these.

I may have made some points for things you have already done as there was a lot of code for me to look over in your source. But the biggest issue I noticed was that your data wasn't normalised at all and that usually creates many issues, so definitely try that first if you haven't already.


text:
Should I use U-net to label keys in a keyboard image?
This is a 600*800 image.

Which algorithm/model should I use to get an image like the one below, in which each key is detected and labeled by a rectangle?
I guess this is some kind of a segmentation problem where U-net is the most popular algorithm, though I don't know how to apply it to this particular problem.

If you just need to draw a rectangle around each key, this is an object detection or template matching problem, so you can use any of the available models for object detection (e.g. YOLO) or any technique for multi-template template matching (e.g. you can use sequential RANSAC or t-linkage). In the first case, you will need a labeled dataset, while, in the second case, you will need the original image and the templates (in your case, a template would be an image of a key).
So, no, this is not a segmentation problem (which would be the task of classifying each pixel in the objects of interest, and not just locating the objects).

There are two related problems for images

Semantic segmentation, where you need to assign each pixel on the image some class. I.e. you have a satellite image and want to segmentate roads/forests/fields and so on
Objects detection, where you need to detect different types of objects and draw a bounding box for each. I.e. there is a popular dataset MSCOCO for the task, where you need to localize all bikes/people/cats/etc on the image

U-net is good for the first task, but I would say you have a second. You can use something like YOLOv3 (if you need fast inference) or fast R-CNN if you need precision. If you need really good performance, you can browse top methods for the task on the paperswithcode.com: semantic segmentation, object detection


text:
How is $v_*(s) = \max_{\pi} v_\pi(s)$ also applicable in the case of stochastic policies?
I am reading Sutton & Bartos's Book "Introduction to reinforcement learning". In this book, the defined the optimal value function as:
$$v_*(s) = \max_{\pi} v_\pi(s),$$ for all $s \in \mathcal{S}$.
Do we take the max over all deterministic policies, or do we also look at stochastic policies (is there an example where a stochastic policy always performs better than a deterministic one?)
My intuition is that the value function of a stochastic policy is more or less a linear combination of the deterministic policies it tries to model, however, there are some self-references, so it is not mathematically true).
If we do look over all stochastic policies, shouldn't we take the supremum? Or do we know, that the supremum is achieved, and therefore it is truly a maximum?

The value function is defined as $v_\pi(s) = \mathbb{E}_\pi[G_t | S_t = s]$ where $G_t$ are the (discounted) returns from time step $t$. The expectation is taken with respect to the policy $\pi$ and the transition dynamics of the MDP.
Now, as you pointed out the optimal value function is defined as $v_*(s) = \max_\pi v_\pi(s)\; ; \;\forall s \in \mathcal{S}$. All we are doing here is choosing a policy $\pi$ that maximises the value function; this can be a deterministic or a stochastic policy, though intuitively it is likely to be deterministic unless for some states that are two (or more) actions with the same expected value, in which case you can take any of said actions with equal probability, thus making the policy stochastic.
For a finite MDP (which is what I assumed above too), we know that an optimal value function exists (this is mentioned in the book) so taking the maximum is fine here.

But do we know that there always exists a deterministic optimal policy?
I think I read somewhere that sometimes only stochastic policies are optimal (for example in a game with an opponent, like Scissors-Stone-Paper, where a deterministic policy would always lead to a loss, when the other opponent can also learn).  But thanks for your answer!

@MaximusMJ In Scissors-Stone-Paper, the optimal policy is definitely not deterministic. You can easily beat any deterministic policy in this case (in the long run), i.e. once you know the opponent chooses e.g. stone, you can choose paper, and so on. So, yes, there are cases where the optimal policy is not deterministic. This should answer your question in parentheses "is there an example where a stochastic policy always performs better than a deterministic one?".

@MaximusMJ We know that there always exists an optimal policy (for a finite MDP), whether it is deterministic or not does not matter in the context of taking the $\max$ (I don't think), as we know the $\max$ exists then it is fine.

@nbro I don't think it is helpful to bring game theory into a discussion on single-agent finite MDPs, although in games the value of a stochastic policies can exceed the value of any deterministic policy. This is not the case in MDP however; if there is an optimal policy in MDP, there is a deterministic optimal policy, additional discussion: https://mathoverflow.net/a/44685/80487


text:
Is there a relationship between Computer Algebra and NLP?
My intuition is that there is some overlap between understanding language and symbolic mathematics (e.g. algebra). The rules of algebra are somewhat like grammar, and the step-by-step arguments get you something like a narrative. If one buys this premise, it might be worth training an AI to do algebra (solve for x, derive this equation, etc).
Moreover, when variables represent "real" numbers (as seen in physics, for example) algebraic equations describe the real world in an abstracted, "linear," way somewhat similar to natural language.
Finally, there are exercises in algebra, like simplifying, deriving useful equations, etcetera which edge into the realm of the subjective, yet it is still much more structured and consistent than language. It seems like this could be a stepping stone towards the ambiguities of natural language.
Can anyone speak to whether this has either (1) been explored or (2) is a totally bogus idea?

There isn't, really. Natural language is way more complex and irregular than algebra, which is far more formalised and unambiguous.
So far, in NLP, most success/progress has been made in little toy domains, which exclude most of the complexities of real life, including many ambiguities.
When you say the rules of algebra are somewhat like grammar, then that is because it is essentially a formal language, for which we can specify a grammar. There is currently no complete grammar for any human language (and I doubt there ever will be), let alone a formal one that can be processed by computer.
This was one of the reasons why the first AI boom, where a lot of over-hyped promises where made about being able to translate Russian into English automatically, failed abysmally: natural languages are more than just formal grammars of lexical items.
Stochastic approaches have gone some way towards pragmatic solutions, but when it comes to understanding language they are basically a fudge. And don't get me started on deep learning approaches to NLP.
So the only relationship is that we use the term 'grammar' for the descriptive formalisms in both cases; a formal grammar of algebra would be very different from a grammar for a human language.
This doesn't mean, however, that approaches developed in the field of NLP cannot be applied to algebra: even those which failed in NLP because they were overly limiting. To find out more about this, look for Chomsky Hierarchy -- that describes the different expressive powers of formal languages.
But I would argue that human language is outside of that, because it is not a formal language.

Gotcha, but it sounds like you kind of agree that there is a relationship - it's just that the NLP problem is drastically harder and specifically that the hardest parts of NLP are not present in doing algebra.

@user37344 Broadly, the only relationship in my view is that we describe both phenomena with formalisms that are called 'grammar', which share a few general features.

It is possible to convert natural-language queries into statements that can be solved by computer algebra systems. Wolfram Alpha does this, for example.

One of the ways to ask if this two problmes are related is to ask, could we solve math/algebra equations with NLP approaches, and the answer is yes, it's an absolutely valid idea and it was approached by many researchers.
For example in the "Deep learning for symbolic mathemathics" paper by facebook researchers, the NLP-based approach was used to solve calculus-level math.
Or in this paper authors propose a method to extract semantics from math problems in words and solve them.
In fact, even very simple approaches like a small LSTM network could work for simple strictly stated problems

Interesting. This is actually the direction that I care more about, so I appreciate the references.

Yes there is a relationship, but it's not an exact one like I think is envisioned in the question.
Fuzzy numbers and fuzzy logic translates natural language expressions into quantitative values and mathematical logic.  See PRUF—a meaning representation language for natural languages by Zadeh.
Dr. Zadeh, the 'inventor' of fuzzy numbers and fuzzy logic, provided a mathematical framework for taking simple language expressions and expressing them in mathematical form. The underlying basis is that natural language is inherently imprecise. To quote this paper, "First, a basic assumption underlying PRUF is that the imprecision that is intrinsic in natural languages is, for the most part, possibilistic rather than probabilistic in nature."
While fuzzy numbers and fuzzy logic are ubiquitous in many applications, for example fuzzy controllers in engineering, I personally have not seen a groundswell of using PRUF in NLP. Not that it isn't possible. I have been pursuing research off-and-on in this area, and I think that there is a possibility of bridging mathematical logic and NLP, particularly when using concept-based NLP (e.g., something along the lines of Cognitive Linguistics).

A lot of natural language processing software are (in 2021) using statistical approaches.
Read The Deep Learning Revolution (by T.Sejnowski), Artificial Beings, the Conscience of a conscient machine (by J.Pitrat), Introduction to Deep Learning (by E.Charniak).
However, mixed approaches (like in RefPerSys) can also be used.
The so-called frame-based approaches can, and have been used, in NLP. From an abstract point of view, they are close to algebra. And probably are still one of the best approaches for generating natural language sentences (in written form).
Pitrat has for dozen of years taught and advocated a symbolic AI approach to NLP. You could read his French book Métaconnaissances, futur de l'IA (metaknowledge, future of AI).
Read also books like Knowledge Representation and Reasoning (Braqueman & Leveque)
You might also read (if you can read French) Nicolas Bourbaki's Théorie des Ensembles or (in English) Interactive Theorem Proving and Program Development (by Bertot & Castéran).
Symbolic AI approaches are definitely "algebra-like". In practice, they don't need powerful GPGPUs, or even don't do a lot of floating point operations.
I assume you want to process human language text available as textual files (e.g. HTML, XML, UTF8 encoded plain text, etc...). If you want to process sounds and speech (for example, an audio signal coming from a microphone), you do need much more signal processing and floating-point operations. If you want to process images (e.g. handwritten text), it is also different. In France, I would recommend Professor Mohamed Daoudi.

This looks like good reading, but it's not clear to me that it is relevant to doing algebra.

Then please improve your question and explain, in at least one paragraph of written English, what do you mean by "doing algebra". Be aware that I am not a native English speaker. If you don't want to write that publicly, send me an email (with the URL of your question) to basile@starynkevitch.net - with several paragraphs in English, French, or Russian

@user37344 My sense is that much of computing in general depends on algebra, and that you can't do much with computational linguistics without some algebraic functions. I was likewise confused by the term "computer algebra".  But it seems like basic algebra is not sufficient for NLP of any meaningful strength.


text:
Do these FOL formula both represent "You can fool some of the people all of the time"?

You can fool some of the people all of the time.

This can be represented in FOL as follows
$$\exists x \; \forall t  \; (\text{person}(x) \land \text{time}(t)) \Rightarrow \text{can-fool}(x,t) \tag{1}\label{1}$$
Is $\exists x \; \forall t \; \text{can-fool}(\text{person}(x), \text{time}(t))$ equivalent to (\ref{1}) ?

(1) can be paraphrased as "There exists an x, and for any t if x is a person and t is a time, then x can be fooled at time t" (I would use fool-able instead of can-fool, as it is closer to the intended meaning).
(2) would be "There exists an x, and for any t, you can fool x is a person and t is a time."
They are not equivalent: person(x)  and time(t) are boolean predicates, which return a truth value: they are true if x is a person, and t is a time, respectively. So in (1) they act as a constraint on the values that x and t can take. If x was a saucepan, then person(x) would be false, and thus you wouldn't be able to claim that you can fool a saucepan.
So can-fool takes two arguments: one for which person is true, and one for which time is true. But in (2), the arguments are actually the boolean truth values: if x was "Falstaff" and t was "yesterday", then in (1) the premise would be true, as person("Falstaff") and time("yesterday") are true, and so you conclude can-fool("Falstaff", "yesterday").
In (2) that becomes can-fool(person("Falstaff"), time("yesterday")), which evaluates to can-fool(true, true), and that won't work.


text:
Could I just choose the other (non-predicted) class when the accuracy is low?
I have a binary classification problem.
My neural network is getting between 10% and 45% accuracy on the validation set and 80% on the training set. Now, if I have a 10% accuracy and I just take the opposite of the predicted class, I will get 90% accuracy.
I am going to add a KNN module that shuts down that process if the inputted data is, or is very similar to the data present in the data set.
Would this be a valid approach for my project (which is going to go on my resume)?

At a guess without more information, this sounds like a relatively severe case of overfitting, or perhaps the test data is distributed rather differently to the training data. Although in principle what you're saying would work, it sounds like a bit of a hack rather than a genuine solution to the problem. Perhaps focus on trying to figure out why you're getting such a dramatic discrepancy? We might be able to help here too.

@htl ok sure I have been working on it for weeks and nothing works. If u wanna find the source code check out this link https://github.com/Julius-R-UI/Heart-Failure-Predictions-Model

Investors do this.  They call it 'contrarian trading'.  George Costanza, on the Seinfeld show, did this quite successfully for many decisions in his life: https://www.youtube.com/watch?v=cKUvKE3bQlY

The part "KNN module" is not clear to me. What do you mean by that sentence? Do you mean that you will use the k-nearest neighbour to compute what exactly?

The short answer is no, you shouldn't do that.
There is a "distribution shift" thing when you have different x-y relation on the validation set then on the train set. The distribution shift would deteriorate your model performance and you should try to avoid that. The reason it's bad - ok, you find the way to fix the model for validation data, but what about novel test data? Will it be like a train? Will it be like validation? You don't know and your model is worthless in fact.
What you can do

You should redefine the train/validation scheme. I would recommend making cross-validation with at least three splits
As mentioned in a comment, your model seems to overfit; try to apply some regularization techniques, like dropout, batch norm, data augmentation, model simplification and so on
Add some data if it's possible, that would better cover all the possible cases.

Here is the source code https://github.com/Julius-R-UI/Heart-Failure-Predictions-Model

Have you tried logistic regression on this data as a baseline? 2. Are you sure your class balanced? If not, use roc-auc score, not accuracy.

And if I answered your question, consider accepting the answer

I'm new to all this so take what I say with a grain of salt and not as fact, I don't have any formal education or training.  I believe when you're referring to inversion predictions, you're not overthinking you're underthinking.  For anything to have value it must also have an inverse or else there's no way to cognitively perceive it (contrast) otherwise you're looking at a white paper against white paper.  Now since you're referring to data set prediction, you need to define it linearly (x, y) or f(x) to scale and plot.  Therefore x and y BOTH must retain inverse proportionate values (I made up that term) in order to, in the context of assigning value, exist.  So you need to have 4 quadrants of data for predictive, so now you're looking at quantum data processing in order to facilitate predictions in a non-linear context.  Use a matrix, I believe Diroches Matrix should be applicable here.  Also, remember that predictions are always changing and updating based on empirical and real-time data, so don't get your programming stuck in an ONLY RIGHT NOW mindset, matrices are designed to be constantly moving and evolving.  Therefore your z-axis should always retain a state of variability, or it should always be Z don't attach a value to it.  Good luck.  I'm jealous I would love to ACTUALLY be working on something cool :/


text:
In the MINE paper, why is $\hat{G}_B$ biased, and how does the exponential moving average reduce the bias?
While reading the Mutual Information Neural Estimation (MINE) paper [1] I came across section 3.2 Correcting the bias from the stochastic gradients. The proposed method requires the computation of the gradient
$$\hat{G}_B = \mathbb{E}_B[\nabla_{\theta}T_{\theta}] - \frac{\mathbb{E_B}[\nabla_{\theta}T_{\theta}e^{T_{\theta}}]}{\mathbb{E}_B[e^{T_{\theta}}]},$$
where $\mathbb{E}_B$ denotes the expectation operation w.r.t. a minibatch $B$, and $T_{\theta}$ is a neural network parameterized by $\theta$. The authors claim that this gradient estimation is biased and that can be reduced by simply performing an exponential moving average filtering.
Can someone give me a hint to understand these two points:

Why is $\hat{G}_B$ biased, and
How does the exponential moving average reduce the bias?

The lower bound in MINE is as follows:
$$\widehat{I(X;Z)}_n = \sup_{\theta\in\Theta} \mathbb{E}_{\mathbb{P}_{XZ}^{(n)}}[T_\theta] - \log{\mathbb{E}_{\mathbb{P}_X^{(n)} \otimes \hat{\mathbb{P}}_Z^{(n)}}[e^{T_\theta}]}$$
Here $\mathbb{\hat{P}^{(n)}}$ denotes the empirical distribution that we get from n i.i.d samples of $\mathbb{P}.$
Note that in the above equation, the first term is calculated from the joint distribution while the second term from the marginals of $X$ and $Z$. In the implementation of MINE, these statistics are calculated over the data from a minibatch. The marginal distribution is obtained by shuffling the values of Z (or X) along the batch dimension. Hence, in this case, the gradient is as follows.
$$\hat{G}_B = \mathbb{E}_B[\nabla_{\theta}T_{\theta}] - \frac{\mathbb{E_B}[\nabla_{\theta}T_{\theta}e^{T_{\theta}}]}{\mathbb{E}_B[e^{T_{\theta}}]},$$

As mentioned, the expectation over the marginals is not calculated
over the true marginal distribution (i.e. over the entire dataset)
but from the shuffled samples in the minibatch. Hence, the above gradient $G_B$ is biased.
When we maintain an exponential moving average of
$\mathbb{E}_B[e^{T_{\theta}}]$, we kind of incorporate statistics from outside the current minibatch also (i.e. over the entire dataset). This is an attempt to get an approximation of the true marginal estimate. The denominator term in the gradient allows for such a computationally inexpensive bias reduction trick.


text:
How to forecast multiple target attributes in Python?
I need to forecast two non-correlated time-series (non-stationary). A sample is presented below:
414049364,21773560
414049656,21773926
414049938,21774287
414050204,21774638
414050453,21774975
414050682,21775296
414050895,21775597
414051093,21775874
414051278,21776125
414051453,21776344
414051620,21776530
414051780,21776678
414051935,21776785
414052089,21776849
414052242,21776865

The above is the input (two attributes) and the output (prediction) is composed of two targets (the same as input) for instance,
414052252,21776765

However, current regression techniques only consider a single attribute (class) forecasting but two or more. I've checked the following site https://machinelearningmastery.com/multi-output-regression-models-with-python/ for multi-target regression or predictive clustering trees. Unfortunately, I don't know how to adapt my data to those techniques. Ideally, I would like to predict multiple steps.
Any idea?

if they are independent it is better considered then as two separate time series and forecast accordingly.

well, they are latitude and longitude values, in this case is also recommended to forecast separately or using MTR/PCT? in that last case, how can I adapt my data to those examples?

In general, multi output models is not that different.
I.e.

As Raghu mentioned in commentary, you could train separate model for each output. There is even helper module in sklearn for that (MultiOutputRegressor)
DecisionTreeRegressor from sklearn allows multiple outputs out-of-the box
Any neural network framework allows any number of outputs

In your particular case there are two much bigger factors.

It's timeseries. Timeseries requires whole another set of models then other data, i.e ANOVA, RNN or transformers.
You data is not well scaled. The reason is that the change between points is insignificant in comparison with their absolute values. You need to deal with it -- but exact solution would depend on your particular task. You could try to substract the mean value on train set or predict the difference, not absolute value.

Edit: here is simple example
import numpy as np
raw = np.array([
[414049364,21773560],
[414049656,21773926],
[414049938,21774287],
[414050204,21774638],
[414050453,21774975],
[414050682,21775296],
[414050895,21775597],
[414051093,21775874],
[414051278,21776125],
[414051453,21776344],
[414051620,21776530],
[414051780,21776678],
[414051935,21776785],
[414052089,21776849],
[414052242,21776865]
])

X = raw[:-1]
y = raw[1:]
from sklearn.linear_model import LinearRegression
model = LinearRegression()
# Train the model
model.fit(X, y)
# Try to predict next step
model.predict([[414051780,21776678]])
# The predicted value [414051933,  21776806] and the true value would be [414051935,21776785], not so far

Next you could normalize the values, use few previous steps as a features, use more complex algorithm then linear regression, and make train/eval split for better model evaluation.

My problem is how to set the algorithms presented in the link above. My data is X however, I don't know what to use as y.

You need to deal with time series, it's core challenge here. You could try following:

Predict step n+1 from few previous steps. I.e. you have 5 steps, that's mean you have ten features and you could use any basic algorithm to predict values on step 6 (just apply regression/tree from you link). You could move the 'window' to generate to points, i.e. predict step 7 by 2-6, predict step 8 by 3-7 and so on.
Use dedicated algorithms for the time series. There is a lot of tutorials, courses and even books in the wild on the topic.

Thanks for the advice, I've using ML since 2012, however, this context is new to me. I've done forecasting before. In previous cases, I had a set (y) of classes, here, I don't have classes. I've search and read a lot unsuccesfully. I don't pretend to get the solution here, just how to adapt my problem to current approaches.

If you had classes, then you had classification problem (to predict class). In this case you have regression problem (to predict value). Algorithm from the link you provided should be doing fine. Simple baseline would be just to predict the next step from the current - I added example to the answer.

Great! I got it know. I was confused on how to treat my data. I guess if I want to predict more than one step ahead, I should use the same "training" but feeding multiple times the prediction


text:
Converting age and sex variables to a 64-unit dense layer
I am studying a preprint for my own learning (https://www.medrxiv.org/content/medrxiv/early/2020/04/27/2020.04.23.20067967.full.pdf) and I am befuddled by the following detail of the neural network architecture:

This is in accord with the paper's description of the architecture (p. 5):

Age and sex were input into a 64-unit hidden layer and was concatenated with the other branches.

How can the two scalars of age and sex be implemented as a 64-unit dense layer?

Convert them into numbers (using one-hot vectors or direct numerical representations) and then concatenate them. Then, you can pass them through the Dense layer.

So, I create a dense component with an input vector of size two, which immediately fans out to a fully connected layer of 64 nodes? What would the benefit of that be? I don't see what advantage this holds over just having a node for the sex and a node for the age.

It will model the relationship between sex and age. You can reduce the number of units though.

That makes sense, in classical modeling there's a max of 1 interaction but I would suspect 3 nodes is all a person needs.


text:
Is it possible to make a neural network to solve this "reaction time test"?
I'm thinking about writing an essay on the comparison between the human nervous system (reaction time) and a neural network that does the same reaction time test. I am very new in this area, so I was wondering if I can build a neural network that can perform a test like this: https://humanbenchmark.com/tests/reactiontime
I just wanted to know how I should approach this problem, and what would be the best way to compare it to the human nervous system.
I have thought about maybe using an image classification neural network, and have it looks for different colors and such, but not too sure about its technical aspects as of yet. Any help is appreciated.

Basically, what you want to do is an anomaly detection at the fastest speed possible. You need to sample the image at two timesteps and if there is a difference you click. The smaller the difference between your timesteps the better it is.
But, you would not need a neural network for it. Since, it is a single colour that will change. A color is represented by a 3-tuple (RGB) so you just need to compare them and if they are different then you click. A single if condition will suffice for it.

Thank you. But would this still fit into the general bubble of AI, and if not, in which scenario (regarding reaction time) would I actually be able to implement a neural network.

To fit your problem into the AI bubble, you need to look at change point detection. Detecting any kind of change in data, occurence of a person in CCTV video, that is something you would require neural network for. Your data should be complex enough to warrant the use of a neural network.

How would I be able to test that against a human reaction time, as that is my primary goal. Compare the reaction times of neural networks and humans. Also, could I still implement a neural network but not necessarily needing it?... Thank you

The reaction time for a human comprises of looking at the picture, processing the picture and clicking the button. Most of a human's time will go in clicking. While the neural network will be able to do the click very fast than a human. This is the going to be the biggest challenge.

You can use ESM (Experience Sampling Method). You can create an app which will at different times in the day ask the user to do the reaction time task. You will then also the neural network to do the same at backend and log that data. You can then compare. This is a possible way to do it. But, the challenge I mentioned in above comment remains.

Ok, thank you man. Can I contact you in the future if I have any other questions? Again thx a lot.!!

Yes, you can contact me in future.

Hey man, do you mind having a look at this question for me: https://ai.stackexchange.com/questions/27161/how-to-improve-a-trained-model-over-timeie-with-more-predictions


text:
How should I implement the state transition when it is a Gaussian distribution?
I am reading this paper Anxiety, Avoidance and Sequential Evaluation and is confused about the implementation of a specific lab study. Namely, the authors model what is called the Balloon task using a simple MDP for which the description is below:

My confusion is the following sentence:

...The probability of this bad transition was modeled using normal density function, with parameters $N(16, 0.5)$

But the fact that this is a continuous, normal distribution makes me stumped. In MDP's, usually there is a nice, discrete transition matrix and so there is no ambiguity as to how to implement it. For instance, if they said the transition to a bad state is modeled by a Bernoulli random variable with parameter $p,$ then it is clear how to implement it. I would do something like:
def step(curr_state, curr_action):
   if uniform random variable(0,1) < p:
      next_state = bad state

But they are using a normal random variable for this "bad" transition, so how do I implement this?

I figured this out by going to the author's publicly available github code. It turned out  the authors were just generating the transition probability $p$ from $\mathcal{N}(\mu,\sigma^2)$ at the beginning of each episode for some reason. Answering it myself for the sake of not leaving this question unanswered.

Wait, but $p$ needs to be limited to $[0,1]$, if it's a probability. However, the Gaussian is not limited to $[0, 1]$. Maybe you could also provide a link to the source code.

@nbro I misworded it - the maximum number of episodes is $T = 20$ and the balloon explodes if the current the number of steps have taken so far $t$ exceeds the normal random variable drawn from $\mathcal{N}(16, 0.5),$ for the low-risk version of the task and $\mathcal{N}(8, 0.5)$ for the high-risk version.


text:
Is it practical to train AlphaZero or MuZero (for indie games) on a personal computer?
Is it practical/affordable to train an AlphaZero/MuZero engine using a residential gaming PC, or would it take thousands of years of training for the AI to learn enough to challenge humans?
I'm having trouble wrapping my head around how much computing power '4 hours of Google DeepMind training' equates to my residential computer running 24/7 trying to build a trained AI.
Basically, are AlphaZero or MuZero practical for indie board games that want a state of the art AI, or is it too expensive to train?

The vast majority of neural networks are now trained on graphics processing units (GPUs) or specialised accelerator hardware such as tensor processing units (TPUs).
In Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, Silver et al. say that the training process involved 5,000 first-generation TPUs generating self-play games and 64 second-generation TPUs for training. This is certainly far beyond what any practical gaming computer is likely to achieve, as you'll likely only have one GPU, and that might not even rival a single TPU. Training on the CPU will be substantially slower again than either a GPU or TPU. Training would be orders of magnitude slower; you might find these benchmarks by Wang et al. of interest.

Not arguing with any part of this answer, and I may add that I think gcp (creator of Leela zero) estimated something like 10000 years in single GPU (estimated 3 years ago). However this training time depends on the game complexity a lot and also if the game has no professional players then a single GPU may well be the best in the world after a few months if not a few weeks or even days.

For board games, you may not need an advanced computer to achieve very good results. TPU and high-end GPUs allow to go one step further of course.
But there is an existing implementation on github alpha-zero-general (not from me). By adding some optimizations tricks from scientific articles, I managed to train alphazero for several board games (Splendor, Santorini, Machi Koro, The Little Prince Make Me A Planet) with my i5 CPU. You may not need GPU as most of the time is spent in self-playing more than training, meaning that computer needs to do lots of network inferences more than gradient. After about a day of training, the training seemed to reach its asymptote. As a result, the computer was winning over me 100% on each game.
I don't want to promote my self github, just telling OP that is feasible.


text:
How is this Pytorch expression equivalent to the KL divergence?
I found the following PyTorch code (from this link)
-0.5 * torch.sum(1 + sigma - mu.pow(2) - sigma.exp())

where mu is the mean parameter that comes out of the model and sigma is the sigma parameter out of the encoder. This expression is apparently equivalent to the KL divergence. But I don't see how this calculates the KL divergence for the latent.

The code is correct. Since OP asked for a proof, one follows.
The usage in the code is straightforward if you observe that the authors are using the symbols unconventionally: sigma is the natural logarithm of the variance, where usually a normal distribution is characterized in terms of a mean $\mu$ and variance. Some of the functions in OP's link even have arguments named log_var.$^*$
If you're not sure how to derive the standard expression for KL Divergence in this case, you can start from the definition of KL divergence and crank through the arithmetic. In this case, $p$ is the normal distribution given by the encoder and $q$ is the standard normal distribution.
$$\begin{align}
D_\text{KL}(P \| Q) &= \int_{-\infty}^{\infty} p(x) \log\left(\frac{p(x)}{q(x)}\right) dx \\
&= \int_{-\infty}^{\infty} p(x) \log(p(x)) dx - \int_{-\infty}^{\infty} p(x) \log(q(x)) dx
\end{align}$$
The first integral is recognizable as almost definition of entropy of a Gaussian (up to a change of sign).
$$
\int_{-\infty}^{\infty} p(x) \log(p(x)) dx = -\frac{1}{2}\left(1 + \log(2\pi\sigma_1^2) \right)
$$
The second one is more involved.
$$
\begin{align}
-\int_{-\infty}^{\infty} p(x) \log(q(x)) dx
&= \frac{1}{2}\log(2\pi\sigma_2^2) - \int p(x) \left(-\frac{\left(x - \mu_2\right)^2}{2 \sigma_2^2}\right)dx \\
&= \frac{1}{2}\log(2\pi\sigma_2^2) + \frac{\mathbb{E}_{x\sim p}[x^2] - 2 \mathbb{E}_{x\sim p}[x]\mu_2 +\mu_2^2} {2\sigma_2^2} \\
&= \frac{1}{2}\log(2\pi\sigma_2^2) + \frac{\sigma_1^2 + \mu_1^2-2\mu_1\mu_2+\mu_2^2}{2\sigma_2^2} \\
&= \frac{1}{2}\log(2\pi\sigma_2^2) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2}
\end{align}
$$
The key is recognizing this gives us a sum of several integrals, and each can apply the law of the unconscious statistician. Then we use the fact that $\text{Var}(x)=\mathbb{E}[x^2]-\mathbb{E}[x]^2$. The rest is just rearranging.
Putting it all together:
$$
\begin{align}
D_\text{KL}(P \| Q) &= -\frac{1}{2}\left(1 + \log(2\pi\sigma_1^2) \right) + \frac{1}{2}\log(2\pi\sigma_2^2) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} \\
&= \log (\sigma_2) - \log(\sigma_1) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} - \frac{1}{2}
\end{align}
$$
In this special case, we know that $q$ is a standard normal, so
$$
\begin{align}
D_\text{KL}(P \| Q) &= -\log \sigma_1 + \frac{1}{2}\left(\sigma_1^2 +
\mu_1^2 - 1 \right) \\
&= - \frac{1}{2}\left(1 + 2\log \sigma_1- \mu_1^2 -\sigma_1^2   \right)
\end{align}
$$
In the case that we have a $k$-variate normal with diagonal covariance for $p$, and a multivariate normal with covariance $I$, this is the sum of $k$ univariate normal distributions because in this case the distributions are independent.
The code is a correct implementation of this expression because $\log(\sigma_1^2) = 2 \log(\sigma_1)$ and in the code, sigma is the logarithm of the variance.

$^*$The reason that it's convenient to work on the scale of the log-variance is that the log-variance can be any real number, but the variance is constrained to be non-negative by definition. It's easier to perform optimization on the unconstrained scale than it is to work on the constrained scale in $\eta^2$. Also, we want to avoid "round-tripping," where we compute $\exp(y)$ in one step and then $\log(\exp(y))$ in a later step, because this incurs a loss of precision. In any case, autograd takes care of all of the messy details with adjustments to gradients resulting from moving from one scale to another.

The part I'm a little confused by is that how do you know the mean of p(x)? Especially in neural network case? Cause it's intractable, or does reducing it down to a multivariate Gauss make it tractable? But if so is the mean and SD based on what the data produces?

The latent representation of a VAE emits a mean and (a transformation of) the variance for each input: the $i$th sample has an associated distribution with parameters $(\mu_i, \sigma_i^2)$. In other words, there's a distribution $p_i(x)$ for each of the $i$ inputs.

This is the analytical form of the KL divergence between two multivariate Gaussian densities with diagonal covariance matrices (i.e. we assume independence). More precisely, it's the KL divergence between the variational distribution
$$
q_{\boldsymbol{\phi}}(\mathbf{z})
=
\mathcal{N}\left(\mathbf{z} ; \boldsymbol{\mu}, \mathbf{\Sigma} = \boldsymbol{\sigma}^{2}\mathbf{I}\right)
=
\frac{\exp \left(-\frac{1}{2}\left(\mathbf{z} - \boldsymbol{\mu}\right)^{\mathrm{T}} \mathbf{\Sigma}^{-1}\left(\mathbf{z}-\boldsymbol{\mu} \right)\right)}{\sqrt{(2 \pi)^{J}\left|\mathbf{\Sigma}\right|}}
\tag{1}\label{1}
$$
and the prior (it's the same as above, but with mean and covariance equal to the zero vector and the identity matrix, respectively)
$$
p(\mathbf{z})=\mathcal{N}(\mathbf{z} ;\boldsymbol{0}, \mathbf{I})
=
\frac{\exp \left(-\frac{1}{2}\mathbf{z}^{\mathrm{T}}\mathbf{z}\right)}{\sqrt{(2 \pi)^{J}}}
\tag{2}\label{2}
$$
where

$\boldsymbol{\mu} \in \mathbb{R}^J$ is the mean vector (we assume column vectors, so $\boldsymbol{\mu}^T$ would be a row vector)
$\mathbf{\Sigma} = \boldsymbol{\sigma}^{2}\mathbf{I} \in \mathbb{R}^{J \times J}$ is a diagonal covariance matrix (with the vector $\boldsymbol{\sigma}^{2}$ on the diagonal of the identity)
$\mathbf{z} \in \mathbb{R}^J$ is a sample (latent vector) from these Gaussians with dimensionality $J$ (or, at the same time, the input variable of the density)
$\left|\mathbf{\Sigma}\right| = \operatorname{det} \mathbf{\Sigma}$ is the determinant (so a number) of the diagonal covariance matrix, which is just the product of the diagonal elements for a diagonal matrix (which is the case); so, in the case of the identity, the determinant is $1$
$\boldsymbol{0} \in \mathbb{R}^J$ is the zero vector
$\mathbf{I} \in \mathbb{R}^{J \times J}$ is an identity matrix
$\mathbf{z}^{\mathrm{T}}\mathbf{z} = \sum_{i=1}^J z_i^2 \in \mathbb{R}$ is the dot product (hence a number)

Now, the (negative of the) KL divergence is defined as follows
\begin{align}
-D_{K L}\left(q_{\boldsymbol{\phi}}(\mathbf{z}) \| p(\mathbf{z})\right)
&=
\int q_{\boldsymbol{\phi}}(\mathbf{z})\left(\log p(\mathbf{z})-\log q_{\boldsymbol{\phi}}(\mathbf{z})\right) d \mathbf{z} \\
&=
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})} \left[ \log p(\mathbf{z})-\log q_{\boldsymbol{\phi}}(\mathbf{z})\right] \label{3}\tag{3}
\end{align}
Given that we have logarithms here, let's compute the logarithm of equations \ref{1} and \ref{2}
\begin{align}
\log
\left(
\mathcal{N}\left(\mathbf{z} ; \boldsymbol{\mu}, \mathbf{\Sigma} \right)
\right)
&=
\dots \\
&=
-\frac{1}{2}(\mathbf{z}-\boldsymbol{\mu})^{\mathrm{T}} \mathbf{\Sigma}^{-1}(\mathbf{z}-\boldsymbol{\mu})-\frac{J}{2} \log (2 \pi)-\frac{1}{2} \log |\mathbf{\Sigma} |
\end{align}
and
\begin{align}
\log
\left(
\mathcal{N}(\mathbf{z} ;\boldsymbol{0}, \mathbf{I})
\right)
&=
\dots \\
&=
-\frac{1}{2}\mathbf{z}^{\mathrm{T}} \mathbf{z}-\frac{J}{2} \log (2 \pi)
\end{align}
We can now replace these in equation \ref{3} (below, I have already performed some simplifications, to remove verbosity, but you can check them!)
\begin{align}
\frac{1}{2}
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})} \left[
-\mathbf{z}^{\mathrm{T}} \mathbf{z}
+ (\mathbf{z}-\boldsymbol{\mu})^{\mathrm{T}} \mathbf{\Sigma}^{-1}(\mathbf{z}-\boldsymbol{\mu})
+  \log |\mathbf{\Sigma} |
\right]
\tag{4}\label{4}
\end{align}
Now, given that $\mathbf{\Sigma}$ is diagonal and the log of a product is just a sum of the logarithms, we have $\log |\mathbf{\Sigma} | = \sum_{i=1}^J \log \sigma_{ii}$, so we can continue
\begin{align}
\frac{1}{2}
\left(
-
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})}
\left[
\mathbf{z}^{\mathrm{T}} \mathbf{z}
\right]
+
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})} \left[
(\mathbf{z}-\boldsymbol{\mu})^{\mathrm{T}} \mathbf{\Sigma}^{-1}(\mathbf{z}-\boldsymbol{\mu})
\right]
+
\sum_{i=1}^J \log \sigma_{ii}
\right)
&= \\
\frac{1}{2}
\left(
-
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})}
\left[
\mathbf{z}^{\mathrm{T}} \mathbf{z}
\right]
+
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})} \left[
\operatorname{tr} \left(
\mathbf{\Sigma}^{-1}(\mathbf{z}-\boldsymbol{\mu})
(\mathbf{z}-\boldsymbol{\mu})^{\mathrm{T}}
\right)
\right]
+
\sum_{i=1}^J \log \sigma_{ii}
\right)
&= \\
\frac{1}{2}
\left(
-
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})}
\left[
\mathbf{z}^{\mathrm{T}} \mathbf{z}
\right]
+
\operatorname{tr} \left(
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})} \left[
\mathbf{\Sigma}^{-1}(\mathbf{z}-\boldsymbol{\mu})
(\mathbf{z}-\boldsymbol{\mu})^{\mathrm{T}}
\right]
\right)
+
\sum_{i=1}^J \log \sigma_{ii}
\right)
&= \\
\frac{1}{2}
\left(
-
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})}
\left[
\mathbf{z}^{\mathrm{T}} \mathbf{z}
\right]
+
\operatorname{tr} \left(
\mathbf{\Sigma}^{-1}
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})} \left[
(\mathbf{z}-\boldsymbol{\mu})
(\mathbf{z}-\boldsymbol{\mu})^{\mathrm{T}}
\right]
\right)
+
\sum_{i=1}^J \log \sigma_{ii}
\right)
&= \\
\frac{1}{2}
\left(
-
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})}
\left[
\mathbf{z}^{\mathrm{T}} \mathbf{z}
\right]
+
\operatorname{tr} \left(
\mathbf{\Sigma}^{-1}
\mathbf{\Sigma}
\right)
+
\sum_{i=1}^J \log \sigma_{ii}
\right)
&= \\
\frac{1}{2}
\left(
-
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})}
\left[
\mathbf{z}^{\mathrm{T}} \mathbf{z}
\right]
+
J
+
\sum_{i=1}^J \log \sigma_{ii}
\right)
&= \\
\frac{1}{2}
\left(
-
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})}
\left[
\operatorname{tr}
 \left(
\mathbf{z} \mathbf{z}^{\mathrm{T}}
\right)
\right]
+
\sum_{i=1}^J 1
+
\sum_{i=1}^J \log \sigma_{ii}
\right)
&= \\
\frac{1}{2}
\left(
-
\operatorname{tr}
\left(
\mathbf{\Sigma}\right)
-
\operatorname{tr}
\left(
\boldsymbol{\mu} \boldsymbol{\mu}^T
\right)
+
\sum_{i=1}^J 1
+
\sum_{i=1}^J \log \sigma_{ii}
\right)
&= \\
\frac{1}{2}
\left(
-
\sum_{i=1}^J \sigma_{ii}
-
\sum_{i=1}^J \mu_{i}^2
+
\sum_{i=1}^J 1
+
\sum_{i=1}^J \log \sigma_{ii}
\right)
&= \\
\frac{1}{2}
\sum_{i=1}^J
\left(
1
+
\log \sigma_{ii}
-
\sigma_{ii}
-
\mu_{i}^2
\right)
\end{align}
In the above simplifications, I also applied the following rules.

the linearity of the expectations,
$\operatorname{tr}(ba^T) = a^Tb$ (where $\operatorname{tr}$ is the trace of a matrix)
you can swap the trace and the expectation
$\mathbf{\Sigma} =
\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})} \left[
(\mathbf{z}-\boldsymbol{\mu})
(\mathbf{z}-\boldsymbol{\mu})^{\mathrm{T}}
\right] $ (this is just the definition of the covariance matrix)
$\operatorname{tr}\left( \mathbf{\Sigma}^{-1}
\mathbf{\Sigma} \right) = \operatorname{tr}\left(\mathbf{I} \right) = J = \sum_{i=1}^J 1$
$ \mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z})}
\left[
\mathbf{z} \mathbf{z}^{\mathrm{T}}
\right] = \mathbf{\Sigma} + \boldsymbol{\mu} \boldsymbol{\mu}^T$ (definition of the correlation matrix)

The official PyTorch implementation of the VAE, which can be found here, also uses this formula. This formula can also be found in Appendix B of the VAE paper, but the long proof that I've just written above is not given. Note that, in my proof above, $\sigma$ is the variance and is denoted by $\sigma^2$ in the paper (as it is usually the case to denote the variance as the square of the standard deviation $\sigma$, but, again, in my proof above $\sigma$ is the variance).


text:
How can we find the value function by solving a system of linear equations without knowing the policy?
An MDP is a Markov Reward Process with decisions, it’s an environment in which all states are Markov. This is what we want to solve. An MDP is a tuple $(S, A, P, R, \gamma)$, where $S$ is our state space, $A$ is a finite set of actions, $P$ is the state transition probability function,
$$P_{ss'}^a = \mathbb{P}[S_{t+1} = s' | S_t = s, \hspace{0.1cm}A_t = a] \label{1}\tag{1}$$
and
$$R_s^a = \mathbb{E}[R_{t+1}| S_t =s, A_t = a]$$
and a discount factor $\gamma$.
This can be seen as a linear equation in $|S|$ unknowns, which is given by,
$$V = R + \gamma PV  \hspace{1mm}  \label{2}\tag{2}$$
$V$ is value of a state vector, $R$ is immediate reward vector, $P$ is transition probability matrix, where each element at $(i,j)$ in $P$ is given by,  $ P[i][j] = P(i \mid j)$ i.e., probability that I am in state $j$ going to state $i$.
As $P$ is given, we treat, equation $\ref{2}$ as a linear equation in $V$.
But $P[i][j] = \sum_a (\pi(a \mid j) \times \mathrm{p}(i \mid j, a) )$. But, $ \pi (a \mid s)$ (i.e., probability that I will take action a in state s) is NOT given.
So, how can we frame this problem as the solution to a system of linear equations in \ref{2}, if we only know $ P^a_{ss'}$ and we do not know $ \pi(a \mid s)$, which is needed to calculate $P[i][j]$?

Your equations all look correct to me.
It is not possible to solve the linear equation for state values in the vector $V$ without knowing the policy.
There are ways of working with MDPs, through sampling of actions, state transitions and rewards, where it is possible to estimate value functions without knowing either $\pi(a|s)$ or $P^{a}_{ss'}$. For instance, Monte Carlo policy evaluation or single-step TD learning can both do this. It is also common to work with $\pi(a|s)$ known but $P^{a}_{ss'}$ and $R^{a}_{s}$ unknown in model-free control algorithms such as Q learning.
However, in your case, you are correct, in order to resolve the simultaneous equations you have presented, you do need to know $\pi(a|s)$
This is not as limiting as you might think. You can construct a control method using simultaneous equations, by starting with the policy set to some arbitrary policy. Either a randomly-chosen deterministic policy or the equiprobable policy are reasonable first guesses. Then, after each solution to linear equations, you improve the policy so that each action choice maximises the expected return. This is essentially the policy iteration algorithm but replacing the policy evaluation step with the linear equations method for calculating the values.

Thanks. I understood. Also one more question I have that is, are we assuming the fact that if there exist a stochastic policy then there exist an optimal deterministic policy?

@Abc1729 For a well-defined MDP with the Markov property, there will always exist one or more optimal deterministic policies. You do not require a stochastic policy for it to be optimal (but you might want one if your optimisation porcess involves exploring by sampling actions to refine estimates). There are other questions about that on this site already, but if you cannot find them or the answers on them do not help, then feel free to ask a new one


text:
NLP: Are hashtags tokenised?
I am exploring a potential NLP project. I was wondering what generally is done with the hashtags words (e.g. #hello). Are those words ignored? is the # removed and the word tokenised? Is it tokenised with the #?

Word including Hash tag need to be tokanised. It has a special meaning and given a context. The word by itself has a different meaning.

So have they been encoded during training by models such as GPT or BERT?


text:
How do we know that the neurons of an artificial neural network start by learning small features?
I'd like to ask you how do we know that neural networks start by learning small, basic features or "parts" of the data and then use them to build up more complex features as we go through the layers. I've heard this a lot and seen it on videos like this one of 3Blue1Brown on neural networks for digit recognition. It says that in the first layer the neurons learn and detect small edges and then the neurons of the second layer get to know more complex patterns like circles... But I can't figure out based on pure maths how it's possible.

We do it experimentally; you're able to look at what each layer is learning by tweaking various values throughout the network and doing gradient ascent. For more detail, watch this lecture: https://www.youtube.com/watch?v=6wcs6szJWMY&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&index=12 it provides many methods used for understanding exactly what your model is doing at a certain layer, and what features it has learnt.

Thank you for the lecture it's so interesting, there a couple of things I didn't get (like in the nearest neighbors part how do we map the images from the training set to the feature space which is comprised of 4096 dimensional vectors) but I got the general idea of each technique. Thank you again it's like a jewel !

The network architecture is relevant to this question.
Convolutional neural network architectures enforce the building up of features because the neurons in earlier layers have access to a small number of input pixels. Neurons in deeper layers are connected (indirectly) to more and more pixels, so it makes sense that they identify larger and larger features. Lots of the visual examples available online which show, for example, a curve, to a circle, to a part of an animal, to a whole animal, are based on convolutional networks. The beautiful examples from the Harvard lecture in the other answer use convolutional networks.
With that being said, increasing complexity with each layer is true generally, including for dense architectures like the 3Blue1Brown one. It's just that this is a more abstract 'increase in nonlinearity' rather than spatial feature size. Depending on the task the network is learning, earlier layers will be more 'basic', but their neurons might use large areas of the input.

Oh I see thank you. So may we say there is a trade-off between the size of the input's areas that the first layers' neurons are using and complexity ? Like Convolutional Neural Networks work better than Dense Neural Networks with images because the first neurons can learn complex detail by looking at small areas of the input while for a dense fully connected network than the neurons look at everything. And are there other interpretations for the other types of architectures or we just say that they find and learn small patterns in the data regardless of what it is by analogy to CNNs ?

Hi @Daviiid. I would not say that there is a trade off as you describe. There are some theorems that deeper layers can learn more complex features, for example the first layer cannot learn XOR but deeper layers can. However, convolutional layers do not learn more complex features than dense layers in that sense. I would say instead that convolutional layers work better because they follow the symmetry of the task. We effective apply some assumed knowledge we have about the task through them. This allows training to be faster and more effective. But not really achieve more complexity.

I see, thank you for your answer. I understand better. I guess it's like with Recurrent Neural Networks. We have this knowledge that time series data is dependent on past data. It's like if we're working with networks that preserve this characteristic structure of the data.


text:
During neural network training, can gradients leak sensitive information in case training data fed is encrypted (homomorphic)?
Some algorithms in the literature allow recovering the input data used to train a neural network. This is done using the gradients (updates) of weights, such as in Deep Leakage from Gradients (2019) by Ligeng Zhu et al.
In case the neural network is trained using encrypted (homomorphic) input data, what could be the output of the above algorithm? Will the algorithm recover the data in clear or encrypted (as it was fed encrypted)?

Just to clarify, although I am not familiar with that paper/work, but are you assuming that the neural network would be trained with an encrypted version of the data?

Yes, exactly the neural network is trained with encrypted data using homomorphic encryption ... but the neural network ie. weights and parameters are not encrypted, only the training data is encrypted.

It will recover the encrypted inputs.
The algorithm starts with dummy data and dummy labels, and then iteratively optimizes the dummy gradients to be close as to the original. This makes the dummy data close to the real training data:
$$\mathbf{x}^{\prime *}, \mathbf{y}^{\prime *}=\underset{\mathbf{x}^{\prime}, \mathbf{y}^{\prime}}{\arg \min }\left\|\nabla W^{\prime}-\nabla W\right\|^{2}=\underset{\mathbf{x}^{\prime}, \mathbf{y}^{\prime}}{\arg \min }\left\|\frac{\partial \ell\left(F\left(\mathbf{x}^{\prime}, W\right), \mathbf{y}^{\prime}\right)}{\partial W}-\nabla W\right\|^{2}$$
As the distance is minimized, the algorithm restores the original training data; in the case of encrypted training data - you should get an encrypted input (up to failures when the resulted 'input' isn't close to the original input).


text:
How do I calculate the probabilities of the BERT model prediction logits?
I might be getting this completely wrong, but please let me first try to explain what I need, and then what's wrong.
I have a classification task. The training data has 50 different labels. The customer wants to differentiate the low probability predictions, meaning that, I have to classify some test data as "Unclassified / Other" depending on the probability (certainty?) of the model.
When I test my code, the prediction result is a numpy array. One example is:
[[-1.7862008  -0.7037363   0.09885322  1.5318055   2.1137428  -0.2216074
   0.18905772 -0.32575375  1.0748093  -0.06001111  0.01083148  0.47495762
   0.27160102  0.13852511 -0.68440574  0.6773654  -2.2712054  -0.2864312
  -0.8428862  -2.1132915  -1.0157436  -1.0340284  -0.35126117 -1.0333195
   9.149789   -0.21288703  0.11455813 -0.32903734  0.10503325 -0.3004114
  -1.3854568  -0.01692022 -0.4388664  -0.42163098 -0.09182278 -0.28269592
  -0.33082992 -1.147654   -0.6703184   0.33038092 -0.50087476  1.1643585
   0.96983343  1.3400391   1.0692116  -0.7623776  -0.6083422  -0.91371405
   0.10002492]]

I'm then using numpy.argmax() to identify the correct label.
My question is, is it possible to define a threshold (say, 0.6), and then compare the probability of the argmax() element so that I can classify the prediction as "other" if the probability is less than the threshold value?

Edit 1:
We are using 2 different models. One is Keras, and the other is BertTransformer. We have no problem in Keras since it gives the probabilities so I'm skipping Keras model.
The Bert model is pretrained. Here is how it is generated:
def model(self, data):
        number_of_categories = len(data['encoded_categories'].unique())
        model = BertForSequenceClassification.from_pretrained(
            "dbmdz/bert-base-turkish-128k-uncased",
            num_labels=number_of_categories,
            output_attentions=False,
            output_hidden_states=False,
        )

        # model.cuda()

        return model

The output given above is the result of model.predict() method. We compare both models, Bert is slightly ahead, therefore we know that the prediction works just fine. However, we are not sure what those numbers signify or represent.
Here is the Bert documentation.

Your call to model.predict() is returning the logits for softmax. This is useful for training purposes.
To get probabilties, you need to apply softmax on the logits.
import torch.nn.functional as F
logits = model.predict()
probabilities = F.softmax(logits, dim=-1)

Now you can apply your threshold same as for the Keras model.

Can you elaborate on how logits are used for training? All of the models I've used apply loss to the output of the softmax layer, I've never seen anything that uses the raw logits for calculating loss

@ATyshka: Both Tensorflow and Pytorch provide this feature as standard, and I expect most other NN libraries do as well. As the original question was about Pytorch, here's a link for the Pytorch loss function for binary cross entropy using the raw logits: https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html  - the OP already was using this or similar, so I don't see any need to put that in the answer. Hope it helps though - if you are running loss on the ouput layer, it is less efficient (and possibly unstable), so switch if you can


text:
Why different images of the same person, under some restrictions, are in a 50 dimension manifold?
In this lecture (starting from 1:31:00) the professor says that the set of all images of a person lives in a low dimensional surface (compared the the set of all possible images). And he says that the dimension of that surface is 50 and that they get this number by adding the three translations of the body, the three rotations of the head and the independent movements of the face's muscles. He also adds that it may be more than 50 but less than 100. How do we get the number 50 ?
The professor previously said (in the same lecture, 1:29:00) that the set of all the images that we could describe as natural and that we could interpret are in a manifold. I try to understand how the number 50 came up like the following: let's take an image of a person, since it's "natural" then it belongs to that manifold. Hence there is an open set to which this image belongs to and there is a homeomorphic map from this open set to an euclidean space. Let's suppose (I don't know why but it's the only possible thing I could come up with to understand) that all the images of that same person, regardless of his position and expressions..., are in that open space then through the homeomorphic mapping we have the "same points" in an euclidean space, do we get the base of it by decomposing all the possible movements of the person?
I hope someone can clarify things for me, it seems this doesn't only work with images but all types of non-structured types of data.

The number 50 is essentially just a guess based on results when compressing and/or generating data of a certain type. The variables such as "the three translations of the body, the three rotations of the head and the independent movements of the face's muscles" are examples only. There is no known formal map with well-defined parameters that defines a well understood manifold of "clear images of this person" in natural images. The lecturer has not constructed such a map as far as I can tell, but has done some related experiments.
Experimentally, it is possible to establish parameter vectors that work, with models like Variational Autoencoders and Generative Adversarial Networks. Depending on the size of the target image, and amount of variation in subject matter that you want to allow for (pose, lighting, clothing, hair style, makeup, camera properties etc), you will end up with different sizes of embedding vectors that appear to capture the important variations. When dealing with multiple people, it is common to see vector sizes of 64, 128, 256.
The lecture suggests compressing images with a clear background, consistent lighting, same person with only changes being in pose. Around 50 dimensions for this relatively simple image space seems reasonable, given facial recognition engines that work well in a more complex domain using 128 dimensional embeddings.
I expect that the lecturer has seen experimental evidence that a vector of 50 dimensions performs well at representing all variations in these images, plus smaller vectors perform measurably worse and larger vectors do not perform better. This experiment is possible by constructing something like a VAE with a specific size of embedding vector, training it, then measuring loss when reconstructing a set of test images.

Thank you for your answer. Can I ask you what are embedding vectors please ? I hear this term a lot but I don't know if I understand it well enough. Are they just the vectors which describe the mapping from the input space to the output space ? Thank you again.

@Daviiid Yes, it is just another term for a point on the manifold.


text:
Face recognition from single image provided
I am working on a computer vision project, based on face detection to record the time spent by a person in an office.
It consists of detecting the face by camera number 1 (input), temporarily storing the detected face, calculating the time spent until this same person leaves and his face is detected by camera number 2. (We don't have a customer database).
Is there a better approach to follow? I would also appreciate articles to read on the topic.

Could you please put your main specific question in the title? "Face recognition..." does not end with ?, so it's not a question. Moreover, the title doesn't seem to describe your actual problem. You actually have 2 cameras, so not just "from a single image".

Matching 2 image of the same person can be done by help of "Siamese Neural network". Here they compare feature of 2 images and if 2 features distance are very close then it's a match. Good thing about this is you do not need person face to match in database. You can use pre-trained network like deepface and use it to compare. However, I guess you will have more trouble connecting real time camera input. As you have to store camera 1 input so that it can be used later for comparison with camera 2 images.


text:
Does stochasticity of an environment necessarily mean non-stationarity in MDPs?
Is a stochastic environment necessarily also non-stationary? To elaborate, consider a two-state environment ($s_1$ and $s_2$), with two actions $a_1$ and $a_2$. In $s_1$, taking action $a_1$ has a certain probability $p_1$ of transitioning you into $s_2$, and a probability $1-p_1$ of keeping you in $s_1$. There is also a similar probability for taking $a_2$ in $s_1$, and taking either action in $s_2$. Let's also say that there is a reward $r$ given only when a transition occurs from either state, and 0 otherwise. This is a stochastic environment. But isn't this non-stationary in one sense and stationary in another? I think it is stationary because the expected return from taking a particular action in a particular state converges to a constant value. But it is non-stationary in the sense that the reward obtained from taking a certain action in a given state may change at a given time. Which is really the case?

I don’t have time to write a full answer atm but I believe that if the $p_i$’s remain fixed then the problem is stationary.

Is a stochastic environment necessarily also non-stationary?

No.
A stochastic environment (i.e. an MDP with a transition model $p(s', r \mid s, a)$) can be stationary (i.e. $p$ does not change over time) or non-stationary ($p$ changes over time). Similarly, a deterministic environment, i.e. the probabilities are $1$ or $0$, can also be either stationary or not. To emphasize that an MDP may be non-stationary, you could write $p$ as a function of time, i.e. $p_t$ (you also do the same thing for the reward function if it's separate from the transition function).
The same idea applies to a stochastic/deterministic policy, which can either be stationary or not.
A non-stationary environment may lead to a non-stationary policy (or may require you to relearn a model of the environment, if you need to learn a model of the environment) [1]. However, note that a stochastic environment (i.e. an MDP) does not necessarily imply a stochastic policy (actually, under some conditions, stationary and stochastic MDPs are known to have a deterministic optimal policy [1]).
In general, if something (e.g. environment, policy, value function or reward function) is non-stationary, it means that it changes over time. This can either be a function or a probability distribution. So, a probability distribution (the stochastic part of an MDP) can change or not over time. If it changes over time, then it makes the MDP non-stationary.

But it is non-stationary in the sense that the reward obtained from taking a certain action in a given state may change at a given time

Informally, you could say that the empirical reward obtained is non-stationary because it changes over time, due to the stochasticity of the reward function, behaviour policy, etc., but the dynamics (transition function and reward function) would still be fixed, so the environment would still be stationary. So, there's a difference between the environment and the experience that you collected so far (with some behaviour policy).

Thank you very much @nbro. That is a well-explained point. In specific, your comment "Informally, you could say that the empirical reward obtained is non-stationary because it changes over time, ..." is very helpful. I believe in short, your point is for me to first identify the nature of the environment (stochastic or deterministic). Then based on the definition of the type of environment, I check to see if there are changes to the underlying feature (probability distribution for stochastic environment, and a single value for the deterministic case).


text:
Is there a document with a list of conjectures or research problems regarding reinforcement learning (like the Millennium Prize Problems)?
Is there a document with a list of conjectures or research problems regarding reinforcement learning like the Millennium Prize Problems?

It's not really an exhaustive list, but Hutter maintains a small list of problems (click on the bullet point "Universal AI Book" here) related to AIXI (a reinforcement learning agent), some of which have already been solved. The money awards are in the range of 50-500 euros, so they are not as financially important as the Millennium Prize Problems.