text:
When do the ensemble methods beat neural networks?
In many applications and domains, computer vision, natural language processing, image segmentation, and many other tasks, neural networks (with a certain architecture) are considered to be by far the most powerful machine learning models.
Nevertheless, algorithms, based on different approaches, such as ensemble models, like random forests and gradient boosting, are not completely abandoned, and actively developed and maintained by some people.
Do I correctly understand that the neural networks, despite being very flexible and universal approximators, for a certain kind of tasks, regardless of the choice of the architecture, are not the optimal models?
For the tasks in computer vision, the core feature, which makes CNNs superior, is the translational invariance and the encoded ability to capture the proximity properties of an image or some sequential data. And the more recent transformer models have the ability to choose which of the neighboring data properties is more important for its output.
But let's say I have a dataset, without a certain structure and patterns, some number of numerical columns, a lot of categorical columns, and in the feature space (for classification task) the classes are separated by some nonlinear hypersurface, would the ensemble models be the optimal choice in terms of performance and computational time?
In this case, I do not see a way to exploit CNNs or attention-based neural networks. The only thing that comes to my head, in this case, is the ordinary MLP. It seems that, on the one hand, it would take significantly more time to train the weights than the trees from the ensemble. On the other hand, both kinds of models work without putting prior knowledge to data and assumptions on its structure. So, given enough amount of time, it should give a comparable quality.
Or can there be some reasoning that neural network is sometimes bound to give rather a poor quality?

Speed:
A classic random forest is O(n) to train and O(1) to run while a feedforward neural network is something like O(n^5) to train and O(n^4) to run, so for many tasks the CART ensemble can train fast and run fast.
Robustness (kinda):
A random forest tends to be natively robust, while GBM and neural networks tend to not be as robust.  There are tweaks to loss functions, and to training sampling, that can make the network less not-robust, but that isn't the same as being robust.  Dropout doesn't bootstrap the domain or target, only the structure.
Basis:
The CART presumes hard edges.  The neural network presumes "continuous".  They are decent at handling hard-edged surfaces, as long as you use them well.
https://stats.stackexchange.com/questions/164048/can-a-random-forest-be-used-for-feature-selection-in-multiple-linear-regression/164250#164250
References:
https://lunalux.io/computational-complexity-of-neural-networks/

This is a great question. Unfortunately the answer is that this is still not very well understood and is an active area of research.
I think doing justice to this problem is beyond the scope of an answer here. Instead I will refer you to some recent research papers that attempt to answer this question.
https://arxiv.org/abs/2106.03253
https://arxiv.org/abs/2207.08815


text:
What is the difference between text-based image retrieval and natural language object retrieval?
I'm working on creating a model that locates the object in the scene (2D image or 3D scene) using a natural language query. I came across this paper on natural language object retrieval, which mentions that this task is different from text-based image retrieval, in the sense that natural language object retrieval requires an understanding of objects in the image, spatial configurations, etc. I am not able to see the difference between these two approaches. Could you please explain it with an example?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How are the lower and upper bound values of the moths determined in the Moth-Flame Optimization algorithm?
I am currently implementing the Moth-Flame Optimization (MFO) Algorithm, based on the paper: Moth-Flame Optimization Algorithm: A Novel Nature-inspired Heuristic Paradigm.
To calculate the values of the Moths, it uses two arrays of values, which contain upper and lower values for each variable. However, as far as I can see. it mentions nothing about what these values are. Quoting from the paper:

As can be seen, there are two other arrays called $ub$ and $lb$. These matrixes define the upper and lower bounds of the variables as follows:
$ub = [ub_1, ub_2, ub_3, \dots,ub_{n-1}, ub_n]$
where $ub_i$ indicates the upper bound of the $i$-th variable.
$lb = [lb_1, lb_2, lb_3, \dots, ub_{n-1}, ub_n]$
where $lb_i$ indicates the lower bound of the $i$-th variable

After that, it says nothing more about this matter
So, if anyone has any idea of how these bound values are determined, please tell me!

I didn't read the paper and I don't have time to read it to help you more now, but if you don't get an answer, you should probably look at an available implementation of MFO online, for example, this or this. I have no idea of the quality of the implementations, so I can't guarantee that they are even correct or contain the answer to your question.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What happens when an opponent a neural network is playing with does not obey the rules of the game (i.e. cheats)?
For example, if AlphaZero plays with an opponent who has a right to move chess figures any way she wants, or make more than 1 move in a turn? Will a neural network adapt to that, as it adapted to an absurd move made by Lee Sedol in 2015?

Are you referring to the game that Lee Sedol won?

yes, round 4. afaik he also tried similar tactic in round 5, but it never worked again

The behaviour when playing against "cheats" depends on how the agent has been trained, and how different the game becomes from the training scenarios. It will also depend on how much of the agent's behaviour is driven by training, and how much by just-in-time planning.
In general, unless game playing bots are written specifically to detect or cope with opponents that are given unfair advantages, they will continue to play in the same style as if the cheating had not occurred, and assuming that the rules are still being followed strictly. If the cheating player only makes one or two rules-breaking moves, and the resulting game state is still something feasible within the game, then the agent should continue to play well. If the agent significantly outclasses the human opponent, it may still win.
A completed, trained agent will not adapt its style to "now my opponent can cheat". An agent still being trained could do so in theory, but it would take many games with cheating allowed for it to learn tactics that cope with an opponent that had an unfair advantage.
Agents that plan by looking ahead during play can cope with more unusual/unseen game state - things that may not have been seen in training. However, they still look ahead on the assumption that game play is as desiged/trained for, they cannot adapt to new rules unless those rules are added to the planning by the bot designers. For instance if the allowed cheating was a limited number of extra moves, but only for the human player, the effects of that could be coded into the planning engine, and the bot would "adapt" with help from its designers.

[AlphaGo] adapted to an absurd move made by Lee Sedol in 2015?

Assuming you are referring to game 4, then as far as I know, AlphaGo did not "adapt" to this play, after Lee Sedol managed to put it in a losing position then it started playing badly as it could not find a winning strategy from the board positions it was in, and could not recover. I don't think any effort was put into refining AlphaGo during this game or afterwards to patch it for game 5.

"Will a neural network adapt to that ?"
No.
The big functional difference between human mind and neural networks : human mind learns by itself, a NN not.
If we call NN the net with its layers, weights, ... this is a static system, unable to learn anything new. The back-propagation algorithm that made intelligent the NN runs outside the NN itself, in a different stage, different hardware and software, software that is not a NN but classic programming.
Thus, a NN never learns nothing while playing, driving, or any other action it is designed for.
If, in the learning stage, some cheats are done, the learning algorithm will learn and adapt to these cheats, thus the resulting NN configuration will be able to react to these cheats in the best way. But this is equivalent, in fact, to learn a different game where these cheats are valid movements.


text:
What are most commons methods to measure improvement rate in a meta-heuristic?
When I run a meta-heuristics, like a Genetic Algorithm or a Simulated Annealing, I want to have a termination criterion that stops the algorithms when there is not any significant fitness improvement.
What are good methods for that?
I tried something like
$$improvement=\frac{fit(Solution_{new})}{fit(Solution_{old})}$$
and
$$improvement={fit(Solution_{new})}-{fit(Solution_{old})}$$
Both options don't seem to be good, because as the old solutions get better and newer solutions even if they are good don't improve so much compare to the old.

You can use one of your suggested methods to calculate the relative improvement, but you need also to define a threshold value $\epsilon$ that determines when a relative improvement is negligible, and so you can terminate the algorithm. To be more concrete, you could terminate the genetic algorithm when, for example, the following condition is met
$$
|f(x_{t+1}) - f(x_{t})| < \epsilon, \tag{1}\label{1}
$$
where

$f$ is the fitness function
$t$ is the iteration number of your iterative algorithm
$x_t$ is a solution at iteration $t$
$|\cdot|$ is the absolute value
$\epsilon$ is the threshold value (a hyper-parameter of your algorithm), which is typically a small number (e.g. $10^{-6}$), but this also depends on the magnitude of the fitness of the solutions

This stopping criterion in \ref{1} (sometimes known as the absolute error) is not the only possible one. For example, you can stop the genetic algorithm when it has run for a certain maximum number of iterations. The stopping criterion (or criteria, i.e. you can use more than one stopping criteria) that you choose probably depends on what you want to achieve and this is not just specific to genetic algorithms, but, in general, this applies to any iterative numerical algorithm.
The MathWorks article How the Genetic Algorithm Works enumerates several stopping criteria for genetic algorithms.
If you want to know more about the topic, maybe take a look at this paper On Stopping Criteria for Genetic Algorithms (2004) by Martín Safe et al.


text:
What are some alternatives to "Papers with Code"?
There are lots of research papers available that are worth reading. We can read papers easily, but the associated code (not necessarily the official one developed by the authors of the paper) is often not available.
Papers with Code (and the associated Github repo) already lists many research papers and often there is a link to the associated Github repo with the code, but sometimes the code is missing. So, are there alternatives to Papers with Code (for such cases)?

I think it mainly depends on the authors if they want to open the source code, they should put it in the footnotes of their paper, or you can easily find it on Github or google. On the other hand, if the source code has too few stars on Github, you can't trust it, right? And paperswithcode also has the slack community where you can ask for code on there (paperswithcode.slack.com)

Thanks, your point makes question make more precise. Point is there are many GitHub repos which are implemented by the developer (not one who published paper) of older research paper. A nearby code also is helpful for starting work. Yes, Slack has community why stack exchange can't have the resources of its own?

what do you mean in this part "why stack exchange can't have the resources of its own?"? Do you want a summary of a good question?

Recently arxiv.org added a Code Tab towards the end of paper descriptions. Which contains links to both the official and community code.

I don't know if this is the case for all the papers or not till know. But I'm sure it'll be extended to all the papers in a short while.

The Community Code actually is a list of links to "Papers with Code".

Another good resource is the CatalyzeX website and free browser extension — it adds in-line links to any relevant code wherever you come across papers on various websites:

Chrome extension
Firefox extension
Edge extension
The corresponding website is catalyzex.com.

You can also see code results from CatalyzeX on arXiv in the Code tab towards the end of paper descriptions.
Full disclosure: I'm one of the creators. It's actively maintained and all feedback and requests are welcome!


text:
To solve chess with deep RL and MCTS, how should I represent the input (the state) to a neural network?
I'm wanting to build a NN that can create a policy for each possible state. I want to combine this with MCTS to eliminate randomness so when expansion occurs, I can get the probability of the move to winning.
I am confident (I believe) in how to code the neural network, but the input shape is the hardest part here. I am firstly wanting to try with 2 player chess and then expand to 3 player chess.
What is the best vector/matrix to use for the input in a chess game? How should the input be fed into a neural network to output the most promising move from the position? In addition, what format should it look like (i.e [001111000], etc.)?

Coincidentally I just started coding this up a few days ago, and I had the same question. I highly recommend reading the AlphaZero paper, they give a decent description of how they encoded the states.


text:
Can someone explain and help to understand this fuzzy diagram?
Could someone help me to understand in detail each step of this fuzzy diagram, because I am lost?

Maybe you should explain where you took that diagram from and what your knowledge of fuzzy logic, etc., is.


text:
What would be a typical pre-processing and data normalization pipeline for time series data (for non-linear models such as neural networks)?
I've started to work on time series. I was wondering what would be the best data normalizing and pre-processing technique for non-linear models, specifically, neural networks.
One I can think of is min-max normalization
$$z = \frac{x - min(x)}{max(x) - min(x)}$$

are we assuming that the min and max do not change over time? specifically, from the training data to the test data. If there is trending or drift then z could end up negative or above 1 for large portions of test data using this normalisation method.

@Leo would you share what neural network model using? At least similar popular model.

I would say any normalization such as min-max or standard deviation is fine as far as the scaling factor is provided as a feature, since time-series of different scale might behave differently.

Could you please elaborate more on that part of "as far as the scaling factor is provided as a feature"? if you have a link with a code example/tutorial it would be great.


text:
How many types of variational auto-encoders are there?
I have been studying about auto-encoders and variational auto-encoders. I would like to know how many variants of VAEs are there today.
If there are many variants, can they be used for feature extraction for complex reinforcement learning tasks like self-driving cars?

There are many variations of the original VAE (proposed in the 2013 paper Auto-Encoding Variational Bayes), with different purposes (such as the generation of discrete data or graphs). Of course, I cannot enumerate all of them, so here I will list only the ones I am currently aware of.

Conditional variational auto-encoder (CVAE) (2015, NeurIPS)
Adversarial autoencoders (AAE) (2015, ICLR)
Ladder variational autoencoder (LVAE) (2016,  NeurIPS)
VAE/GAN (2016, PMLR)
Vector quantised variational autoencoder (VQ-VAE) (2017, NeurIPS)
Denoising variational autoencoder (DVAE) (2017, AAAI)
Grammar variational autoencoder (GVAE) (2017, ICML)
Syntax-directed variational autoencoder (SD-VAE) (2017, ICLR)
PixelVAE (2017)
GraphVAE (2018, ICANN)
Constrained graph variational autoencoder (CGVAE) (2018, NeurIPS)
Character VAE (CVAE) (2018, ACS central science)
Regularized VAE (RVAE) (2018, NeurIPS)
Junction tree variational autoencoder (JT-VAE) (2018, PMLR)
InfoVAE/MMD-VAE (2019, AAAI)
Molecular hypergraph grammar variational autoencoder (MHG-VAE) (2019, ICML)
DAG variational autoencoder (D-VAE) (2019, NeurIPS)
Denoising adversarial autoencoders (iDAAE and DAAE) (2019, IEEE Transactions on Neural Networks and Learning Systems)
MACRo-mIcro Disentangled Variational Auto-Encoder (MacridVAE) (2019, NeurIPS)

VAEs have been used for drug design (see e.g. JT-VAE) and in reinforcement learning (see e.g. world models). I don't know if they have been used for self-driving cars, but it's possible.

TODO (optional): add links to the implementations.


text:
Why one unit in the layers of neural network is not enough?
In a deep connected network, when every unit gets all the input features(X) so it has one parameter for every feature and every unit tweaks its parameters for loss optimization. What if we use only one unit and that one unit will have all the parameters which it can tweak for loss optimization. Is there a reason or benefit of using multiple units in every layer except the output layer?

So if I understand correctly, you're proposing to use a neutral net with $N$ input units (let's say data is in $\mathbf{R}^N$), 1 hidden unit, and whatever the necessary output needs to be.
Let's say we try to do this. Then each unit of the output layer is responsible for computing its output based on a single scalar input. So it's like as if you're saying the problem can be compressed to another problem that is 1-dimensional. Now let's say you're doing regression. By modeling your NN this way, you're saying there's a function that (approximately) maps a scalar (one dimensional object) to the target output. Suppose your output layer is a linear layer (which is usually the case in regression), this is saying that the dimension of the output space is at most 1. This is almost never the case, especially with complex data.
Now suppose you had several hidden units (call this number $M$) and the weights are initialized properly. Then, it's likely that your hidden unit will have dimension larger than one, perhaps even $M$, which allows for a richer output space as $M$ increases.
You can make similar arguments for the classification setting, but the output nonlinearity just makes it a little less clear.

Note that this does not answer the question "why... is not enough?", but it's just an attempt to interpret the situation.

Sorry, about that, just added some elaboration.


text:
Is it possible to express attention as a Fourier convolution?
Convolutions can be expressed as a matrix-multiplication (see e.g. this post) and as an element-wise multiplication using the Fourier domain (https://en.wikipedia.org/wiki/Convolution_theorem).
Attention utilizes matrix multiplications, and is as such $O(n^2)$. So, my question is, is it possible to exploit the Fourier domain for attention mechanisms by turning the matrix multiplication of attention into a large convolution between the query and the key matrices?


text:
What is the best algorithm to solve the regression problem of predicting the number of languages a Wikipedia article can be translated to?
I'm doing a student project where I construct a model predicting the number of languages that a given Wikipedia article is translated into (for example, the article TOYOTA is translated into 93 languages). I've tried extracting basic info (article length, number of links, etc.) to create a simple regression model, but can't get the $R^2$ value above $0.25$ or so.
What's the most appropriate NLP algorithm for regression problems? Almost all examples I find online are classification problems. FYI I'm aware of the basics of NLP preprocessing (tokenization, lemmatization, bag of words, etc).

Random Forest Classifier, RNN, etc. Check out this NLP collection. I am sure you will get your answers

One thing that might not be apparent at the first glance when starting with ML is that many algorithms could be used for more than just classification tasks. As you say, many deep learning models, for example, are used for classification tasks most of the time. However, when changing the loss function, (possibly) the set-up of the output layer, and training a network to predict continuous values, you can turn a deep learning model into a regression model as well. So, often what you get out of a model only depends on how you configure it.

I think it's difficult to tell wich algorithm is "the best" or "the simplest".
I had the same issue of choosing the suited NLP algorithm for my dataset and I used :
https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html
Then I recommanded you to test many algorithms as you can to find the best for your needed.


text:
Why aren't the BERT layers frozen during fine-tuning tasks?
During transfer learning in computer vision, I've seen that the layers of the base model are frozen if the images aren't too different from the model on which the base model is trained on.
However, on the NLP side, I see that the layers of the BERT model aren't ever frozen. What is the reason for this?

Maybe you should provide the link to 2-3 examples (of implementations or papers/models) where the layers are not frozen (just to give more context). In any case, freezing the layers may not be necessarily required if you use a very low learning rate (but I am not an expert on this topic).

I wouldn't say they are never frozen, just rarely. Generally, you fine-tune the entire model if you care more about accuracy (or whatever metric) while you freeze layers and fine-tune only the later layers if you care more about fine-tuning/prediction speed and memory usage. For example, fine-tuning the entire BERT on a GPU can take about 8GB of VRAM in my experience, which can be reduced by a gb or two by freezing layers.

@primussucks if we don't freeze bert new layers will mess up trained weights. isn't it so?

@datonefaridze It will change the weights yes, but "mess up" suggests that the model will perform poorly if they aren't frozen, which just isn't true.

Corrections and other answers are welcome, but here are a few thoughts:
There are several approaches in terms of which weights get frozen (and also other considerations, see for example Fig. 5 in "Galactica: A Large Language Model for Science").
Which of the approaches yields higher-quality results depends on the architecture (and hyperparameters) and dataset.
There can be rules of thumb, for example this old snapshot of a "Documentation" of Transformer architectures at Hugging Face said:

we are directly fine-tuning the whole model without taking any precaution. It actually works better this way for Transformers model

but this explanation apparently was removed from the new vesion of this page. Maybe it turned out that such rules of thumb aren't right in general.
Quality of results is also not the only thing being optimized. Some choices are made due to memory or compute considerations. For example, when freezing the first layers, their output features can be computed only once for all samples, saved, and used thereafter; moreover, computing the gradient of the loss with respect to the weights of the first frozen network block is not necessary.


text:
Is it ok to perform transfer learning with a base model for face recognition to perform one-shot learning for object classification?
I am trying to create a model that is using a one-shot learning approach for a classification task. We do this because we do not have a lot of data and it also seems like a good way to learn this approach (it is going to be a university project). The task would be to classify objects, probably from drone/satellite image (of course zoomed one).
My question is, do you think it would be ok to use a model for face recognition, such as DeepFace or OpenFace, and, using transfer learning, retrain it on my classes?


text:
What is the difference between derivation and entailment?
In section 7.3 of the book Artificial Intelligence: A Modern Approach (3rd edition), it's written

An inference algorithm that derives only entailed sentences is called sound or truth-preserving.

The property of completeness is also desirable: an inference algorithm is complete if it can derive any sentence that is entailed.

However, this does not make much sense to me. I'd like someone to kindly elaborate on this.

I've the same problem, reading same book.
This helped me a little bit: https://www.youtube.com/watch?v=xL0kNw5TudI
let me know if you get something better.


text:
How to find distance between 2 points when dimensions are all of different nature?
I have a dataset with four features:

the x coordinate
the y coordinate
the velocity magnitude
angle

Now, I want to measure the distance between two points in the dataset, taking into account the facts that the angle dimension is toroidal, and taking into account the difference in nature of the dimensions (2 of them are distances, one of them is velocity magnitude, and the other an angle).
What kind of distance function would suit this need?
If I have to go for an $L^p$ norm, can I determine which value of $p$ would be apt by some means?
Also, if you are aware, please, let me know how such problems have been solved in various applications.

Can you explain more about the nature of your data? Although I can imagine that the x and y coordinates and the angle could constitute a pose (e.g. of a robot) and the other feature would represent the magnitude of the velocity when that pose was recorded. I think that computing the distance between points will depend on what you want to achieve. Why do you want to compute the distance between observations in the dataset?

The purpose is for clustering the data points. I want the metric to be robust enough such that if two points have their X and Y coordinates close and their velocity magnitude are almost same, yet their orientation is in the opposite direction, they should be treated to belong to different clusters

So, you want to take into account all 4 properties.

Yes, all four properties should be considered. Even if the opposite is true, that is the data points are far apart in space but have similar orientation, they must be clustered into different clusters


text:
How does the MCTS tree look like?
I have come across the Monte Carlo tree search (MCTS) algorithm, but I can't find what the tree should look like. For example, does it still represent a minimax process, i.e. player 1 from the root has its child nodes as probabilities of moves, then from those child nodes, the next move is player 2, etc.? Is this how the tree looks like, so when we backpropagate we update only the winning player nodes?

In a MCTS (Monte Carlo tree search) the resulting tree is expanded by two policies:

tree-policy (selects next and unknown child nodes by a exploration-exploitation tradeoff)
default policy (selects known child nodes till terminal state e.g. with random uniform moves).

The term terminal state is often considered to be the end a tree. If you consider designing an agent with MCTS that is able to play chess, then the terminal state would a win, lose or draw. One game like this is called a playout (sometimes also roll-out). The main concept of MCTS relies on repeating this playouts many times and updating the tree according to the scores. The tree is built up in an incremental and asymmetric manner.

Imagine designing a chess-playing agent. The tree, or rather how the tree is built up can be like this:

Root Node: Starting with an empty board
Child Nodes: Board configuration after each move (opponent and agent)
Selection of Child Nodes: If the opponent is a human, then the selection of child nodes is performed by his/her own strategy. The agent is selecting moves according to a policy.
Terminal State: Score of the game (win/lose/draw).

So after each move (opponent or agent) the current child node/root node is expanded by the resulting child nodes.
If you want to dive deeper into MCTS, then I would recommend to take a look at A Survey of Monte Carlo Tree Search Methods.

Some small corrections: default policy does not really select nodes, but states (or even more accurately, actions); typically no nodes are added to the tree for the states encountered during rollout (except for one Expansion at start of rollout/end of Selection). If the opponent is a human, of course the MCTS during its tree search still makes decisions "for the human" while traversing the tree. Just in the "real game" you need to wait for the human, but in the search tree process you don't. In Chess, the initial state (root node) is not an empty board :)


text:
Why scaling down the parameter many times during training will help the learning speed be the same for all weights in Progressive GAN?
The title is one of the special things in Progressive GAN, a paper of the NVIDIA team. By using this method, they introduced that

Our approach ensures that the dynamic range, and thus the learning speed, is the same for all weights.

In details, they inited all learnable parameters by normal distribution $N(0,1)$. During training time, each forward time, they will scale the result with per-layer normalization constant from He's initializer
I reproduced the code from pytorch GAN zoo Github's repo
def forward(self, x, equalized):
    # generate He constant depend on the size of tensor W
    size = self.module.weight.size()
    fan_in = prod(size[1:])
    weight = math.sqrt(2.0 / fan_in)
    '''
    A module example:

    import torch.nn as nn
    module = nn.Conv2d(nChannelsPrevious, nChannels, kernelSize, padding=padding, bias=bias)
    '''
    x = self.module(x)

    if equalized:
        x *= self.weight
    return x

At first, I thought the He constant will be $c = \frac{\sqrt{2}}{\sqrt{n_l}}$ as He's paper. Normally, $n_l > 2$ so $w_l$ can be scale up which lead to the gradient in backpropagation is increase as the formula in ProGan's paper $\hat{w}_i=\frac{w_i}{c}$ $\rightarrow$ prevent vanishing gradient.
However, the code shows that $\hat{w}_i=w_i*c$.
In summary, I can't understand why to scale down the parameter many times during training will help the learning speed be more stable.
I asked this question on some communities e.g: StackOverflow, mathematics, Data Science, and still haven't had an answer.
Please help me explain it, thank you!

What do you mean by "He constant"? And which parameter are you referring to when you say "I can't understand why to scale down the parameter many times during"? Moreover, note that I changed the title of your post to be more descriptive. Make sure that's your question in the title.

Thank you for fixing my title, I've not had the experience to ask a correct question. I said "He constant" because the paper mentioned it. At first, I didn't understand so I found the code and got the equation above. About my question, the parameters I mentioned here is all learnable parameters that init by Normal distribution, if you want I can quote a part of the paper into my question.


text:
Can we stop training as soon as epsilon is small?
I'm new to reinforcement learning.
As it is common in RL, $\epsilon$-greedy search for the behavior/exploration is used. So, at the beginning of the training, $\epsilon$ is high, and therefore a lot of random actions are chosen. With time, $\epsilon$ decreases and we often choose the best action.

I was wondering, e.g. in Q-Learning, if $\epsilon$ is small, e.g. 0.1 or 0.01, do the Q-values really still change? Do they just change their direction, i.e. the best action remains the best action but the Q-values diverge further, or do the values really change again so that the best action always changes for a given state?

If the Q-values really do still change strongly, is it because of the remaining random actions, which we still have at $\epsilon>0$ or would it still change at $\epsilon=0$?

The question in the title is different than the questions in the body, although they are related. I suggest that you connect the title's question with the body's questions also in the body, so that we understand what your main concern/question is. For example, the current only answer does not address the question in the body, and that's probably because there's no connection in the body.

@nbro yes, exactly why I didn't answer the title of the question. If OP confirms whether his title is his actual question then I can answer that too.

How much the $Q$-values change does not depend on the value of $\epsilon$, rather the value of $\epsilon$ dictates how likely you are to take a random action and thus take an action that could give rise to a large TD error -- that is a large difference between the returns you expected from taking this action as to what you actually observed. How much the $Q$-value changes depends on the magnitude of this TD error.

$Q$-learning is not guaranteed to converge if there is no exploration. Part of the convergence criteria assumes that each state-action pair will be visited infinitely often in an infinite number of episodes, and so if there is no exploration then this will not happen.


text:
What are the disadvantages of actor-only methods with respect to value-based ones?
While the advantages of actor-only algorithms, the ones that search directly the policy without the use of the value function, are clear (possibility of having a continuous action space, a stochastic policy, etc.), I can't figure out the disadvantages, if not general statements about less stability or bigger variance with respect to critic-only methods (with which I refer to methods that are based on the value function).


text:
Why is it necessary to divide the priority range according to the batch size in Prioritized Experience Replay?
According to DeepMinds's paper Prioritized Experience Replay (2016), specifically Appendix B.2.1 "Proportional prioritization" (p. 13), one should equally divide the priority range $[0, p_\text{total}]$ into $k$ ranges, where $k$ is the size of the batch, and sample a random variable within these sub-ranges. This random variable is then used to sample an experience from the sum-tree according to its priority (probability).
Why do we need to do that? Why not simply sampling $k$ random variables in $[0, p_\text{total}]$ and getting $k$ variables from the sum-tree without dividing the priority range into $k$ different ranges? Isn't this the same?

This question was also asked here.


text:
What is meant by "ground truth" in the context AI?
What does "ground truth" mean in the context of AI especially in the context of machine learning?
I am a little confused because I have read that the ground truth is the same as a label in supervised learning. And I think that's not quite right. I thought that ground truth refers to a model (or maybe the nature) of a problem. I always considered it as something philosophical (and that's what also the vocabulary 'ground truth' implies), because in ML we often don't build a describing model of the problem (like in classical mechanics) but rather some sort of a simulator that behaves like it is a describing model. That's what we/I call sometimes black box.
What is the correct understanding?

See also What is Ground Truth.

In the context of ML, ground truth refers to information provided by direct observation (empirical evidence). If you're training an algorithm to classify your data, then the ground truth will be the actual, true labels which could for example be manually annotated by an domain expert. Please note that the models prediction or the inferred labels, are not considered ground truth.

It really depends on what words you put after "ground truth". Sometimes people will talk about "ground truth labels", for example in the context of classification or regression problems. The "ground truth labels" in such a case would refer to the true labels of instances; the labels that we use as target labels for instances from a training set, or the labels that we expect our models to output (and "punish" them for if they fail to do so) when evaluating/testing a trained model. This basically follows razvanc92's answer.
"Ground truth" can also refer to something more abstract though, something that we know exists in some form or another, but we may not even know how to express it. For example, there may be "ground truth laws of physics", the laws of physics that our world "follows". We may build or train a simulator trying to approximate those ground truth functions / laws, but we may not actually know how to explicitly express all of them.

For the latter consideration, consider also a statistical distribution that generated some training data. You might not be able to observe this underlying distribution directly, but at least you can learn to approximate it.


text:
How are weight matrices in attention learned?
I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, they treat learning the weight matrices (for query, key, and value) as it is the most trivial thing.
So, how are these weight matrices learned? Is the error just backpropagated, and the weights are updated accordingly?

Short answer to an old question. Yes, the error is backpropagated and the weights are updated accordingly. The skip/residual connections in the transformer help to avoid vanishing gradients with larger amounts of decoder blocks in sequence. See also the 'basic training' section of this post.


text:
DQN not learning and step not stepping towards target
I am trying to create a simple Deep Q-Network with 2d convolutional layers.
I can't figure out what I am doing wrong, and the only thing I can see that doesn't seem right is when I get the model prediction for a state after the optimizer step it doesn’t seem to get closer to the target.
I am using pixels from pong in OpenAI's gym with single-channel 90x90 images, a batch size of 32, and replay memory.
As an example, if I try with a batch size of 1, and try running self(states) again right after the optimizer step the output is as follows:
current_q_values -> -0.16351485  0.29163417  0.11192469 -0.08969332  0.11081569  0.37215832
q_target ->         -0.16351485  0.5336551   0.11192469 -0.08969332  0.11081569  0.37215832
self(states) ->     -0.8427617   0.6415581   0.44988257 -0.43897176  0.8693738   0.40007943

Does this look as what would be expected for a single step?
The network with loss and optimizer:
    self.in_layer = Conv2d(channels, 32, 8)
    self.hidden_conv_1 = Conv2d(32, 64, 4)
    self.hidden_conv_2 = Conv2d(64, 128, 3)
    self.hidden_fc1 = Linear(128 * 78 * 78, 64)
    self.hidden_fc2 = Linear(64, 32)
    self.output = Linear(32, action_space)

    self.loss = torch.nn.MSELoss()
    self.optimizer = torch.optim.Adam(
        self.parameters(), lr=learning_rate) # lr is 0.001

def forward(self, state):
    in_out = fn.relu(self.in_layer(state))
    in_out = fn.relu(self.hidden_conv_1(in_out))
    in_out = fn.relu(self.hidden_conv_2(in_out))
    in_out = in_out.view(-1, 128 * 78 * 78)
    in_out = fn.relu(self.hidden_fc1(in_out))
    in_out = fn.relu(self.hidden_fc2(in_out))
    return self.output(in_out)

Then the learning block:
        self.optimizer.zero_grad()

        sample = self.sample(self.batch_size)
        states = torch.stack([i[0] for i in sample])
        actions = torch.tensor([i[1] for i in sample], device=device)
        rewards = torch.tensor([i[2] for i in sample], dtype=torch.float32, device=device)
        next_states = torch.stack([i[3] for i in sample])
        dones = torch.tensor([i[4] for i in sample], dtype=torch.uint8, device=device)

        current_q_vals = self(states)
        next_q_vals = self(next_states)
        q_target = current_q_vals.clone()
        q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (~dones).float()

        loss = fn.smooth_l1_loss(current_q_vals, q_target)
        loss.backward()

        self.optimizer.step()
```

In my experience, neural networks with convolutional layers take much much longer to train, so try increasing the number of iterations (time steps). After running, save the network model (I dont know how to do it in torch, but in tensorflow it was model.save("filename"+".h5") ).
Then, load this saved model file and do a test run to see if it worked. In this case, you should notice pretty easily if it learned or not).

Since you are looking at a single iteration and expect a meaningful change my guess is that you aren't training for long enough. Q-learning can take very long, for many environments it takes millions of iterations.

I found the reason it wasn't learning. The issue was this line of code:
q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (~dones).float()

I had been using the tilde operator before to invert uint8 tensors, but recently I had updated to the latest version of pytorch that seems to have changed how the operator works. It was changing the done values to 255.
Changing to this line fixed it:
q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (1 - dones)


text:
What is the search depth of AlphaGo and AlphaGo Zero?
I cannot find reliable sources but someone says it is 40 moves and someone else says it is 50+ moves. I read their papers and they use value function (NN) and policy function to trim the tree, so more layers can be searched while spending less time searching less different positions.
My question is, is the search depth a fixed preset parameter? If so, approximately how much is it back to 2016 (AlphaGo) and 2018 (AlphaGo Zero)?

For easier visualization, I recommend this video:
https://twitter.com/i/status/1257053365424578565
The more detailed article about GO algorithms:
https://deepmind.com/blog/article/alphago-zero-starting-scratch.
With its breadth of $250$ possible moves each turn (go is played on a $19$ by $19$ board, compared to the much smaller $8$ by $8$ chess field) and a typical game depth of $150$ moves, there are about $250^{150}$, or
$10^{360}$.
After $2$ moves in go, it's $130000$ possible combinations.
Decision tree pruning

Do you know the approximate depth or the average depth?

if it'll be Monte Carlo Tree Search - then hypothesizes 3-5 deepth or 5-15 (with prunning) (depend on CPU, GPU)


text:
How much is currently invested in artificial general intelligence research and development?
How much is currently invested in artificial general intelligence research and development worldwide?
Feel free to add company or VC names, but this is not the point. The point is to get an idea of the economics around artificial general intelligence.

In the last years, there have been big investments in AI technologies. For an overview, maybe take a look at this article Artificial Intelligence: Investment Trends and Selected Industry Uses (2019).
A few companies that have the long-term goal of creating an AGI, although, currently, they mainly do research on specific problems (e.g. video games) or AI techniques (e.g. reinforcement learning), have received many funds. I will only list a few (maybe the most well-known ones) of these companies below, but there are probably many other companies that have this long-term goal and have been funded by other companies or people.
DeepMind
In their site, they write

Like the Hubble telescope that helps us see deeper into space, we aim to build advanced AI - sometimes known as Artificial General Intelligence (AGI) - to expand our knowledge and find new answers. By solving this, we believe we could help people solve thousands of problems.

DeepMind was acquired by Google in 2015 for $500 million, given its success in playing games at a superhuman performance, which is a promising step towards the development of more AI techniques and maybe AGI.
The Wikipedia article on DeepMind contains some information about people or companies that have invested in DeepMind, which includes companies Horizons Ventures and Founders Fund, and people  Scott Banister, Peter Thiel, Elon Musk, and Jaan Tallinn, although I cannot give you the exact numbers in terms of capital. In any case, serious investments have been done in DeepMind, which is definitely one of the promising companies that could develop good insights into the development of AGI systems.
OpenAI
Another company that has a similar goal and is doing research on similar topics (such as reinforcement learning or natural language processing) is OpenAI, which also has the long-term goal of creating AGI systems, as they write in their website

OpenAI’s mission is to ensure that artificial general intelligence (AGI) — by which we mean highly autonomous systems that outperform humans at most economically valuable work — benefits all of humanity. We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.

Their investors include Microsoft, Reid Hoffman's charitable foundation, and Khosla Ventures.
Vicarious
They write in their site

Artificial general intelligence is the finish line on our journey toward progressively more capable robots. Our approach leverages deep expertise in neuroscience and is shaped by a decade of research.

They are apparently backed by more than 150 million dollars from people like Jeff Bezos, Elon Musk and Mark Zuckerberg and companies like Samsung.

Nice answer, it contains interesting examples. Thank you for your contribution.

This blog post What’s So Disturbing about Microsoft’s OpenAI Investment? (2019) by Ben Goertzel may also be interesting and useful.

Forget about OpenAI and DeepMind etc. in that order. They are just doing stupid deep learning and backward propagation which is technology from the 50s. There are virtually no pure AGI companies because it's still early days.
Total investment per year of serious AGI is far below 100 Million a year.
Go read some AGI papers from the AGI conference and adjacent conferences, some entities like to point out publically their funding sources. Sometimes even with the exact amount of funding.
Bottom line: Funding is still ridiculous low compared to what say Google makes in 2 months alone with stupid ads.
The field is still massivly underfunded  https://medium.com/@petervoss/why-arent-more-people-working-on-agi-7c8367bf2615


text:
Is GPT-3 an early example of strong AI in a narrow setting?
In GPT-2, the large achievement was being able to generate coherent text over a long-form while maintaining context. This was very impressive but for GPT-2 to do new language tasks, it had to be explicitly fine-tuned for the new task.
In GPT-3 (From my understanding), this is no longer the case. It can perform a larger array of language tasks from translation, open domain conversation, summarization, etc., with only a few examples. No explicit fine-tuning is needed.
The actual theory behind GPT-3 is fairly simple, which would not suggest any level of ability other than what would be found in common narrow intelligence systems.
However, looking past the media hype and the news coverage, GPT-3 is not explicitly programmed to "know" how to do these wider arrays of tasks. In fact, with limited examples, it can perform many language tasks quite well and "learn on the fly" so to speak. To me, this does seem to align fairly well with what most people would consider strong AI, but in a narrow context, which is language tasks.
Thoughts? Is GPT-3 an early example of strong AI but in a narrower context?

GPT-3 is based on in-context learning. It’s common wisdom one can hope that bigger models will yield better in-context capabilities. And indeed, this holds true, in the case of GPT-3 175B or "GPT-3".
Neverthless GPT-3 is more powerful than it's predecessors. In some of the tasks, GPT-3 failed miserably. This might be due to the choice to use an autoregressive LM, instead of incorporating bidirectional information (similarly to Bert).
While in-context learning is more straightforward with autoregressive LMs, bidirectional models are known to be better at downstream tasks after fine-tuning.
In the end, training a bidirectional model at the scale of GPT-3 or trying to make bidirectional models work with few-shot learning is a promising direction for future research.
Check out this, this and the paper on Scaling Laws for Neural Language Models.

You should probably cite some research papers that support this claim: " It’s common wisdom one can hope that bigger models will yield better in-context capabilities". Moreover, I think you should explain more in detail what "in-context learning" really means.


text:
What is the "Hello World" problem of Reinforcement Learning?
As we all know, "Hello World" is usually the first program that any programmer learns/implements in any language/framework.
As Aurélien Géron mentioned in his book that MNIST is often called the Hello World of Machine Learning, is there any "Hello World" problem of Reinforcement Learning?
A few candidates that I could think of are multi armed bandits problem and Cart Pole Env.

MNIST (along with CIFAR) may be the "Hello World" of supervised learning for image classification, but it is definitely not the "Hello World" of all machine learning techniques, given that RL is also part of ML and MNIST is definitely not the "Hello World" of RL.
I don't think there is a single "Hello World" problem for RL. However, if you are looking for simple problems (or environments) that are usually used as baselines to assess the quality of RL agents, then I would say that the simple grid worlds where you need to move from one place to the other, the CartPole, MountainCar, Pendulum or other environments listed here are often used.
The environment that you choose to train and test your RL agent depends on your goals. For example, if you designed an algorithm that is supposed to deal with continuous action spaces, then an environment where you can take only a discrete number of actions may not be a good option.
The mentioned environments are very simple (i.e. toy problems). In my opinion, we need more serious environments that can show the applicability of RL to other areas other than (relatively simple) games.

While there's no simple Hello World problem of RL, if your aim is to understand the basic working of Reinforcement Learning and see it at play while using as few moving parts as possible, a simple suggestion would be using Tabular Q-Learning in a toy environment (like your suggested Cart-Pole Env).
Here's the reasoning behind this suggestion
Let's say we interpret MNIST's label as a Hello World of Supervised Learning to mean something showing the basic steps of doing Supervised Learning: Create a model, load the data, then train.
If that interpretation is not far off, we can say a simple introductory problem to Reinforcement Learning (RL) should focus on easily demonstrating a working Markov Decision Process (MDP) which is the backbone of the RL decision making process. As such, this minimal working would involve: Observing the world, selecting an action, as shown in this loop:

This picture is missing two important steps in an RL algorithm learning loop:

Estimating the rewards or Fitting the model
Improving how you select actions. (Updating your policy)

How we decide to update the policy, or fit the model is what makes difference in the RL algorithm most of the time.
So a suggested first problem would be one that helps you see the MDP in action, while keeping steps 1 and 2 simple enough so that you understand how the agent learns. Tabular Q-Learning seems clear enough for this because it uses a Q-table represented as a 2D array to do the two steps. This should not suggest Q-learning is a "Hello World" RL algorithm because of the said relative ease in understanding it :)
You will be unable to use it's Tabular version anywhere else than in a toy environment though, typically Frozen-Lake and CartPole. An improvement would be using a neural network instead of a table to estimate Q values.
Here are a few useful resources:

Q-Learning with Tables (Guide)
Q-learning jupyter notebook (Code ~25 lines)
Q-Learning with Frozen-Lake and Taxi (Code)
Reinforcement Learning with Q-Learning (Guide)

A multi-armed bandit would also be great in introducing you to exploration-exploitation trade-off (which Q-learning does too), though it wouldn't be considered a full RL algorithm since it has no context.

So, the ML version of while(!worldOnFire){flames++}?

Although you are talking about tabular Q-learning, I think that saying that Q-learning is the "Hello World" program (not problem, as in my answer) of RL is misleading, because it makes it seem like Q-learning is just a "toy useless algorithm", but, of course, it's not. Q-learning with function approximation (specifically, neural networks) was used to solve/play the Atari games with superhuman performance in many cases. So, Q-learning, although quite simple, can be used to solve non-trivial problems, so it's a powerful algorithm, and that's why your suggestion is misleading.

Definitely, if this answer presents Q-learning as a "useless toy algorithm", that's very misleading. I've noted how the use of program and problem (as you explain it) contributes to this. I've attempted to edit the answer to alleviate that, as much as I could. Thanks @nbro! My intention was not to mislead.


text:
How to train a model to predict the number of people at a certain bus stop before they cumulate in large numbers?
Each person probably uses an app that tracks his/her position periodically and sends it to our servers. What I want is to use these data to train a model to predict the rush hours of each bus-stop on the map, so we can send extra buses to handle the predicted cumulation before it happens.
I have no experience in AI nor machine learning. So, which model should I use to do this?

Welcome to AI Stack Exchange. Do you have any data from the apps, or is this an imagined app that you want to work with a first version the model? The phrasing suggests this is an application/service idea and you are trying to figure out the very first parts of trying to build it. Do you have any data available (as in already possess it) at all, such as general traffic data, a map of bus routes etc?

Maybe you can use a recurrent neural network on saved data to train a predictive model based on past data.

Maybe you should provide more details about your suggestions. For example, why would an RNN be suited for this task?

H2O's AutoML is the thing that you are looking for believe me it will make your life super easy.
So how it generally works is:

You have data and want to make sense out of it by making a prediction/classification and whole other array of things, in your case prediction.
Suppose there are 50 prediction algorithms out there, it's highly unlikely that an individual knows all of them. That's where AutoML comes into the picture.
You give AutoML some part of your processed data and tell it to find a perfect algorithm that he thinks you should use on this data for this type of prediction. See the usage of the AutoML's API in the doc and videos on youtube.
It then gives you a list of best algorithms that AutoML thinks is best based on the loss function that you specify. There are many other parameters that you can specify in the AutoML's API.
Pick top 1-2 and tune the hyperparameters of the algorithm.
That's it.

Even then if you are not happy with the performance of your system try out ensemble learning before you jump for the power of neural nets which comes with lots of complexities and performance issues.


text:
How differentiable programming and programming language supporting it will potentially help the development towards AGI?
After the state of the art Deep Learning techniques/algorithms being implemented in low-level languages like Objective-C, C++, etc to high-level languages like Python, JS, etc. and with the help of huge libraries like Tensorflow, Pytorch, Scikit-Learn, etc.
Now, Swift: Google's bet on differentiable programming, they are making Swift differential programming ready see this manifesto and they are building TensorFlow from the ground up in swift S4TF.
So, how differentiable programming and programming language supporting it will potentially help the development towards AGI?


text:
What's wrong with my answer to this constraint satisfaction problem, which needs to be solved the AC-3 algorithm?
I was watching the video Constraint Satisfaction: the AC-3 algorithm, and I tried to solve this question:

Given the variables A, B, C and D, with domain {1, 2, 3, 4} in each of them and restrictions A> B, B = C and D ≠ A, use the AC algorithm.

But the teacher told me that my answer below is wrong!
He gave me a tip: Domain D will not be changed!
Below, I is my answer step by step. If someone can help me find the error, I appreciate it!
To solve this exercise, it is first necessary to organize the data in order to separate what is the domain, agenda and arc.

Soon after, we will analyze the first item on the agenda “A> B” with domain A, in order to eliminate unnecessary elements from the domain.

Analyze domain B with the agenda item “B <A”

Analyze domain B with the agenda item “B = C” and add the constraint “A> B”

Analyze domain D with the agenda item “D ≠ A” and add the constraint “B <A”

Analyze domain A with the agenda item “A ≠ D”

Analyze domain A with the agenda item “A> B”

Analyze domain B with the agenda item “B = C”

Analyze domain B with the agenda item “B <A”

Result

To satisfy $D\ne A$, for each element in the domain of $D$, say $d$, there must be at least one element in the domain of $A$, say $a$, that $d \ne a$.
Domain of $D$: $ \{ 1, 2, 3, 4 \} $
Domain of $A$: $ \{ 2, 3, 4 \} $
So, for example if $d=2$, there is at least one element in the domain of $A$ like $a$ such that $d \ne\ a$. E.g. $1, 3$ and $4$.
So, $2$ remains in the domain of $D$. This is the case for other elements of $D$.


text:
How could I convolve a 4D image and a 4D filter with stride?
I want to create a CNN in Python, specifically, only with NumPy, if possible. For optimizing the time of convolution (actually correlation) in the network, I wanna try to use FFT-based convolution. The data that needs to be convoluted (correlated) is a 4D image tensor with shape [batch_size, width, height, channels] and 4D filter tensor [filter_width, filter_height, in_channel, out_channel]. I read a lot of articles about FFT-based convolution, but they aren't doing it in my way. Thus, I need your help.
How could I fft-convolve a 4D image and a 4D filter with stride?

Hello. It's not clear what your question is. Please, explicitly ask a specific question about your problem.

Hi, @nbro, I did it.

Ok, thanks for clarifying that. If you're looking for code, this site is not appropriate to ask your question (i.e. programming questions are generally off-topic here), so your question would be off-topic. You can read more about what is on-topic here: https://ai.stackexchange.com/help/on-topic, and I suggest that you read it. Are you asking for code? If yes, then probably Stack Overflow is the best place to ask your question. If you want, I can migrate your question to Stack Overflow.

Actually, No. I want to understand how to FFT-Convolve two 4D tensors with clarifying.

Thanks, @nbro again). You're really good moderator

I think this should work for you:
scipy.signal.correlate | SciPy
I used it myself while I was writing a CNN in numpy.

Mm, it doesn't seem optimal, but I don't know a better way that just to correlate and take elements with the corresponding step.

Anyway, thanks for good answer. I'll use it in my code.

Given that the OP apparently wants to implement the correlation/convolution from scratch, then maybe you should spend some words to explain how that can be conceptually done. By the way, welcome to AI SE! Have a look at our on-topic page https://ai.stackexchange.com/help/on-topic to know more about the type of question we are looking for.

I think what you need to use is 3D convolution operation. Your data is 3D, width, height, and num_channels. Your data is similar to color images with RGB channels. However, since you are trying to consider the correlation amongst channels 2D convolution will not work for you. You can use 3D convolution which is available to use with deep learning tools such as Tensorflow.

P.S. My data is 4D: [ batch_size, width, height, and num_channels ].

Yes, my images are RGB

What is the 4 dimensions in your data, i only see 3, batch_size is not a dimension of data, isnt it the batch size of the input of the neural network or is it somthing else.


text:
What is a "learned emulator"?
In this article, the term "learned emulator" is used.

Recently, scientists have started creating "learned emulators" using
AI neural network approaches, but have not yet fully explored the
advantages and potential pitfalls of these surrogates.

What is a "learned emulator"?  I believe it is related to neural networks. Where can I read more?

In a typical situation, for the emulation of physical environments, you need to define all physical rules and forces. In the "learned emulators", they use some machine learning techniques to learn those rules by supervising and interacting (instead of formalizing all of them). In this case, they do not need any exact formulation of the physical environment to emulate it.
An instance of this simulation can be this article "Realistic Atomistic Structure of Amorphous Silicon from Machine-Learning-Driven Molecular Dynamics", i.e., a machine-learning-driven simulation instead of exact formulation of the environment.


text:
How much should we augment our training data?
I am wondering how much I should extend my training set with data augmentation. Is there somewhere a pre-defined number I can go with?
Suppose I have 10000 images, can I go as far as 10x or 20x times, to get 100000 and 200000, respectively, images? I am wondering how will this impact model training. I am using a mask R-CNN.

If hardware and training time isn't a limitation, just try it and see when validation accuracy starts to decrease.

Maybe you should describe which model you are trying to use (I suppose it's a neural network) and how you are augmenting your training data. There are several ways you can augment the training data.

@nbro Mask RCNN, this is the model.

Literally augment as much as possible as long as a valid image is still produced. The gold standard for CV is human equivalent understanding. Now you dont need to preagument them, they can be augmented at sampling time.


text:
Can entire neural networks be composed of only activation functions?
Inverse Reinforcement Learning based on GAIL and GAN-Guided Cost Learning(GAN-GCL), uses a discriminator to classify between expert demos and policy generated samples.
Adversarial iRL, build upon GAN-GCL, has its discriminator $D_{\theta, \phi}$ as a function of a state-only reward approximator $f_{\theta, \phi}$.
$$
D_{\theta, \phi}\left(s, a, s^{\prime}\right)=\frac{\exp \left\{f_{\theta, \phi}\left(s, a, s^{\prime}\right)\right\}}{\exp \left\{f_{\theta, \phi}\left(s, a, s^{\prime}\right)\right\}+\pi(a \mid s)},
$$
where $f_{\theta,\phi}$ is expressed as:
$$f_{\theta,\phi} = g_{\theta} (s) + γh_φ (s\prime ) − h_φ (s).$$
The optimal $*g(s)$ tries to recover the optimal reward function $r^*(s)$. While the $h(s)$ tries to recover the optimal value funtion $V^*(s)$, which makes $f_{\theta,\phi}$ interpretable as the advantage.
My question comes from the network architecture used for $h(s)$ in the original paper.

... we use a 2-layer ReLU network for the shaping term h. For the policy, we use a two-layer (32 units) ReLU gaussian policy.

What is meant by the quoted text in bold, because my interpretation of that text, (shown below) doesn't seem viable
h = nn.Sequential([nn.ReLu(), nn.ReLu()])

I've seen this short example from pytorch docs. They define a fully-connected ReLU network with one hidden layer as nn.Sequential( nn.Linear(in, H), .nn.ReLU(), nn.Linear(H, out))

The Pytorch docs define a fully connected ReLU network as:
torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)

Neural networks are called are made of neurons. Activation functions only help determine which of these neurons to fire up, meaning they have no learnable nodes themselves through which we can back-propagate gradients. A network with no learnable parameters is therefore not a neural net. So neural nets can't be composed of activation functions only.

What is meant by the quoted text in bold, because my interpretation of that text, (shown below) doesn't seem viable

Yes, what's given here is not a network that can approximate the said function $h(s)$. A two layer ReLU network would resemble:
   x = nn.ReLU(nn.Linear(d_IN, H))
   x = nn.ReLU(nn.Linear(H, H))
   out = nn.Linear(x, d_OUT)

Another way to see it is that a network must have an input and ouput layer, and optional hidden layers. It's not possible to use an activation funtion as an input layer, because then you'll have no way of configuring the number of features to represent your input data. In this context, a ReLU can't represent the features of the observation input s.
To show activation functions have no learnable nodes in them and that this interpretation

h = nn.Sequential([nn.ReLu(), nn.ReLu()])

is not what the authors are driving across, here a script that counts the number of parameters in a network.

import torch.nn as nn
import numpy as np

activation = nn.ReLU

def count_params(module):
    return np.sum([np.prod(x.shape) for x in module.parameters()])

one_linear = nn.Sequential(nn.Linear(32, 10), nn.Linear(10, 1))
linear_act = nn.Sequential(nn.Linear(32, 10), activation(), nn.Linear(10, 1))
act_only = nn.Sequential(activation(), activation())

t_lin = count_params(linear_act)
lin = count_params(one_linear)
act = count_params(act_only)

print(f'Linear only: {lin}, Linear + Activation: {t_lin},' +
        f'Activation only: {act}')

[Out]: Linear only: 341, Linear + Activation: 341, Activation only: 0.0
The activation-function-only module has zero learnable parameters. Likewise, an activation function adds no parameters to the fully connected layers.
Update: Links to implementations
To confirm this answer's interpretation is correct, here are links to GAIL and GAN-GCL example implementations

GAIL : discriminator prediction (Forward call), discriminator architecture (The ReLU net):
GAN-GCL : discriminator prediction, discriminator architecture:


text:
Are mult-adds and FLOPs equivalent?
I am comparing different CNN architectures for edge implementation. Some papers describing architectures refer to mult-adds, like the MobileNet V1 paper, where it is claimed that this net has 569M mult-adds, and others refer to floating-point operations (FLOPs), like the CondenseNet paper claims 274M FLOPs.
Are these comparable? Is 1 multiply-add equivalent to 2 floating-point operations? Any direction will be greatly appreciated.

One of the papers that you mention states "Throughout the paper, FLOPs refers to the number of multiplication-addition operations", and this is consistent with my knowledge of FLOPS, which are pretty standard in computational science (i.e. analysis and implementation of numerical algorithms, etc.). However, it is the first time I hear of "mult-add", to be honest, so I would need to check what those papers actually refer to when they calculate the "mult-adds".

This answer says they are equivalent, but it doesn't explain why. However, this answer seems to provide more insight. Maybe later I will provide a more formal answer.

According to this source, one MAC is roughly equal to two FLOP (multiply accumulate). My guess/understanding would be that the distinction is made because neural nets spend compute overwhelmingly on multiply-accumulate operations, and thus optimizations and statistics over MAC operations would be more significant than FLOPs.

Comments are not for extended discussion; this conversation has been moved to chat.


text:
Why does PPO lead to a worse performance than TRPO in the same task?
I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of the environment and rewards are the same. PPO is just a more efficient method compared to TRPO and has proven to be a state-of-the-art method in RL. So, why shouldn't it work? I just thought to ask if someone knows roughly how to transform configuration parameters from TRPO to PPO.
Here some more details about my configurations.
TRPO

Actor loss:  $-\log(\pi) * A$ where $A$ are advantages
Critic Loss: MSE(predicted_values, discounted return)
Desired KL Divergence for Actor and Critic: 0.005
Conjugate gradient iterations: 20
Residual tolerance in conjugate gradient: 1e-10
Damping coefficient for Fisher Product: 1e-3

PPO

Actor and Critic optimizer and learning rate: Adam with 0.0001
Actor loss: negative minimum of either:

$\frac{\pi}{\pi_{old}} * A$
$clamp(\frac{\pi}{\pi_{old}}, 1-0.1, 1+0.1) * A$

Critic loss: MSE(predicted_values, discounted_rewards)
Optimization iterations: 10

The rest of my problem set-up is absolutely the same. But somehow I get completely different results while training, as you can see on the plots above. I also changed learning rates and optimization iterations, gradient clipping, optimizing with mini-batches and $-log(\pi) * A$ as Loss for PPO, but neither helped. Taking importance sampling $\frac{\pi}{\pi_{old}} * A$  as loss for TRPO gives the same results there.
Can someone please help me to understand where could be the problem? Or which parameters I would need to change in PPO?


text:
Why do we use the tree-search version of breadth-first search or A*?
In Artiﬁcial Intelligence A Modern Approach, search algorithms are divided into tree-search version and graph-search version, where the graph-search version keeps an extra explored set to avoid expanding nodes repeatedly.
However, in breadth-first search or A* search, I think we still need to keep the expanded nodes in the memory so that we can track the path from the root to the goal. (The node structure contains its parent node which can be used to extract the solution path, but only if we have those nodes kept in the memory)
So, if I'm right, why do we need the tree-search version in BFS and A*, given that the expanded nodes still need to be stored? Why not just use the graph-search version then?
If I'm wrong, so how do we track the solution path given that the expanded nodes have been discarded?

The problem (and, at the same time, advantage) of graph search is that you use a closed list, where you store nodes. It's a problem because this list can grow, so the space complexity of the graph search can be bigger than the tree search's space complexity. The use of the graph search or tree search will then probably depend on your requirements. See this answer too. So, I guess that the answer to your question is that the nodes of the path will be fewer than all other nodes that have been visited (i.e. the nodes in the closed list).

@nbro Thanks for answering my question! You said that the nodes of the path will be fewer than all other nodes that have been visited. However, I won't know the solution path until I find a solution, so I still need to store all the nodes though only a small part of them are in the solution path. So that's what confuses me most, both the tree-search version and graph-search version need to store all the nodes to find the path.

Hm, right... Thinking about it a little more, I guess that you can retrieve the path lazily without needing to store all information of the nodes of the path in the frontier or closed list. In other words, assuming that, while searching, you keep track of the parent of each node (in the nodes, which you need to have access to), for each path, you can then recursively follow these pointers/links, but, meanwhile, you don't necessarily need to store all information of these nodes in another list (which is not the list of nodes of your graph), such as the frontier or closed list.

authors: Александр Невский


text:
How does NN follows law of energy conservation?
Communication requires energy, and using energy requires communication. According to Shannon, the entropy value of a piece of information provides an absolute limit on the shortest possible average length of a message without losing information as it is transmitted. (https://towardsdatascience.com/entropy-the-pillar-of-both-thermodynamics-and-information-theory-138d6e4872fa)
I don't know whether Neural Network actually deals with information flow or not. This information flow is taken from the idea of entropy. Since I haven't found any paper or ideas based on the law of energy for neural networks. The law of energy states that energy can neither be created nor destroyed. If it is creating information (energy) (e.g. in the case of a generative model), then some information may be lost while updating weights. How is Neural Network ensuring this energy conservation?

Please, you should still focus on a more specific question. Here you're raising several non-trivial issues. We could start from the question: "Is there an information-theoretic view of neural networks", to which you can find the answer here. Your question about the "conservation of energy" is apparently based on the assumption that there's information in the NNs, but you don't even know what that means, so your post could lead to a big answer. So, again, you should probably start by reading that answer, then formulate your question again.

I personally find this to be an interesting question b/c cost of computation is extremely important (as is waste heat in processing.)


text:
Why are shallow networks so prevalent in RL?
In deep learning, using more layers in a neural network adds the capacity to capture more features. In most RL papers, their experiments use a 2 layer neural network. Learning to Reset, Constrained Policy Optimization, Model-based RL with stability guarantees just to name a few - these are papers I personally remember but there are definitely many others.
I came across this question whose answers generally agree that yes, RL using a shallow network is considered deep RL, but the reason for preference of shallow networks was not part of the question.
In MetaMimic (2018), the authors trained the largest neural net RL algorithm at the time (a residual net with 20 convolution layers) for one-shot imitation learning. The paper demonstrates that larger networks as policy approximators generalize better and represent many behaviours.
So, why are shallow 2 layer networks so widely used?

Because the point of the paper is to show performance and efficiency of the proposed algorithm not the approximation capability of neural network. It's only fair to compare algorithms with similar neural network structures and not "artificially" improve results with better network architectures

Thanks @Brale. When doing fair baseline comparisons, similar net structures are necessary. Apart from comparisons, there's generally no other reason to using shallow networks?

Simple policy parametizations, including linear functions in some cases, can solve continous control tasks in RL. It's therefore not necessary to have a complex approximator for the function to be expressive enough in capturing the desired agent's behaviour in popular RL benchmarks

Towards Generalization and Simplicity in Continous Control tries to answer the question "What are the simplest set of ingredients needed to succeed in some of the popular benchmarks?"
Among them is the representation capacity required of the function approximators.
It shows that in popular OpenAI gym locomotion benchmarks (Swimmer 3D,  3D Humanoid, Ant, Walker), using Linear and RBF policy architectures is just as competitive in the reward attained as using neural nets. Here's a summary:

The more Fourier Featurers RBF has, the better the performance
A 2 layer neural net policy's performance is slightly short of a 500-feature RBF
A linear representation is unable to learn some of the locomotion behaviour, e.g walking

I assume RBF, and its use of Conjugate Gradients, has some shortcomings which make shallow Neural networks more preferrable. (I'm not familiar with RBF, and stand to be corrected).
Overall, the the authors use these result to suggest that complex policy architectures should not be a default choice in RL unless side-by-side comparisons with simpler alternatives suggest otherwise.
This might explain the prevalence of shallow networks. It's also possible that since most RL research (presently) doesn't try to show how we can learn better representations using complex policy architectures, but rather how to better make use of representation capacities by addressing issues like sample efficiency, shallow networks are expressive enough in the adapted benchmarks. As expressed in Brail's comment, this is also useful for setting up fair comparisons.
However, the effect of network architecture design in RL has not been widely pursued.
It's also worth mentioning the findings of this work D2RL, (2020) which shows one can gain significant performance improvement by using input concatenations in the policy and value function, and that more layers in a vanilla neural net decreases the agent performance.

The effect of increasing the number of FCs in SAC. Performance drops after 2 layers (D2RL)
Further Reads:
Why is everyone using shallow networks (Reddit thread)

I've keyed in RBF in Google search. Says it's short for "Resting Bitch Face" :-)


text:
How does a GCN handle new input graphs?
Quick questions to see whether I understand GCNs correctly.
Is it correct that, if I have trained a GCN, it can take arbitrary graphs as input, assuming the feature size is the same?
I can't seem to find explicit literature on this.

Graph neural networks, of which GCNs are a specific type, are able to handle arbitrary graphs as input. GNNs operate first over "neighborhoods" of nodes to compute individual node representations and then optionally apply a pooling function to reduce these to a single graph-level representation that can be used in classification. This means that GNNs work "locally" and do not contain implicit assumptions about the graph topology.
For example, the update equation for a given node's representation in a GCN can be written as
$$h_v^{t+1} = \sigma\left({\bf W}^{t+1} \sum_{u \in \mathcal{N}_v} {\bf L}_{uv}~ h_u^t \right) $$
where $h_v^l$ is the representation of node $v$ at update $t$, $\sigma$ is an activation function, ${\bf W^{t}}$ is a weight matrix, ${\bf L_{uv}}$ is the value of the graph Laplacian (which is a matrix) at nodes $u$ and $v$, and finally $\mathcal{N}_v$ is the neighborhood of $v$. Looking at this expression it becomes clear that the value of the summation is always of the same dimension as $h$, no matter how you define the neighborhood $\mathcal{N}_v$. So you are correct that as long as the node representation size is static, the network can take arbitrary graphs as input.
I highly recommend this tutorial paper on GNNs, which I used when first learning about them. Section 2.3 specifically answers your question and discusses how things like cycles and non-positional graphs are handled.


text:
Which machine learning technique can I use to match one set of data points to another?
I have two measuring devices. Both measure the same thing. One is accurate, the other is not, but does correlate with a non-fixed offset, some outliers, and some noise.
I won't always be using the accurate device. The nonfixed offset makes things difficult, but I'm certain there is sufficient similarity to make a link using a machine learning (or AI) technique and to convert one set of numbers to a good approximation of the other.
One is a footbed power meter and gives power in Watts every second. The other is a crank-based power meter, also outputting Watts at 1Hz. The footbed power is much less than the crank (which I know to be accurate), but it does track the increases and decreases in power, just with more noise and, as I say, a non-fixed offset (and by non-fixed I mean, at low power the offset is different to that at high power, I don't mean it isn't consistent, it is consistent). Both measure cadence which may be a useful metric to help find a pattern.
I will be collecting sets of data from both and hoped to plug the footbed data in as a column of values with the crank data as another column representing the truth, so after training, the model would be able to transform the footbed data to an approximation of the crank data.
Anyway, I'm completely lost as to how to begin. I've tried searching, but, clearly, I'm using the wrong keywords. Does anyone have any pointers, please?

OK, so I found the answer - it is to use multiple linear regression. I think this can be marked a solved, but I don't have enough rep to do that.


text:
Is Deep SARSA learning a feasible approach?
I noticed that SARSA has been rarely used in the deep RL setting. Usually, the training for DQN is done off-policy. I think one of the major reasons for this is due to greater sample efficiency in training due to the reuse of experiences in training off-policy. For SARSA, I would think that at every time step of an update, a stochastic gradient update of that sample would have to be performed and then the sample would have to be thrown away.
The approach while might take longer to train, might still allow the agent to do relatively well. Would deep SARSA implementation perform as well as DQN in terms of final performance (since SARSA would definitely take longer to train)?


text:
When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?
When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We want to learn the target policy $\pi$ while behaving according to policy $\mu$. Therefore, for each transition $(s_t, a_t, r_t, s_{t+1})$, we apply the importance ratio $\frac{\pi(a_t | s_t)}{\mu(a_t | s_t)}$.
My question is: if we are training at every step, the behavior policy may change at each step and therefore the transitions of the trajectory $\tau$ are not obtained from the current behavior policy, but from $n$ behavior policies. Why do we use the current behavior policy in the importance sampling? Should each transition use the probability of the behavior policy of the timestep at which that transition was collected? For example by storing the likelihood $\mu_t(a_t | s_t)$ along with the transition?

According to my understanding, you don't use just the current behavior policy for sampling. The importance sampling ratio is calculated as the product of the probability ratios for both the target and behaviour policy throughout the trajectory.
See the calculation below, where the product is happening for all the probabilities throughout the trajectories. (screenshot from Chapter 5, section 5.5 (page 85) of Sutton & Barto)

I'm not sure whether this formulation from Sutton and Barto assumes that all experiences in the trajectory were generated with a behaviour policy when the value function was fixed (for all those experiences in the trajectory) or not. I think that's the crux of the question. If the value function changes while you're collecting experience for the trajectory, then the behaviour policy, which is derived from the value function, may also change (i.e. take different actions).


text:
How to build a Neural Network to approximate the Q-function?
I am learning reinforcement learning with Q-learning using online resources, like blog posts, youtube videos, and books. At this point, I have learned the underpinning concepts of reinforcement learning and how to update the q values using a lookup table.
Now, I want to create a neural network to replace the lookup table and approximate the Q-function, but I am not sure how to design the neural network. What would be the architecture for my neural network? What are the inputs and outputs?
Here are the two options I can think of.

The input of the neural network is $(s_i, a_i)$ and the output is $Q(s_i,a_i)$

The input is $(s_i)$ and the output is a vector $[Q(s_i,a_1), Q(s_i,a_2), \dots, Q(s_i,a_N)]$

Is there any other alternative architecture?
Also, how to reason about which model would be logically better?

Have you heard of Deep Q-learning or Deep Q-Network (DQN)? It may be a good idea to start looking at the DQN paper. In any case, this question seems to be a duplicate of this one.

I had the same question when I first learned RL. The architectural design may depend on the task you're considering. Since you're moving from a tabular Q-learning to function approximation, I'm suspecting that you are considering a relatively small action space; in this case, you should use option 2 where the input is the state and the number of output nodes matches the number of available actions.
One main reason for this choice: when exploiting the learnt function (i.e. policy is $\arg \max_a Q(s,a)$), for option (1) you would have to repeatedly run a forward pass for all available actions, whereas option (2) requires a single forward pass and run arg max either through numpy or the framework you're using.


text:
Is it possible to flip the features and labels after training a model?
The goal of this program is to predict a game outcome given a game-reference-id, which is a serial number like so:

id,totalGreen,totalBlue,totalRed,totalYellow,sumNumberOnGreen,sumNumberOnBlue,sumNumberOnRed,sumNumberOnYellow,gameReferenceId,createdAt,updatedAt
1,1,3,2,0,33,27,41,0,1963886,2020-08-07 20:27:49,2020-08-07 20:27:49
2,1,4,1,0,36,110,31,0,1963887,2020-08-07 20:28:37,2020-08-07 20:28:37
3,1,3,2,0,6,33,83,0,1963888,2020-08-07 20:29:27,2020-08-07 20:29:27
4,2,2,2,0,45,58,44,0,1963889,2020-08-07 20:30:17,2020-08-07 20:30:17
5,0,2,4,0,0,55,82,0,1963890,2020-08-07 20:31:07,2020-08-07 20:31:07
6,2,4,0,0,36,116,0,0,1963891,2020-08-07 20:31:57,2020-08-07 20:31:57
7,3,2,1,0,93,16,40,0,1963892,2020-08-07 20:32:47,2020-08-07 20:32:47

Here's the link for a full training dataset.
After training the model, it becomes difficult to use the model to predict the game output, since the game-reference-id is the only independent column, while others are random.
Is there a way to flip the features with the labels during prediction?

Your question was indeed a little bit unclear. It seems that you wanted to predict the independent features rather than the dependent ones, but it's not fully clear why you want to do that, if your initial goal is to predict the game output (dependent feature). I tried to clarify your post, but I am not sure if it changed the meaning or not. Please, read your post again and try to clarify what you're really asking, after my feedback.

The body of your post seems to be asking a completely separate question than the title of your post, so I will answer both:
"Body: How do I complete the goal of this program?"
Your dataset does not have the dependent variable, which is the outcome of the game (win/loss/draw). What I am assuming is that you have a way of looking up the outcome of the game from either the "id" field or the "gameReferenceId" field.
So you would have to augment the dataset with a new column, "gameOutcome", which has values (win/loss/draw), by looking up the outcome of each game (each row) and adding that to your dataset.
Once you have this, you have the 12 independent variables (the 12 columns already there), and the 1 dependent variable (the "gameOutcome"), and the prediction task should be straightforward from there.
"Title: Given a label, how do I predict features?"
(note: this section will not help your program)
What you are looking for are generative models. Generative models can generate instances given the label and/or some random seed. This is a completely different model than discriminative models, which given the instance predicts the label (this is the one you have).
The simplest generative model is a Naive Bayes model. While Naive Bayes is normally used as a discriminative model (to classify a label), it has enough information to generate instances given a label as well. Here are some tips on how to turn Naive Bayes into a generative model.
If you are looking for generative deep neural network models, this blog by OpenAI has a nice overview explaining the three modern approaches (generative adversarial networks, variational autoencoders, and autoregressive models). The blog explains at a high level what the strengths and weaknesses are of each approach, and links to some open source projects for you to try it out as well.

When a post is unclear or there are inconsistencies between the title's question and the body's question, the best thing to do is to ask for clarification under the post/question before providing an answer (and, meanwhile, flag the post to be closed as unclear to avoid getting all kinds of answers that differ because of different interpretations of the post/question).

Got it, makes sense!


text:
What would be the AlphaGo's performance in continuous action space?
During my research for Google DeepMind's Go-playing program Alpha Go and its successor Alpha Go Zero, I discovered that the system uses a clever pipeline and an interplay of blocks of both policy and value networks to play the game of Go in such a way, that it is able to outperform even the best players in the world. This is in particular remarkable, because the game of Go was considered to be unsolvable a few years ago. This success gained international attention and it was labeled as a breakthrough in the community of AI. It is also not a secret that the research team behind AlphaGo and AlphaGo Zero used lots of computation power to create such a sophisticated system.
But, since each board configuration is considered as a distinct state, where algorithms can be applied really well, and just consider AlphaGo Zero, which uses no prior knowledge and can figure out how the play the game of go from scratch, my question is the following:
Is there any way to state (theoretically) how the performance of AlphaGo would be in continuous action spaces (e.g. self-driving cars)?

The paper A0C: Alpha Zero in Continuous Action Space (although a pre-print) could be useful.

I'm not familiar with reinforcement learning terminology, but are real-time video games (like tetris, breakout, pong, etc.) considered continuous action spaces? Their action space is technically discrete, but each pixel is very small compared to the total field. If yes, you might be interested in reading MuZero, which is AlphaZero's successor (AlphaZero is AlphaGo Zero's successor). MuZero generalized AlphaZero to play Atari games, which are real time video games. It achieved state of the art. https://arxiv.org/pdf/1911.08265.pdf

Thanks @nbro & @user3667125!
I'll check that out.


text:
Why shouldn't batch normalisation layers be learnable during fine-tuning?
I have been reading this TensorFlow tutorial on transfer learning, where they unfroze the whole model and then they say:

When you unfreeze a model that contains BatchNormalization layers in order to do fine-tuning, you should keep the BatchNormalization layers in inference mode by passing training=False when calling the base model. Otherwise the updates applied to the non-trainable weights will suddenly destroy what the model has learned.

My question is: why? The model's weights are adapting to the new data, so why do we keep the old mean and variance, which was calculated on ImageNet? This is very confusing.


text:
Aside from specific training sets, what distinguishes the capabilities of different AI implementations?
(Disclaimer: I don't know much about ML/AI, besides some basic ideas behind it all.)
It seems like ML/AI models can often be boiled down to statistics, where certain levers (weights) get fine-tuned based on the specific input of a large set of training data.
Clearly, ML/AI models don't only distinguish themselves in their training data alone, otherwise there would not be so many improvements happening in the field all the time. My question therefore is: What does distinguish different models of the same category?
If I have an AI that completes real-life pictures that have some missing parts, and an AI that completes a painting with missing parts, what key concepts separates the two?
If I have an AI detecting text in an image, and an AI detecting... trees in an image, what key concepts separates the two?
In other words, what is stopping me from "taking" an existing implementation of a certain AI category, and just feeding it my specific training set + rewards (i.e. judgement criteria for good vs bad output), in order to solve a specific task?
In yet again other words, if I wanted to use ML/AI to build a new model for a specific task, what concepts and topics would I need to pay extra attention to? (I guess you could say I'm trying to reverse engineer the learning process of the field here. I don't have the time to properly teach myself and become an "expert", but find it all very interesting and would still like to use some of the wonderful things people have done.)

I don't fully understand what you mean by this "ML/AI models don't only distinguish themselves in their training data alone". Are you saying that ML models are not only different in terms of the data they need to deal with (in terms of inputs and outputs)? This post is a bit problematic, because, even though you do not realise it, you're asking us to describe what machine learning is and which types of machine learning are there, from what I understand.

If you're looking for different types of machine learning technique, maybe you should start from this or this answers. I suggest that you read those answers carefully, then try to narrow the scope of this post. Right now, it's broad, because essentially you're asking us how to solve problem with machine learning. I suggest that, as an example, you focus on a specific problem that you would like to solve, then ask a more specific question.

Moreover, note that AI and ML are not the same thing. ML is a subfield of AI concerned with learning from data, as you will understand after having read the answers I linked you to, which you should really read before proceeding.

@nbro Thank you for your answer. I read the answers you linked, and think that I was already aware of those "generalities"; What I'm trying to ask, I guess, is, for a GIVEN model, in what steps of a complete algorithm do two different tasks differ. For example, take two image inpainting tasks, getting rid of wrinkles and removing an object. Both will use a similar model/approach, but what I want to know is what makes them different: Say, if I were given one of the two algorithms, what would I have to change to produce the other?

But there are so many possible differences between a model/solution and another. For example, the number of layers, the number of units, the optimizer, etc., etc., that you use in one case could be different than in another. Or maybe the model is completely different, the data is different. So, I think you should really focus on a more specific problem and ask a more specific question, because, right now, I don't think this question can be reasonably answered.

If I understand what you mean correctly then the answer is basically nothing. Fundamentlaly all ML algorithms are ding the same thing, which is to optimize some weights for a certain output (this is true even for non-parametric methods in an implict way, but lets not dive too deep here). The only differences are:

Dataset the model is trained on
Specific dimensionality of inputs and outputs to make them compatible with the input data and the output labels.
Complexity that can be expressed by the model (the number of weights and their structure in layers in the case of Neural Networks).
Changes to the optimization process (gradient clipping, regularization, ecc.)
Structural changes to the architecture that embed assumptions about the specific problem setting (e.g. 2D-Convulutions embed assumptions for images, softmax activations embeds the assumption for classification with probabilities, hidden states embed the assumptions about memory and "fogetting", ecc...)

Therefore, if you have a "similar" problem setting where you can assume that 3, 4, 5 can be the same without problems, then you can just make the appropriate changes in 2 and change the dataset (1) to get a model that works on something different.
Of course, being able to tell how "similar" a problem is to another and what possible things could be different is quite tricky and requires a lot of knowledge about the domain of Machine Learning and how every algorithm works and why.
In essence, I'm saying that in principle you could take a model and train it for a new setting with very minor and simple changes that don't require a lot of knowledge. However, without the wider knowledge on the field of ML/AI you won't be able to tell if what you are doing is ok and how well can it work in general.

Ah, interesting, thank you for the answer. So I'm assuming that I can't really go around learning the details of each of these steps in order to judge where I can "order" my task into? Do you have a recommendation on what learning approach would be best?


text:
When we translate a text from one language to another, how does the frequency of various POS tags change?
When we translate a text from one language to another, how does the frequency of various POS tags change?
So, let's say we have a text in English, with 10% nouns, 20% adjectives, 15% adverbs, 25% verbs, etc., which we now translate to German, French, or Hindi. Can we say that in these other languages the POS tag frequency will remain the same as earlier?