text:
Why doesn't VAE suffer mode collapse?
Mode collapse is a common problem faced by GANs. I am curious why doesn't VAE suffer mode collapse?

With Generative Adversarial Networks, all the generator cares about is fooling the discriminator. There's no requirement to be clever, or exhaustive, or make efficient use of the input space. As long as the discriminator returns "real" (vs. "fake") the generator "wins".
The hope is that as the generator and discriminator are trained simultaneously, each will attempt to capture the faults of the other. As the discriminator gets better for one mode in telling real from fake, the generator will focus more on the other modes where it will have an easier time fooling the discriminator on.
But that isn't the only way things can happen. When the discriminator gets better on distinguishing one mode, the generator can instead concentrate on that mode. Instead of spreading its training over different modes, it can instead attempt to get really good at producing samples within a particular mode. As long as the discriminator can be fooled into reporting "real" for the samples being produced, the generator gets the same reward as if it were to spread itself across multiple modes.
Think about the forger/detective analogy. Instead of a forger trying to learn how to be good at making a range of paintings (Caravaggios, van Goghs, Warhols, etc.) the forger can concentrate on being really good at making a particular style of painting (e.g. Dutch Masters) or even a particular artist. Someone who spends all their time perfecting their Rembrandt forgeries will do a much better job evading detection than someone who tries to be good at both Picasso and Rembrandt (and Klee and Miró and ...)
Variational Autoencoders are different, though. With a VAE, the latent space is built from all of the input examples. There's no way for a VAE to concentrate on a particular mode. All of the training examples are being mapped to the same latent space. If, for some reason, the decoder network decided to concentrate on a single mode, mapping the latent space to outputs of just that mode, then all the training examples from the other modes would have poor recovery. This means there would be a strong loss and gradient correcting that for the decoder training. The decoder training is balanced across all the input examples/modes, so the decoder needs to be similarly balanced when decoding the latent space. GANs don't have this similar balance, as the generator doesn't necessarily need to be balanced in its output - just so long as it can reliably fool the discriminator.
That doesn't mean that VAEs don't have similar issues, though they're not identical to mode collapse. The problem with VAEs is that each individual example is (in extremis) being mapped to the same random distribution in the latent space. This can encourage the decoder to ignore the latent variable input, generating output more-or-less arbitrarily. There's a number of approaches recently (in particular Adversarial Autoencoders as well as InfoVAE and the closely related Wasserstein Autoencoders) which attempt to remedy this by constraining the latent space distribution on an ensemble level, rather than on an individual training example basis.

Exactly. Vanilla vae collapse to a constant variational posterior too often


text:
Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?
I observed in several papers that the variational autoencoder's output is blurred, while GANs output is crisp and has sharp edges.
Can someone please give some intuition why that is the case? I did think a lot but couldn't find any logic.

The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you should know the result -- VAE can only have a mean and blurry output.
If you increase the bandwidth of the bottleneck, i.e. the size of latent vector, VAE can get a high reconstruction quality, e.g. Spatial-Z-VAE

I don't understand why this answer is correct. GAN also requires a low-dimensional vector to generate images.

In essence, Variational Autoencoders learn an "explicit" distribution of the data by trying to fit the data via a multi-dimensional Gaussian/Normal distribution.
However, Generative Adversarial Networks learn an "implicit" distribution of data meaning that you cannot directly sample them.
Also, due to the deterministic nature of neural networks, GANs tend to learn a Dirac Delta function. If you're lucky and the training of the GAN is successful, you can therefore get sharper images, since the model doesn't have to explicitly deal with the noise injected into it due to samplings, hence this could be a simpler learning problem.
By deterministic, I mean assuming that you have no sampling anywhere in the middle layers of your model and only use the neural network as an input-output mapping function.

Can you share some sources for your statement "deterministic nature of neural networks GANs tend to learn a Dirac Delta function". It is quite interesting and I would like to look more about it

Nice comparison of VAE and GAN. I understand that GAN is trained differently, but what about blurriness of the output of Variational Autoencoder? Why is it so hard to VAE to produce sharper images?

@samutamm, as it is mentioned, VAE learnt the distribution of the data by fitting it into a multi-dimension normal distribution. This results at having the generated samples from VAE models to be blurry because of the independence assumption of the samples given latent variables.
Ref: Page 1 in https://openreview.net/pdf?id=B1ElR4cgg
"...suffer from a well-recognized issue of the maximum likelihood training paradigm when combined with a conditional independence assumption on the output given the latent variables: they tend to distribute probability mass diffusely over the data space.."

The reason is because of L1 (or L2) reconstruction loss used in VAEs.
As is discussed in Image-to-Image Translation with Conditional Adversarial Networks: " If we take a naive approach and ask the CNN to minimize the
Euclidean distance between predicted and ground truth pix-
els, it will tend to produce blurry results. This is because Euclidean distance is minimized by averaging all plausible outputs, which causes blurring". Further: "  GANs learn a loss that tries to classify if the output image is real or fake, while simultaneously training a generative model to minimize this loss. Blurry images will not be tolerated since they look obviously fake."
For further details read the ablation study in 4.2 of linked paper.

The reason is simple: because VAE's & GANs are almost opposite in their strengths and weaknesses.
VAE's (even VQ-VAEs!) always introduce some kind of noise to their encoder's output which has two consequences: it makes the latent space interoperable which increases generation diversity (the goal), but the added noise introduces some error/blur (unintended side-effect).
In contrast GAN's always uses adversaries to detect bad (e.g. blurry) images which has two consequences: the generated fake images are nearly indistinguishable from real images (the goal), the generator tends drop modes (aka reduce diversity) to make fooling the adversary easier (unintended side effect).
P.S. for VQ-VAEs: The noise in introduced by the vector-quantization & limited codebook size which causes a many-to-one (i.e. noisy) mapping.

Fundamentally speaking, the optimization target of a VAE has two parts: minimizing reconstruction loss (similar to an AutoEncoder) and minimizing KL divergence loss (ensuring that the latent z follows a normal distribution so we can sample from it). However, these two objectives are antagonistic. The former aims to clearly distinguish different classes in the latent space to facilitate the final reconstruction, while the latter aims to mix different classes in the latent space to conform to a standard normal distribution. Therefore, after optimization, the images generated by a VAE are unlikely to be very clear and sharp.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Do GANs come under supervised learning or unsupervised learning?
Do GANs come under supervised learning or unsupervised learning?
My guess is that they come under supervised learning, as we have labeled dataset of images, but I am not sure as there might be other aspects in GANs which might come into play in the determination of the class of algorithms GAN falls under.

GANs are usually trained in a self-supervised fashion, i.e. they use the unlabelled data as the supervisory signal. Note that some self-supervised learning methods are unsupervised learning techniques, given that no human-annotated data is needed. However, not all SSL techniques are used for solving an unsupervised learning task. In fact, there are SSL techniques that are specifically used to generate labeled data, which can then be used to train a model in a supervised fashion.

The terms Supervised Learning and Unsupervised Learning predate the invention of the application of artificial networks to a generative and discriminative network pair, which was the first popular generative topology.  The existence of labeling is the key distinction between the two.  Even partial labeling indicates supervision, as odd as that jargon is, since the supervisor does no learning and the labels are constants.
Based on the original description of the discriminative network in a GAN, that it consumes examples without labels, GANs are a type of unsupervised learning.  That fact does not eliminate the use of labels as part of an extension of the original design that generates based on some labeled element in the examples or the use of other labels to indicate the fitness of each image for a class into which generated images are expected to fall.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How do I predict if it is rainy or not?
I'm building a weather station, where I'm sensing temperature, humidity, air pressure, brightness, $CO_2$, but I don't have a raindrop sensor.
Is it possible to create an AI which can say if it's raining or not, with the help of the given data above and maybe analyzing the slope from the last hour or something? Which specific technology should I use and how can I train it?

I think this question would be better suited for https://stats.stackexchange.com/, even though, there, this question may sound too basic, given that this question is about "prediction"/"inference", in general. There you will find a lot more competent people that can answer this question. But, honestly, this is the type of questions that I would like to see more on this website, because they are actually useful.

Anyway, just to give you an idea. What you're sensing, e.g. the temperature, the humidity, are usually called, in the context of machine learning (and AI, in general), features. In statistics, these are called variables. These variables can be independent or confounding. Anyway, the idea is to combine these variables in certain way, so as to create an output which needs to correspond to the output of the raindrop sensor. Which model to use and how to deal with your variables, it's more technical, and sometimes the choice of model, etc., requires a little domain knowledge.

This sounds like a great project, although this exact setup limits your options somewhat.  Supervised machine learning approaches are effectively ruled out because you don't have the necessary training data to develop a model (i.e. the dependent variable: whether it is raining or not).  You could look at accessing similar data (from what source depends on your geographical location) that includes precipitation information, and attempt to train a model based on that, before applying it to your own data to 'predict' the current conditions.
If you want a model to determine whether it's raining or not (rather than attempting to define the type, rate or other characteristics of precipitation) then you need a binary classification model.  If you can get the data to train a supervised model, there are many such models available.  Which one to choose depends on a number of factors – enough factors that it is as much an art as a science (and often the best way to choose is just to try several and pick the best one).  XGBoost is a popular choice for a wide range of problems, including binary classification.  State Vector Classifiers (SVCs, a subtype of State Vector Machine, SVM) are also common.  Of course there are neural networks also.  Depending on how the dependent variable correlates with the independent ones, you may find a simpler model to be more effective.  If you choose to try a neural network, I suggest keeping it simple and starting with a Multi-Layer Perceptron (MLP).
Alternatively, you could try an unsupervised approach, which commonly takes the form of clustering.  In this instance, you would look to try clustering algorithms, such as DBSCAN, to see if the independent variables naturally form groups (clusters) which differentiate between when it is raining or not.  Of course you can cluster the data using an unsupervised model, but you would still need to then correlate that with rainfall in order to make sense of the model.
There are a number of challenges you may need overcome.  For example, depending on the frequency with which the data is sampled and stored, you may need to downsample it to a manageable size.  Presumably you also have the time at which the data was collected.  If not, you would also need to make use of the brightness sensor to differentiate between night and day, which may confound or over-power any cooling effect from rainfall.  (You could look at clustering the data, based on temperature and brightness, in order to create a new feature which indicates night and day.)
To summarise, there are numerous approaches you could take, all of which will, at some point, require data on whether it is/was raining or not in order to train a model.  If you have suitably localised weather data available, you may be able to train a model using that, synchronised via time and/or the other data you'll be collecting.  The problem, as stated, is a binary classification one (as opposed to a time series prediction one, i.e. weather forecasting, for example).  So look into classification and clustering models for supervised and unsupervised machine learning respectively.  Scikit Learn for python or Caret for R are commonly used packages for the types of models I've described.


text:
Are all ant routing algorithms the same?
Are all ant routing algorithms the same? If no, what is the common properties of all of them? In other words, how we can detect a routing algorithm is an ant routing algorithm?


text:
How Dempster-Shafer theory work in AI?
How does Dempster-Shafer theory work in representing ignorance in the AI field?

Dempster-Shafer Theory was studied fairly seriously in AI through the 70's and 80's. However, Dempster-Shafer theory has some serious shortcomings, that will cause an agent that uses it to make irrational decisions. These were uncovered by Pearl and others in the 80's, with more problems emerging in later years.
See Cheeseman for a summary of the arguments that probability is the only suitable framework for reasoning under-uncertainty in AI (which is the current orthodox view), along with Josang & Harkin 2012 (which develops a theory that subsumes Dempster-Shafer, and under which DS can be shown to lead to poor decisions), and Pearls' On Probability Intervals.


text:
Is there an efficiency swarm Intelligence algorithm for off-chain channels routing in blockchain?
One of the solutions to scale blockchain is to use off-chain channels. You can find its definition here: https://en.bitcoin.it/wiki/Off-Chain_Transactions.
However, one of the problems of off-chain channels is finding a suitable decentralized routing mechanism.
Since in Bitcoin there is no routing table and transactions are only broadcast and also, in general, we need to avoid centralized approaches for routing, is it practical to use swarm intelligence algorithms, such as ant colony optimization ones, for off-chain channels?
I refer you to a proposed ant routing algorithm in the paper Ant routing algorithm for the Lightning Network for an instance of employing ACO algorithms for routing in _lightning network. However, the paper has not been evaluated to demonstrate its performance.


text:
How is the cost of the path to each node computed in A*?
How is the cost of the path to each node $n$ computed in the A* algorithm? Do we need to add the cost of the path to the parent node $p$ to the cost of the path of the child node $n$?

The evaluation function in A* is $f(n) = g(n) + h(n)$, where $g(n)$ is the cost of the path from the starting node to $n$ and $h(n)$ is an estimate of the distance from $n$ to the goal node. To compute $g(n)$, you simply do $g(n) = g(p) + c(p, n)$, where $p$ is the parent node, and $c(p, n)$ is the $c$ost of the edge between $p$ and $n$. So, yes, to compute $g(n)$ you use $g(p)$.


text:
Why do we apply the mutation operation after generating the offspring?
Why do we apply the mutation operation after generating the offspring, in genetic algorithms?

The mutation operation is (usually) needed to introduce new genes not found in the population.
For example, suppose that you have 4 possible genes $A$, $B$, $C$, and $D$, and that your chromosomes have a non-binary encoding. In that case, if no member of your population has the gene $D$, then no amount of crossover operations will result in the introduction of that gene.
However, if your chromosomes use a binary encoding, then new genes could be introduced as a side effect of crossover operations. But it is always safer to have some kind of mutation to ensure that all genes can be accessed.
See also the paper Genetic Algorithm for Traveling Salesman Problem with Modified Cycle Crossover Operator, which mentions several non-binary encodings for the TSP.

Mutation is used to maintain diversity in the solutions. Crossover alone cannot do this.

This might be correct, but you should provide more details about why that is the case, because it's not obvious why the cross-over alone doesn't introduce diversity. In fact, it does, but to what degree? The other answer actually provide an example of when the mutation could be useful, but you should also extend this answer to provide more details. Right now, this answer looks like a comment, so it should be converted to a comment.


text:
Why does the hill climbing algorithm only produce a local maximum?
Apparently, the hill climbing algorithm just produces a local maximum, and not necessarily a global optimum. It's stuck on a local maximum. Why does hill climbing algorithm only produce a local maximum?

This part of your sentence is not always true "and not reached to final goal/solution". If you have just one maximum at all and it is finite, hill climbing (HL) can reach to it and it is a global maximum too (for example, if the function is a parabola).
To answer your question is back to the stop criteria of HL. It will be stopped when reaching to a maximum point. The maximum point means a point which the objective function has a greater value in that point than on its $\epsilon$-neighborhood of that point. Therefore, if objective function has some local maxima, depends on the initial point of HL, it might be trapped in one of the local maxima as their definition is satisfied in the explained definition.
In sum, depends on the initial point of HL, it might not reach to the global maxima and trapped in local maxima if there exists.

authors: НИКОЛАЙ  КЕРЕМЕДЧИЕВ


text:
Are these steps to get a final linear regression model correct?
I am new to machine learning. I know Logistic Regression (LR) is a supervised learning technique. Therefore, we need training data to train the model.
I tried to understand the basic steps to get the final RL model.
According to my understanding, here are the steps.

We define the LR model, that is, $y = \text{sigmoid}(W x + B)$. Set $W$ and $B$ to zero or another value.

Given the training data (the inputs are $x_1, x_2, \dots, x_m$, and the outputs are $y_1, y_2, \dots, y_m$), we find $W$ and $B$ values by minimizing a cost function using gradient descent.

Then we use the found $W$ and $B$ values. We then again apply a known sample from the training data $\hat{x}$ to get the predicting of $\hat{y}$, that is, $\hat{y} = \text{sigmoid}(W \hat{x} + B)$.

We test the final model on unknown data.

Are these steps correct?
Please, if you could also give me the basic idea behind the supervised technique, I would appreciate it.

In general these steps are correct, but are few clarifications would be good. For supervised learning, you can:

pick model to use
split data into training and test set
train model, or determine weights by optimization of our error function (inputting in training data)
use trained model to predict on test data
utilize metrics of interest to evaluate our model (e.g. MSE for continuous outcomes, AUC for binary)

This however is a very elementary pipeline to train a supervised machine learning model. There are many other steps taken to ensure our model is as robust as possible (like cross validation, feature selection, hyperparameter optimization, model selection...). I would strongly recommend learning about these other steps and seeing how they integrate with the above pipeline.

For unsupervised learning, the process depends a lot on the task, but in general it involves optimizing some objective function using the data provided. This is usually done using all the data in one go (not split into training and test) unlike supervised learning. This can be seen more clearly with examples, take methods like PCA and k clustering; the steps taken to apply these methods very much depend on the method used.

Going further, I would recommend Andrew Ng's coursera course on machine learning. It is very accessible and a great introduction to this field, covering both unsupervised/supervised learning.


text:
How to improve the efficiency of the backtracking search in CSPs?
Backtracking search is the basic uninformed search algorithm for constraint satisfaction problems (CSPs). How could we improve the efficiency of the backtracking search in CSPs?

Backtracking algorithms can be improved by adding heuristics methods.
You could add filtering and ordering techniques.

@Cecelia - If you have an answer, can you please post it as an answer below instead of as a comment? Thank you!

This question may do better on or.stackexchange.com


text:
Can intelligent agents have personalities and emotions?
Can intelligent agents (and chatbots) have personalities and emotions, given a properly defined ontology?

It can depend on how you define "emotion" and "personality":

Personality

Wikipedia defines personality as "the characteristic set of behaviors, cognitions, and emotional patterns that evolve from biological and environmental factors."  (This is a serviceable definition we can use.)
Since intelligent agents make decisions (take actions), it is safe to say they have a set of behaviors.  Cognition is easy here as well--regardless of the structure of the algorithm, we can say it utilizes a logical process that serves the function of cognition.
Emotion is more difficult.

Emotions

How exactly human emotions work is still partly obscure, and it seems to be a safe bet that chemicals are involved.  But, in a reductionist sense, going back to Ancient Greek Philosophy, we can use a binary "love/hate" (attraction/repulsion) and apply it generally, where these feelings are the prime motivators of natural forces:

And Hesiod says, “First of all things was Chaos made, and then/Broad-bosomed Earth . . . and Love, the foremost of immortal beings,” thus implying that there must be in the world some cause to move things and combine them.
Aristotle (Metaphysics 1.984b)

The real issue comes with interpretations of emotion that involve consciousness, such as on Wikipedia, but even the wiki immediately issues the disclaimer: "Scientific discourse has drifted to other meanings and there is no consensus on a definition," so it may be more useful to look to definitions rooted in cognition and the presumed utility of emotions from an evolutionary standpoint.
If reinforcement learning is being used, an automata can be said to like/dislike (love/hate) certain outcomes, which can be expressed via methods such as reward functions.
Certainly this is not emotion as we humans understand it, but it can be seen as an analog. (Regarding humans chemical response to entities/objects and events/outcomes as reinforce future decisions.)
ANSWER:
In no way do contemporary bots and agents have personalities and emotions in the sense of humans--we are probably still very from this and don't know if it's even possible. It may never be possible validate artificial consciousness, regardless of how conscious a system appears to be.
But, depending on how we define these terms "personality" and "emotion", we can choose to regard automata as having some reduced form of analogous qualities and functions.
Example: If I engineer a game AI to have certain tendencies and behaviors (high-risk-tolerance, partisan, aggressive) that are distinct from other agents (low-risk-tolerance, cooperative, defensive), such agents are commonly referred to as having personalities.
Example: We actually do use terms like "goodness/badness" and informally refer to the automata as "liking" or "not liking" nodes in a graph.  This is a function of anthropomorphization.  (Whether you consider these more than analogies depends on the definitions of feelings, but, generally, there needs to be something to feel. So you also need to reduce consciousness/awareness to the most basic requirement of simply "receiving of input".)


text:
How to create a good fitness function?
In genetic algorithms, a function called "fitness" (or "evaluation") function is used to determine the "fitness" of the chromosomes. Creating a good fitness function is one of the challenging tasks in genetic algorithms. How would you create a good fitness function?

No magical formula
As already stated in this answer, the definition of the fitness function depends on the problem, given that it essentially determines the solutions that you are looking for, and it raises similar issues to the ones you would encounter while defining a reward function in reinforcement learning, such as fitness misspecification (in fact, the concepts of a reward function and fitness functions are similar, although, in some cases, a fitness function is more similar to a cost function in supervised learning problems: as an example, take a look at the fitness functions used to solve symbolic regression). The design of a fitness function is an engineering problem, in the sense that you need to think about the solutions you are looking for, and what represents a good and bad solution. So, I cannot give you the magical formula to define the fitness function for all problems, but I can give you some info that can be useful to guide the design of a fitness function.
What is a fitness function?
Let's start with the definition. A fitness function is a function that maps the chromosome representation into a scalar value (a real number) that quantifies the quality of the chromosome (i.e. solution), so it's a function of the form
$$
f : \Gamma \rightarrow \mathbb{R},
$$
where $\Gamma$ is the space of chromosomes (where a chromosome is a numerical encoding of a solution for your problem that is suitable for evaluation and modification).
Different types of optimization problems
There are different optimization problems that affect the definition of the fitness function [1], such as

Unconstrained: this is the simplest case where the fitness function corresponds to your objective function
Constrained: in this case, your fitness function could be composed of two terms: the original objective and a penalty term (that penalizes solutions that do not satisfy the constraints)
Multi-objective: where you have multiple fitness functions (one for each objective) and the final fitness function is a combination of these multiple fitness functions; in this context, you will often encounter terms like Pareto-optimality
Dynamic (or noisy), where the fitness of solutions can change over time or depend on some noise component (e.g. Gaussian noise)

Encodings and fitness functions
In genetic algorithms, a form of evolutionary algorithms, the chromosomes are often assumed to be binary (i.e. $\Gamma$ is a space of binary arrays), so this can limit the way you can evaluate them.
In other evolutionary approaches, the solutions may be encoded differently and represent something different than just a collection of numbers. In particular, in genetic programming, the solutions are programs, so the fitness should correspond to something that the program you are looking for is supposed to accomplish. For example, if you are looking for an analytical expression (e.g. a polynomial) that minimises the squared error with another expression, then the fitness would be e.g. the squared error.
See section 10.3 (p. 180) of [1] for more details.
Cheap fitness evaluation
One of the most desirable properties that you should look for while designing a fitness function is how cheap it is to evaluate the fitness of an individual. Ideally, the fitness evaluation should be quite cheap in order for the evolutionary algorithm to be feasible and practically useful. If it takes 1 year to perform an evaluation of an individual, then you don't get anything done. Alternatively, you can approximate the fitness of an individual, if the computation of the exact fitness is too expensive.
Absolute and relative measures
The fitness function can provide an absolute or relative (to other chromosomes in the population or other competing populations) measure of the quality of a chromosome (or solution). Relative fitness functions are used in co-evolutionary algorithms and are suited for situations where an absolute measure of the quality of a solution is not possible [1].
Fitness sharing
There is also the concept of fitness sharing (see section 9.6.1 of [1], p. 165) where the fitness of an individual can be adjusted based on the fitness of other individuals
Further reading
Apart from the [1], which has several chapters on evolutionary computation, you could take a look at Evolutionary Computation 2: Advanced Algorithms and Operators. The paper Fitness functions in evolutionary robotics: A survey and analysis (2009) also seems useful from the title and abstract (and other parts that I quickly read).

In my experience, the fitness function is a way to define the goal of a genetic algorithm. It provides a way to compare how "good" two solutions are, for example, for mate selection and for deleting "bad" solutions from the population.
The fitness function can also be a way to incorporate constraints, prior knowledge you may have about the shape of the fitness landscape, or the way your crossover/recombination operators will work in that fitness landscape.
For example, the fitness function can include hard constraints like "Genes x,y, and z must all stay on one side of the surface $Ax +By +Cz = k$" by assigning the fitness value at zero if the gene values are on the wrong side of the surface.  However, it's often better in a case like that soften the boundary by assigning a fitness penalty that is zero at the surface and grows larger as the gene values move farther from the surface on the wrong side of the surface.
Different fitness functions can be used for mate selection vs deleting "bad" trial solutions.  For example, "mating fitness" between two potential parents A and B can be a function of how different the two parents are.  By providing a mating advantage to pairs that are significantly different, the population can be forced to remain fairly diverse and thus explore a larger region of solution space, or to avoid converging to local (sub-optimum) fitness maxima.  Meanwhile, the usual kind of fitness will cull the low-fitness individuals from the population and drive evolution toward high fitness.
What is often much more important is the set of variables ("genes") used to represent a trial solution, how the genes are arranged in the "chromosome", and the ways genes from two parents can be combined to form a new trial solution. Since you didn't ask about those things I won't go into detail in this answer, but if you ask in a separate question I will provide a detailed answer.

Maybe you should provide references to papers that describe specific examples (just for reliability), but, after the first reading, this answer seems reasonable. I suggest that you take a look at the book "Evolutionary Computation 2: Advanced Algorithms and Operators". There you have some info that could support some of your claims. You could also talk about multi-objective problems.

To be perfectly frank, I have not looked at a paper about evolutionary computation in a LONG time.  My answer was based entirely on about 30 years' experience creating and using genetic algorithms.  I would be willing to try to answer a question about multi-objective problems if it were asked separately, but one of the first things I learned in a Calculus of Variations class many, many years ago was that it is in general not possible to simultaneously maximize multiple independent functions.  The route to success is to find reasonable ways to couple or constrain the objective functions.


text:
What kind of distributions can be used to model discrete latent variables?
If we take the vanilla variational auto-encoder (VAE), we $p(z)$ is a Gaussian distribution with zero mean and unit variance and we approximate $p(z|x) \approx q(z|x)$ to be a Gaussian distribution as well, for each latent variable $z$.
But what if $z$ is a discrete variable? What kind of distributions can be used to model discrete latent variables? For example, what kind of distribution can be used to model $p(z)$ and $p(z|x) \approx q(z|x)$?


text:
How to generalize finite MDP to general MDP?
Suppose, for simplicity sake, to be in a discrete time domain with the action set being the same for all states $S \in \mathcal{S}$. Thus, in a finite Markov Decision Process, the sets $\mathcal{A}$, $\mathcal{S}$, and $\mathcal{R}$ have a finite number of elements. We could then say the following
$$p(s',r | s,a) = P\{S_t=s',R_t=r | S_{t-1}=s,A_t=a\} ~~~ \forall s',s \in \mathcal{S}, r \in \mathcal{R} \subset \mathbb{R}, a \in \mathcal{A}$$
where the function $p$ defines the dynamics of the finite MDP and $P$ defines the probability.

How could I extend this to a general MDP? That is, an MDP where the sets $\mathcal{A}$, $\mathcal{S}$, and $\mathcal{R}$ haven't a finite number of elements? To be more precise, in my case $\mathcal{A} \subset \mathbb{R}^n$, $\mathcal{S} \subset \mathbb{R}^m$, and $\mathcal{R} \subset \mathbb{R}$. My thought is that the equation above is still true, however, the probability is zero for each tuple $s',r,s,a$.
Is it sufficient to say that for finite MDP we have
$$\sum_{s'\in\mathcal{S}}\sum_{r\in\mathcal{R}}p(s',r|s,a)=1 ~~~ \forall s\in\mathcal{S},a\in\mathcal{A}$$
while in non-finite MDP (supposing that the sets $\mathcal{s}$ and $\mathcal{A}$ are continuous) we have
$$\int_{s'\in\mathcal{S}}\int_{r\in\mathcal{R}}p(s',r|s,a)=1 ~~~ \forall s\in\mathcal{S},a\in\mathcal{A}$$
or is it more complex than this?


text:
Why are Q values updated according to the greedy policy?
Apparently, in the Q-learning algorithm, the Q values are not updated according to the "current policy", but according to a "greedy policy". Why is that the case? I think this is related to the fact that Q-learning is off-policy, but I am also not familiar with this concept.

Short answer
The Q values are updated using a greedy policy because, in the Q-learning algorithm, the $\max$ operator is used to determine the target, which is denoted by
$$\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$$
Intuitively, the $\max$ operator is used because we assume that the target policy (the policy associated with the optimal value function that we want to learn) takes a greedy action, which is defined, in this context, as the action associated with the highest Q value: $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ means that we are selecting the $Q$ value, associated with the (next) state $S_{t+1}$, which corresponds to the action $a$, such that $Q(S_{t+1}, a)$ is the highest (with respect to other possible actions from $S_{t+1}$).
Note that the $Q$ function receives as input a state and an action. So, for each state $s$, we have an action (among all possible actions, $a_1, a_2, \dots$, which we can take from the state $s$), denote it by $a^*$, such that $Q(s, a^*) > Q(s, a_1)$, $Q(s, a^*) > Q(s, a_2)$, etc. In the expression $\color{blue}{\max_{a}Q(S_{t+1}, a)}$, we are basically selecting $Q(s, a^*)$ for $s = S_{t+1}$.
More explanations
To explain all the components of this target, I will explain the Q-learning algorithm. Hopefully, after this explanation, you will be able to understand why Q-learning uses the greedy policy to update the Q values and why it is an off-policy algorithm (which is, IMHO, a quite unintuitive and confusing term to describe what off-policy actually means).
How does Q-learning work?
Here's the Q-learning algorithm

The Q-learning algorithm estimates a value function, known as the Q-function, associated with a policy $\pi$. Intuitively, it does that by simulating an agent which takes actions in the environment, observes the impact of those actions on the environment in terms of the received rewards and the new states where the agent ends up in after taking those actions. Meanwhile, during this exploration of the environment, it attempts to estimate the optimal $Q$ function (i.e. the value function associated with the optimal policy, which, if followed, will give the agent the highest amount of reward in the long run, aka return).
Q-learning proceeds in episodes. So, initially, you need to pass the number of episodes as input. You can think of episodes as iterations (like in any iterative optimisation algorithm). However, in the context of RL, an episode is a little bit more specific: the start and end of an episode are associated with specific states of the environment: the episode starts when the agent is in a starting state $S_0$ (which can be sampled from a probability distribution over $S_0$, if there is more than one) and ends when it is in a terminal state.
In the pseudocode above, at the beginning of each episode, we initialise $t=0$, where $t$ represents the time step of a specific episode.
Inner/Episode loop
We then have the following loop, which terminates when the agent reaches a terminal state:

So, at each episode, we run the loop above. The block of code inside this loop contains the main logic of the Q-learning algorithm.
Behaviour policy
On each iteration of this inner loop, the agent chooses an action $A_t$ (the action at time step $t$ of the current episode) using a policy (which is known, in this context, as the behaviour policy, which should ensure that all states are sufficiently visited, in order for tabular Q-learning to converge). In this case, the $\epsilon$-greedy policy is used.
How does this $\epsilon$-greedy policy work?
If you look at the pseudocode above, $\epsilon$ is initialised at beginning of each episode. In the pseudocode above, $\epsilon$ can change from episode to episode, but assume, for simplicity, that, at every episode, it is a fixed small number (e.g. $0.01$). The statement

Choose action $A_t$ using policy derived from $Q$ (e.g., $\epsilon$-greedy)

means that, with probability $1 - \epsilon$, the greedy action is chosen, and, with probability $\epsilon$, a random action is taken.
What is the greedy action in this case?
In this case, the greedy action is the action, in the current state $S_t$, which is associated with the highest Q value (given the current estimate of the Q value). It is exactly the same action as the action $a^*$ (as I explained above). The difference is that, in this case, we choose $A_t$ using the $\epsilon$-greedy policy: so, most of the time, we choose the greedy action, but, sometimes, we can also choose a random action.
The agent then executes the just chosen action $A_t$ in the environment, and it observes the impact of this action on the environment, which is determined by how the environment responds to this action: the response consists of a reward, $R_{t+1}$, and a next state, $S_{t+1}$.
To recapitulate, the agent chooses an action using the $\epsilon$-greedy policy, executes this action on the environment, and it observes the response (that is, a reward and a next state) of the environment to this action. This is the part of the Q-learning algorithm where the agent interacts with the environment in order to gather some info about it, so as to be able to estimate the Q function.
Q-learning update
After that, the agent can update its estimate of the Q function using the following update rule
$$\color{orange}{Q(S_t, A_t)} \leftarrow \color{red}{Q(S_t, A_t)} + \alpha ([\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}] - \color{red}{Q(S_t, A_t)})$$
where $S_t$ is the current state (of the current episode) the agent is in, $A_t$ is the action chosen using the $\epsilon$-greedy policy (as described above), and $S_{t+1}$ and $R_{t+1}$ are respectively the next state and rewards, which, collectively, are the response of the environment to the just taken action $A_t$.
So, how is the estimate of this $Q$ function updated?
First of all, I would like to note that, if you look at the beginning of the pseudocode above, $Q(s, a)$ is initialized arbitrarily for all states $s \in \mathcal{S}$ and for all actions $a \in \mathcal{A}$: it can e.g. be initialised to $0$. $Q(s, a)$ can e.g. be implemented as a matrix (or 2-dimensional array) $M \in \mathbb{R}^{|\mathcal{S}| \times |\mathcal{A}|}$, where $M[s, a] = Q(s, a)$, $|\mathcal{S}|$ is the number of states in your problem and $|\mathcal{A}|$ the number of actions.
Furthermore, note that the symbol $\leftarrow$ means "assignment" (like assignment to a variable, in the context of programming). So, in the update rule above, we are assigning to $\color{orange}{Q(S_t, A_t)}$ (which will be the next or updated estimate of the Q value for the current state $S_t$ and the just taken action from that state $A_t$) the value $\color{red}{Q(S_t, A_t)} + \alpha (\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)} - \color{red}{Q(S_t, A_t)})$. Let's break this value down.
$\color{red}{Q(S_t, A_t)}$ (on the right side of the assignment) is the estimate of the Q value for the state $S_t$ and action $A_t$ before the assignment. So, we are summing $\color{red}{Q(S_t, A_t)}$ and $\alpha (\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)} - \color{red}{Q(S_t, A_t)})$, and then we assign it to $\color{orange}{Q(S_t, A_t)}$ again.
$\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$ is what is often called the target. Q-learning is a temporal-difference (TD) algorithm, and TD algorithms update estimates of the value or action-value functions based on the difference between the current estimate, in the case of Q-learning it is denoted by $\color{red}{Q(S_t, A_t)}$ (on the right side of the $\leftarrow$), and a "target". So, in the Q-learning algorithm, $\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$ is the target. We can roughly think of it as "the value that $\color{red}{Q(S_t, A_t)}$ should have been". So, in a certain way, we are performing supervised learning, where $\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$ would be the ground-truth label and $\color{red}{Q(S_t, A_t)}$ the current estimate, and so $[\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}] - \color{red}{Q(S_t, A_t)}$ would be the error (or loss): in fact, it is often called the TD error. However, note that this is not really supervised learning, because $\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$ is not a ground-truth (it is partially an estimate, because of the part $\gamma \color{blue}{\max_{a}Q(S_{t+1}, a)} $, and it partially a ground-truth, because of $\color{green}{R_{t+1}}$).
To recapitulate, $\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$ is the target, $\color{red}{Q(S_t, A_t)}$ is the current estimate, and $[\color{green}{R_{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}] - \color{red}{Q(S_t, A_t)}$ is the TD error. We are thus summing the "error" (weighted by the hyper-parameter $\alpha$, which is, in this case, often called the learning rate) and the current estimate $\color{red}{Q(S_t, A_t)}$ in order to produce the new estimate $\color{orange}{Q(S_t, A_t)}$.
In the target, you can see that we are multiplying the $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ by $\gamma$. This is a hyper-parameter (a parameter which often needs to be chosen by the programmer before the algorithm is executed), known as the discount factor. It controls the contribution of $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ to the target: that is, how much of $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ we want to include in the target. Recall that I've just said above that the target is composed of the reward $\color{green}{R_{t+1}}$ (which is a ground-truth or real-world experience, because it is directly received from the environment) and $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ (which actually uses an estimate of the Q function, that is, it uses $Q(S_{t+1}, a)$). So, $\gamma$ controls the contribution of an estimate to the ground-truth.
As I said at the beginning of this answer, $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ can be thought of as the $Q$ value associated with the next state $S_{t+1}$ (which was observed by the agent after he has taken the action $A_t$) and associated with the action $a$, such that $Q(S_{t+1}, a)$ is the highest among all other possible actions from state $S_{t+1}$. In other words, $\color{blue}{\max_{a}Q(S_{t+1}, a)}$ can be thought of the estimate of the Q value associated with the next state $S_{t+1}$ and the greedy action taken from that same state.
Q-learning is off-policy
Note that, when we update the value function, the agent is not really taking actions in the environment (the only action taken is $A_t$, and it was taken, using the behavior policy, before the update!). Nonetheless, people often call Q-learning an off-policy algorithm because

It uses the $\epsilon$-greedy policy to interact with the environment (aka a behavior policy). In this case, actions are really taken, and the responses of the environment are really produced, observed, and used to update estimates of the $Q$ function.

It uses a target that is based on an estimate which is greedy (i.e. it uses $\color{blue}{\max_{a}Q(S_{t+1}, a)}$).

Given that Q-learning uses estimates of the form $\color{blue}{\max_{a}Q(S_{t+1}, a)}$, Q-learning is often considered to be performing updates to the Q values, as if those Q values were associated with the greedy policy, that is, the policy that always chooses the action associated with highest Q value. So, you will often hear that Q-learning finds a target policy (i.e. the policy that is derived from the last estimate of the Q function) that is greedy (so, usually, different from the behavior policy).

Small note: start and end of episodes do not necessarily have to be associated with specific states. In theory we can have a probability distribution over initial states $S_0$, although in practice the most common case certainly is that the starting state is always the same. It is relatively common that in practice that there are many different states that can all end the episode, though I believe that is already consistent with what you wrote :)

@nbro: Thanks for your answer. I have a couple of questions to it as several things are unclear. 1) How can we actually calculate the max Q( S(t+1), a)? You write "We can roughly think of it as "the value that Q(St,At) should have been"". What is exactly meat by that? How can we know, what value it should have been? 2) Can you give me an example on how the Q-values are updated in each iteration (or a link where an example is explained in a comprehensive way)? 3) You claim that Q-learning used e-greedy policy for taking the actions and greedy policy for updating. Can other policies be used?

If we initialise all Q(s, a) to 0, will then not every max Q (s(t+1), a) be also 0 in each iteration? 5) If we always use a greedy policy to udate the Q values, I would assume that the chances are quite high that we will be stuck in a local (but not global) optimum. Why can we for example not also use the e-greedy policy for updating the Q values?

@PeterBe I am not sure I understand your first question. $\max Q( S(t+1), a)$ is just the maximum state-action value function with respect to $a$. For example, say you have a function $f: {1, 2 } \rightarrow {10, 5}$, and define it like this: $f(1) = 10$ and $f(2) = 5$, then $\max_x f(x) = 10$ and $\text{argmax}x f(x) = 1$. To answer your second question, remember that, during learning, $Q$ is an estimate of the true value function $q^*$. $\color{green}{R{t+1}} + \gamma \color{blue}{\max_{a}Q(S_{t+1}, a)}$ is an estimate of the true value $q^*(S_t, A_t)$.

To answer your 3rd question, we don't know $q^*(S_t, A_t)$. @PeterBe This post is already too long and I already spent too much effort on it, so I don't plan to give you a step-by-step explanation of Q-learning, because that was not even the question originally. I explained Q-learning here, but I should not have done that. I recommend that you ask your other questions in separate posts, but make sure to search for answers on our site first, because I think some of your questions are already answered by my post above and other posts on the site. Please, ask only one question per post!


text:
Is an algorithm that is no longer actively learning an AI?
This question assumes a definition of AI based on machine learning, and was inspired by this fun Technology Review post:

SOURCE: Is this AI? We drew you a flowchart to work it out (Karen Hao, MIT Technology Review)
As the definition of artificial intelligence has been a continual subject of discussion on this stack, I wanted to bring it to the community for perspectives.
The formal question here is:

Is an algorithm that is no longer actively learning an AI?

Specifically, can applied algorithms that are not actively learning be said to reason?

Interesting question, but this boils down to the definition of AI.

@nbro it does, but asks specifically if learning is the requirement. For instance, is an NN that has been trained but is no longer actively learning an AI?  (Interesting DeepMind post on Enabling Continual Learning in Neural Networks.)

A usual there needs to be lines drawn between "classic AI", maybe "classic CI (computational intelligence)", "modern AI in practice", the (very vague) AGI. An AGI that could not learn would be severely hobbled - Human: I am Bob. AGI: Sorry, I forgot, what was your name? - unless you want to include memory but exclude learning, and the possible difference between the two would be an interesting question in its own right.

In my view learning is a useful aspect of an AI system, which makes it more resilient towards a changing environment, but it is not a requirement. Early AI systems (Expert systems, theorem solvers) would usually not use learning algorithms, but had a fixed knowledge base.
It also works a lot better with numerical models than with symbolic processing; the proliferation of training data which boosts the most recent AI wave (which is based more on statistical and numerical methods) has made it somewhat necessary for modern systems to have a learning/training component.
However, one needs to distinguish between the initial training, which leads to a production system which then becomes fixed (and will not learn from new input) and those which have a learning component built in as an intrinsic part and will continue to adapt throughout their lifetime.
A cautionary example is Microsoft's "Tay" chatbot. This was still actively learning from conversations and was quickly abused by subjecting it to undesirable input. Switching off the learning component would have been prudent in this case, but it would not have taken the AI out of it.
So, to answer the question: learning is not a required part of an AI system once it is operational. While modern systems generally have some element of learning to build their initial models, older systems were simply hand-coded without any automated learning at all. But one would still regard them as AI system.


text:
Which of these two numerical methods for z-score normalisation is preferable, in multivariate linear regression?
In the exercise Exercise 3: Multivariate Linear Regression, by Andrew Ng, the author suggests to "scale both types of inputs by their standard deviations and set their means to zero".
$$x_{n e w}=\frac{x-\mu}{\sigma}$$
Method 1
The author provides the following Matlab (and Octave) code to scale the inputs.
x = [ones(m, 1), x];
sigma = std(x);
mu = mean(x);
x(:,2) = (x(:,2) - mu(2))./ sigma(2);
x(:,3) = (x(:,3) - mu(3))./ sigma(3);

Method 2
But why not simply scale the inputs between zero and one, or, divide by the maximum?
x_range=max(x)
x(:,2) = (x(:,2)/x_range(2));
x(:,3) = (x(:,3)/x_range(3));

I have done the exercise with method 2 and these are the results.

Question
Is there a computational advantage with the first method over the second method?

Clarifications
Before directly answering the question, there are a few things to note.

The word range is actually applicable to a dependent variable.  On the input side, it is the domain that needs to be normalized.
In the second method, since the minimum vector may not be the origin, the zeroth order term should be included in the normalization.  The minimum must be subtracted from the maximum to determine the domain and must also be subtracted as an offset to the independent, pre-normalized values.
The standard deviation is always less than half the difference between the max and min, so there is a difference in proportion between the two methods.

The crux of the question centers around the statistics of normalization and its potential impact on these methods.

Multivariate linear regression (in the question introduction)
Gradient descent (selected as a tag)

Although these two are not technically mutually exclusive, practically they are if by linear is meant conforming to $\vec{y} = a \vec{x} + \vec{b}$, where $a$ is a matrix dimensioned appropriately for $\vec{x}$ and $\vec{y}$.  For that first degree polynomial cases, there is a closed form (formula) to acquire $(\vec{a}, \vec{b}, c)$ from a set of pairs $(\vec{x}, \vec{y})$, where $c$ is the correlation coefficient.  It would be grossly wasteful to use iterative convergence methods, such as gradient descent, to calculate a mean squares regression result for a first degree polynomial model of the relation between $\vec{x}$ and $\vec{y}$.
If there is no closed form for the model or no known model at all, and the complexity of the function to be approximated is tenable in relation to the convergence methods available, the data available for training, and the computing resources and time that can be delegated to the task, then gradient descent is a method through which a least squares convergence on minimal loss may be possible.
The significance of artificial networks is that, through the interconnection of a sequence of layers, each with an array of like activation functions, a network can approximate a useful function the mathematical model of which is not explicitly known.  Of the statistical functional approximation problems where no closed form is known and for which convergence methods are possible, artificial network training has become the appropriate device for a growing set of cases.
Normalization
The metrics of mean and standard deviation are designed specifically for data that can be approximated by a Gaussian distribution.  The term normalization originated from the belief that the Gaussian distribution was normal, however Gauss did not hold that belief.  He presented it as a distribution resulting from the limit of an accumulation of random perturbations as the mean of the square of the perturbation size decreases to zero as the length of the series of perturbations approaches infinity.
Because of this, if the loss (or error) function is a sum of the squares of difference between the current output values of an artificial network and the labels providing the accurate and correct values, a near Gaussian distribution of input is likely to produce a faster and complete convergence.  In the multivariate case, there is an equivalent of the Gaussian distribution.
In the real world, data distribution very close to Gaussian is not the norm.  The distribution of land surface temperatures at a specific latitude and longitude on earth for a particular month reflect two maxima in the temperature distribution, one for nighttime and one for daytime.  The model of the relationship between these two maxima and the time of year, in chaos theory, are called attractors.  Points seem attracted to those norms.
The distribution of signal amplitudes in common circuits and natural sounds in the air are not Gaussian distributed either, but the $\log(P)$ of the power of common signals and sounds are close to a Gaussian distribution, which is partly why decibels are a common measure of signal strength and loudness.  The other part is that the logarithm in calculating decibels causes them to indicate relative geometric difference (ratio) in power rather than a less representative arithmatic difference.
Normalization is often oversimplified to mean ensuring values fall within a specific domain convenient to an algorithm's numerical representation or around a mean with a given standard deviation to feed a least squares strategy for statistical analysis or correlation.  However, data is not often symmetric and nicely distributed.
In the case of land surface temperatures the nighttime distribution has a lower standard deviation than the daytime distribution, so the two peaks have different adjacent slopes in the distribution.  How that is reflected in normalization varies depending on goals and algorithms receiving the normalized data.  Normalization strategies for signal power and loudness would include running the data through a logarithm to reflect the near exponential Gaussian distribution of power values.
Outliers
Outliers are data points that fall outside an expected trend or relationship, either because they were incorrectly measured in one or more dimensions of measurement, the measurement was corrupted, they were strongly influenced by a phenomenon outside the intended range of study, or representative of a significant but previously undiscovered phenomenon.  This last cause is why it is not wise to dismiss and exclude all outliers from functional analysis.
When outliers are valid data points that inconveniently lie more than a few standard deviations from mean, normalizing from zero to one causes a very narrow distribution peak in one place within that input domain.  If the outlier is of the kind that should not be dismissed and legitimately representative of the phenomena under study or being controlled, then some normalization scheme other than simply a first degree polynomial translation of the data is indicated.
Housing Predictor
In the case of square footage of living space, the data is not symmetric.  An increase of 10 square meters for a large house will not produce the same price differential as 10 square meter increase for a small one.  A properly configured neural network may learn to perform something similar to a logarithm in the first few layers during training, or the engineer can explicitly perform a logarithm to un-skew the data and present it in a form with closer to a Gaussian distribution to the network for training and then also later for actual use.
At the exit, prior to presenting the network output to the loss function, applying an exponential function may also produce faster and more reliable convergence, since a monetary unit has different values to different buyers.  A person buying a large house is less apt to be put off by cost associated with a fourth bedroom or more area than a person buying a small house.
Summary
The comparative advantages of different normalization methods are dependent upon a number of factors, including but not limited to these.

The resolution of the data types used to represent forward and backward signal propagation in the network, which affects the conditions of signal saturation
Skew of data set used for training
The number of standard deviations to the min and max in the data set used for training
Presence of multiple attractors in the distribution, which can occur in any of the features (dimensions of input) or combinations of them

Method one is best for a scenario that involves a Gaussian distribution and may be best for the housing prediction.  Method two might be best if the distribution is relatively flat almost all the way to the extremes.
Running each method and comparing them may reveal advantages for one or the other that is not predicted in this answer and further knowledge can be gained about why by performing further experiments.


text:
Is the singularity something to be taken seriously?
The term Singularity is often used in mainstream media for describing visionary technology. It was introduced by Ray Kurzweil in a popular book The Singularity Is Near: When Humans Transcend Biology (2005).
In his book, Kurzweil gives an outlook to a potential future of mankind which includes nanotechnology, computers, genetic modification and artificial intelligence. He argues that Moore's law will allow computers an exponential growth which results in a superintelligence.
Is the technological singularity something that is taken seriously by A.I. developers or is this theory just a load of popular hype?

I think you will have a really hard time finding any solid AI-related scientific baseis that suggests Singularity is even remotely possible.

I can say that among AI researchers I interact with, it far more common to view it as wild speculation than as settled fact.
This is borne out by surveys of AI researchers, with 80% thinking strong forms of AI will emerge in "more than 50 years" or "never", and just a few percent thinking that such forms of AI are "near".
Software Developers are not the same as AI researchers, and I have found the Singularity myth to be much more widespread among developers. It has a nice ring to it: Computers keep getting faster, at some point they'll be faster than brains, at that point we just simulate brains. Soon after, we simulate something better than brains.
I suspect that the reasons AI researchers are less optimistic are rooted in the fact that we still don't have a good understanding of human intelligence, or even enough of an understanding of the brain to simulate it. For example, in the last two weeks we have discovered previously unknown types of brain cells. This gives the (correct) impression that even if we had a fast enough computer, we are not at all close to being able to accurately simulate a human brain. We don't really know what a human brain is.
Even if we did know that, simulations are necessarily lossy. We may not have good simulation techniques. Even if we did have good techniques, we may simulate the brain and discover our simulation does not behave as expected for reasons that we don't understand. This is very probable when simulating new systems. In some sense, proponents of the Singularity resemble people predicting that weather control was near in the 1940s. After all, we could simulate simple weather patterns already then, and generate forecasts that sort of worked. How much more complex could it really be to generate perfect forecasts?

Writing this from 2023... Strong AI is already here...

There is at least one very important and serious AI scientist that apparently believes in the creation of true artificial general intelligence and possibly superintelligence: Jürgen Schmidhuber, who is the co-author of the LSTM, among many other important contributions. In fact, he recently founded NNAISENSE for this ultimate purpose, that is, to build a general-purpose artificial general intelligence. In his talk When creative machines overtake man, at TEDxLausanne, Schmidhuber talks about the singularity (also known as omega). See also his web article Is History Converging? Again? (2012) or the paper 2006: Celebrating 75 years of AI - History and Outlook: the Next 25 Years (2007).

This is true, but he's definitely in the minority of AI researchers on this. It's true Schmidhuber is well credentialed, but I don't think most AI researchers take his views on this element of AI seriously.

@JohnDoucette Yes, I agree with you. I just wanted to point out this interesting exceptional case.

In order to be on the same page, you should give references about "technological singularity", as it comprises multiple fields (mathematics, statistics, philosophy of science, epistemology, sociology, politics, economics, to mention a few).
Generally, when you consider concepts related to adj + ai (where adj = {weak, strong, full, narrow, ...}), the breath of speculation is quite large and in fieri, so as a developer (where for developer I assume you work on coding-related problems, not the project manager at google x and the like) I would not be worried, unless you are enjoying a cup of tea with your colleagues during a break.

Yes, it must be taken seriously. There are two main reasons:

There is no sharp argument or no-go theorem against the existence of a singularity. It's unclear how fast the singularity could develop, but many authors given a non zero probability to this event (see this reference, it contains different points of view on the singularity by leading experts).
The consequences of a singularity would be dramatic. So even if the perceived probability in point 1 is very small, it's worth studying possible mitigations routes.


text:
Why is baseline conditional on state at some timestep unbiased?
In the homework for the Berkeley RL class, problem 1, it asks you to show that the policy gradient is still unbiased if the baseline subtracted is a function of the state at time step $t$.
$$ \triangledown _\theta \sum_{t=1}^T \mathbb{E}_{(s_t,a_t) \sim p(s_t,a_t)} [b(s_t)] = 0   $$
I am struggling through what the first step of such a proof might be.
Can someone point me in the right direction? My initial thought was to somehow use the law of total expectation to make the expectation of $b(s_t)$ conditional on $T$, but I am not sure.

Don't really have much time to write down the exact equations and format it (maybe later if it is still not answered) with LaTeX but here is a hint. You want to have that the sum does not depend on the policy so that the derivative would be 0. So you somehow try to express things using the policy p(s,a). The answer btw can also be found in Sutton's RL Intro book in the policy gradient chapter.

Thank you very much! I will use that hint to get started, as well as thank you for telling me about it being in Sutton RL. I am reading that book and it is quite excellent!

@LauraC could you please elaborate how you arrived at this formulation? Because in the homework (at least as of now), the original equation is different and I don't know how to get to your version. Thanks!

Using the law of iterated expectations one has:
$\triangledown _\theta \sum_{t=1}^T \mathbb{E}_{(s_t,a_t) \sim p(s_t,a_t)} [b(s_t)] = \nabla_\theta \sum_{t=1}^T \mathbb{E}_{s_t \sim p(s_t)} \left[ \mathbb{E}_{a_t \sim \pi_\theta(a_t | s_t)}
 \left[ b(s_t) \right]\right] =$
written with integrals and moving the gradient inside (linearity) you get
$= \sum_{t=1}^T \int_{s_t} p(s_t) \left(\int_{a_t} \nabla_\theta b(s_t) \pi_\theta(a_t | s_t) da_t \right)ds_t =$
you can now move $\nabla_\theta$ (due to linearity) and $b(s_t)$ (does not depend on $a_t$) form the inner integral to the outer one:
$=
\sum_{t=1}^T \int_{s_t} p(s_t) b(s_t) \nabla_\theta \left(\int_{a_t} \pi_\theta(a_t | s_t)  da_t \right)ds_t= $
$\pi_\theta(a_t | s_t)$ is a (conditional) probability density function, so integrating over all $a_t$ for a given fixed state $s_t$ equals $1$:
$=
\sum_{t=1}^T \int_{s_t} p(s_t) b(s_t) \nabla_\theta 1 ds_t = $
Now $\nabla_\theta1 = 0$, which concludes the proof.

It appears that the homework was due two days prior to this answer's writing, but in case it is still relevant in some way, the relevant class notes (which would have been useful if provided in the question along with the homework) are here.
The first instance of expectation placed on the student is, "Please show equation 12 by using the law of iterated expectations, breaking $\mathbb{E}_{\tau \sim p \theta(\tau)}$ by decoupling the state-action marginal from the rest of the trajectory."  Equation 12 is this.
$\sum_{t = 1}^{T} E_{\tau \sim p \theta(\tau)} [\nabla_\theta \log \pi_\theta(a_t|s_t)(b(s_t))] = 0$
The class notes identifies $\pi_\theta(a_t|s_t)$ as the state-action marginal.  It is not a proof sought, but a sequence of algebraic steps to perform the decoupling and show the degree to which independence of the state-action marginal can be achieved.
This exercise is a preparation for the next step in the homework and draws only on the review of CS189, Burkeley's Introduction to Machine Learning course, which does not contain the Law of Total Expectation in its syllabus or class notes.
All the relevant information is in the above link for class notes and requires only intermediate algebra.


text:
Are there dynamic neural networks?
Are there neural networks that can decide to add/delete neurons (or change the neuron models/activation functions or change the assigned meaning for neurons), links or even complete layers during execution time?
I guess that such neural networks overcome the usual separation of learning/inference phases and they continuously live their lives in which learning and self-improving occurs alongside performing inference and actual decision making for which these neural networks were built. Effectively, it could be a neural network that acts as a Gödel machine.
I have found the term dynamic neural network but it is connected to adding some delay functions and nothing more.
Of course, such self-improving networks completely redefine the learning strategy, possibly, single shot gradient methods can not be applicable to them.
My question is connected to the neural-symbolic integration, e.g. Neural-Symbolic Cognitive Reasoning by Artur S. D'Avila Garcez, 2009. Usually this approach assigns individual neurons to the variables (or groups of neurons to the formula/rule) in the set of formulas in some knowledge base. Of course, if knowledge base expands (e.g. from sensor readings or from inner nonmonotonic inference) then new variables should be added and hence the neural network should be expanded (or contracted) as well.

as far as my experience takes me, you are looking for genetic algorithms, they keep a population of neural networks and merge them together at random points given how good they do

But such external GAs are external controllers which are incapable of learning. I am most interested in self-aware neural networks that themselves can take those decisions.

maybe look at this : https://arxiv.org/ftp/cs/papers/0504/0504055.pdf

This article on Dynamically Expandable Neural Networks (DEN) (by Harshvardhan Gupta) is based on this paper Lifelong Learning with Dynamically Expandable Networks (by Jeongtae Lee, Jaehong Yoon, Eunho Yang, Sung Ju Hwang)
This presents 3 solutions to increase the capacity of the network if needed retaining whatever useful information from the old model and train the new model:

Selective retraining
Dynamic Network Expansion
Network Split/Duplication

To me, it seems that such neural network is dynamic and improving. As such, they answer partially your question. If they don't sorry about that.

I mostly studied HMMs and such models are called Infinite HMMs in that specific domain.
I believe that what you are looking for is called Infinite Neural Networks. Not having access to scientific publications, I cannot really refer any work here. However, I found this GitHub repository: https://github.com/kutoga/going_deeper that provides some implementation and a document with multiple references.


text:
How can I minimize the number of answers that are relevant to a machine learning model?
Problem:
We have a fairly big database that is built up by our own users.
The way this data is entered is by asking the users 30ish questions that all have around 12 answers (x, a, A, B, C, ..., H). The letters stand for values that we can later interpret.
I have already tried and implemented some very basic predictors, like random forest, a small NN, a simple decision tree etc.
But all these models use the full dataset to do one final prediction. (fairly well already).
What I want to create is a system that will eliminate 7 to 10 of the possible answers a user can give at any question. This will reduce the amount of data we need to collect, store, or use to re-train future models.
I have already found several methods to decide what are the most discriminative variables in the full dataset.  Except, when a user starts filling the questions I start to get lost on what to do. None of the models I have calculate the next question given some previous information.
It feels like I should use a Naive Bayes Classifier, but I'm not sure. Other approaches include recalculating the Gini or entropy value at every step. But as far as my knowledge goes, we can't take into account the answers given before the recalculating.

The Problem Statement
It does not appear from the wording in the question that the semantics of the language in the questionnaire will be processed, so no knowledge of the associations in questionnaire questions and answers will be derived prior to its administration.
There are $M$ questions, with $N_m$ undifferentiated answers for each question, presented to human users.  The administration of this questionnaire builds a set of data that will later be used to train an independent learning system.
This question pertains to both some subsequent training and the learning that may need to occur to achieve stated goals in the acquisition of data for the subsequent training.

Reduce the burden on the system users that answer the questions.
Reduce the storage requirements.
Remove redundancy from the data to be used as examples for subsequent training.
Utilize in real time early answers within a single user session to achieve the above three objectives.

Analysis
The assumption in the question that the above four can and should be achieved through the same mechanism may or may not prove true during the design process.  These are some alternative ways to achieve goals.

Goal 2. can be achieved by employing bit-wise operations to pack the answer data into an innocuous payload.
Goal 3. can be achieved by auto-encoding independently of 1.

Goal 1. is the key to this problem, and its solution, utilizing goal 4., should be addressed first as the predominant technical risk.  Afterward, if byproducts of the solution to goal 1. can be employed to assist in the solution of goals 2. and 3., then so be it.
There are at least three meanings of models in machine learning.

The model of the concept being learned
The model thorough which learning occurs
The model of what is considered sufficient

This last model defines how accuracy and reliability are measured and what quality acceptance criteria are given regarding them.
It appears that none of the three models for either the data acquisition learning or the later learning using the data acquired are yet defined, however some architectural considerations can be addressed and some conclusions can be reached.
Structure of Information
Consider these structures of data that can aid learning.

The $M$ features of user linguistic response can be saved within a user session as a time series of vectors $\sigma(m, n, t)$.
Those series can be collected across $U$ users, each with an user ID $i$ indexed sequentially for mathematical purposes by $u$, as a structure of elements $\gamma(m, n, u, t)$.

It is the the second structure of $\gamma$ elements that contains information that can assist in the presentation of questions and answers to subsequent users, given the $\sigma$ time series in the current user's session state.
The Key Challenge
The challenge is not the application of probability and statistics to a single session, which is straightforward, outlined below.  The difficulty is that once those of the $M$ questions are rated for their likely usefulness in later training, by reordering questions, defaulting them to the most likely response, or eliminating any answers, the system has changed the conditions driving the associations.  This is the classic scenario when the measurement changes the item being measured.  In such a feedback condition, the knowledge acquired about usefulness can produce unstable conditions, including oscillations or chaotic system behavior.
Important Early Design Choice
Real time learning is indicated, and the system must be stabilized using the same kind of stabilization techniques involved in electrical engineering, robotics, and balancing.  Perhaps the best choice for current designs is Q-learning, pioneered by Watson and developed further by many others.

Watkin's 1989 PhD thesis Learning from Delayed Rewards which introduces the Q learning function in context — It is an excellent work that models learning in a way that is independent from whether the learning is biological or programmed on a computer, drawing on MDP (Markov decision process).
Human-level control through deep reinforcement learning, Mnih et. al., 2015

Applicable Probability and Statistics
The straightforward application of probability and statistics mentioned above is this.  Using Bayes' Theorem (not the naive categorization precisely), one can calculate the probability that any given $\sigma$ will be selected, based on the multi-user sequence of $\gamma(m, n, u)$.  However the most effective solution can be achieved via the application of Q-learning to the ordering, defaulting, and elimination of $\sigma$, given both the multi-user sequence of $\gamma$ and current user sessions sequence of $\sigma$.  Where naive Bayesian categorization or feature extraction may be most useful is in the profiling of users to further enhance user experience.  Even this can be contained within the Q-learning process.  The key to Q-learning is the inverse exponential decay of past knowledge.
Application Architecture
The most scalable architecture would be to use JavaScript in the web browser to acquire learned behavior via AJAX and adjust answers order, defaulting, and hiding in real time with the must updated learned state.
A more sophisticated approach would be to encapsulate the learned information in fuzzy logic rules to essentially remove redundancy and compress the learned information.
Further Requirements Analysis Indicated
This may be the extent to which design can proceed without more analysis of the requirements of the system.
What must be defined first is how the training data will be labeled, if at all.

Is the goal to provide example data for supervised learning, where expected results are provided to the training process?
Or is the goal to provide example data for unsupervised learning, where no expected results are given, and the training is to proceed toward some defined goal that is universal across the examples used in training?

Another consideration is to decide whether intelligent defaulting will be employed.

Should the system guess what the next answer is going to be and intelligently select it, allowing the user to change the selection if the guess was wrong?

You don't need to re-train on the fly. What you're looking for is an embedded feature selection algorithm, and even more precisely, one that minimizes the number of responses required.
I think this might be one of the rare cases where genetic and evolutionary approaches are the obviously correct choice.
Genetic Programming is a technique for finding models that are simply computer programs. You generate a bunch of computer programs at random, and then breed the "better" ones together. Repeating this process over time leads to highly optimized programs.
A nice feature of GP is that it is extremely flexible when picking what to optimize. So instead of "better" meaning just "more accurate", "better" can mean the sum of accuracy and $\frac{1}{\#answers\_used}$. The algorithm works the same way, and, with carefully chosen rewards, you may be able to get the best of both worlds.
There are lots of variations on this. I would probably start with a simple toolkit like EJC, and standard, boring, genetic programming.
There are specialized techniques for things like your problem too, but you'll probably get 80% of the benefit without needing to pursue them.

You definitely don't want Naive Bayes, but you could also look at modeling this with a Bayesian Network. That requires you to have a fairly deep understanding of the problem domain though.


text:
What is the dimensionality of the output map, given the dimensionality of the input map, number of filters, stride and padding?
I am trying to understand the dimensionality of the outputs of convolution operations. Suppose a convolutional layer with the following characteristics:

Input map $\textbf{x} \in R^{H\times W\times D}$
A set of $F$ filters, each of dimension $\textbf{f} \in R^{H'\times W'\times D}$
A stride of $<s_x, s_y>$ for the corresponding $x$ and $y$ dimensions of the input map
Either valid or same padding (explain for both if possible)

What should be the expected dimensionality of the output map expressed in terms of $H, W, D, F, H', W', s_x, s_y$?

where, $p_{x}$ and $p_{y}$ are padding values. (equal on both sides).
You can have different padding on left and right side. (similarly top and bottom). If same padding, equation will have $2*p_{x}$ or $2*p_{y}$, else you can just add values of both padding and replace in the equations. (For example $p_{xLeft} + p_{xRight}$)
Padding is optional. Generally thumb rule is add padding such that the equations above produce integers and also add padding such that if stride is 1, output is same as input and if stride is 2, output is half of input.


text:
Is it necessary to know the details behind the AI algorithms and models?
I am interested in the field of artificial intelligence. I began by learning the various machine learning algorithms. The maths behind some were quite hard. For example, back-propagation in convolutional neural networks.
Then when getting to the implementation part, I learnt about TensorFlow, Keras, PyTorch, etc. If these provide much faster and more robust results,  will there be a necessity to code a neural network (say) from scratch using the knowledge of the maths behind back-prop, activation functions, dimensions of layers, etc., or is the role of a data scientist only to tune the hyper-parameters?
Further, as of now the field of AI does not seem to have any way to solve for these hyperparameters, and they are arrived at through trial and error. Which begs the question, can a person with just basic intuition about what the algorithms do be able to make a model just as good as a person who knows the detailed mathematics of these algorithms?

This is a good question. I tend to think the answer is yes it is necessary to know the details, because a person without mathematical understanding of these algorithms cannot consistently make a model as good as someone who does have that understanding.
The reason is right at the core of computer science: abstractions are useful, but usually obscure details. When those details matter, someone who only knows the abstraction and not the details that lie beneath can't understand what's going on.
As an example, if you don't understand the math behind optimizing the weights of a neural network, it might not be apparent how parameters like the learning rate are impacted by properties like network depth when some of the inputs have not been properly normalized. If you understand the optimization process mathematically, you can reason through the effects even if you are trying to work on an unfamiliar problem. This ability to reason through the probable effects of parameter decisions in new domains is the main thing that you miss by working from intuition.


text:
Historical weakness of GOFAI in relation to partisan combinatorial games?
I was recently perusing the paper Some Studies in Machine Learning Using the Game of Checkers II--Recent Progress (A.L. Samuel, 1967), which is interesting historically.
I was looking at this figure, which involved Alpha-Beta pruning.

It occurred to me that the types of non-trivial, non-chance, perfect information, zero-sum, sequential, partisan games utilized (Chess, Checkers, Go) involve game states that cannot be precisely quantified. For instance, there is no way to ascribe an objective value to a piece in Chess, or any given board state.  In some sense, the assignment of values is arbitrary, consisting of estimates.
The combinatorial games I'm working on are forms of partisan Sudoku, which are bidding/scoring (economic) games involving territory control.  In these models, any given board state produces an array of ratios allowing precise quantification of player status. Token values and positions can be precisely quantified.
This project involves a consumer product, and the approach we're taking currently is to utilize a series of agents of increasing sophistication to provide different levels challenge for human players.  These agents also reflect what is known as a "strategy ladder".
Reflex Agents (beginner)
Model-based Reflex Agents (intermediate)
Model-based Utility Agents (advanced)
Goals may also be incorporated to these agents such as desired margin of victory (regional outcome ratios) which will likely have an effect on performance in that narrower margins of victory appear to entail less risk.
The "respectably weak" vs. human performance of the first generation of reflex agents suggests that strong GOFAI might be possible. (The branching factors are extreme in the early and mid-game due to the factorial nature of the models, but initial calculations suggest that even a naive minimax lookahead will be able to look farther more effectively than humans.) Alpha-Beta pruning in partisan Sudoku, even sans a learning algorithm, should provide greater utility than in previous combinatorial game models where the values are estimates.

Is the historical weakness of GOFAI in relation to non-trivial combinatorial games partly a function of the structure of the games studied, where game states and token values cannot be precisely quantified?

Looking for any papers that might comment on this subject, research into combinatorial games where precise quantification is possible, and thoughts in general.
I'm trying to determine if it might be worth attempting to develop a strong GOFAI for these models prior to moving up the ladder to learning algorithms, and, if such a result would have research value.
There would definitely be commercial value in that strong GOFAI with no long-term memory would allow minimal local file size for the apps, which must run on lowest-common-denominator smartphones with no assumption of connectivity.
PS- My previous work on this has involved defining the core heuristics that emerge from the structure of the models, and I'm slowly dipping my toes into the look ahead pool.  Please don't hesitate to let me know if I've made any incorrect assumptions.

I'm wondering about these "precisely quantifiable game states"... are these terminal game states? If so, that's no different from games like chess/go. If they're not terminal... are they really "precisely quantifiable"? Can there not be subsequent actions that change the value? If a game state is not terminal, it's game-theoretic value is normally defined in terms of game-theoretic values of successor states.

Precise quantification is possible theoretically for game states in chess and go. It would consist of -1, 0, 1 labels for each game state, referring to eventual lose, win, draw when playing perfectly versus another perfect-playing agent. Agents like AlphaGo estimate this function. I think you need to explain more about what you mean by quantifying a game state - from what I can tell, this is something to do with bidding values and a scoring system related to that, that is within your game?

@DennisSoemers This helps to clarify.  Not necessarily terminal values. Here I'm thinking just of a given present state, without regard to future states.

@NeilSlater thanks for your comments.  I'm going to think on it and look to edit the question to include what I mean by quantification.  (In these forms of partisan sudoku, the elements confer points in the regions of their placement. You can look at any given board state and determine the regional score as a ratio.  In the two player game with 9 regions, this can range from 0/9 to 9/0.)  The bidding element involves a more advanced mechanic which introduces regional stability based on the disparity between players' points-in-region, used to order a endgame cascade that determines the outcome.

@NeilSlater The endgame cascade can cause regions to flip between players, changing the controlling player.  When that happens, opposing elements defect. (Elements can influence neighboring regions, and defection can alter the deltas in those regions.) This renders any given element placement a bid, because the placing player may not ultimately control an element they place. But the AIs can run this cascade from any give state to discern its actual status as opposed to the apparent status.

Regarding weakness or strength of GOFAI, read Jacques Pitrat's books and see RefPerSys.

Nice question!
I think there are a couple of issues at work here.

Is the historical weakness of GOFAI in relation to non-trivial
  combinatorial games partly a function of the structure of the games
  studied, where game states and token values cannot be precisely
  quantified?

I think the short answer is yes. The real issue is in the last part:

token values cannot be precisely quantified

The most successful GOFAI approaches to these games were all some variation on A* search, combining combinatorial search with some form of heuristic function that estimated the value of the pieces and their positions in any given state. Piece counting is probably a better heuristic than not counting anything at all, but it's still clearly incorrect, because a player with less material may still have an overwhelming positional advantage. Some heuristics can try to estimate this positional advantage as well however.
The real problem that GOFAI encounters in these games is that positional advantage can be emergent in ways that require incredible heuristic power to detect. Checkers is a good example. In the 1990's, the Chinook project at the University of Alberta set out to solve it completely. Checkers is notable because it had the same world champion for more than 15 consecutive years, Marion Tinsley. Tinsley lost a total of 7 competitive matches over 40 years of play. This makes him an especially interesting person to examine when we look at combinatorial games. Figuring out how Tinsley plays can help us understand how human intelligence works in games like this. In the course of solving checkers, the researchers noted that Tinsley was making moves that required up to 42 move lookaheads to reveal an advantage (See Schaeffer et al., AI Magazine, Vol. 17, Issue 1).
This strongly suggests that Tinsley was not methodically considering each possible move. Instead, by his own admission, his thinking was guided by a combination of memory over his 40 year career (in one match against Chinook in 1992, he indicated he was trying to recall a sequence from a match 30 years prior when making a move (AI Magazine Volume 14, Number 2); and of attentional heuristics (i.e. not thinking about every move sequence, and being able to reliably rule out parts of the search space without looking at them).
The key is that for GOFAI to solve checkers without heuristics (i.e. to solve it exactly), required enormous amounts of computational power, because some moves yield positional advantages that require 40+ moves of followthrough. Even an incredibly simple game (branching factor of 2) would be hard under that constraint.
In contrast though, self-play techniques like those pioneered in Backgammon with TD-Gammon (Tesauro, Comm. of the ACM 1995) mimic the process through which Tinsley became so good: they play lots and lots of games, learn a good heuristic estimate of position and material value, and more importantly, can learn to remember odd circumstances that require careful play. TD-Gammon achieved worldclass play despite only explicitly looking 2 moves ahead. GOFAI search techniques weren't even close despite searching much more deeply.
Modern research on attention could salvage the GOFAI approach however. If you can learn to tell what's important, you might be able to get a lot more value out of deeper lookaheads. This seems even closer to how Tinsley played: great ability to estimate value was used to guide an explicit analysis of a specific chain of moves.

Thanks for this answer, and the TD-Gammon link in particular!  (There's a nice little writeup on it in Matthew Lai's Giraffe Chess paper for those interested.)


text:
Should the input to the negative log likelihood loss function be probabilities?
I am trying to train a supervised model where the output from the model is output of a linear function $WX + b$. Kindly note that I'm not using any softmax or $\log$ softmax on the result of the linear. I am using negative log-likelihood loss function, which takes the input as the linear output from the model and the true labels. I am getting decent accuracy by doing this, but I have read that the input to negative log-likelihood function must be probabilities. Am I doing something wrong?

log softmax is used for classification problems only. Are you solving a classification problem? If not you should use a MSEloss or something for regression task. Apart from that softmax takes the network output and transforms it to probabilities. NLL just takes the negative logarithm. So for classification you usually combine a softmax followed by NLL. However, in praxis you often do this in one go that avoids explicit calculation of steps inbetween for better numerical stability, such combined function is often called CrossEntropy loss. So take that for a classifica. task with raw net output.

This seems pretty reasonable to me. You can optimize for any function that is proportionate to the negative log-likelihood. Conventionally, we assume that the likelihood of piece of data under a linear model is proportionate to some sort of gaussian function of the difference between the predicted value and the observed value. If you're a Bayesian you'd say this is a probability. If you're a hardcore frequentist, you might quibble about that, but it's still a number between 0 and 1.
If you take the log of this likelihood function however, you'll get a quadratic function of the difference, with some scalers that you don't need to worry about. So you ought to minimize:
$$\sum_y (y-\hat{y})^2$$
This is not a probability, but since it is proportionate to the log of the original likelihood function (which was, in some sense), minimizing it will also minimize the original function.
Hope that helps!

Thanks for the help. It made some sense to me but still, I am not able to understand completely. It would be nice if you could provide some sources/supplements that would help me in having a better understanding of your answer.

This is a reasonable starting place: https://en.wikipedia.org/wiki/Least_squares#Solving_the_least_squares_problem