text:
What is the intuition behind importance sampling for off-policy value evaluation?
The technique for off-policy value evaluation comes from importance sampling, which states that
$$E_{x \sim q}[f(x)] \approx \frac{1}{n}\sum_{i=1}^n f(x_i)\frac{q(x_i)}{p(x_i)},$$ where $x_i$ is sampled from $p$.
In the application of importance sampling to RL, is the expectation of the function $f$ equivalent to the value of the trajectories, which is represented by the trajectories $x$?
The distributions $p$ represent the probability of sampling trajectories from the behavior policy and the distribution $q$ represents the probability of sampling trajectories from the target policy $q$?
How would the trajectories from distribution $q$ be better than that of $p$? I know from the equation how it is better, but it is hard to understand intuitively why this could be so.

could you also elaborate on this sentence - 'is it correct to equate the expectation of the function  is equivalent to the value of the trajectories, which is represented by the trajectories ' please? I'm not quite sure what you mean.

Sorry i didn't explain it that well. I wanted to know how the terms in importance sampling are being applied in the off policy evaluation RL context.

Recall that our goal is to be able to accurately estimate the true value of each state by computing a sample average over returns starting from that state:
$$v_{q}(s) \doteq \mathbb{E}_{q}\left[G_{t} | S_{t}=s\right] \approx \frac{1}{n} \sum_{i=1}^{n} Return_i $$
where $Return_i$ is the return obtained from the $i^{th}$ trajectory.
The problem is that the $\approx $ does not hold, since in off-policy learning, we got those returns by following the behavior policy, $p$, and not the target policy, $q$.
To address that, we have to correct each return in the sample average by multiplying by the importance sampling ratio.
$$v_{q}(s) \doteq \mathbb{E}_{q}\left[G_{t} | S_{t}=s\right] \approx \frac{1}{n} \sum_{i=1}^{n} \rho_i Return_i$$
where the importance sampling ratio is : $\rho=\frac{\mathbb{P}(\text { trajectory under } q)}{\mathbb{P}(\text { trajectory under } p)}$
What this multiplication does is that it increases the importance of returns that were more likely to be seen under the target policy $q$ and it decreases those that were less likely. So, at the end, in expectation, it would be as if the returns were averaged following $q$.
(A side note: To avoid the risks of mixing $p$ and $q$, it might be a good idea to denote/think of the behavior policy as $b$ and the target policy as $\pi$, following the convention in Sutton and Barto's RL book.)

Let's fix some notation: we're collecting data from behavior policy $\pi_0$ and we want to evaluate a policy $\pi$.
Of course, if we had plenty of data from policy $\pi$ that would be the best way to evaluate $\pi$ as we just take the empirical average (without any importance sampling) and CLT gives us confidence intervals that shrink at $\frac{1}{\sqrt n}$ rates.
However, collecting data from $\pi$ is often time-consuming and costly: you may need to productionize it at a company, and if $\pi$ were dangerous, some damage could be done during rollouts. So how can we make the best use of our data from any policies, not necessarily $\pi$, to evaluate $\pi$? This is the question of off-policy evaluation, and you're right that IS is one approach.
This picture from a great talk by Thorsten provides nice intuition on why the weighting is unbiased .

In the application of importance sampling to RL, is the expectation of the function $f$ equivalent to the value of the trajectories, which is represented by the trajectories $x$?

I believe what you are asking here is if when using importance sampling in the off-policy RL setting that we set $f(x)$ from the general importance sampling formula to be our returns - the answer to this is yes. As always we are interested in calculating our expected returns.

How would the trajectories from the distribution $q$ be better than that
  of $p$? I know from the equation how it is better but it is hard to
  understand intuitively why this could be so.

I think here you got your $p$ and $q$ the wrong way around as we are using samples from $p$ to approximate our policy $q$. We typically will use importance sampling to generate samples from a different policy to our target policy for a few reasons - one reason might be that our target policy is hard to sample from whereas sampling from our behaviour policy $p$ might be relatively easy to sample from. Another reason is that we generally want to learn an optimal policy, but this could be difficult to learn if we don't explore enough. So we can follow some other policy that will explore sufficiently and still learn about our optimal target policy through the importance sampling ratio.


text:
Can we increase the speed of training a reinforcement learning algorithm?
I am new in reinforcement learning. I started reading the PyTorch's documentation about the cart pole control. Whenever an agent fails, they restart the environment.
When I run the code, the time in the game is the same as time in real life. Can we train models quicker? Can we make the game faster so that model will be training faster?

Can we make the game faster so that model will be training faster?

It depends on how much processing is required to run the simulation, how efficient that is implemented in whichever library you have loaded, and whether there is anything non-necessary for training that you can disable. Some environments for instance deliberately run "real time" so humans can appreciate the video output, and that is not necessary for training purposes (unless you want to experiment with real-time robotics).
For OpenAI Gym, there is one thing you can usually do: Switch off the rendering. The rendering for environments like CartPole slows down each time step considerably, and unless you are learning using computer vision processing on the pixels output, the agent does not need the pictures. You may even notice during training that moving the rendering window so it is not visible will speed up the training process considerably.
What I did for CartPole, LunarLander and a couple of similar environments is turn off rendering for 99 out of 100 episodes, and render just one of them in 100 to help me monitor progress. For Q learning, I also picked that to be a "test" episode where I stopped exploration.
Another option for speeding up training is to run a distributed system with multiple simulations at once. You will need mechanisms to share collected data too, so it is more work, but it is another approach to take if the simulation steps are the bottleneck for training speed.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How estimate the minimum size of an autoencoder to overfit the training data?
Given e.g. $1$M vectors of $1000$ floating points each, where every point in vectors is sampled from a uniform distribution between $-1$ to $1$, how to estimate the minimum network size required between input ($1000$ units), bottleneck (preferably $1$ unit), and output ($1000$ units) which is cable of overfitting the training data perfectly?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How can Cat Swarm Algorithm (CSO) used for feature selection?
Cat swarm optimization (CSO) is a novel metaheuristic for evolutionary optimization algorithms based on swarm intelligence which proposed in 2006. See Feature Selection of Support Vector Machine Based on Harmonious Cat Swarm Optimization.
According to Modified Cat Swarm Optimization Algorithm for Feature Selection of Support Vector Machines

CSO imitates the behavior of cats through two sub-modes: seeking and tracing. Previous studies have indicated that CSO algorithms outperform other well-known meta-heuristics, such as genetic algorithms and particle swarm optimization. This study presents a modified version of cat swarm optimization (MCSO), capable of improving search efficiency within the problem space. The basic CSO algorithm was integrated with a local search procedure as well as the feature selection of support vector machines (SVMs).

Can someone explain how exactly Cat Swarm Algorithm (CSO) is used for feature selection?


text:
How do CNNs or RNNs "stack the feature of nodes by a specific order"?
I am trying to understand the following statement taken from the paper Graph Neural Networks: A Review of Methods and Applications (2019).

Standard neural networks like CNNs and RNNs cannot handle the graph input properly in that they stack the feature of nodes by a specific order.

This statement is confusing to me. I have not used CNNs/RNNs for non-Euclidean data before, so perhaps that's where my understanding falls off.
How do CNNs/RNNs stack the feature of nodes by a specific order?


text:
What is a example showing that the tree-based variant for the greedy best-first search is incomplete?
I understand that a tree-based variant will have nodes repeatedly added to the frontier. How do I craft an example where a particular goal node is never found. Is this example valid.

On the other hand, how do I explain that the graph-based version of the greedy best-first search is complete?

Hi and welcome to AI SE! Do you know the difference between tree search and graph search? If not, please, read this answer.


text:
How do I set up rewards to account for unmanned aerial vehicle crashes?
I am working on a project to implement a collision avoidance algorithm on a real unmanned aerial vehicle (UAV).
I'm interested in understanding the process to set up a negative reward to account for scenarios wherein there is a UAV crash. This can be done very easily during the simulation (if the UAV touches any object, the episode stops giving a negative reward). In the real world, a UAV crash would usually entail it hitting a wall or an obstacle, which is difficult to model.
My initial plan is to stop the RL episode and manually input a negative reward (to the algorithm) each time a crash occurs. Any improvements to this plan would be highly appreciated!

This is a bad idea. Primarily because RL takes many, many episodes which will are not practical on a real device. You're better off planning using classical MPC for trajectory generation.


text:
What are the most common feedforward neural networks?
What are the most common feedforward neural networks? What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics? Is there such a taxonomy?

What is a neural network?
Many neural networks can be defined as a function $f: \mathbb{R}^n \rightarrow \mathbb{R}^m$, where $n, m \geq 1$.
Equivalently, many neural networks can also be defined as a set of interconnected units (aka neurons or nodes) $f^i$ that receive some input and produce output, i.e. $f^i(\mathbf{x}^i) = y^i$, where $\mathbf{x}^i \in \mathbb{R}^k$. The actual function $f^i$ is variable and depends on the application or problem you want to solve. For example, $f^i$ can just be a linear combination of the inputs, i.e. $f^i(\mathbf{x}^i) = \sum_{j} \mathbf{w}_j^i \mathbf{x}_{j}^i$, where $\mathbf{w}^i \in \mathbb{R}^k$ is a vector of weights (aka parameters or coefficients). The linear combination can also be followed by a non-linear function, such as a sigmoid function.
If the neural network (more precisely, its units) doesn't contain recurrent (aka cyclic) connections, then it can be called a feedforward neural network (FFNN).
What are the most common feedforward neural networks?
Perceptron
The simplest (non-binary) FFNN is the perceptron, where the inputs are directly connected to the outputs. The perceptron performs a linear combination of the inputs followed by a thresholding operation, so perceptrons can only represent straight-line functions, so they can only be used for classification or regression problems where your data is linearly separable. In fact, the perceptron cannot solve the XOR problem.
Before the perceptron, McCulloch and Pitts had introduced simplified models of biological neurons, where all signals are binary, in an attempt to closely mimic their biological counterpart. The perceptron can actually be seen as an extension of this work. In fact, a perceptron can be viewed as a single artificial neuron.
Multi-layer perceptron
An FFNN with more layers (of units) between the input and the output is often called a multi-layer perceptron (MLP). The layers in the middle are often denoted as hidden layers. The MLP can represent not only linear functions (i.e. straight lines), but also more complicated functions by making use of non-linear functions, such as the sigmoid.
Convolution neural network
You can have other forms of FFNNs that perform other operations.
For example, a convolution neural network (CNN), provided it doesn't contain recurrent connections, is an FFNN that performs the convolution operation (and often also a sub-sampling operation). For this reason, they are particularly suited to deal with images (and videos). (This shouldn't be surprising if you are familiar with the basics of image processing and computer vision, which I don't think it's the case)
However, note that CNNs can also have recurrent connections, but this is not usually the case.
Residual neural network
There are also residual neural networks, i.e. neural networks where a node in a certain layer $l$ can be connected to other nodes in layers $l+j$, for $j \geq 1$, as opposed to being connected only to the nodes in layer $l+1$, which is the typical case.
Auto-encoders
Auto-encoders are neural networks that compress the input and then decompress it. The answers to this question may help you to understand why AEs would be useful.
What kind of inputs do they receive?

What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics?

In principle, each of these FFNNs can receive either binary or real numbers or vectors (either of real or binary numbers). However, certain NNs are more appropriate to deal with certain inputs. For example, CNNs are more appropriate for images (which are typically represented as matrices or tensors).
How can you further classify the NNs?
Based on chapter 2 of the book Neural Networks - A Systematic Introduction (1996) by Raul Rojas, you can also divide neural networks into other categories

Unweighted (i.e. binary, such as the McCulloch and Pitts' model) vs weighted (e.g. the perceptron)
Synchronous vs asynchronous (e.g. Hopfield networks, which are recurrent neural networks, though)
Neural networks that store states vs NNs that don't store states

You could also distinguish between FFNNs based on the learning algorithm. Nowadays, the widely used NNs are trained with gradient descent (and back-propagation to compute the gradients), but there are other approaches to train NNs, such as evolutionary algorithms or Hebbian learning. Moreover, you could also distinguish between neural networks that compute a deterministic function and neural networks that have some randomness or stochasticity inside them (e.g. Bayesian neural networks). There are probably many more possible subdivisions.


text:
What are the procedures to get RL paper results?
I finished working on a new algorithm in Reinforcement Learning, I need to compare it to some well-known algorithms. That's why I need to know the step-by-step procedures that RL researchers usually take in order to get their results and compare them to other papers' results. (e.g. running the algorithm multiple times with different random seeds, saving results to .csv files, plotting).
Anyone can help?
(I am working on Pytorch, PyBullet Environments.)

I am not sure exactly what you are looking for. To compare your procedures with well known paper, maybe use https://paperswithcode.com/

This question is indeed a little bit unclear. Are you looking for methods of how RL researchers do research and write their research papers?


text:
Do smaller loss values during DQN training produce better policies?
During the training of DQN, I noticed that the model with prioritized experience replay (PER) had a smaller loss in general compared to a DQN without PER. The mean squared loss was an order of magnitude $10^{-5}$ for the DQN with PER, whereas the mean squared loss was an order of magnitude $10^{-2}$.
Do the smaller training errors have any effect on executing the final policy learned by the DQN?

I think it says something about the training progress, while another approach you can make sure is to look at the gradient norm. Sometimes, the training loss is really noisy while the gradient norm is much more clear.

u mean the gradient of the weights of the model ? what would the gradient norm say about the training ?

well smaller norm would indicate that training is about to converge i guess. At a local minima, all the dimensions of a gradient would be 0.


text:
What is the difference between fuzzy neural networks and adaptive neuro fuzzy inference systems?
I have, like you see, just a general question about the combination of fuzziness and neural networks. I understood it as follows

Fuzzy neural networks as a hybrid system: the neural network helps me to find the optimal parameters related to the fuzzy system, for example, the rules or the membership function
Adaptive neural fuzzy inference systems (ANFIS): the NN helps me to find the optimal parameters related to the fuzzy inference system. What are some examples here?

I cannot intuitively grasp the difference between these two.

I don't see any difference between these two. The question leads to what you actually mean by a "fuzzy neural network" and an ANFIS; don't you mean the same, that is, layer 1 are the inputs, layer 2 the application of the antecedents membership functions, layer 3 the rules, layer 4 the consequents (apply inverse membership function to rule strength) and layer 5 the final combined output?


text:
Are my steps correct for a proper classification of a sick brain?
I have a dataset with MRI of patients with a specific disease that affects the brain and another dataset with MRI of healthy patients.
I want to create a classifier (using neural networks) to classify if the MRI of a new patient shows the presence of the illness or not.
First of all, I extracted the brain from all the MRIs (the so-called skull stripping) using BET tool found in FSL.
I have three questions for you

As the input to the training phase, I want to give the whole extracted brains (possibly in the nii format). What kind of preprocessing steps do I need to apply once I've extracted the brains (before passing it to the classifier)?
Do you know any better tool for skull stripping?
Do you know a tool (or library) that takes as input a nii files and allows me to create a classifier that uses neural networks?

Please, focus on one question only. I would remove at least the second question and maybe leave the 3rd question as a side note.

It looks like everything you want is available with the Deep Learning Toolkit (DLTK) for Medical Imaging
There is also a blog: An Introduction to Biomedical Image Analysis with TensorFlow and DLTK
There is a DataCamp course that walks you through most of the process but instead of a classifier they use deep learning to reconstruct brain images.  They provide a link to their MNIST classifier example which should be easy to adapt for your purpose.  See: Reconstructing Brain MRI Images Using Deep Learning (Convolutional Autoencoder)
ResearchGate has a thread that may help:
What is the appropriate way to use Nifti files in deep learning?


text:
Why is ANFIS important in general?
I am actually working with the iris dataset from sklearn and try to understand the ANFIS-Package for python. But that does not really matter! I have a more general question.
During thinking about adaptive neuro-fuzzy inference system (ANFIS), a general question came into my mind. I don't really understand: in general, why is ANFIS necessary?
So, for example, if I want to predict classes for this iris dataset, I also can use a supervised learning method or a neural network and I get the result.
In ANFIS, I do nothing other than splitting the input attributes into linguistic terms and give membership functions to it. At the end of the day, I will receive "predictions" for the input values, which are classes.
But - with the ANFIS-Package in Python - I cannot see, if my membership function has changed during the learning time or what rules the network constructed. So, I cannot really see why this is useful. Maybe it is just because I am usually using the iris dataset for supervised learning.


text:
Is there an AI system that, given a patient's symptoms, produces a diagnosis and suggests a treatment?
Is there an AI system (preferably, one that interacts with the human, such as a chatbot like this one) that, given some input (e.g. entered into the system by writing text), such as a person's physical history and symptoms (of a certain disease), produces a diagnosis of the disease and/or suggests medications or a treatment to improve the condition of the patient?

Maybe have a look at this question: https://ai.stackexchange.com/q/18584/2444.


text:
How many spectrogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?
I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguishable from humans) using the GitHub https://github.com/Rayhane-mamah/Tacotron-2.
I'm very confused about a simple aspect of text-to-speech even after reading the paper several times. Tacotron-2 generates spectrogram frames for a given input-text. During training, the dataset is a text sentence and its generated spectrogram (it seems at a rate of 12.5 ms per spectogram frame).

If the input is provided as a character string, then how many spectogram frames does it predict for each character?
How does training supply which frames form the expected output from the dataset? Because the training dataset is simply a thousand of frames for a sentences, how does it know which frames are ideal output for a given character?

This basic aspect seems just not mentioned clearly anywhere and I'm having a hard time figuring this one out.


text:
Is my understanding of how AI works correct?
In my discussion over my question on Math SE, I explained to a user, how I think AI works, I wrote that with the sigmoid(logistic) function, features of a data set are identified, many such iterations provide learning.
Is my understanding of how this works correct?

Artificial Intelligence as a field is not only neural networks, you are loosely describing a neural network, not describing AI in itself. Please do not call neural networks as "AI".

What other purpose does neural networks other than to create AI?

Neural networks can do curve fitting, would you call that AI? Please do not be naive, AI is much bigger than just neural networks. If you look at Alpha Go, you can easily see that it uses more than just neural networks (it uses search trees), and that is closer to AI than a plain neural network.

There's some useful information in your description, but that's just a very vague description of how neural networks with sigmoid activation functions are trained.
Moreover, there are many other AI systems apart from neural networks (such as support vector machines, expert systems, etc.), which, of course, I cannot exhaustively list here.

Is my understanding of how AI works correct?

I would say it's not completely incorrect, but, as I said, it's a very vague description and it only refers to a subset of techniques in the AI field. With that description, no newbie would probably understand how neural networks are really trained, apart from knowing that you will train them iteratively and there are sigmoids involved.

He claimed to be working in AI, so I guess, he might know about it. I'm asking to see if my understanding is correct.

@noviceFedora I am also studying and working in AI, so I know what I am talking about. You didn't say what that user told you about your description.

I think he said "pretty much" as in pretty much what I said happens.

@noviceFedora As I say in my answer, what you say is not incorrect, but it's very very vague! A newbie would require a better explanation and would not really understand how neural networks are trained with your explanation. But you're on the right track. You just need to study more now! Anyway, if your friend said that your explanation is correct, he probably doesn't know much about the topic or maybe he's just careless. Training a neural network is quite more complex than what you said. There are many other details.

Actually he didn't say pretty much, I misremembered it, he actually didn't answer anything. You can view the discussion here: https://math.stackexchange.com/questions/3658539/any-resources-which-introduce-mathematical-concepts-without-much-emphasis-on-cal


text:
In what RL algorithm category is MiniMax?
Q-learning is a temporal-difference method and Monte Carlo tree search is a Monte Carlo method. In what category is MiniMax?

I think you are looking at it from the wrong direction, min-max is just a planning algorithm, decision strategy, in the sense that you are describing other algorithms/methods it does not have a category. For example, you have negamax algorithm which is in a sense the same thing the Monte Carlo Search Tree is to Monte Carlo. Min-max category is game theory really.
Now you should be thinking about RL algorithms in another way, and this is taxonomy:

So if you think about methods, you mentioned, let's put them in the right place:

TD methods in general - model free
Monte Carlo methods - model free
MinMax - model-based (that could be discussed but it definitely needs access to a world model)


text:
How can I make ELIZA more realistic?
I’ve coded a simple ELIZA chatbot for a high school coding competition. The chatbot is part of an app that’s designed to help its user cope with depression, anxiety, and similar mental health disorders. It uses sentiment analysis to identify signs of mental illness, and to track it's user's progress toward "happiness" over time.
My question is, what steps can I take to make it more realistic (without using some pre-existing software, library, etc, which isn't allowed)? Also, are there any existing tables of questions/responses I can add to my ELIZA bot's repertoire so that it can handle more conversations?

One 'easy' way would be to have some sort of conversational memory, where you track what the user has said already. I don't know how complex your patterns are, but if you could recognise names and track references, you could try and build up a mental model of the user's relationships with other people, and perhaps refer to that in your bots responses.
The latter will be quite advanced, but keeping track of things said earlier and referring back to them on occasion might make it appear a lot more capable.
As an added bonus, track changes in the user's sentiment scores, and see if you spot a pattern in the conversation (maybe over the course of multiple conversations) to see which bot utterance have the biggest (positive or negative) effect on the user's mood.

Google Home already tries to do this to a very small extent, i.e. it tries to take into account the context or previously asked question. Although my knowledge of NLP and NLU is limited, I guess this type of capability will improve even further with time.


text:
Does Gödel's second incompleteness theorem put a limitation on artificial intelligence systems?
According to Brian Cantwell Smith

no calculation without representation

Therefore, computers depend on models. So, we can say that AI is limited internally by the model and externally by the environment. This problem is discussed here in a previous question I have asked.
Now, consider Gödel's second incompleteness theorem

a coherent theory does not demonstrate its own coherence

Can we say that Gödel's second incompleteness theorem puts a limitation on artificial intelligence? How could AI bypass Gödel's second incompleteness theorem?

This and this are two very related questions.

Why should AI systems be consistent? Intelligent humans are not!

I think the colloquial understanding of Gödel's incompleteness theorems allows them to be too broadly applied. Gödel's second incompleteness regards the consistency of a formal system, which is a technical concept of formal systems that means the system cannot prove every formula. It is commonly framed as a system not being able to prove both a formula and its negation (e.g. $2+2=4$ and $2+2 \neq 4$), since many logical systems allow anything to be proven from a contradiction.
The second incompleteness theorem states that if a consistent formal system is expressive enough to encode basic arithmetic (Peano arithmetic), then that system cannot prove its own consistency. This implies that we must use a stronger system B to prove the consistency of A. The system needs to be able to represent arithmetic because that is what is used to define the representability conditions of Gödel's proof that allowed him to formally construct the self-referential formulae central to the incompleteness theorems.
Here I diverge with my own opinion on this, I feel the concept of consistency in formal systems has no obvious bearing on the limits of artificial intelligence. An intelligent agent need not know anything about formal consistency to reach its level of intelligence -- the vast majority of humans have never encountered this concept, and yet they are still intelligent. Even many mathematicians don't give it a second thought unless they are in the trenches of mathematical logic. One would have to take an overly narrow view of artificial intelligence to allow Gödel's second incompleteness to serve as a limitation to it.
I caution against the popular informal restatements of Gödel's incompleteness theorems. These theorems were undoubtedly earth-shattering in the study of foundational mathematics and still have grand implications today, but projecting those results too far away from their rigorous origins is going to lead to many stray conclusions.

authors: Алексей Распопов


text:
Is this dataset with only two features suitable for clustering with k-means?
I am working with the K-means clustering algorithm for unsupervised learning.
Is the following dataset suitable for the k-means clustering task or not? Why or why not? The dataset has only two features.

One problem with clustering algorithms is that they will typically find you a solution, ie they will split your data set into clusters, but it will find you a structure even if there isn't one. Your data looks like it could consist of about 5 to 7 clusters, but it could equally well just be 2 or only 1.
What you need to do after the clustering is to assess the quality of the result. I recommend having a look at Finding Groups in Data by Kaufman & Rousseeuw. They discuss various clustering algorithms and also a procedure that works out how cohesive your clusters are. Though it is 30 years old, it is an excellent book on the topic.
You also have the issue of choosing a value for k in your clustering: I usually start with two, and increase it from there; at each step I compute the cohesion of the result using their method, until I get the best score. This is an objective way of finding a good value for k and usually a reasonable clustering result.
The ultimate test, of course, is then if looking at the result makes sense to you. No cluster algorithm can do that for you.

How can i answer this question that if this dataset is suitable for k-means clustering task or not? considering what measures?

There is a cohesion measure described in Kaufman & Rousseeuw. IIRC you calculate for each element the distant to its own cluster centre and the ones of the other clusters; if it's closest to its own, then it's a positive score. But it's been a while since I last did this.

Okay but clustering is the next step, i just need to answer by looking at the dataset and it's statistical measures that is it suitable for k-means clustering task or not? so i am confused how can i answer this?


text:
How to best make use of learning rate scheduling in reinforcement learning?
How to best make use of learning rate scheduling in reinforcement learning?
To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't see why over training time this should be linearly brought down. Wouldn't this increase overfitting too, as it promotes an early adopted policy to get further and further finetuned for the rest of the training?
Wouldn't it be better to keep it constant over the entire training so that when the agent finds novel experiences later, it still has a high enough learning rate to update its model?
I also don't really know how these modern deep RL papers do it. The starcraft II paper by DeepMind, and the OpenAI hide and seek paper don't mention learning rate schedules for instance.
Or are there certain RL environments where it's actually best to use something like a linear learning rate schedule?

I have not used learning rate schedules, but I do have experience with adjustable learning rates.
The Keras callback ReduceLROnPlateau is useful for adjusting the learning rate. If you use it to monitor the validation loss versus training loss, you will avoid the danger of overfitting. Also, you can use the ModelCheckpoint callback to save the model with the lowest validation loss and use that to make predictions. The documentation is here.
I look at the validation loss as a deep valley in $N$ space, where $N$ is the number of trainable parameters. As you progress down the valley, it becomes increasingly narrower, so it is best to reduce the learning rate to get further down the valley (closer to the minimum). With the adjustable learning rate, you can start with a larger initial rate that converges faster, then reduces as needed to achieve a minimum loss.
I wrote a custom callback that initially monitors the training loss and adjusts the learning rate based on that until the training accuracy achieves 95%, then it switches to adjusting the learning rate based on validation loss.
I also am experimenting with a slightly different approach to training. On a given epoch, assume the quantity you are monitoring does NOT improve. That means that you have moved to a point in $N$ space (value of the weights) that is NOT as "good" as the point you were at in the previous epoch. So, instead of training from the point you are at in the current epoch, I set the weights back to what they were for the previous (better) epoch, reduce the learning rate then continue training from there. This appears to work rather well.

Yet, you answer holds for settings that are not reinforcement learning. In RL, we have the problem of "moving target" mitigated by using two neural networks.


text:
What are the differences between a deep belief network, a restricted Boltzmann machine and a deep Boltzmann machine?
Can anyone list the differences between deep Belief network (DBN), restricted Boltzmann machine (RBM), deep Boltzmann machine (DBM) using simple examples?
Links to other resources are also appreciated.

There's already a question about the difference between DBN and DBM (https://ai.stackexchange.com/q/130/2444) and the difference between stacked DBN and DBM (https://ai.stackexchange.com/q/2583/2444).


text:
Does the concept of validation loss apply to training deep Q networks?
In deep learning, the concept of validation loss is to ensure that the model being trained is not currently overfitting the data. Is there a similar concept of overfitting in deep q learning?
Given that I have a fixed number of experiences already in a replay buffer and I train a q network by sampling from this buffer, would computing the validation loss (separate from the experiences in the replay buffer) help me to decide whether I should stop training the network?
For example, If my validation loss increases even though my train loss continues to decrease, I should stop training the training. Does deep learning validation loss also apply in the deep q network case?
Just to clarify again, no experiences are collected during the training of the DQN.

This question may be helpful How can I handle overfitting in reinforcement learning problems?.


text:
What is the advantage of using Google's Coral over Nvidia's Xavier?
I was reading about the possibility of using Google's Coral for deep learning-based object detection and image classification. I heard it has a good speed in terms of frames/sec.
I also read that Google's Coral is only compatible with quantized models. What does this mean? How will this affect the performance of object detection or classification in terms of accuracy and speed?
What is the advantage of using Google's Coral over Nvidia's Xavier?

Quantization is a technique used to make deep learning models smaller and faster to run.
Deep learning models are essentially collections of real-valued numbers. Because there are infinitely many real numbers, computers represent them using a format call 'floating point' numbers, which are not completely accurate. For example, a 32-bit floating point number can only represent at most $2^{32}$ distinct values. In contrast, a 64-bit floating point number can represent $2^{64}$ distinct values.
Most CPUs and GPUs cannot operate directly on large floating point numbers. This means that to do something like multiply two floating point numbers, the CPU might have to work on half of each number at a time, and do some tricky work to combine the results. Some CPUs and GPUs can operate directly on large floating point numbers, but only by using more than one core to work on a single number.
To get around this, you might chose to take a model you have that was trained with high-precision floating point weights, and reduce it to lower precision. The weights won't be exactly the same, but they'll be very close. Doing this will make the models run much faster, but you might lose some accuracy.
So the advantage of using a tool that only supports Quantized models is that models will run faster, but might be slightly less accurate.


text:
Can I find a mapping that minimizes the maximum distance ratio of certain vectors?
Let's say we have several vector points. My goal is to distinguish the vectors, so I want to make them far from each other.  Some of them are already far from each other, but some of them can be positioned very closely.
I want to get a certain mapping function that can separate such points that are close to each other, while still preserving the points that are already far away from each other.
I do not care what is the form of the mapping. Since the mapping will be employed as pre-processing, it does not have to be differentiable or even continuous.
I think this problem is somewhat similar to 'minimizing the maximum distance ratio between the points'. Maybe this problem can be understood as stretching the crushed graph to a sphere-like isotropic graph.
I googled it for an hour, but it seems that the people are usually interested in selecting the points that have such nice characteristics from a bunch of data, rather than mapping an existing vector points to a better one.
So, in conclusion, I could not find anything useful.
Maybe you can think 'the neural network will naturally learn it while solving classification problem'. But it failed. Because it is already struggling with too many burdens. So, this is why I want to help my network with pre-processing.

isn't it what kernels do?

which kernel do you mean?

The general concept of kernel trick in kernel methods in machine learning. such as SVM. the kernels do what you're saying: transforming to a new hyperspace which a discriminative hyperplane can easier separate classes.

this is interesting, but, why do you want to do this?

"vector points" means "set of points described as vectors" ? Is a 1D, 2D, 3D, ... problem ? The maximum distance will be reached send each point to the infinite (except one of them at origin). Do you have restrictions for the base space ?

related: https://math.stackexchange.com/questions/339649/maximizing-distance-between-points

An interesting question. I would start by finding n nearest neighbors of each data point, then calculate their center of mass c and the point's distance d to its nth nearest neighbor. The smaller the d is the larger the density is around a given point. You could then iteratively step every point away from their c in an inverse proportion to the distance d with a suitable step size. This would spread out the clusters.
But this won't help you transform any new points in the dataset, maybe you can learn this arbitrary mapping R^n -> R^n by using an other neural network and apply it to new samples?
This is the first ad-hoc idea which came to my mind. It would be interesting to see a 2D animation of this.
A more rigorous approach might be a variational autoencoder, you can embed the data in a lower dimensional space with approximately normal distribution. But it doesn't guarantee that clusters would be as spread out as you'd like. An alternative loss function would help with that, for example every point's distance to their original nth closest neighbor should be as close to one as possible.

Thanks for your advise. I also considered about using k-nearest neighbor(kNN) based scheme, but as you know it includes max (or min) function in it. So I think it is very hard to use kNN related metric as a loss function to train VAE, even though the goal mapping itself do not have to be a differentiable one. But maybe some relaxed version of it, such as stochastic NN could be a naive replacement.

After studying neural networks more I recently came up with a new idea (might not be novel): train an autoencoder (it doesn't matter if the encoded "code" dimension is smaller or same size as the input) and add extra penalty to the "code" activations if they are too close on known dissimilar inputs. I am about to write a blog article about this at some point.

Too late to edit the previous comment. Surely this is not a novel idea, it combines an autoencoder with a contrastive loss. But wikipedia's article on contractive autoencoder mentions only positive samples, which makes it more of a denoising autoencoder IMO.


text:
What are mono-variable and multi-variable neural networks?
In this document, the terms "Redes Neuronales estáticas monovariables" and "Redes Neuronales estáticas multivariables" are mentioned.
What are mono-variable and multi-variable neural networks? Is it the same as a multi-layer or uni-layer NN?
I have searched about multivariable/mono-variable static/dynamic neural networks in some books, but at least in those books there's no information about these topics.
I have the idea it refers to the inputs/outputs, but I'm not sure.

What are mono-variable and multi-variable neural networks?

I am not sure about this, because most (if not all useful) neural networks are multivariable neural networks (i.e. they contain multiple parameters). Even the perceptron usually contains more than one parameter, so that terminology isn't clear even to me. Maybe they are referring to the number of inputs (sometimes called variables), but I don't see why this distinction in this context would make sense.

What are static and dynamic neural networks?

To answer this question, I will first quote an excerpt from this document (written in Spanish) to provide some context to Spanish speakers (I am not a Spanish speaker, but I understand 99% of it).

Un primer intento de clasificación puede separ estáticos y dinámicos o recurrentes (fig.2.1)
Los modelos estáticos realizan un mapeo entre entrada y salida. Despreciando el tiempo de procesamiento interno, la salida se obtiene en forma inmediata en función de la entrada, no existe memoria ni dinámica de estados en el sistema neuronal.
Por el contrario los sistemas recurrentes si la poseen, son sistemas realimentados que ante un estimulo de entrada evolucionan hasta converger a una salida estable.
Casos tipicos de ambos sistemas son el Perceptrón (Rosemblatt, 1960a) (de una o múltiples capas) y la memoria asociativa de Hopfield, respectivamente
(Tank, 1987).

So, in this document, the word "dynamic" and "recurrent" are being used interchangeably. An example of a static (i.e. non-recurrent) neural network is the perceptron. An example of recurrent (or dynamic) neural network is the Hopfield network.
Anyway, I recommend you contact the author of that article to ask for clarification (especially, about the mono-variable NNs)!

i just asked to some people involved in that, and they said it refers to the kind of value the network receives in the variable, (Binary values) for monovariable, or Real values for multivariable : 1,2,3,4,5...

@CaesarCruez Yes, that makes more sense (and it's definitely possible to have networks that work with binary values) than the assumption that the network has only 1 variable!


text:
Does the reduction of the dimensions over multiple layers allow more details to be stored within the final representation?
From : https://debuggercafe.com/implementing-deep-autoencoder-in-pytorch/
the following autoencoder is defined
class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()

        # encoder
        self.enc1 = nn.Linear(in_features=784, out_features=256)
        self.enc2 = nn.Linear(in_features=256, out_features=128)
        self.enc3 = nn.Linear(in_features=128, out_features=64)
        self.enc4 = nn.Linear(in_features=64, out_features=32)
        self.enc5 = nn.Linear(in_features=32, out_features=16)

        # decoder
        self.dec1 = nn.Linear(in_features=16, out_features=32)
        self.dec2 = nn.Linear(in_features=32, out_features=64)
        self.dec3 = nn.Linear(in_features=64, out_features=128)
        self.dec4 = nn.Linear(in_features=128, out_features=256)
        self.dec5 = nn.Linear(in_features=256, out_features=784)

    def forward(self, x):
        x = F.relu(self.enc1(x))
        x = F.relu(self.enc2(x))
        x = F.relu(self.enc3(x))
        x = F.relu(self.enc4(x))
        x = F.relu(self.enc5(x))

        x = F.relu(self.dec1(x))
        x = F.relu(self.dec2(x))
        x = F.relu(self.dec3(x))
        x = F.relu(self.dec4(x))
        x = F.relu(self.dec5(x))
        return x

net = Autoencoder()

From the Autoencoder class, we can see that 784 features are passed through a series of transformations and are converted to 16 features.
The transformations (in_features to out_features) for each layer are:
784 to 256
256 to 128
128 to 64
64 to 32
32 to 16

Why do we perform this sequence of operations? For example, why don't we perform the following sequence of operations instead?
784 to 256
256 to 128

Or maybe
784 to 512
512 to 256
256 to 128

Or maybe just encode in two layers:
784 to 16

Does the reduction of the dimensions over multiple layers (instead of a single layer) allow more details to be stored within the final representation? For example, if we used only the transformation $784 \rightarrow 16$, may this cause some detail not to be encoded? If so, why is this the case?


text:
How do you manage negative rewards in policy gradients?
This old question has no definitive answer yet, that's why I am asking it here again. I also asked this same question here.
If I'm doing policy gradient in Keras, using a loss of the form:
rewards*cross_entropy(action_pdf, selected_action_one_hot)

How do I manage negative rewards?
I've had success with this form in cases where the reward is always positive, but it does not train with negative rewards. The failure mode is for it to drive itself to very confident predictions all the time, which results in very large negative losses due to induced deviation for exploration. I can get it to train by clipping rewards at zero, but this throws a lot of valuable information on the table (only carrots, no sticks).

If you have a question that has already been asked but no good answer has been given, one way to drive attention to that question is to start a bounty.

You don't need to manage negative rewards separately, if you implemented the algorithm correctly it will work regardless if the rewards are negative or not. You seem to be using rewards for the loss but you should be using the return which is the sum of the rewards for some state action pair from that point until the end of trajectory.
You also seem to be missing $-$ sign from the loss. The objective function for the vanilla policy gradient algorithm (REINFORCE) which we want to maximize is
\begin{equation}
J = \sum_a \pi(a|s) q_{\pi}(s, a)
\end{equation}
It can be shown that the gradient sample for this policy gradient method is
\begin{equation}
\nabla J = G_t \nabla \log (\pi(A_t|S_t))
\end{equation}
so in TensorFlow you should define your loss as
\begin{equation}
J = - G_t \pi(A_t|S_t)
\end{equation}
We need the $-$ because in TensorFlow you use minimizers, but we want to maximize this function so minimizing this loss is same as maximizing the objective function. In conclusion, the code similar to what you wrote, should be
-return * cross_entropy(action_pdf, selected_action_one_hot)
EDIT
As pointed out in the comment we don't actually need $-$ because it is already included in cross_entropy function.

Thanks.  I'm not sure about the sign as you have it.  cross_entropy loss in keras has an implied negative sign relative to the normal definition of crossentropy.  My models train without the minus sign.  See also: https://stackoverflow.com/a/56893454/2364295

But regardless of the above.  My issue is with negative rewards (or returns).  I implement exploration by deviating from the model PDF output in some cases.  If the model outputs nearly 100% confident actions and the exploration logic deviates, a negative return can lead to a hugely negative reward.  Perhaps my exploration approach is invalid.

You're right about the $-$, I edited the answer, as for the returns are you saying that the model drops performance when you choose an action that's not the one given by the model ? How do you choose it, do you sample it from the output distribution ?

Also, make sure you ciip the gradients, because the larger return will cause large parameter updates and the policy may break with too large updates because it will deviate from the current one too much.

When training I allow the model to explore by doing a proportional draw from the model provided softmax PDF.  The detail is that I also broaden the PDF a bit from the model version so it doesn't collapse and stop exploring (by adding a small value and renormalizing).  If the action I choose happens to have zero probability from the model, and negative return, the loss will be hugely negative.  I'm experimenting now with putting the PDF softening into the model itself, controlled by an input, so I can have the selected action always be consistent with the model.


text:
How does sampling works in case of imbalanced image datasets?
I am solving a problem of image classification of the image dataset for 3 classes. Dataset is highly imbalanced.
How will sampling (either over- or under-sampling) work in that case? Should I remove (or add) any random number of images, or should I follow some pattern?
In the case of CSV data, the general rule is to do PCA, and then remove the data points, but how to do it in the image dataset? Is there any other way to handle this problem?


text:
Is there any published research on the information-carrying capacity of the human face?

Is there any published research on the information-carrying capacity of the human face?

Here I mean "how much information can be conveyed via facial expressions & micro-expressions".
This is a subject of interest because the human face is arguably "the most interesting" single thing for humans, since it's likely the first real pattern we recognize as infants, and conveys so much non-verbal communication than can relate to achievement of a goal or identification of a mortal threat.  (Dogs similarly are said to have co-evolved to read human faces. Film acting as "the art of the closeup" also validates this viewpoint.)
Essentially, I'm tying to get a sense of how complex is the set of human facial expressions, what is the computational complexity of the range of problems related to identification of the range of possible expressions, and the emulation of such expressions to imitate a human agent.  (i.e. these techniques can be used to "read" a human subject or manipulate a human subject.)
Well researched articles & blogs would also be welcome.


text:
Can you use transformer models to do autocomplete tasks?
I've researched online and seen many papers on the use of RNNs (like LSTMs or GRUs) to autocomplete for, say, a search engine, character by character. Which makes sense since it inherently predicts character-by-character in a sequential manner.
Would it be possible to use the transformer architecture instead to do search autocomplete? If so, how might such a model be adapted?


text:
If the output of a model is a ridge function, what should the activation functions at all the nodes be?
I have the following assignment.

I can't understand the b part of this question in my assignment. I have completed the 1st part and understand the maths behind it, but the 2nd part has me stumped.
I looked up ridge functions and they basically map real vectors to a single real value, from what I understood. For that reason, I considered that the activation function has to be one that ranges over the real numbers, but that still doesn't clear my doubts.
I don't need a full answer just an explanation of the question will be very helpful, here's some text from the book I'm referring( Russel and Norvig), though I couldn't really grasp how this would help me choose an activation function.

Before delving into learning rules, let us look at the ways in which networks generate complicated functions. First, remember that each unit in a sigmoid network represents a soft threshold in its input space, as shown in Figure 18.17(c) (page 726). With one hidden layer and one output layer, as in Figure 18.20(b), each output unit computes a soft-thresholded linear combination of several such functions. For example, by adding two opposite-facing soft threshold functions and thresholding the result, we can obtain a “ridge” function as shown in Figure 18.23(a). Combining two such ridges at right angles to each other (i.e., combining the outputs from four hidden units), we obtain a “bump” as shown in Figure 18.23(b).


text:
Why are the value functions sometimes written with capital letters and other times with lower-case letters?
Why are the state-value and action-value functions are sometimes written in small letters and other times in capitals? For instance, why in the Q-learning algorithm (page 131 of Barto and Sutton's book but not only), we the capitals are used $Q(S, A)$, while the Bellman equation it is $q(s,a)$?

In the Sutton and Barto book $q(s,a)$ is used to denote the true expected value of taking action $a$ in state $s$, whereas capital $Q(s,a)$ is used to denote an estimate of $q(s,a)$. However, there is likely to be a lot of inconsistency in the literature as each author has their own preference on how to denote things. I would encourage you to consider whether the value you are reading is to denote an estimate or the true value.

Ordinary variables vs Random Variables
The difference is whether you're talking about a ordinary variable or a random variable.
For instance, the q-function (lowercase) is an expectation value (i.e. not a random variable), conditioned on a specific state-action pair:
$$
q(s,a)\ =\ \mathbb{E}_t\left\{ R_t+\gamma R_{t+1} + \gamma^2R_{t+2}+\dots\,\Big|\, S_t=s, A_t=a \right\}
$$
Then, in some case, some authors may abuse notation slightly by feeding in a random variable into the q-function, e.g. $q(S_t,a)$, $q(s,A_t)$ or even $q(S_t,A_t)$, thereby undoing some or all of the conditioning in the definition of the q-function as an expectation value.
Feeding a random variable into a function like the q-function results in an output that is a random variable in its own right. It is for this reason that some authors choose to give the function itself an uppercase letter as well.
My advice would be to think to yourself, is this a random variable? For the rest, I would interpret upper/lowercase as no more than a hint to the reader.

to add to your answer, capital Q is typically used when it is an estimate of the true q-value function. this follows from what you are saying as if it is an estimate then it is a stochastic approximation i.e. a random variable.

It's also worth mentioning that the notation in the first edition of Barto and Sutton's wasn't very consistent, but it should have improved in the second edition.

@Kris, could you maybe clarify what would passing a random variable into a function q mean?


text:
Is subsection generation $O(n^4)$
When I say template matching, I'm referring to finding occurrences of a small image (the template) in a larger image.
The OpenCV library provides the trivial solution, that slides the template over every possible location of the image. While this implementation provides translation invariance, it provides no stretching/scaling invariance. To get stretch and translation invariance, one would need to stretch the template (or shrink the main image) until iteratively, running the original template check over the image. This increases complexity to $O(S * n^2)$ where S is the number of different resolutions one checks - if one want's to check every possible resolution, the overall complexity is $O(n^4)$. Effectively, you generate $O(n^4)$ subsections and check if they're equal to the template.
From what I was taught, Image Segmentation Networks do just this - however, instead of using the basic template matching (i.e. checking the pixels match) the generated subsection is put through a classifier network - so this would be more expensive than standard templating.

My question is, are my calculations correct - for complete subsection generation, is the complexity $O(n^4)$ and is there really no better algorithm for generating these subsections - used by both image detection algorithms?

Hi and welcome to AI SE! OpenCV also provides other tools to perform template matching (such as a combination of SIFT, FLANN and findHomography), which, in theory, should be translation and scale-invariant.


text:
What is the proof that "reward-to-go" reduces variance of policy gradient?
I am following the OpenAI's spinning up tutorial Part 3: Intro to Policy Optimization. It is mentioned there that the reward-to-go reduces the variance of the policy gradient. While I understand the intuition behind it, I struggle to find a proof in the literature.

Does the answer to this question answer yours as well?

No, the linked question only proofs that the reward-to-go does not introduce any bias to the gradient estimate.

This is nontrivial to prove, actually anything involving stochastic function approximation is nontrivial. You can search research papers, you won't find it in any book right now


text:
What is the reason for different learned features in upper and lower half in AlexNet?
I was reading AlexNet paper and the authors quoted

the kernels on one GPU were "largely color agnostic," whereas the kernels on the other GPU were largely "color-specific."

The upper GPU takes operates on filters on the top and lower GPU deals with the lower half. But what is the reason for each of them learning a different set of features, i.e. the top half of kernels learning the edges mostly and the bottom kernels learning color variation? Is there any reason behind it?


text:
How GAN generator produce integer RGB colored picture?
For traditional neural networks, I know that we can't constraint the output to be strict integers. My question is what technique does GANs use to produce integer outputs, that can be then converted to RGB colored pictures?

I think the value is rounded to the nearest integer. Or whatever rounding scheme is preferred.

@PyWalker27 Thanks, I was thinking perhaps there might be some technique in there.


text:
Should I just use exploitation after I have trained the Q agent?
When using a trained Q-learning algorithm in an actual game, would I just use exploitation and no longer use exploration? Should I use exploration only during the training phase?

Once you have estimated the $Q$ function, you can derive the policy from it in different ways. For example, you can act greedily with respect to it (see this answer), which can be formally denoted as
$$
\pi(s) = \operatorname{argmax}_{a^*}Q(s, a), \; \forall s \in \mathcal{S}
$$
where $Q(s, a)$ is your estimated value function and $\pi$ the policy greedily derived from it.
This means that you would just exploit your current knowledge of the return. This is probably a good thing to do if you believe your value function is optimal and the dynamics of the environment don't change.
Of course, if your policy is not optimal, you may not want to always execute the greedy action. In that case, you could still perform some form of exploration (e.g. with the $\epsilon$-greedy policy).
Moreover, if the dynamics (e.g. the reward function) of your environment change over time, you could continually train your RL agent. If you are interested in continual RL, the paper Continual Reinforcement Learning with Complex Synapses (2018) could potentially be useful.


text:
What happens if the opponent doesn't play optimally in minimax?
I just read an article about the minimax algorithm. When you design the algorithm, you assume that your opponent is a perfect player, i.e. it plays optimally.
Let's consider the game of chess. What happens if the opponent plays irrationally or sub-optimally? Do you still have a guarantee that you are going to win?

What happens if the opponent plays irrationally or sub-optimally? Do you still have a guarantee that you are going to win?

If your search is deep enough to guarantee optimal play in all cases, then yes. Optimal play is such that the opponent's decision causes least impact to your agent. In fact, if the opponent makes a mistake, often that can make the search easier/faster, and the agent will win more convincingly.
What it may mean is that optimisations you may have taken - e.g. pruning game tree segments that lead to non-optimal decisions by either player - might not be as useful. This might impact decision time if you keep some partial game tree or cache branch evaluations between moves to help speed up the agent.
Actual optimal agents for games as complex as chess are not possible. In these games, you will not have a truly optimal agent, but approximately optimal. You will be relying on some heuristic to guide the minimax search when it cannot force an end game win. If the opponent manages to control play into a state where the heuristics are not accurate, they could cause minimax search to fail and misdirect your agent into making mistakes.
A combination of effects is also possible if you have implemented a tree caching mechanism for performance improvements and made the system playable by limiting planning time - e.g. you limit computer search time to 3 seconds max - an irrational opponent may cause your agent's performance to degrade to the point where it too starts to make mistakes. Whether or not this is enough for a smart opponent to take advantage of it and beat an agent which is capable of otherwise playing a "perfect" game depends on details of the game, and how narrow the agent's measure of "perfect" is.
An extreme case might be an agent that has memorised a single perfect game (by scoring high heuristics for any state on a single tarjectory through the game that has been pre-calculated for perfect moves by both players), and has poor heuristics otherwise - once state moves away from what it can evaluate directly, the agent will be limited by how well it can search for completed game wins, and can easily be manipulated by a more generally smart opponent into a losing position that is beyond the depth of its search.
In practice, well-coded agents will not suffer too much from this effect. If you make a mistake, or try a random probably bad move in an attempt to confuse the agent when playing against Stockfish, you will lose.


text:
What are the common pitfalls that we could face when training neural networks?
Apart from the vanishing or exploding gradient problems, what are other problems or pitfalls that we could face when training neural networks?

There are several pitfalls or issues that require your attention when or before training or using neural networks. I will list some of them below, along with some questions you need to ask yourself before or while using neural networks.

Over-fitting and under-fitting problems, and the related generalization problem. Is your neural network generalizing to unseen data?
Availability of training and test data

Do you have enough data to train your neural network so that it generalizes well (i.e. it neither over-fits or under-fits)?
Is your test dataset big enough to assess the generalization ability of your neural network?
Is your data representative of the problem you are trying to solve?
Do you need to augment or normalize your data?
Do you need to use cross-validation?
Is your data independent and identically distributed (i.i.d.)? If your data is correlated, training could be unstable. Shuffling your data may be a feasible solution when your data is initially correlated.

Do you have enough computational resources (i.e. GPUs) for training and testing your neural network?
Are you solving a regression or classification problem? The type of the outputs and the loss function will typically be different in both cases
Do you need explainability and transparency? If yes, neural networks aren't probably the best model to use, as the connections between the neurons are quite obscure and don't really represent any meaningful interaction. That's why neural networks are called black-boxes.
Do you need uncertainty estimation? If yes, you may want to try Bayesian neural networks. Typical neural networks are not very appropriate for uncertainty estimation!
If you use a neural network for function approximation (e.g. in reinforcement learning), you will lose certain convergence guarantees.

I can't say that it is the biggest problem of deep neural network but it is one of the big problem with deep neural network.
Other issue which happens a lot is overfitting on the training data hence network behaves badly on test set which can be solved using regression. So you have to make sure that network is generalised enough.
If the network is trying to have a classification then if the outputs are mutually inclusive or exclusive, depending upon that you define your loss function which will be categorical cross entropy for mutually inclusive output and binary cross entropy for mutually exclusive.
Other thing which I can think is about all zero initialization. Initializing the network weights as zero can lead to same gradient computation and due to which network does not learn most of the times.

The question wasn't about what causes the VGP or how to solve it. So, I suggest you edit your answer so that it mainly addresses the main question.  Also, I suggest you do not provide a link to external resources. In your words, describe the issues that are mentioned in the linked article!


text:
Why do we update $W$ with $\frac{1}{\mu (A_t | S_t)}$ instead of $\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$ in off-policy Monte Carlo control?
I had the same question when I am reading the RL textbook from Sutton Bartol as posted here.

Why do we update $W$ with $\frac{1}{\mu (A_t | S_t)}$ instead of $\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$?
It seems that, with the updating rule from the textbook, whatever action $\mu$ decides to choose, we automatically assume that $\pi$ will choose it with 100% probability. But $\pi$ is greedy with respect to Q. How does this assumption make sense?

It says there that $\pi$ is a deterministic policy

Ah I see! It could be 0 but in case of 0, loop already breaks before updating W. Should have read the algorithms a few more times. Thank you very much!


text:
What does a self-improving artificial general intelligence with finite resources and infinite time do?
What would happen when an artificial general intelligence can improve itself over a long time, with limited resources?
The assumption is that it has a large but finite amount of computing power, and can not escape that limit to find resources. Let's assume the limit to be on the order 100 human brains or so - the limits are not important, only that it is limited.
Now, we let it run to self-improve as much as it can with the given resources, until it goes to some stable or periodic state.
What it can do is at least one of the following:

converge to some state
oscillate between two or more states
go into a chaotic state
stop doing anything
disable itself in some other way than just stop doing anything

There are certainly more complex behaviors possible:

become maximally happy, and then periodically redefine what happy means
hacking its reward function in some other way

I would expect it to converges to a single state - but that seems naive.
Are there any ideas on how it would end up?

To be clear, you're assuming that you have an AGI. Right? So, this question assumes that there's an AGI already that is able to self-improve. Right? You talk about states, but you don't define states, so this is a bit dangerous.
I suggest you simplify as much as possible your post and question and try to make it clear what you want to ask and define the terms you are using. This is clearly a philosophical, question, but not only.

Also, you talk about "finite resources" but also about "infinite time". Isn't time also a resource? So, you should probably clearly define what you mean by "resource". I am trying to make this question a very good question. I think that it's already interesting, but, hopefully, we can make it more scientifically answerable.

@nbro Yes, the term "resource" is too general there. In terms of finite ressources, but infinite time: I do not want to limit how long it takes, I' want to wait until it gets stable or periodic, basically. "Wait until nothing new happens". Yes, that are all good points, thanks!


text:
Can a typical supervised learning problem be solved with reinforcement learning methods?
Let's say I want to teach a neural to classify images, and, for some reason, I insist on using reinforcement learning rather than supervised learning.
I have a dataset of images and their matching classes. Then, for each image, I could define a reward function which is $1$ for classifying it right and $-1$ for classifying it wrong (or perhaps even define a more complicated reward function where some mistakes are less costly than others).  For each image $x^i$, I can loop through each class $c$ and use a vanilla REINFORCE step: $\theta = \theta + \alpha \nabla_{\theta}log \pi_{\theta}(c|x^i)r$.
Would that be different than using standard supervised learning methods (for example, the cross-entropy loss)? Should I expect different results?
This method actually seems better since I could define a custom reward for each misclassification, but I've never seen anyone use something like that

Is your question a duplicate of https://ai.stackexchange.com/q/14167/2444? I don't think it's an exact duplicate, but these two questions are related. Can you clarify how these two questions are different?

authors: شاكر ألكالدي كوريا2


text:
What are the most popular and effective approaches to leveraging AI for stock price prediction?
Currently, what are the most popular and effective approaches to leveraging AI for stock price prediction?
It seems like there could be several approaches and problem formulations:

Supervised learning:
Regression: predict the stock price directly
Classification: predict whether the stock price goes up or down
Unsupervised learning: find clusters of stocks that move together
Reinforcement learning: let the agent directly maximize its stock market return
Other AI methods: rules, symbolic systems, etc.

Which are most popular/performant? Are there other ways that people are using machine learning in stock trading (sentiment analysis on financial statements, news, etc.)?

Your list is complete for what is considered 'popular' by most practitioners who apply AI for stock trading.  Supervised learning and rule learning are at the top for accuracy.  There are more academic papers published on classifiers than on regression approaches;
classifiers are typically more accurate than regressors.

One technique I've used in the past (I don't know how mainstream this is) is to use unsupervised learning (self-organizing maps) to identify commonly occuring features in stock charts, and then use those features as input to a traditional back propagation network.  I found I was able to get equivalent results with slightly smaller network size having done this, but it wasn't a huge gain.

NLP is one of the tools you may use for stock prediction. Here, are a couple of articles to help you get started:

Using NLP and deep learning to predict the stock market
Here's how I predicted apple's stock price using NLP

Do you know any company that has published the way they are using NLP to predict the price of a stock (in the short team I guess)?

Usecases I came across:

As mentioned by Saurav, the use of NLP is definitely a use case. Adding to his source, you can check out the Two Sigma's Kaggle competition. Going through their careers page it is evident that use of NLP is prominent. Cofounder of Kavout, Alex Lu reconfirms that in
https://emerj.com/ai-podcast-interviews/artificial-intelligence-in-stock-trading-future-trends-and-applications/

Two Sigma Challenge Kaggle: https://www.kaggle.com/c/two-sigma-financial-news

ML to determine whether some large market player is rebalancing or liquidating his or her portfolio.
Source: https://qr.ae/pGnoGx
I have heard about ML being used in Statistical Arbitrage(Pairs trading) also.
Example case: http://cs229.stanford.edu/proj2019spr/report/26.pdf
Further, I am aware of use of ML for hedging purposes in a couple of banks. It's called deep hedging. Source: https://www.maths.ox.ac.uk/system/files/attachments/2019%2004%2024%20Deep%20Hedging%20Frontiers%20Imperial%202.1.pdf


text:
What are preferences and preference functions in multi-objective reinforcement learning?
In RL (reinforcement learning) or MARL (multi-agent reinforcement learning), we have the usual tuple:
(state, action, transition_probabilities, reward, next_state)

In MORL (multi-objective reinforcement learning), we have two more additions to the tuple, namely, "preferences" and "preference functions".
What are they? What do we do with them? Can someone provide an intuitive example?

In MORL the reward component is a vector rather than a scalar, with an element for each objective. So if we are using a multiobjective version of an algorithm like Q-learning, the Q-values stored for each state-action pair will also be vectors.
Q-learning requires the agent to be able to identify the greedy action in any state (the action expected to lead to the highest long-term return). For scalar rewards this is easy, but for vector values it is more complicated as one vector may be higher for objective 1, while another is higher for objective 2, and so on.
We need a means to order the vector values in terms of how well they meet the user's desired trade-offs between the different objectives. That is the role of the preference function and preferences. The function defines a general operation for either converting the vector values to a scalar value so they can be compared, or for performing some sort of ordering of the vectors (some types of orderings such as lexicographic ordering can't readily be defined in terms of scalarisation). So, for example, our preference function might be a weighted sum of the components of the vector. The preferences specify the parameters of the preference function which define a specific ordering (i.e. based on the needs of the current user). So, in the case of a weighted sum for the preference function, the preferences would be specified in terms of the values of the weights.
The choice of preference function can have implications for the types of solutions which can be found, or for whether additional information needs to be included in the state in order to ensure convergence.
I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a useful introduction to this area)

Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113.


text:
How are the classical MDP and the object-oriented MDP views different?
I've been reading the attached paper - which aims to model entities in the world as objects, including the learning agent itself!
To say the least, the goal is to navigate through what seems like a maze (path-planning problem) - and drop off passengers in desired destinations, while avoiding walls in the map of the world (5x5 grid for now). The objects involved are, a taxi, passengers, walls and a destination.
Now, a particular paragraph says the following:

"Whereas in the classical MDP
  model, the effect of encountering walls is felt as a property of specific locations in the grid, the OO-MDP view is that wall interactions are the same regardless of their location. As such, agents’ experience can transfer gracefully throughout the state space."

What does this mean? How are the classical MDP and the object-oriented MDP views different?
I can't make sense of the above extract, at all. Any help would be appreciated!
P.S. I did not consider posting parts of the extract as separate questions since my problem has more to do with understanding the extract as a whole which inevitably relies on understanding the parts.

Maybe it doesn't define the state in terms of the location but instead the local properties. Thus the mdp only considers the agents past local properties. This kind of makes sense.

it should be also noted this approach will always result in a finite horizon solution, you can't do global planning with it since it's always operating locally,

Could you elaborate a bit on what you mentioned in the comments, @FourierFlux? Also, what is meant by "local properties"? How does it help in making sense of the line - "wall interactions are same regardless of location" and "agents' experience can transfer gracefully throughout the state space"?

The state space im assuming only consists of what the agent sees and directly interacts with.


text:
How should I define the loss function for a multi-object detection problem?
I'm trying to create a text recognition project using CNN. I need help regarding the text detection task.
I have the training images and bounding box details for them. But I'm unable to figure out how to create the loss function.
Can anyone help me by telling how to take the output from the CNN model and compare it to the bounding box labels?

Okay so your CNN model is taking an Image and outputting the bounding boxes for them. That means the last layer of CNN model must be having four outputs which are generating real numbers. This is a regression problem.
In that case, you can take L1 loss (mean absolute error) or L2 loss (mean square error) as your loss function. I have created a similar project, I have used L1 loss.
Suppose your input image is x and predicted_bb is the model output and real_bb is your original bounding box for image x. Then you should proceed as follows
predicted_bb = model(x)

# CALCULATE LOSS BETWEEN THE ACTUAL & PREDICTED BB COORDINATES

loss_bb = torch.nn.functional.l1_loss(predicted_bb, real_bb, reduction="none").sum(1)

# SET GRADIENTS TO ZERO

optimizer.zero_grad()

# BACKPROPOGATE THE LOSS

loss.backward()

For tensorflow keras
predicted_bb = Dense(4, activation='relu')(x)

model = Model(inputs = image_input, outputs = [predicted_bb])
model.compile(loss=['mae'], optimizer='adam', metrics =['accuracy'])

this answer doesn't explain how to deal when you have more than one bounding boxes for the same class and your network is outputting say (100, 4) shaped output


text:
What is the point of using 1D and 2D convolutions with a kernel size of 1 and 1x1 respectively?
I understand the gist of what convolutional neural networks do and what they are used for, but I still wrestle a bit with how they function on a conceptual level. For example, I get that filters with kernel size greater than 1 are used as feature detectors, and that number of filters is equal to the number of output channels for a convolutional layer, and the number of features being detected scales with the number of filters/channels.
However, recently, I've been encountering an increasing number of models that employ 1- or 2D convolutions with kernel sizes of 1 or 1x1, and I can't quite grasp why. It feels to me like they defeat the purpose of performing a convolution in the first place.
What is the advantage of using such layers? Are they not just equivalent to multiplying each channel by a trainable, scalar value?

Traditional CNNs used for image classification (and related tasks) are composed of 1 or more fully connected layers (FCs), after the convolutional and pooling layers, which take as input the features extracted from the convolutional and pooling layers, in order to perform classification or regression.
One problem with FCs in CNNs is that the number of parameters can be very big, with respect to the number of parameters in the convolutional layers.
There are tasks, such as image segmentation, where this big number of parameters is not really needed. An example of a neural network that does not make use of fully connected layers but only uses convolutions, downsampling (aka pooling), and upsampling operations is the U-net, which is used for image segmentation. A neural network that only uses convolutions is known as a fully convolutional network (FCN). Here I give a detailed description of FCNs and $1 \times 1$, which should also answer your question.
In any case, to answer your question more directly, $1 \times 1$ convolutions have been used for image segmentation tasks, i.e. dense classification tasks, i.e. tasks where you want to assign a label to each pixel (or a group of pixels), as opposed to sparse classification tasks such as image classification (where the goal is to assign 1 label to the whole image). Moreover, in comparison with FC layers, they have fewer parameters and, more importantly, the number of parameters in an FCN does not depend on the dimensions of the images (as in the case of traditional CNNs), which is a good thing (especially, when your images have high resolutions), but typically it depends on the number of kernels and instances (of objects), in the case of instance segmentation.
The FCN paper discusses this reduction of the number of parameters (and computation time), so you should probably read this paper for more details.

Typically 1x1 convolutions are used for changing the number of channels. Each output channel is a linear combination of the input channels.
For example, if you perform a 1x1 convolution with only one output channel on an RGB image, then you get a grayscale image, whose intensity is a linear combination of the red, green, and blue values of the corresponding pixel (plus bias).
If you perform a 1x1 convolution with more than one output channel, then each channel is formed in the same way, as a linear combination of the input channels. You can think of it as multiple convolutions, whose output is stacked on top of each other. All these filters have different parameters.
Notice that if the output was equivalent to multiplying each channel by a scalar value, then you would always have the same number of inputs and outputs.


text:
Are there any good resources (preferably books) about techniques used for entity extraction?
Given some natural language sentences like

I would like to talk to Mr. Smith

I would like to extract entities, like the person "Smith".
I know that frameworks, which are capable of doing so (f. e. RASA or spaCy), exist, but I would like to dive deeper and understand the theory behind all this.
At university, I learned a few of the basic models like CRF's or SVM's used for this task, but I wonder if there are any good resources (preferably books) about this topic.


text:
If two objects are too close to each other, would an object detector do a poor job of correctly classifying them?
Suppose we have an object detector that is trained to detect $20$ products. If two objects are too close to each other, in general, would an object detector do a poor job of correctly classifying them? If they were far apart in the scene, would the object detector to a better job of correctly classifying them?

As you ask, "in general...", I will answer generally, however this changes a lot from model to model and the way they handle close objects.
In general, yes, they would do a poor job detecting very close objects, switch to segmentation models for that (for class or better, instance segmentation).
In general, objects detectors learn to tell an object from other based in 2 criterion:

Intersection over union: for object of the same class
Class probability: for objects of different class

So, if two objects of the same class are very close, the 2 detected bounding boxes will be highly overlapping, then, the Non Maximal Suppression filter will remove one of them. This is where objects detector, in general, perform worse.
Similarly, if two objects belong to different classes the 2 detected bounding boxes will be highly overlapping but the NMS filter won't remove them (again, in general, NMS is set only for same class objects). However when 2 objects are very close, there is a high chance they are partially occluded. Objects detectors, in general, don't handle occlusions very well.
So, in conclusion, objects detectors will perform better detecting far-away objects.


text:
Should neural nets be deeper the more complex the learning problem is?
I know it's not an exact science. But would you say that generally for more complicated tasks, deeper nets are required?

The gist is that you can't just pick any numbers for width and depth and expect optimal performance. Picking the right size network is somewhat trial-and-error, and somewhat of an art.

Deeper models can have advantages (in certain cases)
Most people will answer "yes" to your question, see e.g. Why are neural networks becoming deeper, but not wider? and Why do deep neural networks work well?.
In fact, there are cases where deep neural networks have certain advantages compared to shallow ones. For example, see the following papers

The Power of Depth for Feedforward Neural Networks (2016) by Ronen Eldan and Ohad Shamir
Benefits of depth in neural networks (2016) by Matus Telgarsky.
Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks (2017) by Safran and Shamir
Optimal approximation of piecewise smooth functions using deep ReLU neural networks (2018) by Petersen and Voigtlaender

What about the width?
The following papers may be relevant

Wide Residual Networks (2017) by Sergey Zagoruyko and Nikos Komodakis
The Expressive Power of Neural Networks: A View from the Width (2017) by Zhou Lu et al.

Bigger models have bigger capacity but also have disadvantages
Vladimir Vapnik (co-inventor of VC theory and SVMs, and one of the most influential contributors to learning theory), who is not a fan of neural networks, will probably tell you that you should look for the smallest model (set of functions) that is consistent with your data (i.e. an admissible set of functions).
For example, watch this podcast Vladimir Vapnik: Statistical Learning | Artificial Intelligence (AI) Podcast (2018), where he says this. His new learning theory framework based on statistical invariants and predicates can be found in the paper Rethinking statistical learning theory: learning using statistical invariants (2019). You should also read "Learning Has Just Started" – an interview with Prof. Vladimir Vapnik (2014).
Bigger models have a bigger capacity (i.e. a bigger VC dimension), which means that you will more likely overfit the training data, i.e., the model may not really be able to generalize to unseen data. So, in order not to overfit, models with more parameters (and thus capacity) will also require more data. You should also ask yourself why people use regularisation techniques.
In practice, models that achieve state-of-the-art performance can be very deep, but they are also computationally inefficient to train and they require huge amounts of training data (either manually labeled or automatically generated).
Moreover, there are many other technical complications with deeper neural networks, for example, problems such as the vanishing (and exploding) gradient problem.
Complex tasks may not require bigger models
Some people will tell you that you require deep models because, empirically, some deep models have achieved state-of-the-art results, but that's probably because we haven't found cleverer and more efficient ways of solving these problems.
Therefore, I would not say that "complex tasks" (whatever the definition is) necessarily require deeper or, in general, bigger models. While designing our models, it may be a good idea to always keep in mind principles like Occam's razor!
A side note
As a side note, I think that more people should focus more on the mathematical aspects of machine learning, i.e. computational and statistical learning theory. There are too many practitioners, who don't really understand the underlying learning theory, and too few theorists, and the progress could soon stagnate because of a lack of understanding of the underlying mathematical concepts.
To give you a more concrete idea of the current mentality of the deep learning community, in this lesson, a person like Ilya Sutskever, who is considered an "important and leading" researcher in deep learning, talks about NP-complete problems as if he doesn't really know what he's talking about. NP-complete problems aren't just "hard problems". NP-completeness has a very specific definition in computational complexity theory!

I'm having trouble following this answer. Are you saying that the answer might be "yes" in practice, but in theory we might find ways to solve more complex problems with shallower nets?

@TheGuywithTheHat Roughly, yes. Current methods that have solved "complex tasks" aren't particularly clever and don't scale well. If you have to use huge amounts of data or spend so much time and resources to train a net, or if training pollutes the environment so much, you're probably not doing the cleverest thing. We still don't understand many things about NNs. With a more solid understanding of learning theory, we will be able to come up with cleverer solutions.

Actually there is evidence that increasing the number of parameters can, paradoxically, reduce the overfitting error. One paper I looked at presented a theory that the smallest model that can be trained to fit the data represents a local maximum in generalization error: you either want a smaller model (to reduce VC dimension) or a larger model (for reasons that are not entirely clear; perhaps an overparametrized model ends up behaving a bit like an ensemble when trained with current methods). But mostly this chain of reasoning is concerned with the width rather than the depth of models.

@CharlesStaats Can you provide the title of and link to the paper?

DEEP DOUBLE DESCENT:
WHERE BIGGER MODELS AND MORE DATA HURT

My experience from a tactical standpoint is to start out with a smaller simple model first. Train the model and observe the training accuracy and validation loss and validation accuracy. My observation is that, to be a good model, your training accuracy should achieve a value of at least 95%. If it does not, then try to optimize some of the hyper-parameters. If the training accuracy does not improve, then you may try to incrementally add more complexity to the model. As you add more complexity the risk of overfitting, vanishing or exploding gradients becomes higher.
You can detect overfitting by monitoring the validation loss. If as the model accuracy goes up the validation loss on later epochs starts to go up you are overfitting. At that point, you will have to take remedial action in your model like adding dropout layers and use regularizers. Keras documentation is here.
As pointed out in the answer by nbro, the theory addressing this issue is complex. I highly recommend the excellent tutorial on this subject which can be found on YouTube here.

Deeper networks have more learning capacity in the sense that they can fit to more complex data. But at the same time, they are also more prone to overfitting the training data and therefore fails to generalize to the test set.
Apart from overfitting, exploding/vanishing gradients is another problem which hampers convergence. This can be addressed by normalizing the initialization and normalizing the intermediate layers. Then you can do backpropagation with stochastic gradient descent (SGD).
When deeper networks are able to converge, another problem of 'degradation' has been detected. The accuracy saturates and then starts to degrade. This is not caused by overfitting. In fact, adding more layers here leads to higher training error. A possible fix is to use ResNets (residual networks), which have been shown to decrease 'degradation'

Speaking very generally, I would say that with the current state of machine learning, a "more complicated" task requires more trainable parameters. You can increase parameter count by either increasing width and also by increasing depth. Again, speaking very generally, I would say that in practice, people have found more success by increasing depth than by increasing width.
However, this depends a lot on what you mean by "more complicated". I would argue that generating something is a fundamentally more complicated problem than just identifying something. However, a GAN to generate a 4-pixel image will probably be far more shallow than the shallowest ImageNet network.
One could also make an argument that the definition of complexity of a deep learning task is "more layers needed == more complicated", in which case it's obvious that by definition, a more complicated task requires a deeper net.


text:
A model for each sub-problem vs one model for the whole problem
Let's say one wants to use a neural net to learn some function $g(x)$. Let's say that we know that $g$ is a combination of two functions (or two sub-problems), $g(x)=f_2(f_1(x))$, and that we have two datasets

composed of $x$ samples and their corresponding $g(x)$ labels, and
composed of $x$ samples and their corresponding $f_1(x)$ labels.

Should we use two nets, one to learn the mapping from $x$ samples to $f_1(x)$ using dataset 1 and another net to learn the mapping from $f_1(x)$ to $g(x)$ (note that we can build a dataset composed of $f_1(x)$ samples and $g(x)$ labels with the trained net), or just one net to learn mappings from $x$ to $g(x)$ using dataset 1?
Intuitively, the first option seems to be better since we take advantage of our knowledge that $f_1$ is a "sub-problem" of $g$.

The tendency in literature in the last years (at least for computer vision problems) seems to point towards the single model option (I'll try to remember to come back and add some links to papers mentioning this when I find them), although this IMO is really data- and problem-dependent.
In your case, I would set up a network for the mapping $x$ to $g(x)$, with a training-only auxiliary loss calculated on the mapping $x$ to $f1(x)$ and compare this with a model trained only on "$x$ to $g(x)$".

I didn't understand your second paragraph. Did you mean to say that you'd try both options and see which one is better?

No. I'd try the second option and compare it with a model that predicts both F1(X) and g(X). The model has two regression heads, one "earlier" in the network for F1(X) and one deeper that regresses g(X). You train as a multi-task problem using a combined weighed loss on both regression targets, but then for inference you only use the output from the head that predicts g(X). The idea is to use the first regression head to steer the training in the "easier" direction by incorporating the knowledge that predicting F1(X) helps predicting G(X).


text:
What does the Fourier transformed image mean?
I have been trying to figure out what the Fourier transformed image represents. I am aware of Fourier transformation in general, but I can't explain myself the image it forms after transformation.
In the given image, what does the outlined white sort of lines mean?

Have a look at this explanation of the two dimensional Fourier Transform applied to an image.
The way I read it is that it somewhat mirrors the diagonal structure of the 'X', as that is where most of the 'image energy' is, so that are the lines you highlighted. Because the image is fairly uniform in colour, most of the components are low-frequency (ie close to the centre of the 2DFT image).
Disclaimer: I'm not really that familiar with a 2D version either, as I mainly have experience with using FFTs in speech processing.


text:
Is there a theory that captures the following ideas?
A big class of problems that are relevant in today's society are full of uncertainty and are also sometimes computationally intractable. Along our lives we come to realize that we are solving the same type of problem multiple times, sometimes with different strategies and mixed results. I would like to close in on three main problem types: pattern recognition, regression and density estimation.
An agent (computer program or even a human) that identifies the type of a problem and applies a systematic procedure for finding its solution. A solution is understood in the classical sense for each of the problem types, thus, the solution does not have to be a global optima. This procedure must be implementable.
Bonus points

Uses metadata about the problem itself to 'gain insight' about the nature of the problem.
Verifies that its solution is correct in some sense.
The types or classes of problems can be expanded later on.
Works with very limited resources.
Works with little information about the problem, or with "small data".

So far, I've found Statistical Learning theory and Bayesian Inference as candidates that implement some of those ideas, but I was wondering if there's something else out there or I just need to take the best of both of those worlds.