text:
Which neural network should I use to approximate a specific but unknown function?
We have convolutional neural networks and recurrent neural networks for analyzing, respectively, images and sequential data.
Now, suppose I want to approximate the unknown function $f(x,y) = \sin(2\pi x)\sin(2\pi y)$, with domain $\Omega = [0,1]\times [0,1]$, that is, $x$ and $y$ can be between $0$ and $1$ (inclusive).
How do I determine which neural network architecture is more appropriate to approximate this function? Which kind of activation functions would be better suited for this?
Note that, generally, I don't know a priori which function the neural network has to learn. I am just asking for this specific $f(x, y)$, as it could be a solution for a differential equation. And $\Omega$ is the domain, i.e., I don't care about the output of the neural network outside $\Omega$.

If the concept class specified is
$$f(x, y) = k \, \sin(2 \pi f_x x) \, sin(2 \pi f_y y) \\ \land 0 < x < 1 \\ \land 0 < y < 1 \; \text{,}$$
and the optimum fit to example data is expected occur when $k \approx 1 \land f_x \approx 1 \land f_y \approx 1$, then it is not an AI problem. It is a problem that can be solved with a least squares convergence, probably in conjunction with a Fourier transform.
If nothing is known about $f(x, y)$ except continuity and that it is single valued with respect to $(x, y)$, then few conclusions can be drawn about best approach. In such a case, the domain of $x$ and $y$ are irrelevant because they can be normalized. Furthermore, the tree of operations, such as $\sin()$ and multiplication, are irrelevant too, because the function could just as easily be
$$f(x, y) = \ln(x) + \Gamma(y) - k \, \text{.}$$
The question indicates the design involves CNN and RNN components for analyzing images and sequential data. It is not clear whether the CNN is for the discovery of objects or waves (given the $\sin()$ in the function mentioned) and whether those objects move between frames so that the RNN must detect motion.
Nothing is given about the pool of example data available or planned to be available or the expected outputs of the system. If data is sequential, where is $t$ in the function? What is the objective of image analysis?
Although a deep MLP (multilayer perceptron) with SGD can learn an arbitrary function, it is hardly an architecture, the mention of images, CNN, RNN, and sequential data, MLP with SGD does not seem to match.
Regarding activation functions, the inner layer functions would depend on the higher level design requirements. The activation functions of the last layer of a single artificial network is usually chosen to match the data type and range of desired output for each output channel (dimension).
If the objective of this question is to take images and sequential data and produce something useful without a priori defining what useful means, then it is an unsolved AI problem thus far and no known topology comprised of artificial networks and other AI building blocks provide a solution. The autonomous development of internal concepts of usefulness would need to be developed mathematically and algorithmically and become practically speed optimized in hardware and software first.

For example, suppose I want to approximate the function
mashine learning is used to approximate an UNKNOWN function. If you do want to do this with NN, you will have a regression task with small full connected network with like:

Input is 2 values of range 0 ..1
a few hidden layers, just start trying with one, of size like 3-6 neurons , activation  sigmoid or tanh
need a non-linear function here to approx non linear sin * sin, whatever
last layer with no activation function just making a difference from... known value f(x,y)


text:
Which problems in AI are not machine learning?
Which problems in AI are not machine learning? Which problems involve both AI and machine learning?

This is a common question and it has already been answered in many different places:
What is the difference between artificial intelligence and machine learning?

Artificial Intelligence vs. Machine Learning vs. Deep Learning

Artificial intelligence is a broader concept than machine learning, which addresses the use of computers to mimic the cognitive functions of humans. When machines carry out tasks based on algorithms in an "intelligent" manner, that is AI. Machine learning is a subset of AI and focuses on the ability of machines to receive a set of data and learn for themselves, changing algorithms as they learn more about the information they are processing.

More specific to your question:
AI without machine learning

If you insert a small amount of knowledge into a machine, you can call it an engineering product. But if you instill a sufficiently large amount of knowledge such that the machine makes better decisions than a human, that can be could AI. For example, if you take hundreds of medical doctors and each spends hundreds of hours detailing correlations between symptoms and disease, if you then pack that knowledge into an easy-to-use machine, that's AI without machine-learning

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
If we use a perceptron with a non-monotonic activation function, can it solve the XOR problem?
I found several papers about how to build a perceptron able to solve the XOR problem. The papers describe a solution where the heaviside step function is replaced by a non-monotonic activation function. Here are the papers:

Single Layer Neural Network Solution for XOR Problem
Solving the XOR and parity N problems using a single universal binary neuron

I also found this related post on Stackoverflow.
Can we really solve the XOR problem with a simple perceptron?
The use of a non-monotonic activation function is not really common, so I don't really know. Papers about this idea are scarce. Generally, the main solution is to build a multilayer perceptron.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is there any research on the identification of a person's feelings using features such as facial expressions or body temperature?
People could be sad, happy, depressive, angry, nervous, calm, relaxed, bored, etc. I don't know how to express all of these feelings and emotions in English terms (I'm not an English native speaker), which would enable me to search for research papers about this topic (e.g. in IEEE Xplore, Scopus, or ScienceDirect).
So, is there any research on the identification/recognition of a person's feelings/emotions using facial expressions, heartbeat, body temperature, sweating, or nervous behavior (using one or all of them)?

Yes, there is research on this topic. The field that studies it is known as affective computing (AC). Emotion recognition seems to be a specific problem in affective computing, i.e. the recognition of emotions, while AC is also concerned with giving machines the ability to convey emotions (in fact, this paper differentiates the two). There's also sentiment analysis, which refers to the analysis of the sentiment of people e.g. in social media (e.g. in comments) using natural language processing techniques, which doesn't seem exactly what you are looking for. These fields/tasks are all very related/similar, but I don't think they are exactly synonymous (although I'm not an expert in this topic).
Apart from the linked Wikipedia articles, you can find another list of related resources (including important papers and books, and software) here. If you're interested in an easy overview of the AC field, you should probably read the paper Affective computing: challenges, by Rosalind W. Picard (who is one of the leading researchers in this field). In this paper, she differentiates between emotions (what you observe/express with gestures, facial expressions, etc.) and feelings (internal states, which sometimes are not even clear to the person that is having them), and compares emotions to the weather: more precisely, when you say that the weather is windy, you're just giving a label to a fuzzy set of conditions: for example, there's the wind, it could start raining soon, etc., but these conditions are not always clear: for instance, how strong should the wind be in order for you to consider the weather windy? I found this is a nice analogy that conveys what emotions are and why interpreting emotions seems to be a difficult task.
Additionally, you may also be interested in the loving AI project.


text:
Should I remove the units of a neural network or increase dropout?
When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is $p$, then we are effectively training with a neural network of size $(1−p)N$, where $N$ is the total number of units in the neural network.
Using this logic, there is no limit how big I can make a network, as long as I proportionately increase dropout, I can always effectively train with the same sized network, and thereby just increasing the number of "independent" models working together, making a larger ensemble model. Thereby improving generalization of the model.
For example, if a network with 2 units already achieves good results in the training set (but not in unseen data -i.e validation or test sets-), also a network with 4 units + dropout 0.5 (ensemble of 2 models), and also a network with 8 units + dropout 0.75 (ensemble of 4 models)... and also a network with 1000 units with a dropout of 0.998 (ensemble of 500 models)!
In practice, it is recommended to keep dropout at $0.5$, which advises against the approach mentioned above. So there seem to be reasons for this.
What speaks against blowing up a model together with an adjusted dropout parameter?

There is no incentive to increase the size of the model for not reason. If a model of size x gives the best possible performance, there is no reason to use a model of size 2*x with 0.5 dropout during training. Usually we want to find the smallest possible model with the best performance. Inflating the model just results in higher computational requirements.
You are basically suggesting to use dropout to allow the network to learn the same features more than once (creating a redundancies in the model). That is not the purpose of dropout.
Dropout is used to enable the network break unnecessary correlations which occur in the training set. For example if class 1 is the only one that contains in it features A and B, but all the training samples always feature them together. The dropout process can make the model realize that even just one of them is enough to point to class 1.

Right, but I stated only good performance on the training set. The original purpose of Dropout as a regularization method is to improve generalization performance (val/ test sets). In this case, the model expansion has a very good reason to be done (or not? That is my question. But the argument 'the model already performs good' was not a motivation of this question, rather, "the model is not performing well, and overfitting").

Increasing layer size while increasing dropout is possible, but will not lead to more effective learning when taken to the extreme. Dropout is a regularization technique
which actually makes the model "worse" for the sake of increasing generalization capabilities. With increased dropout you increase the negative impacts of dropout (learning less), while not significantly increasing the upsides (reducing overfitting).
When using double the neurons and increasing dropout respectively, you will have to show the model double the amount of data to learn the same amount, since only half the neurons get updated each iteration. If you always use the same amount of data, this means you would have to show the model each sample twice to achieve the same results. This is much worse than simply training one iteration with less dropout.
Additionally, with high dropout the rate of neurons "firing together" is reduced to a minimum, meaning the neurons are almost independent of each other, resulting in a great loss of computational power.
The highest Dropout values reccommended are no higher than 0.8:

A good value for dropout in a hidden layer is between 0.5 and 0.8. Input layers use a larger dropout rate, such as of 0.8.

https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/
Also check out this question for a more detailed answer.


text:
Is it possible to clean up an audio recording of a lecture using some type of AI system?
Is it possible to clean up an audio recording of a lecture from a smartphone (i.e. remove the background noise) using some type of AI system?

Yes, it is possible. Usually, the noise reduction is done using regular signal processing methods, such as spectral subtraction due to demand for low latency. But, of course, modern methods of deep learning are applicable to this problem. For example, the variational autoencoder is the first that comes to my mind, you can check this  project. Another example you can look at is this project, which is based on WaveNet.

Thanks for the links, I'll check those out .Think it's what I'm looking for, yeah.
I don't really need realtime processing. I just have a bunch of audio files from lecture recordings that I'd like to "extract the speaker's voice" from as much as possible. So hopefully I can do that with this.
I'll mark your answer as the best answer when I've tried it out

Hello @igrinis thank you very much for your answer. It would be great if you can make a short video or a list of operations needed to use it. I'm not familiar with github project.


text:
Is there a mathematical example for Conditional Random Fields?
I am learning about probabilistic graphical models and I was wondering if there is an example explaining the math behind conditional random fields. Looking solely on the formula, I have no idea what we actually do. I found a lot of examples for the hidden Markov model. There is a part speech-tag task where we have to find the tags for the sentence "flies like a flower". On these slides (slide 8) Ambiguity Resolution: Statistical Method-Prof. Ahmed Rafea, an HMM is used to find the correct tags. How would I transform this model into a CRF and how would I apply the math?

CRF is for HMM in sequences classification as Logistic Regression is for Naive Bayes for simple classification. For a in depth difference between them I strongly suggest you to read the classical Sutton article "An Introduction to Conditional
Random Fields". For a more practical example explaining how to apply the math I suggest you to read this article.


text:
What is the difference between continuous domains and discrete combinatorial optimization?
According to this website: http://yarpiz.com/67/ypea104-acor (in the website it is mentioned that it is a project aiming to be a resource of academic and professional scientific source codes and tutorials.):

"Originally, the Ant Algorithms are used to solve discrete and
  combinatorial optimization problems. Various extensions of Ant Colony
  Optimization (ACO) are proposed to deal with optimization problems,
  defined in continuous domains. One of the most useful algorithms of
  this type, is ACOR, the Ant Colony Optimization for Continuous
  Domains, proposed by Socha and Dorigo, in 2008 (here)."

What is the difference between continuous domains and discrete combinatorial optimization? I appreciate if you could also mention some examples for each type.

A continous domain can be imagined as a space in which the axes of the coordinate systems are the parameters of the continous domain. If we take 2D Cartesian space as an example, it is a continous domain where there is an infininite amount of possibilities of placing an object in this space. Let's call the position of the object its state. Constraining the space to e.g. a sheet of paper does not change its continuity. There is still an infinte amount of posibilites of placing a dot (or any object) on that sheet of paper or with other words the state space of the dot is still infietly large.
Computational methods are much better in dealing with descretet spaces (mainly because the possiblities to take the next step is limited). In a discrete space the possibilities are limited. If we take the example above and discretize the sheet of paper, e.g. use a grid on the paper and say that we are not interested in the exact location of the point, just in which grid it is in we have succesfully defined a discrete space.
A different example for a problem in descrete domain can be navigation on a map. If we assume that the exact rajectory is not important, then we can make descrete decisions of turning left, right or moving forwards or backwards by one step. Even if add further possibilites (like moving up one step or down) until the number of possibilities in one state is not infinite the domain is descret.


text:
What are examples of daily life applications that use simulated annealing?
In AIMA, 3rd Edition on Page 125, Simulated Annealing is described as:

Hill-climbing algorithm that never makes “downhill” moves toward states with lower value (or higher cost) is guaranteed to be incomplete, because it can get stuck on a local maximum. In contrast, a purely random walk—that is, moving to a successor chosen uniformly at random from the set of successors—is complete but extremely inefficient. Therefore, it seems reasonable to try to combine hill climbing with a random walk in some way that yields both efficiency and completeness. Simulated annealing is such an algorithm. In metallurgy, annealing is the process used to temper or harden metals and glass by heating them to a high temperature and then gradually cooling them, thus allowing the material to reach a lowenergy crystalline state. To explain simulated annealing, we switch our point of view from hill climbing to gradient descent (i.e., minimizing cost) and imagine the task of getting a
  ping-pong ball into the deepest crevice in a bumpy surface. If we just let the ball roll, it will come to rest at a local minimum. If we shake the surface, we can bounce the ball out of the local minimum. The trick is to shake just hard enough to bounce the ball out of local minima but not hard enough to dislodge it from the global minimum. The simulated-annealing solution is to start by shaking hard (i.e., at a high temperature) and then gradually reduce the intensity of the shaking (i.e., lower the temperature)

I know its about its example, but I just want more examples where Stimulated Annealing used in daily life

Can you clarify which examples you already know so that answerers get a better idea of what other examples you might be looking for?

Examples of simulated annealing in the 2010s
These are a few examples.

Optimised simulated annealing for Ising spin glasses, 2015, S.V. Isakov et. al.
A parallel simulated annealing method for the vehicle routing problem with simultaneous pickup–delivery and time windows, 2014, Chao Wang et. al.
Feature-based tuning of simulated annealing applied to the curriculum-based course timetabling problem, 2015, Ruggero Bellio et. al.

Analysis of Quote in the Question
The quote in the question is from page 125 of Artificial Intelligence: A Modern Approach, Stuart Russell and Peter Norvig, 2010, 3rd Edition.
The quote begins with the statement, "A hill-climbing algorithm that never makes 'downhill' moves toward states with lower value (or higher cost) is guaranteed to be incomplete," which is not strictly true. The stated guarantee has at least two important exceptions.

The class of hills being climbed have only one peak
The search is executed using parallel computing architecture scaled such that search granularity includes no more than one peak per processing unit

Describing simulated annealing using the analogy of combining inverted hill climbing and random walk, as Stuart and Norvig explains it, is not as clear and technically accurate as the original analogy. It is often better to find original sources of ideas and read them than to read simplified explanations. The original ideas may be just as easy or easier to understand than the textbook paraphrases (which is why the best textbooks simply quote the originating authors).
Source of the Term
The original algorithm termed simulated annealing is introduced in Optimization by Simulated Annealing, Kirkpatrick et. al.1, which may not qualify as one one explicitly employed by AI researchers or practitioners on a daily basis. Simulating the atomic motion of metals during annealing is the first of several approaches to injecting pseudo-random perturbations into convergence mechanisms. An approximation of a ${\chi}^2$ distribution with $k$ degrees of freedom is a more progressive injection source.
These approaches share the common goal of avoiding the mistaking of a local minimum in loss surface as its global minimum. Such mistaken conclusion causes a failure to achieve an optimal learning state according to the criteria the loss function represents.
As can be seen in the below excerpts, the original article is a good read and sets a foundation of understanding. Such a foundation will assist in understanding all methods used avoid local minima using stochastic injection.

Iterative improvement consists of a search in this coordinate space for
  rearrangement steps which lead downhill. Since this search usually gets stuck in a local but not a global optimum, it is customary to carry out the process several times, starting from different randomly generated configurations, and save the best result.
...
Metropolis et al 2, in the earliest days of scientific computing, introduced a simple algorithm that can be used to provide an efficient simulation of a collection of atoms in equilibrium at a given temperature.
...
Random numbers uniformly distributed in the interval (0,l) are a convenient means of implementing the random part of the algorithm.
...
Using the cost function in place of the energy and defining configurations by a set of parameters {$x_i$}, it is straightforward with the Metropolis procedure to generate a population of configurations of a given optimization problem at some effective temperature. This temperature is simply a control parameter in the same units as the cost function. The simulated annealing process consists of first "melting" the system being optimized at a high effective temperature, then lowering the temperature by slow stages until the system "freezes" and no further changes occur.
...
Simulated annealing with Z-moves improved the random routing by 57 percent, averaging results for both x and y links.

References
[1] Optimization by Simulated Annealing, Science S. Kirkpatrick, C. D. Gelatt and M. P. Vecchi, Science New Series, Vol. 220, No. 4598, May 1983, pp. 671-680
[2] N. Metropolis, A. Rosenbluth, M. Rosenbluth., A. Teller. E. Teller, J. Chem. Phys. 21. 1087 110511

Simulated annealing is just one of the approaches for an optimization problem:

Given a function f(X), you want to find an X where f(X) is optimal (has maximum or minimum value).

Shaking (a methaphor for introducing a degree of randomness to annealing process) is used to prevent the algorithm from stopping prematurely, when only a sub-optimal solution (local minimum or local maximum) has been found.
I've once used it for finding the optimal schedule of lessons for teacher. The optimization function took into consideration "time windows" between windows, so that teachers do not have to wait too much.

http://www.persion.info/projects/antenna_annealing/
Using simulated annealing to evolve ideal antennas.


text:
Are there other approaches to deal with variable action spaces?
This question is about Reinforcement Learning and variable action spaces for every/some states.
Variable action space
Let's say you have an MDP, where the number of actions varies between states (for example like in Figure 1 or Figure 2). We can express a variable action space formally as $$\forall s \in S: \exists s' \in S: A(s) \neq A(s') \wedge s \neq s'$$
That is, for every state, there exists some other state which does not have the same action set.
In figures 1 and 2, there's a relatively small amount of actions per state. Instead imagine states $s \in S$ with $m_s$ number of actions, where $1 \leq m_s \leq n$ and $n$ is a really large integer.

Environment
To get a better grasp of the question, here's an environment example. Take Figure 1 and let it explode into a really large directed acyclic graph with a source node, huge action space and a target node. The goal is to traverse a path, starting at any start node, such that we'll maximize the reward which we'll only receive at the target node. At every state, we can call a function $M : s \rightarrow A'$ that takes a state as input and returns a valid number of actions.
Approches

A naive approach to this problem (discussed here and here) is to define the action set equally for every state, return a negative reward whenever the performed action $a \notin A(s)$ and move the agent into the same state, thus letting the agent "learn" what actions are valid in each state. This approach has two obvious drawbacks:

Learning $A$ takes time, especially when the Q-values are not updated until either termination or some statement is fulfilled (like in experience replay)

We know $A$, why learn it?

Another approach (first answer here, also very much alike proposals from papers such as Deep Reinforcement Learning in Large Discrete Action Spaces and Discrete Sequential Prediction of continuous action for Deep RL) is to instead predict some scalar in continuous space and, by some method, map it into a valid action. The papers are discussing how to deal with large discrete action spaces and the proposed models seem to be a somewhat solution for this problem as well.

Another approach that came across was to, assuming the number of different action set $n$ is quite small, have functions $f_{\theta_1}$, $f_{\theta_2}$, ..., $f_{\theta_n}$ that returns the action regarding that perticular state with $n$ valid actions. In other words, the performed action of a state $s$ with 3 number of actions will be predicted by $\underset{a}{\text{argmax}} \ f_{\theta_3}(s, a)$.

None of the approaches (1, 2 or 3) are found in papers, just pure speculations. I've searched a lot, but I cannot find papers directly regarding this matter.
Does anyone know any paper regarding this subject? Are there other approaches to deal with variable action spaces?

Have a similar issue, and my immediate thoughts are to perform some transformation of the problem into a domain where the action space is fixed. For instance, if I am working in active learning, where the action is to select an example from a unlabelled training dataset (without replacement), then perhaps a different formulation where the action is to select a class, or point in data space might work as well (this will have a fixed/static action space)

I have the same question. The best answer I have found is the following paper:
Learning Action Representation for Reinforcement Learning

How about your master thesis? Have you figured out methods to solve your question or anything else you want to share? I found the most related works as listed below are large action spaces.

@DongDongChen we used a modified version of the https://arxiv.org/pdf/1512.07679.pdf paper, along with an ILP solver, which worked ok for us in the thesis. However, I think our version was very specific to our instance of the problem and was hard to generalize.

Does anyone know any paper regarding this subject?

I'm not familiar with any off the top of my head. I do know that the vast majority of Reinforcement Learning literature focuses on settings with a fixed action space (like robotics where your actions determine how you attempt to move/rotate a particular part of the robot, or simple games where you always have the same set of actions to move and maybe ''shoot'' or ''use'' etc.). Another common class of settings is where the action space can easily be treated as if it always were the same (by enumerating all actions that ever could be legal in some state), and filtering out illegal actions in some sort of post-processing steps (e.g. RL work in board games).
So, there might be something out there, but it's certainly not common. Most RL people like to involve as little domain knowledge as possible, and I suppose that a function that generates a legal set of actions given a particular state can very much be considered to be domain knowledge.

Are there other approaches to deal with variable action spaces?

The problem you describe is mostly a problem in Reinforcement Learning with function approximation, in particular, function approximation using Neural Networks. If you can get away with using tabular RL algorithms, the problem instantly disappears. For example, a table of $Q(s, a)$ values as commonly used in the tabular, value-based algorithms does not need to contain entries for all possible $(s, a)$ pairs; it's fine if it only contains entries for $(s, a)$ pairs such that $a$ is legal in $s$.
Variable action spaces primarily turn into a problem in Deep RL approaches, because we normally work with a fixed neural network architecture. A DQN-style algorithm involves neural networks that take feature vectors describing states $s$ as inputs, and provide $Q(s, a)$ estimates as outputs. This immediately implies that we need one output node for every action, which means you have to enumerate all the actions, which is where your problem comes in. Similarly, policy gradient methods traditionally also require one output node per action, which again means you have to be able to enumerate all the actions in advance (when determining the network architecture).
If you still want to use Neural Networks (or other kinds of function approximators with similar kinds of inputs and outputs), the key to addressing your problem (if none of the suggestions you've already listed in the question are to your liking) is to realize that you'll have to find a different way to formulate your inputs and outputs, such that you are no longer required to enumerate all actions in advance.
The only way I can think of doing that really is if you are able to compute meaningful input features for complete state-action pairs $(s, a)$. If you can do that, then you could, for example, build neural networks which:

Take a feature vector $x(s, a)$ as input, which describes (hopefully in some meaningful way) the full pair of the state $s$ and the action $a$
Produce a single $\hat{Q}(s, a)$ estimate as output, for the specific pair of state + action given as input, rather than producing multiple outputs.

If you can do that, then in any given state $s$ you can simply loop through all the legal actions $A(s)$, compute $\hat{Q}(s, a)$ estimates for them all (note: we now require $\lvert A(s) \rvert$ passes through the network rather than just a single pass as would normally be required in DQN-style algorithms), and otherwise proceed similarly to standard DQN-style algorithms.
Obviously, the requirement of having good input features for actions is not always going to be satisfied... but I doubt there are many good ways to get around that. It's very similar to the situation with states really. In tabular RL, we enumerate all states (and all actions). With function approximation, we usually still enumerate all actions, but avoid the enumeration of all states by replacing them with meaningful feature vectors (which enables generalization across states). If you want to avoid enumerating actions, you'll in a very similar way have to have some way of generalizing across actions, which again means you need features to describe actions.

This is great feedback and interesting thoughts, thanks for that. I find many papers regarding "large action space"'s, which is in pretty much the same issue. I think the paper "Discrete Sequential Prediction of continuous action for Deep RL" is very interesting since it predicts a sequence of actions using Recurrent Neural Networks instead, which solves the fixed network issue. We will do our master thesis in this subject and I hope we'll collect more information in this matter.

I am trying to do this state-action representation and create a DQN-like network that would output the Q-value of the x(s,a), like a regression problem. Having read the tf_agents documentation, I do not think that I can model it with this. Is there a way to model it with tf_agents or should I do it from scratch ?

@ddaedalus Hmmm I'm not sure, I'm not super familiar with tf_agents. I must say that I don't believe I've ever seen anyone in the literature actually doing this thing with state-action pairs being the inputs, rather than states inputs and actions outputs. Almost everyone in all the literature always seems to assume they can just predetermine the number of actions. So, since this is so rarely done in the literature, it would indeed be very well possible that it's not easily supported by such frameworks and requires more work from scratch

In my case, every action is different from the other. If you choose an action, then it cannot be appeared again. Also from each state, you always have different actions or actions that you have seen again but not taken yet. Do you think that I could create an environment that inherits from gym environment? I doubt that because of what you said.

@ddaedalus Again not 100% sure... I'm thinking that might be possible though. If I remember correctly, you do have to predefine your action space in advance, but it does not have to be discrete; you can specify continuous action spaces, which is used for all these continuous control envs (the ones where you have to make all sorts of robots learn how to walk etc.). I suppose that in your situation you'll also have to find some action representation (probably with binary or numeric features). If you have upper/lower bounds on your feature values, you can define your action space with that?

@DennisSoemers Both my state and actions are texts so I have word embeddings with a mean pooling layer for each text and I concatenate state and action embeddings. So it's numeric features for x(s,a).

Another approach that came across was to, assuming the number of different action set $n$ is quite small, have functions $f_{\theta_1}$, $f_{\theta_2}$, ..., $f_{\theta_n}$ that returns the action regarding that perticular state with $n$ valid actions. In other words, the performed action of a state $s$ with 3 number of actions will be predicted by $\underset{a}{\text{argmax}} \ f_{\theta_3}(s, a)$.

That sounds pretty complicated and the number of different action sets is usually very high, even for the simplest games. Imagine checkers, ignore promotions and jumping, for simplicity, and there are some $7 \cdot 4 \cdot 2=56$ possible actions (which is fine), but the number of different sets of these actions is much higher. It's actually difficult to compute how many such sets are possible in a real game - it's surely much less than $2^{56}$, but also surely far too big for being practical.

Are there other approaches to deal with variable action spaces?

Assuming the number of actions is not too big, you can simply ignore actions which don't apply in a given state. That's different from learning - you don't have to learn to return negative reward for illegal actions, you simply don't care and select the legal action returning the best award.

Note that your expression
$$\forall s \in S: \exists s' \in S: A(s) \neq A(s') \wedge s \neq s'$$
can be simplified to
$$\forall s \in S: \exists s' \in S: A(s) \neq A(s')$$
or even
$$|A(s)|_{s \in S} > 1$$

authors: Денис Демидов


text:
When should I use simulated annealing as opposed to a genetic algorithm?
What kind of problems is simulated annealing better suited for compared to genetic algorithms?
From my experience, genetic algorithms seem to perform better than simulated annealing for most problems.

Could you clarify what you mean by "perform better" - e.g. does GA find the better solution, is it faster to run, does it use less memory, storage or CPU resources? Also name the problems you have tried SA vs GA on . . .

See also this SO question: https://stackoverflow.com/q/4092774/3924118.

Simulated Annealing vs genetic algorithm?

Simulated annealing is a materials science analogy and involves the introduction of noise to avoid search failure due to local minima. See images below. To improve the odds of finding the global minimum rather than a sub-optimal local one, a stochastic element is introduced by simulating Brownian (thermal) motion. Variants of simulated annealing include injection of random numbers with various distributions and the averaging effect of mini-batching (dividing a batch into segments and performing network parameter adjustment after each).
Genetic algorithms are search methods based on principles of mutation, meiosis, symbiosis, test, elimination of inadequacy, and recursion. The advantages of such approaches is the simulation of sexual reproduction, where the possibility of dominant genetic features from two individuals producing a child individual containing the best of both exists. In a population of children, the probability of such a hybrid emerging is higher. Over several generations, even higher than that.

What kind of problems does simulated annealing perform better than genetic algorithms if any?

This question can only be answered well if comparing original, pure versions of both, not the variants that have developed since their introduction.
Simulated annealing or other stochastic gradient descent methods usually work better with continuous function approximation requiring high accuracy, since pure genetic algorithms can only select one of two genes at any given position.

From my experience, genetic algorithm seems to perform better than simulated annealing for most problems.

Those performance results would be of value to others and should be published as a paper, presented as an open source project, or published creative commons in the appropriate AI venue. At the very minimum, the results could be placed in the above question or an answer to it.

References
[1] Optimization by Simulated Annealing, Science S. Kirkpatrick, C. D. Gelatt and M. P. Vecchi, Science New Series, Vol. 220, No. 4598, May 1983, pp. 671-680
[2] N. Metropolis, A. Rosenbluth, M. Rosenbluth., A. Teller. E. Teller, J. Chem. Phys. 21. 1087 110511

Optimised simulated annealing for Ising spin glasses, 2015, S.V. Isakov et. al.
A parallel simulated annealing method for the vehicle routing problem with simultaneous pickup–delivery and time windows, 2014, Chao Wang et. al.
Feature-based tuning of simulated annealing applied to the curriculum-based course timetabling problem, 2015, Ruggero Bellio et. al.

Images from Other Answers

If digital values are mere estimates, why not return to analog for AI?
Method to check goodness of combinatorial optimization algorithm implementation, plotted from https://github.com/facebookresearch/nevergrad/blob/master/nevergrad/functions/corefuncs.py

Simulated annealing algorithms are generally better at solving mazes, because they are less likely to get suck in a local minima because of their probabilistic "mutation" method. See here. Genetic algorithms are better at training neural networks, because of their genetically inspired training algorithm. This makes them more versatile and efficient in more complex situations.


text:
What does it mean to do multi-dimensional processing with tensors in tensor cores?
In some tweets about NeurIPS 2018, this video from NVIDIA appeared. At around 0.37, she says:

If you think about the current computations in our deep learning systems, they are all based on Linear Algebra. Can we come up with better paradigms to do multi-dimensional processing? can we do truly tensor-algebraic techniques in our tensor cores?

I was wondering what she is talking about. I'm not an expert so I'd like to understand better this specific point.

Both PR and Theoretically Valid
Although Anima Anandkumar's presentation is a puff piece for NVidia, her representation is not contrary to theory.

... next level [above NVidia's GPU success] ... means new algorithmic research. So, if you think about the current computations in our deep learning systems, they are all based on linear algebra. Can we come up with better paradigms to do multi-dimensional processing. Can we do truly tensor algebraic techniques in our tensor cores, and what kind of new architectures will this realize?

Tensors
A tensor is the extension of the idea of functions and vectors. Vectors emerged allow the encapsulation of multiple variables of the same type into a single unit to express multidimensional properties such as position or force. Variables that are not encapsulations of multiple variables are then referred to as scalars. When such multidimensional properties are compositions of functions rather than variables because their values are dependent upon other variables, they are vector fields.
To more cleanly model electromagnetism and gravity in mathematics, higher levels of variable and functional encapsulation were required. Scalar variables and functions that return scalar variables are thus tensors of rank one (1). Vectors and vector fields are tensors of rank two (2), and higher ranks are extensions of the pattern created by scalars and vectors. A matrix containing functions can be represented as a tensor of rank three (3), encapsulating its elements into a single unit.
Digital Processing and Integration Trend
CUDA cores, which are the signal processing units used in current generation NVidia GPUs (graphical processing units) can be used for production of two or three dimensional rendering or leveraged to produce parallel processing of signals through an artificial network. This is following the trend to delegate to digital circuitry the most significant bottlenecks in algorithm execution. VLSI (very large scale integration) technology is the logical result of this trend.

Industrial and military calculators with tests and branching running on rows of racks of relays
Add speed with tubes and load programs with punch cards
Reduce power and improve reliability and speed with transistors, magnetic core memory, and paper tape
Integrated circuits on motherboards with vinyl tape
Handheld programmable calculators for students
Microprocessors with magnetic disks
Floating point, digital signal, and graphics processors to extend microprocessors
Multiple cores (repeating large scale circuit patterns on a single substrate)
Re-purposing and extending graphics rendering circuits to offload artificial network computations to GPUs

With all these, speed, size, energy thrift, and convenience are not really a shift in the original paradigm that started with Norbert Wiener, Claude Shannon, Alan Turing, John von Neumann, and others. In fact, computing is still catching up in several ways with the perspectives of these pioneers and a long way off from producing in VLSI common concepts in science. Software is the solution, which is why it is called soft, meaning flexible, not necessarily weak. However, flexibility sacrifices speed and capability, thus the trend above.
The Parallel Computing Challenge
Part of what Anima Anandkumar is stating that the signal paths in current VLSI processors are still a much lower level of abstraction than the ideas of mathematicians, physicists, and AI engineers. Scientific theory describes probability, statistical distribution, expectation, force, loss, gain, pain, reward, memory, momentum, semantics, combination, and correlation at a much higher level than current digital circuitry.
The use of sequential algorithms is the splaying out in time what could be a massively parallel operation. The serial algorithm limits the rate of processing. The finding of ways of dealing with things in parallel in mathematics can be done with a pencil. In computers, paralleling algorithms and finding parallel processing structures in VLSI form that are as flexible as software in some respects is much more challenging and thus far behind.
This is one problem that has been the focus of research in at least twenty large corporate and government labs since for half a century and has been the intention of VLSI from the onset. The work is not specific to NVidia. It is not a new problem and the approach to solving it has been along this paradigm.

Add abstraction and encapsulation in the mind.
Express it in mathematics.
Write it as a serial algorithm, leveraging whatever parallel constructs are supported by programming languages and libraries that can leverage VLSI parallelism or computing clusters.

The last seventy years of computing machinery development has been bringing hardware, operating systems, and software closer to the level of mathematical expressions that were twenty to two hundred years old. That may change, and everyone wants to ride the new wave.
Anima Anandkumar and her counterparts in IBM, Intel, Google, Microsoft, the U.S. Navy, Amazon, Alibaba, and the other corporate and government laboratories do not state (because it is either classified or company confidential) what they intend to do to further the parallelization of computing. Whatever they do along those particular lines would not be a paradigm shift but rather a next step along the current paradigm.
Enters Corporate Strategy
They would also not state what they may be doing that is not in that paradigm. They are constrained to only give hints without theoretical substance. If they were working on a chip that exhibited what the human brain exhibits when neurons grow and connect according to DNA based propensity, they wouldn't say that in a technically precise way. Corporate secrecy is part of the global economic game play, not tipping the hand. They have their game face on.
The idea of moving from linear to non-linear is a good public relations theme and when used for puff pieces is not conclusively technical. The pitch usually goes along this line.

What they did was very linear. We are moving into a non-linear space.

It is an attempt to claim that what was done was primitive and the impending game changing advancement is coming from the speaker and their people. Sometimes it works to create a temporary hike in the value of tradable securities, which that company needs as of this writing. If there is a true change in the game, it will be known when it is released. Those who have worked in laboratories for years know to wait until something is released that demonstrates, when the example code is configured and ran, what game change actually occurred, if any. Or they develop the game changer themselves, which is why it is a lab.
Ambiguity of 'Linear' Even in Mathematics
Also note that curved lines are still lines and the term linear can mean two things depending on the context.

Conformant to the linear equation $\vec{Y} = V \vec{X}$, such that it graphs as a line, plane, or higher dimensional flat surface, with constant gradient and without curvature.
Conformant to the principles of linear algebra, which would include spaces, eigenvalues, regression with high-order polynomials, and a number of constructs that involve varying gradient and curvature.

Predicting Paradigm Shifts
The skepticism developed after watching tech company PR for some time does not necessarily dismiss game changing technology advancements of the past and potentials of the future. Any open source project team, individual, corporation, or government lab might do something that shifts the paradigm, usually over a period of years. Classic examples:

Rope
Wheel
$F = m a$
Oxygen (that air is not an element)
Electromagnetism
Alternating current energy transmission
Relativity
Internal combustion
Radio

Computing examples:

Information theory
Cybernetics
First transistor
LISP and FORTRAN
C and UNIX
TCP/IP

Shifts do not need to be as far reaching and game changing as these to have an impact. Who will achieve the next with regard to AI concepts, information structure, algorithm, execution environment is not something very easy to predict. Consider those that lived before any of the above and try to imagine them trying to predict that the projects of Isaac Newton, Antoine Lavoisier, Michael Faraday, Nicola Tesla, Norbert Wiener, Claude Shannon, Ken Thompson, Dennis Ritchie, or any of the others were the seeds of the next paradigm shift.
There is research into analog artificial networks, neuromorphic hardware, semantic modelling, graph algorithms, and other potential game changers, each of which has impressive conceptual foundations and is discussed in some of the Q&A here. These are a few.

What topologies are largely unexplored in machine learning?
Novelty Search Mutation Algorithm
Can deep networks be trained to prove theorems?
If digital values are mere estimates, why not return to analog for AI?
Are commercially available neural ICs digital?

What of these may point to the beginnings of paradigm shift cannot be known, and even if an idea first posted here or referenced from here is the seed, it may not be known later. The multilayer perceptron may be the seed of future streets and highways being dominated by automated vehicles in 2090, but no one in a million people will realize in seventy years that the trend toward AV research was seeded by MLP enthusiasm from the first decade of this century.
Gaps to be Filled and NVidia is a Contendere
All this aside, we use NVidia hardware every day for robotics and analysis, so they have credibility resulting from past success. If they produced a chip that does something remarkably clever before Intel or the IBM-MIT collaboration, it would be a small surprise, but not it is not a completely unbelievable possibility. Certainly the comprehension of Hilbert spaces, semantics, and topology are limited in the computer science field, and a paradigm shift toward greater comprehension of them or some new thing that is not even part of mathematical thought today would add some needed diversity to the computing industry.


text:
Is there a mechanism in the human brain that works analog to LSTMs?
Is there a mechanism in the human brain that works analog to LSTMs? Is there a biological/neuroscientific interpretation of LSTMs and recurrent neural networks? How do long-term and short-term memories work in the brain, on a neuron level?
I would also appreciate links to more in-depth explanations in the literature.


text:
Is it feasible to use minimax to solve a board game with a large number of moves?
I have to build a KI for a made-up game similar to chess. As I did research for a proper solution, I came upon the MinMax algorithm, but I'm not sure it will work with the given game dynamics.
The challenge is that we have far more permutations per turn than in chess because of these game rules.

Six pieces on the board, with different ranges.
On average, there are 8 possible moves for a piece per turn.
The player can choose as many pieces to move as he likes. For example none, all of them, or some number in between (whereas in chess you can only move one.)

Actual questions:

Is it feasible to implement MinMax for the described game?
Can alpha-beta-pruning and a refined evaluation function help (despite the large number of possible moves)?
If no, is there a proper alternative?

By similar to Chess do you mean 2-player, non-chance, perfect information, sequential (turn-based) games involving moving and capturing tokens?  How big is the gameboard and what accounts for the higher number of branches? Is the game natively finite, or can it get "loopy" (potentially infinite loops)?

-The player can choose as many pieces to move as he likes. For example none, all of them, or some number inbetween. (Whereas in chess you can only move one)

That quote specifically is the part that really causes the size of your legal action set to blow up. You have a combinatorial action space here. If each of your pieces has 8 legal moves, then that is:

8 legal moves for the first piece (or 9 if that count didn't already include the "do nothing" option)
for each of those, there are again 8 or 9 different choices for the second piece (leading to e.g. $8 \times 8 = 64$ possible combinations for just the first two pieces)
for each of those, again 8 choices for the third piece (leading to $64 \times 8 = 512$ possible combinations for just the first three pieces).
etc.

This blows up way too quickly, and there's really no hope of ever getting a decent player for this using any MiniMax-based algorithm (including things like alpha-beta pruning, principal variation search etc.).
In the kinds of games that you describe, you'll want to use algorithms that can exploit the "structure" of your action space. A raw enumeration of all possible combinations blows up quickly, but many algorithms can do reasonably well by re-phrasing the problem in such a way that you have more "depth" rather than "breadth". For example, instead of viewing a full combination of choices for all pieces as a single "action", you can treat the choices per piece as a separate "action".
Rather than making a single choice out of $8 \times 8 \times 8 \times \dots$ possibilities every turn, you want to have a search tree where your player makes one choice out of $8$ (for the first piece), followed immediately by another choice out of $8$ (for the second piece), etc. The opposing player only gets to make a choice after the current player has made choices for all pieces. With such a strategy, the breadth of your search tree will no longer be a problem, but the depth will become a problem. To address this, you'll additionally want to make sure that your methods can generalize across different depth levels.
A good place to look would be combinatorial versions of Monte-Carlo Tree Search, such as those described in:

https://project.dke.maastrichtuniversity.nl/games/files/msc/Roelofs_thesis.pdf
https://www.jair.org/index.php/jair/article/view/11053
(probably a few other publications by the author of that second link)

These algorithms are quite a bit more complicated than MiniMax though, MiniMax is a very basic algorithm in comparison.

All games are algorithms, and "combinatorial game" is a fuzzy term that expands as the scope of CGT expands (all games are also combinatorial.)  For more info see Constraint Logic: A Uniform Framework for Modeling Computation as Games and [Playing Games with Algorithms:
Algorithmic Combinatorial Game Theory]
(http://erikdemaine.org/papers/AlgGameTheory_GONC3/paper.pdf)  (Demaine/Hearn)


text:
When should I use Genetic Algorithms as opposed to Particle Swarm Optimization?
For which problems are Genetic Algorithms more suitable than Particle Swarm Optimization, and vice-versa? Are there any guidelines?

These kind of questions cannot be answered without looking at a particular project. Each algorithm has its particular strengths and weaknesses; and trade-offs in terms of use of resources (processing power and/or storage space, for example). If there was an objective answer, then the worse algorithm would surely fall in disuse.
It also depends what you mean by "better". Faster? Better score according to some evaluation measure? More robust (ie works with many diverse data sets)?
I would recommend looking at both algorithms in more detail, and trying to understand how they work. Then you should be able to find out which best fits your problem.
However, one problem with Particle Swarm Optimisation is that it is not well understood, so you might have to resort to trial-and-error.

This should have been a comment (for the most part). You should have encouraged the asker to change his question to something like: when should I use genetic algorithms as opposed to PSO?

In order to save the original post (not sure if it is a good idea), I've reformulated the question, so that it's more "acceptable". You may want to review this answer to reflect those changes, as the question "when to use one over the other?" may be useful to some people. You may also consider the deletion of this answer, as it only partially addresses the current answer now (and, as I had said in a previous comment, this should probably have been a comment).

The paper Comparison between genetic algorithms and particle swarm optimization (1998, by Eberhart and Shi) does not really answer the question of when to use one over the other (this may be an open question), but at least it provides a comparison of how the methods work and what could affect their performance (i.e. which parameters or operators they use, and what the typical values are), so it may be worth reading it.


text:
What is the role of biology in AI?
Biology is used in AI terminology. What are the reasons? What does biology have to do with AI? For instance, why is the genetic algorithm used in AI? Does it fully belong to biology?

Here's my attempt to answer this question: https://arxiv.org/abs/2209.04425

Biological organisms (such as animals or plants) are the main examples of intelligent systems that we are aware of (excluding artificially intelligent systems, so as not to discuss whether current AI systems are really intelligent or not). Consequently, biological life is often an inspiration for AI researchers to develop AI systems.
There are numerous examples of AI systems that have been introduced  (at least, partially) based on or just inspired by the biology. Here are a few examples.

Reinforcement learning is based on a similar way that animals (such as dogs or pigeons) can learn. For more details, see Sutton & Barto's book (especially chapters 14 and 15).

Artificial neural networks are very approximative models of human neural networks.

Genetic algorithms are roughly based on Darwin's theory of evolution.

Ant colony optimization algorithms (and, in general, swarm intelligence) are based on the way real ants (and, respectively, biological swarms) behave. (There is even a rap song dedicated to ants).

There are probably other examples that don't come to my mind right now. See also this and this questions.
There are cases where AI discoveries have also helped the development of biology or related fields (such as neuroscience and psychology). For instance, Sutton & Barto (on page 4) write

Of all the forms of machine learning, reinforcement learning is the closest to the kind of learning that humans and other animals do, and many of the core algorithms of reinforcement learning were originally inspired by biological learning systems. Reinforcement learning has also given back, both through a psychological model of animal learning that better matches some of the empirical data, and through an influential model of parts of the brain's reward system.

Evolutionary game theory and evolutionary algorithms
I see the connection arising mostly thought Evolutionary Game Theory and Evolutionary Algorithms.  Evolutionary algorithms are an analog of natural selection, where successive generations of a given decision making agent are more optimized than previous generations.  Like organisms in nature, this process uses "reproduction, mutation, recombination and selection".
There are a couple of recent articles from Quanta Magazine.  One, "The Math That Tells Cells What They Are" discusses mathematical optimization as the core function of fundamental biological systems.

"Through evolution, these cells have figured out how to implement Bayes' trick using regulatory DNA."
"Natural selection [seems to be] pushing the system hard enough so that it ... reaches a point where the cells are performing at the limit of what physics allows."

This second quote is exactly the goal of Artificial Intelligence, where utility is limited by physics (computing resources).  One way for an algorithm to increase utility is to increase computing power, but the other method is to refine the algorithm to make strong decisions more efficiently.  (MCTS vs. Brute Force where a model is intractable, as an example.)
A second article "Mathematical Simplicity May Drive Evolution’s Speed" talks about Genetic Algorithms

"Creationists love to insist that evolution had to assemble upward of 300 amino acids in the right order to create just one medium-size human protein. With 20 possible amino acids to occupy each of those positions, there would seemingly have been more than 20300 possibilities to sift through, a quantity that renders the number of atoms in the observable universe inconsequential."

The game of Go on a 19x19 board has a similar quality--the number of potential gamestates is vastly exceeds the number of atoms in the universe, and, even if the entire universe were converted to computronium, the game would still be intractable.

"The fatal flaw in their argument is that evolution didn’t just test sequences randomly: The process of natural selection winnowed the field. Moreover, it seems likely that nature somehow also found other shortcuts, ways to narrow down the vast space of possibilities to smaller, explorable subsets more likely to yield useful solutions."

This would also be an accurate description the process of pruning a search space.  The article concludes that, although there is still much research to be conducted:

“The idea of thinking about life as evolving software is fertile.”

The process of optimization in nature and in computer science is similar in spirit, if not in fact.
Automata as a form of artificial life
The second factor may arise out of the mythology of AI, via speculative fiction.  In science fiction, the idea of automata as a form of artificial life is persistent.  Shows & films like Westworld, BladeRunner, and the Alien franchise, with David the Android as a prime example of a superior, artificial species that may supplant humanity, are extremely popular. These are all based on Phillip K Dick's ideas explicated in Do Androids Dream of Electric Sheep, the plot of which turns on evolutionary game theory, written about 5 years before the field was formalized! (Dick's influence can even be seen in Google's "Nexus" naming convention for their Android phone;)  Underneath all of this is also the idea that Artificial Intelligence itself is a function of nature, with humans as merely the vehicle for the next form of dominant life.

I edited this answer to structure it into 2 distinct parts, given that I think your original answer was divided into 2 parts, so this should make it clear, but feel free to revert this edit (and delete this comment afterwards).


text:
When will we have computer programs that can compose mathematical proofs?
When will it be possible to give a computer program a bunch of assumptions and ask it if a certain statement is true or false, giving a proof or a counterexample respectively?

Arguably we have had this since 1957, with the General Problem Solver. However, it is a thorny problem, and like so much in AI, it works fine in toy domains (like the Towers of Hanoi problem), but fails in real life, as real life is too complex for it to cope.
There is a list of solvers (the category of programs dealing with this issue) on Wikipedia. As of today, it comprises about 40 systems (split into two sub-categories). These work with mathematical equations and presumably also more general problems, assuming they are not too complex.


text:
Do we need multiple parallel environments to train in batches an on-policy algorithm?
When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that to apply batching to an on-policy method you have to have multiple parallel environments?
As an extension of this, if only one environment is available when using on-policy methods, does that mean batching isn't possible? Doesn't that limit the power of such algorithms in certain cases?

Here, here and here are 3 related questions.

We don't need multiple environments.
On-policy algorithms require that new training samples are collected with the newest policy, so we can't use an experience buffer. However we can use the newest policy to collect multiple samples, even over multiple epochs, before updating the weights. This update can be a batch update.


text:
How does best-first search differ from hill-climbing?
How does best-first search differ from hill-climbing?

See also this post.

You should accept the answer below, if it answers your question, which I think it does.

Best-first search
BFS is a search approach and not just a single algorithm, so there are many best-first (BFS) algorithms, such as greedy BFS, A* and B*. BFS algorithms are informed search algorithms, as opposed to uninformed search algorithms (such as breadth-first search, depth-first search, etc.), i.e. BFS algorithms make use of domain knowledge that can be encoded into a so-called heuristic function (that's why they are informed!).
Every BFS algorithm defines a so-called evaluation function $f$ of the following form
$$f(n) = g(n) + h(n)$$
for all nodes $n \in V$ (where $V$ is the set of nodes or states of the search space), where

$g(n)$ is the cost from the start node (of the search) to the node $n$, and

$h(n)$ (the so-called heuristic, which is the way of including domain knowledge to solve the search problem) is an estimate of the cost of the cheapest path from $n$ to the goal node.

Depending on how you define $f$ and, in particular, the heuristic function $h$, you get different BFS algorithms. For instance, A* is a BFS algorithm where $h$ is an admissible heuristic (i.e. it never overestimates the cost to the goal node). Because of this admissibility property, A* is guaranteed to find the globally optimal solution (i.e. the cheapest path from a start node to the goal node among all paths), but you can ignore this detail.
To apply a BFS algorithm, you need to define the evaluation function and the search space, i.e. the states (or nodes) and their connections. For example, if you want to find the cheapest path from Paris to Madrid, you need to define that Paris is the start node, Madrid the goal node and then you need to define all intermediate nodes between Paris and Madrid, but you also need to define $g$ and $h$.
Hill climbing
Hill climbing (HC) is a general search strategy (so it's also not just an algorithm!). HC algorithms are greedy local search algorithms, i.e. they typically only find local optima (as opposed to global optima) and they do that greedily (i.e. they do not look ahead). The idea behind HC algorithms is that of moving (or climbing) in the direction of increasing value. HC algorithms can be used to solve optimization problems and not just well-defined search problems, i.e. you start from some solution and you move to the best neighboring solution, and then loop.
Best-first search vs hill climbing

BFS algorithms are informed search algorithms (as opposed to uninformed)
BFS algorithms need to define the search space and the evaluation function
Some BFS algorithms (such as A*) are guaranteed to find the best global solution
HC algorithms are general (i.e. widely applicable) but local and greedy search and optimization algorithms; consequently, they are not generally guaranteed to find the global optimum (but, in practice, they may work well, depending also on the problem).
HC algorithms do not need to define the search space explicitly (i.e. you do not need to define the start and goal nodes, and so on), but you just need a way of determining the best neighboring solution

Further reading
The book Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig provides more details about these two search approaches. (You can find free copies of this book online).


text:
What does the notation "for t=T to 1,−1 do" in terms of time steps, in deep recurrent q network?
In looking at an algorithm in the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning.
Here is the full algorithm:

What does the notation for t=T to 1,−1 do: refer to in terms of time steps?
The network structure is a deep recurrent q network.
Secondly, why do the gradients need to be reset to zero?

That notation should mean to go from time step $T$ to time step $1$ by a negative step $-1$, i.e. backward, so $T$, then $T-1$, then $T-2$, and so on until $1$. If you know Python, this should be familiar. However, note that this is just a guess because I am not familiar with this algorithm.

Yes, this makes sense, and fits the problem. Thank you!

That notation should mean to go from time step $T$ to time step $1$ by a negative step $-1$, i.e. backward, so $T$, then $T-1$, then $T-2$, and so on until $1$. If you know Python, this should be familiar. However, note that this is just a guess, because I am not familiar with this algorithm.


text:
Should the importance sampling ratio be updated at the end of the for loop in the off-policy Monte Carlo control algorithm?
I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below.

Here, $W$ (importance sampling ratio) is updated at the end of the For loop.
But, I think that updating should be located after calculating $G$ (the return) and before updating $C(s,a)$ (cumulative of $W$). This seems to be right considering the second picture below, which I found in http://incompleteideas.net/book/first/ebook/node56.html.

Is Sutton and Barto's book wrong? Or the two algorithms in Sutton and Barto's book and seconds picture are actually the same, and I am wrong? Is there any difference between the two when implemented? If I am wrong, can you explain the reason?

I had the same question too, but finally, I found it was just a mismatch between the explained context and the algorithm given.
Note that in context, it explains importance sampling ratio for state value function (V) instead of state-action value function (Q) which is the target of the boxed algorithm. Following exercise 5.6, you will know that.


text:
What is the score used to visualize attention in this paper?
I'm reading this paper Global-Locally Self-Attentive Dialogue State Tracker and follow through the implementation published in GLAD.
I was wondering if someone can clarify what variable or score is used to calculate the global and local self-attention scores in Figure 4 (the heatmap).
For me, it is not really clear how to derive these scores. The only score that would match the given dimension would be in the scoring module  $p_{utt}=softmax(a_{utt})$. However, I do not see in their implementation that anything is done with this value.
So, what I did was the following:
 q_utts = []
 a_utts=[]
 for c_val in C_vals:
     q_utt, a_utt = attend(H_utt, c_val.unsqueeze(0).expand(len(batch), *c_val.size()), lens=utterance_len)
     q_utts.append(q_utt)
     a_utts.append(a_utt)
attention_score= torch.mean(torch.stack(a_utts,dim=1),dim=1)

But the resulting attention score differs very much from what I expect.


text:
In PyTorch, why does the sequence length need to be provided as the first dimension of the input tensor for an RNN?
I am confused as to why the sequence length is the first dimension of the input tensor for an RNN, while the batch size is the first dimension for any other kind of network (linear, CNN, etc.).
This makes me think that I haven't fully grasped the concept of RNN batches. Is each independent batch a different sequence? And is the same hidden state across batches? Is the hidden state maintained between timesteps for a given sequence (for vanilla/truncated BPTT)?

As it says in the documentation, you can simply reverse the order of dimensions by providing the argument batch_first=True when constructing the RNN.
Then, the dimensionality will be: (batch, seq, feature), i.e. batch-size times sequence length times the dimension of your input (however dimensional that may be). Then, everything is gonna work as you are used to it.
To answer the second part of your question, normally, each sequence in a batch is independent of the others (since they commonly get sampled at random). So, there is no direct dependence between any two inputs in a batch (except, of course, for the fact that they are commonly expected to stem from some underlying shared data generating process which you want to approximate by the RNN).
And a hidden state is commonly maintained per batch element, i.e. there is one hidden state per batch element (i.e. per sequence).

annoyingly this can mess things up. I was doing attention type stuff where I was interested in hidden state of encoders, but later when I did linear layers I had to have batch first. I thought that solving this would be to use batch_first = True but the output dimension of hidden state was still time x batch x features, much to my annoyance.

Thank you, that explains it. But what is the logic behind this change in order which, as @david points out, can be rather inconvenient?

@rboz22 imo they probably added the batch_first argument before people started doing stuff where you explicitly need the hidden state, originally this was just a 'behind the scenes' thing that you would not be particularly interested in. This is of course just a guess.

Also, I think it might be motivated by how real-life datasets are set up. I could imagine that there are many sets where you have for each time point a set of measurements (i.e. your batch elements) and then want to have sequence first, consequently.


text:
How should we interpret all the different metrics in reinforcement learning?
I'm trying to train some deep RL agents using policy gradient methods like AC and PPO. While training, I have a ton of different metrics being monitored.
I understand that the ultimate goal is to maximize the reward or return per episode.
But there are a ton of other metrics that I don't understand what they are used for.
In particular, how should one interpret the mean and standard deviation curves of the policy loss, value, value loss, entropy, and reward/return over time while training?
What does it mean when these values increase or decrease over time? Given these curves, how would one decide how to tune hyperparameters, see where the training is succeeding and failing, and the like?

I think you should focus on one metric and one algorithm. In any case, I think you should provide a definition of "policy loss" and "value loss". And are you calculating the entropy of what exactly?

As you said, generally the most important one is reward per episode. If this isn't increasing overall, there's a problem (of course this metric can fluctuate, I mean to say that macroscopically it should increase).
Policy loss (I assume you mean the "actor loss"?) is generally harder to interpret. You should think of this more as a source of gradients and not necessarily a good indicator of how well your agent is performing.
I'm not really sure why you'd be monitoring the value during training. Value loss, however, is basically equivalent to value loss in value based methods like Q-learning, for example. So this one should be decreasing overall. Otherwise the baselines you compute to reduce variance in the policy gradient will either be less effective, or even harmful.
Entropy is a nice quantity to measure, because it's a good indicator of how much your agent is exploring. If you see that your agent is not achieving high returns and the entropy is really low, this means that your policy has converged to a suboptimal one. If the entropy is really high, this means the agent is acting fairly randomly (so it's basically exploring a lot). Ideally the entropy should decrease over time, so your policy becomes more deterministic (less exploration) as it reaches an optimum.


text:
How can I create an embedding layer to convert words to a vector space from scratch?
For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some vector representation. I understand the gist, but I can't find any deep explanations or tutorials that don't start with importing TensorFlow. All I'm really told is that it works by context using a few surrounding words, but I don't understand exactly how.
Is it much different from a classic network, with weights and biases? How does it figure out the loss?
If someone could point me towards a guide to how these things work exactly I would be very grateful.

Word2vec embeddings are trained using a simple auto-encoder model that takes a word and tries to predict one word from the window of surrounding words.

You could define it like this:
num_of_words = 50000
# one hot encoded word
input = Input(num_of_words)
# You could use non linear activation
w2v = Dense(300, activation=”linear”)(input)
output = Dense(num_of_words, activation=”softmax”)(w2v)

But in practice, the model is redefined and takes two words as input and predicts the next words. It outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words).

It is trained both ways from the beginning to the end of a sentence and in the reverse direction. The loss used is categorical_crossentropy. A detailed explanation can be found here [http://jalammar.github.io/illustrated-word2vec/]

thank you for this, ill read the article you have mentioned, but if I could also ask how many words is a good amount to use to train such a network?

It depends what you want to achieve And what kind of hardware you have. I trained  it for 1 million words on a regular laptop with gpu.  I did not want to reinvent the wheel so I have used gensim. If There is a pretrained model for your task/language then I would go for it as it usually is trained on bigger datasets.


text:
If the transition model is available, why would we use sample-based algorithms?
Sample-based algorithms, like Monte Carlo Algorithms and TD-Learning, are often presented as useful since they do not require a transition model.
Assuming I do have access to a transition model, are there any reasons one might want to use sample-based methods instead of performing a full Bellman update?

A full Bellman update can be intractable. For instance, if your state space or action space are continuous, the full Bellman update is intractable. You can try to solve this by discretizing, but if your state space is large this will also be intractable.

Would it also make sense to prefer MC in MDPs where episodes run for a very long time, since it will take Bellman updates far longer to propagate back to the start state?

Not necessarily, I believe you'd experience the same issue with MC. I believe in general if you can do bellman updates, you probably should.

Interesting. In MC, all states in an episode are updated based on the terminal state. So if the reward signal is sparse and only present in terminal states, like in a game, I think the signal would propogate faster, since in Bellman updates, the reward propogates back one state at a time

Don't forget, in MC you have to wait an entire episode before an update. Credit assignment will be difficult one way or the other, I believe.


text:
Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?
Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?
Max basically zeros out the gradients of all the non-maximum values. Especially at the beginning of training, this means it is zeroing out potentially useful features simply because of random weight initialization. Wouldn't this drastically slow down the training in the beginning?

Could you give more details of the substitution that you are considering? There's a few different ways you could use max in place of softmax. Some won't work at all, let alone slowly. But there might be some limited ways in which you could use max and still be able to train a classifier


text:
How to fine tune BERT for question answering?
I wish to train two domain-specific models:

Domain 1: Constitution and related Legal Documents
Domain 2: Technical and related documents.

For Domain 1, I've access to a text-corpus with texts from the constitution and no question-context-answer tuples. For Domain 2, I've access to Question-Answer pairs.
Is it possible to fine-tune a light-weight BERT model for Question-Answering using just the data mentioned above?
If yes, what are the resources to achieve this task?
Some examples, from the huggingface/models library would be mrm8488/bert-tiny-5-finetuned-squadv2, sshleifer/tiny-distilbert-base-cased-distilled-squad, /twmkn9/albert-base-v2-squad2.

The answer is yes but 'lightweight' will require a 'lightweight' model.
Your application for 'domain one' is called open domain question answering (ODQA).  Here is a demonstration of ODQA using BERT: https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/
Your application for 'domain two' is a little different.  It is about learning sequences from sequences.  More specifically these are called sequence to sequence models.  Here is an example using a pre-trained BERT model fine-tuned on the Stanford Question Answering (SQuAD) dataset.
Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.
In both applications, the resources required are going to rely on the performance you require.  There are many sizes of BERT models.  Generally, the larger the model, the larger the GPU memory requirements, and the higher the performance (i.e. accuracy, precision, recall, F1-score, etc.).  For example, I can run BERT Base on a particular data set on a GTX 1080Ti and a RTX 2080Ti but not BERT Large.
This article, NVIDIA Quadro RTX 6000 BERT Large Fine-tune Benchmarks with SQuAD Dataset shows performance for BERT using TensorFlow on four NVIDIA Quadro RTX 6000 GPUs.
There is a 'mobile' version of BERT called MobileBERT for running on small devices like smartphones.  Here is an article on using that with SQuAD: https://www.tensorflow.org/lite/models/bert_qa/overview
cdQA-suite is a good package.  The following should help in fine-tuning on your own corpus:

How to create your own Question-Answering system easily with python
cdQA examples

For Domain 1, I have a list of articles from which I want the model to answer questions, so it is Closed Domain, or rather I want it that way.
For Domain 2, yes I'm up to date with BERT and the memory issues, what I want to know specifically, is whether just a text corpus can be used to fine-tune a model.

Yes, you only need a text corpus.

Can you point me to a resource, using which I can fine-tune a model only using a text-corpus and then use this model for Question-Answering.

I added some information on the cdQA Suite which should help.

I've also used cdQA extensively, the problem with the same is the cdQA-annotator tool depends on us manually highlighting answer passages, which would be difficult to achieve when we have close to 5000 question-answer pairs.

That is not how I understand their architecture.  For example, their PDF tutorial shows that their data frame requires a title of the document and the 'paragraphs' field is the content of the entire document.  There is no training on discretely defined questions and answer pairs.  https://github.com/cdqa-suite/cdQA/blob/master/examples/tutorial-use-pdf-converter.ipynb

Let us continue this discussion in chat.


text:
Is it necessary to label the background when generating the labelled dataset for semantic segmentation?
When I label images for semantic segmentation (using u-net, if that matters), is labeling the background (anything I am not interested in) necessary? Will it improve the network's performance?

I am not sure I understand your concern. In image segmentation, you want to label each pixel. So, you will have the labels for each of your objects of interest. I think the remaining objects will just be considered background. If you explicitly label the background, then I suspect that this could mean that you force the background to look in a certain way, but I am not sure about the implications. To be honest, I am not sure what the benchmark datasets like COCO do. You may want to investigate that.

Maybe this dataset ADE20K is worth investigating too.

Thanks, that's interesting. I'll see if I can find the time to benchmark that


text:
How is depth perception (e.g. in autonomous driving) addressed without using a Lidar or Radar unit?
For practical applications, like autonomous driving, depth perception is needed to make useful decisions.
How is this normally addressed without using a LIDAR or RADAR unit (but using a camera)?

Usually you need two cameras, rt to observe the same point from two different angles, like our eyes do to create a stereo image.

https://docs.opencv.org/master/dd/d53/tutorial_py_depthmap.html

I don't know about any single camera-based depth perception based thing but there is software used by some game devs  for large buildings that involves a drone or some camera that photographs a location from every angle possible.

Is that what you were looking for?

Thanks, now how would one separate the ground plane from objects laying on it? The simplest idea is to angle your cameras to now include video right infront of the bottom of the robot but that could miss things.

I am afraid I don't really know. I haven't worked on that specific stuff in CV.


text:
How to quickly change hand-drawn shapes to symmetrical polished shapes?
Given a hand-drawn shape, I'd like to generate the corresponding symmetrical polished shapes such as circle, rectangle, triangle,  trapezoid, square, parallelogram, etc.
A short video demonstration
Here below we can see a parallelogram, trapezoid, triangle and a circle.
I was wondering how can I transform it into symmetrical polished shapes?

At first, I tried a simple approach with traditional computer vision algorithms, with OpenCV (no neural networks were involved), by counting the number of corners, but it failed miserably, since there are many edge cases within a user's doodler.
So, I was thinking to delve into CNN specifically U-Net for segmentation.
Can somebody please give me some suggestions on how to approach this kind of problem? I'd like to read some relevant articles and code about this subject for getting a better grasp of this kind of problem.

Depending on how you want to do this problem it could be really easy. Do you just want the ability to have simple shapes drawn? Because if you only care about a small sample of shapes, you can make a super simple classifier network that would have no trouble getting very high accuracy, then if it detects a square, just return an image of a square. But again, this depends on the complexity of the task. Will there be multiple shapes? Does the output need to resemble the dimensions of the input?

@Recessive thank you for your questions, yes I would like to generate those polished shapes, But the actual position is important for me, and moreover the shapes can reside inside other shapes, therefore it's not only a classification problem. I have updated the question with more details. and have uploaded an image.

From how it looks, the most reliable method to try out is using Hough transform.
The Hough transform can be used to detect e.g. lines and circles in images (depending on which variant you are using; in this case it would amount to a combination of variants for both lines and circles obviously).
So, given some input image, the Hough transform tells you what the line/circle parameters are that have created a line in the input image. For example, given a line, it would tell you the intersection with the $y$-axis and the $slope$ of the line detected in the image.
Then, you could use these parameter information to reconstruct lines and circles detected in the image.
The last remaining problem to be solved then is to check where a detected line starts and stops (since this is not obvious from line parameters like $m$ (=$slope$) and $b$ (=intersection with $y$-axis) in the equation $y = mx+b$ describing some line).
But for that, you could "walk" along a line in the image space and check where the line is present or not. Then, you can draw line segments in the reconstruction image when the corresponding elements are also present in the original image.
The problem with (C)NNs would be that they are sensitive to rotation and scale etc. You could of course take a tremendously large number of filters to account for shapes of different rotation and scale, but that would increase the demand for labeled training data again (which could of course be automatically be generated a priori in this simple case).
Anyway, I'd suggest checking out Hough transform. To get some feeling for it, there are lots of libraries available implementing it for Python or MatLab, for example.
For further information, check out Wikipedia or YouTube.


text:
Why does adding another network help in double DQN?
What is the idea behind double DQN?
The target in double DQN is computed as follows
$$
Y_{t}^{\text {DoubleQ }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right) ; \boldsymbol{\theta}_{t}^{\prime}\right),
$$
where

$\boldsymbol{\theta}_{t}^{\prime}$ are the weights of the target network
$\boldsymbol{\theta}_{t}$ are the weights of the online value network
$\gamma$ is the discount factor

On the other hand, the target in DQN is computed as
$$Y_{t}^{\mathrm{DQN}} \equiv R_{t+1}+\gamma \max _{a} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}^{-}\right),$$
where $\boldsymbol{\theta}_{t}^{-}$ are the weights of the target network.
The target network for evaluating the action is updated using weights of the online network and the value fed to the target value is basically the old q value of the action.
Any ideas on how or why adding another network based on weights from the first network helps? Any example?

As the authors of this paper state it:

In $Q$-learning, the agent updates the value of executing an action in the current state, using the values of executing actions in a successive state. This procedure often results in an instability because the values change simultaneously on both sides of the update equation.  A target network is a copy of the estimated value function that is held fixed to serve as a stable target for some number of steps.

If I remember it correctly, the main concern is that the network could end up in a positive feedback loop, making sufficient exploration of various action and state combinations less likely to occur, which could be detrimental to the learning task.


text:
Is my understanding of back-propogation correct?
I am trying to learn backpropagation and this is what I know so far.
To update the weights of the neural network you have to figure out the partial derivative of each of the parameters on the loss function using the chain rule. List all of these partial derivatives in a column vector and you have your gradient vector of your current parameter's on the loss function. Then by taking the negative of the gradient vector to descend the loss function and multiplying it by the learning rate (step size) and adding it to your original gradient vector, you have your new weights.
Is my understanding correct? Also, how can this be done in iterations over training examples?

Your understanding seems to be correct (although your explanation isn't completely precise), apart from "adding it to your original gradient vector". You add the gradient vector to the parameters/weights vector.
(Note that back-propagation is just the algorithm that computes the gradient vector. The update of the parameters with the gradient of the loss function is the gradient descent/ascent step, even though it's true that some people, at least in the context of deep learning, refer to the combination of gradient descent and the computation of the gradients as the back-propagation algorithm, but this is just a terminology issue, which you should not dwell too much on.)

Also, how can this be done in iterations over training examples?

If I understand this question correctly, you want to know how we would update the parameters when there's more than one training example. In that case, you compute a gradient vector for each training example. Then the average of the gradient vectors (which is also a vector) is the actual gradient vector that you use to update the parameters. See this answer for more info.


text:
Why does this Keras implementation of the DDPG algorithm update the critic's network using the gradient but the pseudocode doesn't?
I'm trying to understand the DDPG algorithm using Keras
I found the site and started analyzing the code, I can't understand 2 things.
The algorithm used to write the code presented on the page

In the algorithm image, updating the critic's network does not require gradient
But the gradient is implemented in the code, why?
with tf.GradientTape() as tape:
    target_actions = target_actor(next_state_batch)
    y = reward_batch + gamma * target_critic([next_state_batch, target_actions])
    critic_value = critic_model([state_batch, action_batch])
    critic_loss = tf.math.reduce_mean(tf.math.square(y - critic_value))

critic_grad = tape.gradient(critic_loss, critic_model.trainable_variables)
critic_optimizer.apply_gradients(zip(critic_grad, critic_model.trainable_variables))

The second question is why in the photo of the algorithm when calculating the actor's policy gradient are 2 gradients multiplied by themselves and in the code only one gradient is calculated for the critic's network and it's not multiplied by the second gradient?
with tf.GradientTape() as tape:
    actions = actor_model(state_batch)
    critic_value = critic_model([state_batch, actions])
    # Used `-value` as we want to maximize the value given
    # by the critic for our actions
    actor_loss = -tf.math.reduce_mean(critic_value)

actor_grad = tape.gradient(actor_loss, actor_model.trainable_variables)
actor_optimizer.apply_gradients(zip(actor_grad, actor_model.trainable_variables))

I'm not 100% sure about this but it looks like to me that for the first question "minimizing the loss" means making a gradient descent step, thus we need to calculate the gradient. As for the second question, the algorithm looks like the expanded chain rule to me for the gradient of critic_value(state_batch, actor_model(state_batch)).

The answer to your first question is because the line 'update the critic by minimising the loss $L = \frac{1}{N} \sum_i \left( y_i - Q(s_i, a_i |\theta^Q)\right)^2$ is implying that you will do this by using a gradient, i.e. you calculate the gradient of the loss wrt the parameters and perform a gradient descent step.
For the second question, I am not 100% sure because I don't use tensorflow but I would assume that in the automatic differentiation used it will do the multiplication automatically behind the scenes. In PyTorch I know that it automatically calculates the chain rule when you take the derivative, so it is likely to be a reason similar to this.


text:
How can I formulate a prediction problem (given labeled data) as an RL problem and solve it with Q-learning?
One of my friends sent me a problem he was working on lately, and I couldn't help but I wonder how could it be solved using Q-learning. The statement is as follows:

Given the following datasets, the objective is to find a suitable strategy per customer contract to maximize Gain and minimize Cost according to the characteristics of the customer.

train.csv: 5000 independent rows, 33 columns.

Columns description:

Day (1, 2 or 3): on which day the strategy was applied.
28 variables (A, B, C, ..., Z, AA, BB): characteristics of the individual;
Gain: the gain for this individual for the corresponding strategy;
Cost: the cost for this individual for the corresponding strategy;
Strategy (0, 1 or 2): the strategy applied on this individual;
Success: 1 if the strategy succeeded, 0 otherwise.
If Success is 1, then the net gain is Gain - Cost, and if Success is 0, consider a standardized cost of 5.

test.csv: 2 000 independent rows, 31 columns.

Columns description:

Index: 0 to 1999, unique for each row.
Day (4): on which day the strategy will be applied.
28 variables (A, B, C, ..., Z, AA, BB): characteristics of the client;
Gain: the gain for this individual for the corresponding strategy;
Cost: the cost for this individual for the corresponding strategy;

From what I understood, the train.csv file is used to build a Q-Learning model, and the test one for generating a strategy and predicting a Success.
My main question is:
How to formulate this problem as an RL problem? How to define an episode? Since the training data is labeled, this could be clearly a classification problem (predicting the strategy), but I have no idea how to solve it using RL (Q-learning ideally). Any ideas will be helpful.


text:
Can text-independent writer identification be done without multi-sentence training datasets for each writer?
I am trying to learn more about text-independent writer identification and was hoping for some advice.
I have a folder with 100k images, each of them with a different handwritten sentence. All of the images have sentences of different lengths. They range from about 30 to 80 English characters. The file names start at 1.png and go up to 100k.png. That's it, as far as input data. 95% of the sentences are written by different writers. 5% are written by the same writers. Some writers might have written 2 sentences, while others 300+.
Does anyone know of an identification method that would be able to determine what images were written by the same writer?
I know that most methods require each writer to have provided a full page of sample writing for training, but, of course, I do not have that.

I have edited your post to clarify it. Make sure that I did not change the meaning of your sentences.

Is the data labeled, i.e. do you know which sentence has been produced by which writer? If so, I'd start out training a Recurrent Neural Network, which is specialized on working with sequence data.


text:
How much data do we need for making a successful de-noising auto-encoder?
Is there a guide how much data do you need for making successful denoising model using autoencoders?
Or the rule is, the more data, the better it is?
I tried with small dataset 350 samples,  to see what I will get as an output. And I failed. :D