text:
Does coarse coding with radial basis function generate fewer features?
I am learning about discretization of the state space when applying reinforcement learning to continuous state space. In this video the instructor, at 2:02, the instructor says that one benefit of this approach (radial basis functions over tile coding) is that "it drastically reduces the number of features". I am not able to deduce this in the case of a simple 2D continuous state space.
Suppose we are dealing with 2D continuous state space, so any state is a pair $(x,y)$ in the Cartesian space. If we use Tile Coding and select $n$ tiles, the resulting encoding will have $2n$ features, consisting of $n$ discrete valued pairs $(u_1, v_1) \dots (u_n, v_n)$ representing the approximate position of $(x,y)$ in the frames of the $n$ 2-D tiles. If instead we use $m$ 2-D circles and encode using the distance of $(x,y)$ from the center of each circle, we have $m$ (continuous) features.
Is there a reason to assume that $m < 2n$?
Furthermore, the $m$-dimensional feature vector will again need discretization, so it is unclear to me how this approach uses fewer features.


text:
What is the difference between return and expected return?
At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$.
If an agent is following a policy (which in itself is a probability distribution of choosing a next state $S_{t+1}$ from $S_{t}$), the agent wants to find the value at $S_{t}$ by calculating sort of "weighted average" of all the returns from $S_{t}.$ This is called the expected return.
Is my understanding correct?

You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations.

Formally, the return (also known as the cumulative future discounted reward) can be defined as
$$
G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1},
$$
where $0 \leq \gamma \leq 1$ is the discount factor and $R_{i}$ is the reward at time step $i$. Here $G_t$ and $R_i$ are considered random variables (and r.v.s are usually denoted with capital letters, so I am using the notation used in the book Reinforcement Learning: An Introduction, 2nd edition).
The expected return is defined as
\begin{align}
v^\pi(s)
&= \mathbb{E}\left[G_t \mid S_t = s \right] \\
&= \mathbb{E}\left[\sum_{k=0}^\infty \gamma^k R_{t+k+1} \bigm\vert S_t = s \right]
\end{align}
In other words, the value of a state $s$ (associated with a policy $\pi$) is equal to the expectation of the return $G_t$ given that $S_t = s$, so $v^\pi(s)$ is defined as a conditional expectation. Note also that the expected value is usually defined with respect to a random variable, which is the case. Note also that $S_t$ is a random variable, while $s$ is a realization of this random variable.
A policy is not a probability distribution of choosing the next state. A stochastic policy is a family of a conditional probability distribution over actions given states. There are also deterministic policies. Have a look at this question What is the difference between a stochastic and a deterministic policy? for more details about the definition of stochastic and deterministic policies.

If an agent is following a policy, the agent wants to find the value at $S_{t}$ by calculating a sort of "weighted average" of all the returns from $S_{t}.$ This is called the expected return.

In the case of Monte Carlo Prediction, the value of a state associated with a specific policy, that is, the expected value of the return given a state is approximated with a finite (weighted) average. See e.g. What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?. Furthermore, note that the expectation of a discrete random variable is defined as a weighted average.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why isn't the evolutionary Turing machine mainstream?
Given that recurrent neural networks are equivalent to a Turing machine, then why isn't the evolutionary Turing machine, e.g. described in the paper Evolution of evolution: Self-constructing Evolutionary Turing Machine case study (2007), mainstream?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is there a rigorous proof for finding Hopfield minima?
I am looking for a rigorous mathematical proof for finding the several local minima of the Hopfield networks. I am searching for something rigorous, a demonstration, not just letting the network keep updating its neurons and wait for noticing a stable state of the network.
I have looked virtually everywhere, but I found nothing.
Is there a rigorous proof for Hopfield minima? Could you give me ideas or references?

See the paper On the Convergence Properties of the Hopfield Model (1990), by Jehoshua Bruck.
In the first section of the paper, J. Bruck describes the Hopfield network (popularized by J. J. Hopfield in 1982 in his paper Neural networks and physical systems with emergent collective computational abilities, hence the name of the network), then he describes the notation that is used throughout the paper and he gives some examples where a simple Hopfield network (with two nodes) converges to stable states (of the network) and cycles (of which the author also gives a definition).
The usual proofs of the convergence properties of Hopfield networks involve the concept of an energy function, but, in this paper, J. Bruck uses an approach (based on an equivalent formulation of the Hopfield network as an undirected graph) that does not involve an energy function, and he unifies three apparently different convergence properties related to Hopfield networks (described in part $C$ of the section Introduction). More specifically, finding the global maximum of the energy function associated with the Hopfield network operating in a serial mode (which is defined in part $A$ of the Introduction section of the paper) is equivalent to find a minimum cut in the undirected graph associated with this Hopfield network.
Furthermore, note that the proofs of convergence of the Hopfield networks actually depend on the structure of the network (more specifically, its weight matrix). For example, if the weight matrix $W \in \mathbb{R}^{n \times n}$ (where $n$ is the number of nodes in the network) associated with the Hopfield network is a symmetric matrix with the elements of the diagonal being non-negative, then the network will always converge to a stable state.
See also the chapter 13 The Hopfield Model of the book Neural Networks - A Systematic Introduction (1996) by Raul Rojas.

Thank you for your answer. Sorry I if was not clear but I intend my question the way I asked it. I understand why Hopfield net  (the proof) converge to a local minima, a stable state etc... Precisely I already read the resources that you mentionned and they only deal with that issue (I found Rojas book very useful thou!)

@ladangvu Ok, so what's your question? What do you want more than this? The resources I provided are already quite rigorous, so I really don't get your question.

What I am looking for is a demonstration for finding all these stable states. Of course everybody states that they are the patterns learned, their opposite (negative counterpart) and spurious states. But nowhere did I find proof that the patterns learned are local minima of the energy function.

@ladangvu What do you mean by "demonstration for finding all these stable states"? Have you read the mentioned paper "On the Convergence Properties of the Hopfield Model"? Even in the initial examples, the author gives examples of states that are the local minimum of the energy function (that is, where the Hopfield network reaches a stable state)? Also, have you read the part of my answer where I say "note that the proofs of convergence of the Hopfield networks actually depend on the structure of the network".

A mathematical demonstration (as we regularly do in multivariable functions optimization) for determining the local minimizer and minimum of the energy function E. A demonstration like any function that we would like to minimize, for example from my course : f(x,y) = x^3+y^3-3cxy where c belongs to IR*. After calculate the partial derivatives, finding the critical points and and calculate the determinant of the hessian at these points, we found that :
(0,0) is a saddle point and (c,c) a local maximizer if c < 0 or a local minimizer if c > 0.

Yes I read it, I understood these examples. Yes I read it, as I am dealing with Hopfield networks, I assume that W is symetric and with a diagonal full of zero (this is part of the definition of a Hopfield network, the weights w_ij = 0 when i = j.

$W$ does not necessarily have to be symmetric or the diagonal elements do not necessarily have to be zero (See part $B$ of the Introduction section). However, some proofs assume that e.g. the elements of the diagonal are non-negative (but it doesn't mean that they can't be positive) and the matrix is symmetric (see appendix 5 of the paper).

This is a good job @nbro


text:
Is convergence to a local minima more likely with transfer learning?
While doing transfer learning where my two problems are face-generation and car-generation is it likely that, if I use the weights of one problem as the initialization of the weights for the other problem, the model will converge to a local minima? In any problem is it better to train from scratch over transfer learning? (especially for GAN training?)


text:
What AI technologies were used in earlier versions of StarCraft?
I remember back in the 2000s, it was, and still is possible to play against computers in StarCraft BroodWar. They were not as good as pro players, but still reasonably smart.
What AI technologies were used in earlier versions of StarCraft: Brood War?

See https://github.com/bwapi/bwapi and https://github.com/jncraton/BWMetaAI, which may be useful, although I think they are related to recent versions of the game. See also http://www.cs.mun.ca/~dchurchill/starcraftaicomp/history.shtml and https://www.gamasutra.com/view/news/300931/Without_code_from_the_original_Blizzard_had_to_build_StarCraft_Remastered_from_scratch.php.

@nbro the links are good for an answer.

Ok, maybe later I will provide one or you can provide one for yourself and I will upvote it if you provide details about the AI in the game ;)

But are you only interested in the AIs used in the old versions of the game? I only quickly skimmed through those links, but I don't think they provide any info about the AIs used in those old games. If you are interested in an overview of the AIs used also in current and more recent versions, then the links above should definitely be useful.

Overview will do fine.


text:
How should I deal with variable input sizes for a neural network classifier?
I am currently working on a project, where I have a sensor in a shoe that records the $X, Y, Z$ axes, from an acceleration and gyroscope sensor. Every millisecond, I get 6 data points. Now, the goal is, if I do an action, such a jumping or kicking, I would use the sensor's output to predict that action being done.
The issue: If I jump, for example, one time I may get 1000 data points, but, in another, I get 1200 amounts, meaning the size of the input is different.
The neural networks I've studied so far require the input size to be constant to predict a $Y$ value, however, in this case, it isn't. I've done some research on how to make a neural network with variable sizes, but haven't been able to find one which works. It's not a good idea to crop the input to a certain size, because then I am losing data. In addition, if I just resize the smaller trials by putting extra $0$s, it skews the model.
Any suggestions on a model that would work or how to better clean the data?

the accelerometer gives the acceleration in the x,y,z. Then the gyroscope does the same.

It is much simpler to process the data in a different way. Since you're using temporal data a common practice is to define a priori a minimum time-step, usually called $\textit{granularity}$, which must be bigger than you're sensor responsiveness. Using this granularity value you'll then be able split your data into intervals, and you can then combine each instance belonging to an interval with a function that you like. Most common choice is obviously averaging the values, but also summing then could be a choice or a moving average.
Don't think that in this way you will loose information, preprocessing is also called data cleaning for a reason, not aways raw data are better.
As a last note, I would suggest you to look into 'Machine learning for the quantified self', there is growing literature going on about using sensors to train predictive models about movements and body measures like heart-rate. You might find something about preprocessing for this specific sensor-applications.


text:
How does the optimization process in hindsight experience replay exactly work?
I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the infamous sparse reward problem. It is based on the intuition that human beings constantly try and learn something useful even from their past failed experiences.
I have almost completely understood the concept. But in the algorithm posited in the paper, I don't really understand how the optimization works. Once the fictitious trajectories are added, we have a state-goal-action dependency. This means our DQN should predict Q-Values based on an input state and the goal we're pursuing (The paper mentions how HER is extremely useful for multi-RL as well).
Does this mean I need to add another input feature (goal) to my DQN? An input state and an input goal, as two input features to my DQN, which is basically a CNN?
Because in the optimization step they have mentioned that we need to randomly sample trajectories from the replay buffer and use those for computing the gradients. It wouldn't make sense to compute the Q-Values without the goal now, because then we'd wind up with duplicate values.
Could someone help me understand how exactly does the optimization take place here?
I am training Atari's "Montezuma's Revenge" using a double DQN with Hindsight Experience Replay (HER).

Yes, you need to add goal state as an input otherwise you won't know what goal you are trying to pursue. It might take a lot of memory to run HER on ATARI games though, also defining final goal is not straightforward for such environments

Maybe you should clarify why "defining final goal is not straightforward for such environments". What goal are you talking about? Are you talking about the fake goals that are needed to augment the ER?

authors: Анатолий Глянцев


text:
Which model should I choose to maximise reward of having chosen two numbers from a list?
I am looking for a technique to train a machine learning model to choose two items from a list.
So, given a list $x=[x_1, x_2, x_3, x_4, \dots, x_n]$, the model needs to choose two elements $(x_i, x_j)$. I have a function $R(x, x_i, x_j)$, which will output the reward of choosing $(x_i, x_j)$ given $x$.
What type of models should I use, and how should I train it to maximize the reward?
I've tried using deep reinforcement learning, but I ran into the following problems with implementing the Q-Network:

Variable-length inputs (fixed by using RNN, I think)
The output size grows factorially (for an input set of n elements, there are n choose 2 ways to pick 2 elements, so the network needs to output n choose 2 expected rewards)


text:
Which activation functions can lead to the vanishing gradient problem?
From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem.
What other activation functions can lead to the vanishing gradient problem?

Possibly, the saturating activation functions (i.e. activation functions that squash their input to a finite interval). Both sigmoid and hyperbolic tangents are saturating activation functions.


text:
What are the main differences between skip-gram and continuous bag of words?
The skip-gram and continuous bag of words (CBOW) are two different types of word2vec models.
What are the main differences between them? What are the pros and cons of both methods?

So as you're probably already aware of, CBOW and Skip-gram are just mirrored versions of each other. CBOW is trained to predict a single word from a fixed window size of context words, whereas Skip-gram does the opposite, and tries to predict several context words from a single input word.
Intuitively, the first task is much simpler, this implies a much faster convergence for CBOW than for Skip-gram, in the original paper (link below) they wrote that CBOW took hours to train, Skip-gram 3 days.
For the same logic regarding the task difficulty, CBOW learn better syntactic relationships between words while Skip-gram is better in capturing better semantic relationships. In practice, this means that for the word 'cat' CBOW would retrive as closest vectors morphologically similar words like plurals, i.e. 'cats' while Skip-gram would consider morphologically different words (but semantically relevant) like 'dog' much closer to 'cat' in comparison.
A final consideration to make deals instead with the sensitivity to rare and frequent words. Because Skip-gram rely on single words input, it is less sensitive to overfit frequent words, because even if frequent words are presented more times that rare words during training, they still appear individually, while CBOW is prone to overfit frequent words because they appear several time along with the same context. This advantage over frequent words overfitting leads Skip-gram to be also more efficient in term of documents required to achieve good performances, much less than CBOW (and it's also the reason of the better performances of Skip-gram in capturing semantical relationships).
Anyway, you can find some comparisons in the original paper (section 4.3).
Mikolov et al. 2013
About the architecture, there's not much to say. They just randomly initialised word embedding for each word, then a projection matrix NxD (number of context words times embeddings dimension) is generated at each iteration, there is no hidden layer, the vectors are just averaged together and then fed into an activation function to predict index probabilities in a vector of dimension V (the size of the vocabulary).
For a more specific explanation (even Mikolov's paper lack some detail) I suggest checking this blog page (Words as Vectors), even though the model described there do apply a hidden layer, unlike the original architecture.

Word embeddings are the results of learning from deep learning algorithms, which can learn characters from data through feature extraction. One implementation of word embedding is word2vec.
Word2vec has two models, namely

Continuous Bag of Word (CBOW) and
Skip Gram Model.

Both of these methods use the concept of a neural network that maps words to target variables, which are also words. In these techniques, "weights" are used as word vector representations. CBOW tries to predict a word on the basis of its neighbors, while Skip Gram tries to predict the neighbors of a word.
In simpler words, CBOW tends to find the probability of a word occurring in a context. So, it generalizes over all the different contexts in which a word can be used. Whereas SkipGram tends to study different contexts separately. Skip Gram needs more data to be trained contains more knowledge about the context.
For further explanation, you can read the paper from Mikolov about word embedding and word2vec.


text:
How should I select the features for predicting diseases (in particular when patients specify their health issues)?
My aim is to train a model for predicting diseases. Now, according to this Wikipedia article, diseases are classified based on the following criteria in general:

Causes (of the disease)
Pathogenesis (the mechanism by which the disease progresses)
Age
Gender
Symptoms (of the disease)
Damage (caused by the disease)
Organ type (e.g. heart disease, liver disease, etc.)

Are these features used for predicting diseases universally (i.e. all types of diseases)? I don't think so. There can be other attributes as well. For example, traveling in the case of coronavirus.
So, are there better features for predicting diseases?
Or which ones among them are better than the others, when patients specify their health issues?

You are not asking an AI question but a subject domain question.  I suggest you ask the same question on a disease research site.  When you build your classifier you can determine which features are dominant.  For example, with random forests see this article: https://towardsdatascience.com/running-random-forests-inspect-the-feature-importances-with-this-code-2b00dd72b92e.

So for Medical Prognosis, there are some variables that commonly come up like Age, Sex, Ascites, Hepato, Spider, Status of the disease and many others but it depends on the disease. You'll commonly encounter these variables if you're doing regression or classification.
Also, if you're reading Radiology Reports for getting the input for the model then you also have to take care of jargons. The same symptoms can be written in various ways but all point towards the same prognosis i.e., there can be synonyms for labels. Try reading this to get more information on how we can do information extraction from Radiology Reports. This is the famous CheXpert paper

To begin from the scratch, and in order to keep approach simple we have to analyze the input text(clinical narration) for the following data:

Is input a word or a group of words or a sentence?

Is input a meaningful sentence? By meaningful, I mean grammatically correct.

Does the word, group of words or a sentence contain symptoms or health issues?

Does the sentence contain data about a person’s age and gender?

Does the sentence contain data about a person’s diet, medical history, work routine,  travelling history or getting in contact with
any ill person?

If there are any other attributes that one has to look for then I would be keen to find out from the subject matter experts.


text:
What is the purpose of argmax in the PPO algorithm?
I'm kinda new to machine learning and still not too solid on math and particularly calculus. I'm currently trying to implement PPO algorithm as described in the spiningUp website :

This line is giving me a hard time :

What does the $\operatorname{argmax}$ mean, in this context? They are also talking about updating the policy with a gradient ascent. So, is taking argmax with respect to $\theta$ the same as doing:

where $J$ is the min() function?

In this case yes, $J$ is the big $\min$ expression and you apply Adam on that. But be careful because they say they do ascent, but automatic differentiation software usually minimizes given function so your $J$ would be $−\min(⋅)$.


text:
What is the simplest classification problem which cannot be solved by a perceptron?
What is the simplest classification problem which cannot be solved by a perceptron (that is a single-layered feed-forward neural network, with no hidden layers and step activation function), but it can be solved by the same network if the activation function is swapped out to a differentiable activation function (e.g. sigmoid, tanh)?
In the first case, the training would be done with the perceptron training rule, in the second case with the delta rule.
Note that regression problems cannot be solved by perceptrons, so I'm interested in classification only.

Anything with a non-step output, perhaps? I thought that a perceptron with a different activation function is still a perceptron.

Anything that is not linearly separable cant be solved perceptrons, unless you use feature maps on data to map them to a higher dimension in which it is linearly separable.
As a simple, concrete example, perceptron cant learn the XOR function.
This page might help you further.

Someone had already provided an answer to this question in the past, but it was deleted by the author after I commented saying that they were not addressing this part "but it can be solved by the same network if the activation function is swapped out to a differentiable activation function (e.g. sigmoid, tanh)?". Maybe you should address this part too.

I am not saying changing the activation function changes or not the set of problems that the perceptron can solve. I am just asking you to explain why.


text:
Is this a good approach to solving Atari's "Montezuma's Revenge"?
I'm new to Reinforcement Learning. For an internship, I am currently training Atari's "Montezuma's Revenge" using a double Deep Q-Network with Hindsight Experience Replay (HER) (see also this article).
HER is supposed to alleviate the reward sparseness problem. But since the reward is annoyingly too sparse, I have also added a Random Network Distillation (RND) (see also this article) to encourage the agent to explore new states, by giving it a higher reward when it reaches a previously undiscovered state and a lower reward when it reaches a state it has previously visited multiple times. This is the intrinsic reward I add to the extrinsic reward the game itself gives. I have also used a decaying greedy epsilon policy.
How well should this approach work? Because I've set it to run for 10,000 episodes, and the simulation is quite slow, because of the mini-batch gradient descent step in HER. There are multiple hyperparameters here. Before implementing RND, I considered shaping the reward, but that is just impractical in this case. What can I expect from my current approach? OpenAI's paper on RND cites brilliant results with RND on Montezuma's Revenge. But they obviously used PPO.


text:
What are the advantages and disadvantages of extrinsic and perplexity model evaluation in NLP?
In the video Evaluation and Perplexity by Dan Jurafsky, the author talks about extrinsic and perplexity evaluation in the context of natural language processing (NLP).
What are the advantages and disadvantages of extrinsic and perplexity model evaluation in NLP? Which evaluation method is preferred usually, and why?

Maybe this podcast would be helpful.


text:
Can denoising auto-encoders be convolutional and fully connected?
I have been reading lately on autoencoders a lot. I just wanted to summarize my understanding of denoising autoencoders. As far as I understand they can be

Fully connected (in which case, they will be over-complete autoencoders)
Convolutional

The reason I say it should be over-complete is that the objective is to learn new features and I think extra neurons in the latent layer would help. There is no reason to have a lesser number of neurons because compressing is not the objective. I just want to understand is this the right thinking.


text:
How does the number of stacked LSTM layers or units in each layer affect the model complexity?
I playing around sequence modeling to forecast the weather using LSTM.
How does the number of layers or units in each layer exactly affect the model complexity (in an LSTM)? For example, if I increase the number of layers and decrease the number of units, how will the model complexity be affected?
I am not interested in rules of thumb for choosing the number of layers or units. I am interested in theoretical guarantees or bounds.

In computational learning theory, the VC dimension is a formal measure of the capacity of a model. The VC dimension is defined in terms of the concept of shattering, so have a look at the related Wikipedia article, which briefly describes the fundamental concept of shattering. See also my answer to the question How to estimate the capacity of a neural network? for more details.
The paper Vapnik-Chervonenkis dimension of recurrent neural networks (1998), by Pascal Koirana and Eduardo D. Sontag, partially (because they do not take into account more advanced recurrent neural network architectures, such as the LSTM) answers your question.
In the paper, the authors show and prove different theorems that state the VC dimension of (standard) recurrent neural networks (RNNs), with different activation functions, such as non-linear polynomials, piecewise polynomials and the sigmoid function.
For example, Theorem 5 (page 70) states

Let $\sigma$ be an arbitrary sigmoid. The VC dimension of recurrent architectures with activation $\sigma$, with $w$ weights and receiving inputs of length $k$, is $\Omega(wk)$.

The proof of this theorem is given on page 75.
What does this theorem intuitively tell you? If you are familiar with big-O notation, then you are also familiar with the notation $\Omega(wk)$, which means that $wk$ is, asymptotically, a lower bound on the capacity of the RNN. In other words, asymptotically, the capacity of an RNN with $w$ weights receiving inputs of length $k$ is at least $wk$. How does the capacity of the RNN increase as a function of $w$?
Of course, this is a specific result, which only holds for RNNs with the sigmoid activation function. However, this at least gives you an idea of the potential capacity of an RNN. This theorem will hopefully stimulate your appetite to know more computational learning theory!
The paper On Generalization Bounds of a Family of Recurrent Neural Networks may also be useful, although it has been rejected for ICLR 2019.

Hmm, so it scales with number of weights? Therefore, can we say VC-dim (RNN with k layers) = k * VC-dim (RNN with 1 layer)?

Yes, it scales with weights and length of the input. But, again, note that $wk$ is a lower bound, so the capacity of the RNN may actually be higher (but only asymptotically).

authors: سوار محمد عبدالله القضاه


text:
Is the Cognitive Approach (SOAR) equivalent to the Chinese Room argument?
Soar is a cognitive architecture.

https://en.wikipedia.org/wiki/Soar_(cognitive_architecture)
https://soar.eecs.umich.edu/

There is something called "the Chinese box" or "Chinese room" argument:

https://en.wikipedia.org/wiki/Chinese_room
https://plato.stanford.edu/entries/chinese-room/

The "Chinese room" seems to be begging its question, but that is not what I am asking.  I am asking if there is any literal difference between a tool like "SOAR" and the formalism of the "Chinese box".  Is SOAR identical or equivalent to a "Chinese Box"?

Interesting question. However, the question in the title seems to be slightly different than the question in the body. Maybe clarify this!

Searle's Chinese room is analogical and is intended to present an easy-to-understand picture of the essential elements and processes of the digital computer. In the room the man (CPU) has a book of intructions (program) for responding to Chinese input questions. That is just one program of many possible programs the room could run. Each different program would be a different instruction book. SOAR would be just one of those books.

Yes, this seems reasonable, but I think you should say something about what SOAR is trying to do, i.e. is SOAR only manipulating symbols, etc.? I think this is the crux of the question.

I can emulate the hardware of a computer, on a computer.  In that emulation I can run a program.  If I wanted to be a little recursive the program itself could be emulation, arguably of much simpler hardware.  Together this system might comprise an inception-like box within a box within a box.  Can I understand your answer to mean that any computer is the Chinese box, an anti-Turing?


text:
How can I solve the zero subset sum problem with hill climbing?
I want to solve the zero subset sum problem with the hill-climbing algorithm, but I am not sure I found a good state space for this.
Here is the problem: consider we have a set of numbers and we want to find a subset of this set such that the sum of the elements in this subset is zero.
My own idea to solve this by hill-climbing is that in the first step, we can choose a random subset of the set (for example, the main set is $X= \{X_1,\dots,X_n\}$ and we chose $X'=\{X_{i_1},\dots,X_{i_k}\}$ randomly), then the children of this state can be built by adding an element from $X-X'$ to $X'$ or deleting an element from $X'$ itself. This means that each state has $n$ children. and the objective function could be the sum of the elements in $X'$ that we want to minimize.
Is this a good modeling? Are there better modelings or objective functions that can work more intelligently?

The hill-climbing algorithm to implement is as follows:

The algorithm should take four inputs: as always, there will be a multiset S and integer
k, which are the Subset and Sum for the Subset Sum problem; in addition, there
will be two integers q and r, with roles defined below.
Do the following q times:

(a) Choose a random subset (multiset) $S_0$ of S as the current subset.
(b) Do the following (hill climbing) r times:
i. Find a random neighbor T (see definition of neighbor below) of the current
subset.
ii. If neighbor T has smaller residue, then make T the current subset.
(c) Keep track of the residue of the final current subset when starting with subset
$S_0$.

Return the smallest residue of the q subsets tested by the algorithm.

Definition: Subset (multiset) B ⊆ S is a neighbor of a subset A of S if you can transform
A into B by moving one or two integers from A to B, or by moving one or two integers from
B to A, or by swapping one integer in A with one integer in B.
An easy way to generate a random neighbor B of a subset A of S is as follows:

Order the elements of S as $x_1, x_2, ..., x_n$.
Initialize B to be a clone of A.
Choose two distinct random indices i and j, where $1 ≤ i; j ≤ n$.
if $x_i$ is in A, remove it from B. Otherwise, add xi to B.
if $x_j$ is in A, then with probability 0.5, remove it from B. If $x_j$ is not in A, then with
probability 0.5, add $x_j$ to B.

The state space itself seems OK but your approach has some disadvantages. First, you need to pick an initial state of size $n$. What if the optimal solution is of size $\frac{n}{4}$ or $10n$? Then the search could take a long time using this approach because you are starting far away. If there are multiple solutions, do you care which one you find or is the smallest zero subset preferred? I'm going to assume that the empty set $X'=\{\}$ is not a solution and that solutions with smaller cardinality are preferred.
I would suggest using a generative heuristic to construct the initial state and then running your hill climbing procedure from that state. For example, a generative heuristic could be something like: "pick the pair of elements ($x_1$,$x_2$) from $X$ such that $x_1 + x_2$ is as close to zero as possible, and set $X' =\{x_1, x_2\}$ as the initial state". To apply this heuristic you will need to initially check $n(n-1)$ subsets of pairs. If this is not a solution, it will hopefully be very close to the solution in the state space and therefore a good place to start.
Another problem is the hill climbing does not remember previously visited states, and you could end up infinitely looping when you allow moves to both add and delete elements from $X'$. For example, suppose $X'=\{1\},X=\{-3,-50,-50,100\}$. Clearly the optimal solution is $X''=\{-50,-50,100\}$ but your method will add element $-3$ to $X'$ first, then remove $-3$, then pick $-3$ again etc, because these states have a lower objective value than any state with one of $-50$ or $100$ in it. So I think you need to use a more sophisticated approach that prevents visiting the same state twice.


text:
How to determine the target value when using ReLU as activation function?
Consider the following simple neural network with only one neuron.

The input is $x_1$ and $y_2$, where $-250 < x < 250$ and $-250 < y < 250$
The weights of the only neuron are $w_1$ and $w_1$
The output of the neuron is given by $o = \sigma(x_1w_1 + x_2w_2 + b)$, where $\sigma$ is the ReLU activation function and $b$ the bias.
Thus the cost should be $(o - y)^2$.

When using the sigmoid activation function, the target for each point is usually $0$ or $1$.
But I'm a little confused witch target to use when the activation function is the ReLU, given that it can output numbers greater than 1.

Hi and welcome to AI SE! Why do you want to use the ReLU as the activation function of the last layer of your neural network?

Thanks for editing! I'm a beginner with neural network. Actually, I read that ReLU is better than sigmoid because it have a non-linear behavior. Should I use sigmoid at the output layer?

ReLU and sigmoid have different properties, as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but I don't exclude people to use it for some reason, I just never saw it). A sigmoid is often used because the sigmoid squashes its inputs to the range $[0, 1]$, so, by doing this, the network will produce numbers that can be interpreted as probabilities, which can be useful in certain cases. In general, the targets don't necessarily need to be restricted to be 0 or 1. Do you want me to provide this information as a formal answer below?

Got it! You were very clear.

ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some reason, e.g. regression tasks where the output needs to be positive). ReLU is usually used as the activation function of a hidden layer. However, in your case, you don't have hidden layers.
The sigmoid function is used as the activation function of the output layer when you need to interpret the output of the neural network as a probability, i.e. a number between $0$ and $1$, given that the sigmoid function does exactly this, i.e. it squashes its input to the range $[0, 1]$, i.e. $\text{sigmoid}(x) = p \in [0, 1]$. When do you need the output of the network to be a probability?  For example, if you decide to use the cross-entropy loss function (which is equivalent to the negative log-likelihood), then the output of your network should be a probability. For example, if you need to solve a binary classification task, then the combination of a sigmoid as the activation function of the output layer and the binary cross-entropy as the loss function is probably what you need.
You could also have a classification problem with more than 2 classes (multi-class classification problem). In that case, you probably need to use a softmax as the activation function of your network combined with a cross-entropy loss function.
See this question How to choose cross-entropy loss in TensorFlow? on Stack Overflow for more info about different cross-entropy functions.
By the way, in general, the targets don't necessarily need to be restricted to be 0 or 1. For example, if you are solving a regression task, your target may just be any number. However, in that case, you may need another loss function (which is often the mean squared error).

You are misunderstanding something. You are mixing up inner layers with the output layer. But the question was very good.
Fist of all, with the only one layer and one neuron neural networks it does not exist. Only one layer can not bring nonlinearity in the network.
One neuron network means it's a linear regression or logistic regression if it passes through a sigmoid activation.
You have to look at NN like below.
The output of a neural networks is the final layer output.
There are two possible situations in the model.

Classification model: For classification people generally use softmax and it has more than one output and the maximum value always below 1(It's a probability distribution).
Regression model: A regression model has a continuous output like the output you mentioned in your problem statement. It has only one output channel. In the regression model, you can make the output as a linear combination of the previous layers (without defining the output range). You can use ReLU if you are sure your prediction is always positive.


text:
What is the difference between the concepts "known environment" and "deterministic environment"?
According to the book "Artificial Intelligence: A Modern Approach", "In a known environment, the outcomes (or outcome probabilities if the environment is stochastic) for all actions are given.", and in a deterministic environment, "the next state of the environment is completely determined by the current state and the action executed by the agent...".
What's the difference between the two terms? Don't they mean the same thing?

What's the difference between the two terms? Don't they mean the same thing?

They mean different things, and can occur in any combination.
A known, deterministic environment
This is an environment where the researcher knows how to calculate all the transitions in advance of observing them, and the transition from state $s$ given action $a$ is always to the same next state $s'$ with the same reward $r$.
Example: Any classic board game against a fixed opponent (where any situation allows choice of action, the opponent always picks the same choice in the same situation)
An unknown, deterministic environment
This is an environment where the researcher does not have the knowledge to calculate all the transitions in advance of observing them, but any observation the transition from state $s$ given action $a$ is always to the same next state $s'$ with the same reward $r$.
Example: Simple mechanical physics environments, where initial measurements are unknown, imprecise or the researcher does not want to code knowledge of them into the agent. E.g. pole balancing.
A known, stochastic environment
This is an environment where the researcher knows all the rules about transitions, but those rules include transitions with random elements. Transition from state $s$ given action $a$ varies according to some probability function $p(s'|s,a)$ possibly so does the reward $p(r|s,a)$ - sometimes combined into joint probability function $p(r, s'|s,a)$.
Example: Any board game involving dice, e.g. Backgammon.
An unknown, stochastic environment
This is an environment where the researcher does not know all the rules, or can only calculate expected results with difficulty or to a low level of precision. Transition from state $s$ given action $a$ varies according to some unknown probability function $p(r, s'|s,a)$. Learning the transition function may require many samples of $s, a$, and this will be a statistical approximation.
Example: In practice, most complex environments, including real-world physics with friction, fluids, non-perfect measurements.
It is quite common for purpose of experiments to have a simulation where the environment is technically known (because the researcher wrote it, or has access to the code and underlying models), but that agents are written to treat it like this last, more challenging case. Agents that can figure out how to act without prior knowledge of the environment are often of interest.

so, for example - chess are deterministic, while games with probability are stochastic?

@PiotrŻak: Yes. Backgammon is a classic board game that is stochastic, due to probilities introduced by the dice.


text:
What does "episodic training" mean?
I'm reading the book Hands-On Meta Learning with Python, and in Prototypical networks said:

So, we use episodic training—for each episode, we randomly sample a
few data points from each class in our dataset and we call that a
support set and train the network using only the support set, instead
of the whole dataset.

I think, but I'm not sure, I have understood what "episodic training" is, but what is the meaning of "episodic" or "episode" here?
I'm sorry, I'm not English and I can't guess what it is meaning searching in a dictionary. I know what an episode is, but I don't know what an episode, in this context of training, means.

It consists of organizing training in a series of learning problems, each relying on small "support" and "query" sets to mimic the few-shot circumstances encountered during evaluation.(an episode is a single task)


text:
What's the best solution to find distance of an object to camera?
I have an object with known size and I want to know that's the distance from the camera and camera angle. Is there any way to do this? I have a single source (camera).

Isn't this a trig. problem given the focal length?

i've seen ways to obtain focal length but i didn't implement this.


text:
How can we combine different deep learning models?
I know that ensembles can be made by combining sklearn models with a VotingClassifier, but is it possible to combine different deep learning models? Will I have to make something similar to Voting Classifiers?


text:
Can the importance sampling estimator have a non-stationary behaviour policy even if the target policy is stationary?
The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization.
The old policy $\mu$, or the behavior policy, is okay to be non-stationary in the IPS estimator even if the new policy $\nu$, or the target policy, should be stationary.
Is this true for the importance sampling (IS) estimator, which seems to be a variant of IPS, for off-policy evaluation in a reinforcement learning problem?
IS estimator is explained in this paper Doubly Robust Off-policy Value Evaluation for Reinforcement Learning.
The target policy should be stationary, but can the old policy be non-stationary in the IS estimator?


text:
What is the difference between the $\epsilon$-greedy and softmax policies?
Could someone explain to me which is the key difference between the $\epsilon$-greedy policy and the softmax policy? In particular, in the contest of SARSA and Q-Learning algorithms. I understood the main difference between these two algorithms, but I didn't understand all the combinations between algorithm and policy

SARSA + $\epsilon$-greedy
SARSA + Softmax
Q-Learning + $\epsilon$-greedy
Q-Learning + Softmax

The $\epsilon$-greedy policy is a policy that chooses the best action (i.e. the action associated with the highest value) with probability $1-\epsilon \in [0, 1]$ and a random action with probability $\epsilon $. The problem with $\epsilon$-greedy is that, when it chooses the random actions (i.e. with probability $\epsilon$), it chooses them uniformly (i.e. it considers all actions equally good), even though certain actions (even excluding the currently best one) are better than others. Of course, this approach is not ideal in the case certain actions are extremely worse than others. Therefore, a natural solution to this problem is to select the random actions with probabilities proportional to their current values. These policies are called softmax policies.
Q-learning is an off-policy algorithm, which means that, while learning a so-called target policy, it uses a so-called behaviour policy to select actions. The behaviour policy can either be an $\epsilon$-greedy, a softmax policy or any other policy that can sufficiently explore the environment while learning.
The figure below shows the pseudocode of the Q-learning algorithm. In this case, the $\epsilon$-greedy policy is actually derived from the current estimate of the $Q$ function. The target policy, in this context, is represented by the $\operatorname{max}$ operator, which is used to select the highest $Q$ value of the future state $s'$, which is the state the RL agent ends up in after having taken the action $a$ selected by the $\epsilon$-greedy behaviour policy, with respect to another action $a'$ from state $s'$. This may sound complicated, but if you read the pseudocode several times, you will understand that there are two different actions (and states). The target policy (i.e. the policy that the RL agent wants to learn) is represented by the $\operatorname{max}$ operator in the sense that the so-called target of the Q-learning update step, i.e. $r + \gamma \operatorname{max}_{a'} Q(s', a')$, assumes that the greedy action is taken from the next state $s'$. For this reason, Q-learning is said to learn the greedy policy (as a target policy), while using an exploratory policy, usually, the $\epsilon$-greedy, but it can also be the softmax. Note that, in both cases, the policies are derived from the current estimate of the Q function.

On the other hand, SARSA is often considered an on-policy algorithm, given that there aren't necessarily two distinct policies, i.e. the target policy is not necessarily different than the behaviour policy, like in Q-learning (where the target policy is the greedy policy and the behaviour policy is e.g. the softmax policy derived from the current estimate of the Q function). This can more easily be seen from the pseudocode.
.
In this case, no $\operatorname{max}$ operator is used and the $\epsilon$-greedy policy is mentioned twice: in the first case, it is used to choose the action $a$ and indirectly $s'$, and, in the second case, to select the action $a'$ from $s'$. In Q-learning, $a'$ is the action that corresponds to the highest Q value from $s'$ (i.e. the greedy action). Clearly, you are free to choose a different policy than the $\epsilon$-greedy (in both cases), but this will possibly have a different effect.
To conclude, to understand the difference between Q-learning and SARSA and the places where the $\epsilon$-greedy or softmax policies can be used, it is better to look at the pseudocode.

If I train my agent with softmax policy, is the learning process more unstable w.r.t the epsilon greedy?

@FraMan http://incompleteideas.net/book/first/ebook/node17.html says that it is unclear whether softmax is better or not than $\epsilon$-greedy and it may depend on the problem.

does Q-Learning + epsilon greedy makes sense? Q-Learning use a greedy policy as default.

@FraMan I think this is a good new question you could ask on the website ;)


text:
Why is this Monte Carlo approach scalable for a growing number of states variables and action variables?
I am reading a research paper on the formulation of MDP problems to ICU treatment decision making: Treatment Recommendation in Critical Care: A Scalable and Interpretable Approach in Partially Observable Health States. The paper applies a Monte Carlo approach to approximate the value function. Below is a screenshot of the excerpt that I came across.

The last sentence of the excerpt reads "The approach is scalable for growing number of states variables and action variables".
What does it mean when the author says that the Monte Carlo approach is scalable for a growing number of states variables and action variables? Wouldn't the amount of data needed to approximate the value function increase with the higher dimensionality of states? Or does the Monte Carlo approach scale better in time complexity as compared to traditional Q-learning methods?


text:
How is clustering used in the unsupervised training of a neural network?
How is clustering used in the unsupervised training of a neural network? Can you provide an example?

What is the topic for you?

@DuttaA I wouldn't say that homework questions are necessarily off-topic here. In this case, the user is asking for an example where clustering is involved in the unsupervised training of an NN. I've reformulated the question to make it more acceptable.

@DuttaA If he's asking only for one example, then this should be ok. If he's asking to describe all cases, yes, this is too broad.

@nbro Can you help me bro? =(

@RamedlawVocinnas Maybe later I will try to give an answer, but I wouldn't surely count on this.

@nbro Why?! =((


text:
What is the expected value of an IOU in this case?
I have a detection problem. An object with a probability of 0.5 is in a box with coordinates ((0,0), (2, 2)) and with a probability of 0.5 a box with coordinates ((2,0), (4,2)).
What is the maximum expected value of an intersection over union (IOU) with an object that a constant detection algorithm that produces one box can reach? I can't understand how I should find the expected value here. P.S. IOU is 0 cos intersection is empty.

Hi and welcome to Artificial Intelligence SE! To clarify (given that I am not familiar with this abbreviation), what do you mean by "IOU"? Is it an abbreviation for "Intersection over Union"?

Yes it is short form of "intersection over Union" . I was given this on the interview and still dont know what I should start with to do this)

So you are talking about the Jaccard similarity?!

Could you please give more details about what you mean by the object's probability of appearing in one box or another? I would take it to mean that you have a low level detection algorithm that is giving you these probabilities, and you want your higher level display algorithm to display a box with maximum IoU given that detection (in which case max IoU here is 0.5 because the boxes combine nicely to make a single box without gaps)? However, your first part of text does not mention any detector, just probability of an object appearing - in which case theoretical maximum IoU would be 1.0.


text:
What activation functions are better for what problems?
I’ve been reading about neural network architectures. In certain cases, people say that the sigmoid "more accurately reflects real-life" and, in other cases, functions like hard limits reflect "the brain neural networks more accurately".
What activation functions are better for what problems?


text:
What's the function that SGD takes to calculate the gradient?
I'm struggling to fully understand the stochastic gradient descent algorithm.
I know that gradient descent allows you to find the local minimum of a function. What I don't know is what exactly that function IS.
More specifically, the algorithm should work by initializing the network with random weights. Then, if I'm not mistaken, it forward-propagates $n$ times (where $n$ is the mini-batch size). At this point, I've no idea about what function should I search for, with over hundreds of neurons each having hundreds of parameters.

Welcome to AI Stack exchange!
You're right, as the network is initialised randomly, the resultant function is essentially impossible to get your head around. This is because most of the time the network has >4 dimensions (4 can be graphed with some effort and a lot of color), and as such is literally beyond human comprehension via graphing.
So what do we do? Well, conveniently, it is possible to find the gradient of segments of a function, without having to know the entirety of the function itself (it's worth noting that it actually is possible to find the resultant function and with a lot of effort find it's derivative. This proves to be much more work than it's worth though, as we don't need the general derivative that tells us what the gradient is for whatever input we give it, we only need the derivative at the specific input we just fed through the network).
This might be hard to understand, but if you're familiar with the chain rule, it might make a bit more sense. The chain rule essentially allows you to split a function into components, and find the gradient of those specific components. By combining all of that, you end up with some nice gradients at each weight/bias with respect to the loss function. Take the negative of the gradient, and you now have the change required to decrease the loss function.
This is obviously quite hard to understand without an example, so here's the best one I've ever found, that helped me very much.
Also, as a side note, the whole mini batch thing is used to minimise catastrophic forgetting (where the network begins to "unlearn" old inputs). To deal with minibatchs, what you do is take input individually, then find the gradient for all weights and bias' for that specific input and remember the changes you want to make. Do that for all inputs in the minibatch, then finally add all the changes together to get the resultant best change  for each weight and bias. Only then do you update the weights and bias'.
Let me know if you have any further questions

I know that gradient descent allows you to find the local minimum of a function. What I don't know is what exactly that function IS.

It's usually called the loss function (and, in general, objective function) and often denoted as $\mathcal{L}$ or $L$ (or something like that, i.e. it is not really important how you denote it). The specific function used as a loss function depends on the problem (ask another question if you want to know the details). For example, in the case of regression, the loss function may be the mean squared error. In classification, the loss function may be the cross-entropy. However, the most important thing to note is that the loss function depends on the parameters of the neural network (NN), so you can differentiate it with respect to the parameters of the NN (i.e. you can take the partial derivative of the loss function with respect to each of the parameters of the NN).
Let's take the example of the mean squared error function, which is defined as
$$
\mathcal{L}(\boldsymbol{\theta}) ={\frac {1}{n}}\sum _{i=1}^{n}(y_i-f_{\boldsymbol{\theta}}(x_i))^{2}.
$$
where $n$ is the number of training examples used (the batch size), $y_i$ is the true class (or target) of the input example $x_i$ and $f(x_i)$ is the prediction of the neural network $f_{\boldsymbol{\theta}}$ with parameters (or weights) $\boldsymbol{\theta} = [\theta_i, \dots, \theta_M] \in \mathbb{R}^M$, where $M$ is the number of parameters.
Given the loss function $\mathcal{L}(\boldsymbol{\theta})$, we can now take the derivative of $\mathcal{L}$, with respect to $\boldsymbol{\theta}$, using the famous back-propagation (BP) algorithm, which essentially applies the chain rule of calculus. The BP algorithm produces the gradient of the loss function $\mathcal{L}(\boldsymbol{\theta})$. The gradient can be denoted as $\nabla \mathcal{L}(\boldsymbol{\theta})$ and it contains the partial derivatives of $\mathcal{L}(\boldsymbol{\theta})$ with respect to each parameter $\theta_i$, that is, $\nabla \mathcal{L}(\boldsymbol{\theta}) = \left[ \frac{\partial \mathcal{L}(\boldsymbol{\theta})}{\partial \theta_i}, \dots, \frac{\partial \mathcal{L}(\boldsymbol{\theta})}{\partial \theta_M} \right] \in \mathbb{R}^M$. (If you want to know the details of the back-propagation algorithm, you should ask another question, but make sure you get informed first, because it may not be easy to fully explain it in an answer.)
Afterward, we just apply the gradient descent step
$$
\boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \gamma \nabla \mathcal{L}(\boldsymbol{\theta})
$$
where $\gamma \in \mathbb{R}$ is often called the learning rate and is used to weight the contribution of the gradient $\nabla \mathcal{L}(\boldsymbol{\theta})$ to the new values of the parameters, and $\leftarrow$ represents an assignment (like in programming).
It is worth emphasizing that both $\boldsymbol{\theta}$ and $\nabla \mathcal{L}(\boldsymbol{\theta})$ are vectors and have the same dimensionality ($M$).
Have also a look at this answer where I explain the difference between gradient descent and stochastic gradient descent.

The OP is not actually asking about the loss function, but about the target function. That's a difference in perspective, I don't see anything factually wrong in this answer, but I think it doesn't address the question as phrased (whether that is OP's intent of course is a different matter). The loss function measures the difference at a set of discrete points, between the target function and the current function that the NN produces. If you combine your and Recessive's answers you get the full picture, which may help the OP.

@NeilSlater My answer addresses this sentence "I know that gradient descent allows you to find the local minimum of a function. What I don't know is what exactly that function IS".

Ah, sorry, the OP actually asks about both and is confused. because they also say  at this point, I've no idea about what function should I search for. the function that is "searched for" is the target function. I think it would help the OP if you addressed the difference

@NeilSlater Don't worry. Maybe I should have clarified which part of the post I was addressing. I've just edited my answer to do that ;)

@NeilSlater Actually, this answer really helped me out xD. I was confused because I didn't even know what to look for, since I didn't understand the learning algorithm in general, but guess what, I just got a clear explanation about that! ;D
I figured out that the "missing piece" was the backpropagation, wich "connects" the loss function to the actual gradient.
Now, I only need to read this answer over, over and over so that it gets stuck in my head forever. Thanks :D

After I've learned a little bit more about the topic, I think I figured out the exact sequence of the algorithm. So, here's my own answer. Please, correct me if I'm wrong.

Give an input, forward-propagate it, and generate an output
For each output neuron:
  for each weight connected to the neuron:
      Given the function C = f(w) (which represents the cost in function to the weight value), calculate the derivative of that function at the point where  the current weight actually is)
Calculate the actual derivative of all the weights by combining all the partial derivatives in respect to the weight: now you have a gradient of the weights
Repeat this process to calculate the gradient for each of the batch elements. If you have a batch size of 8, then you'll have 8 gradients.
Find the average gradient ((gradient_1+gradient2+gradient3...)/n_gradients)
Move the weights of that gradient

Am I right? How does this apply to deeper layers?

@nbro I'm asking for your greatness :)

Thanks for the compliment. I also try to do my best :) Your answer seems roughly correct, but I am not sure why you distinguish the weights connected to the output neurons from the other weights. Anyway, if you have a batch of size 8 (or any other number), you will not compute the gradient with respect to each element of the batch, sum these gradients and then divide by 8.

Instead, you'll compute the loss with these 8 elements (see the loss function I defined in my answer), then compute a single gradient (which is often called a "stochastic estimate" of the actual gradient, where, by "actual gradient", I mean a gradient computed with all elements of the dataset). How do you compute it? You just take the derivative of the loss function (I defined in my answer) with respect to each parameter of the network. Try to take the derivative of that loss function with respect to one parameter of the network, for a simple network. You will understand better the process!

Then I suggest you try to implement a basic version of a simple feed-forward neural network (e.g. with 1 or 2 hidden layers), then implement a basic version of back-propagation and gradient descent to train this network. You will understand all the details after having implemented it. See these implementations https://github.com/mnielsen/neural-networks-and-deep-learning. I think they may be useful ;)


text:
Are there clever (fitness-based) crossover operators for binary chromosomes?
While studying genetic algorithms, I've come across different crossover operations used for binary chromosomes, such as the 1-point crossover, the uniform crossover, etc. These methods usually don't use any "intelligence".
I found methods like the fitness-based crossover and Boltzmann crossover, which use fitness value so that the child will be created from better parents with a better probability.
Is there any other similar method that uses fitness or any other way for an intelligent crossover for binary chromosomes?

It's not obvious what you mean by "intelligent crossover".
However, it is common to use fitness-based selection of parents: individuals in the current population who have higher fitness are assigned a higher probability of being selected to mate and produce offspring.  This will increase the likelihood that "good" combinations of genes in members of the current population will be passed along to the next generation, and that independent "good" combinations will be combined in some members of the next generation.
The "best" crossover operator depends dramatically on the structure of the problem being solved, and on the mapping of gene "vectors" to the salient features of a solution.
Edit #1:
In some cases it is important to increase diversity in order to avoid convergence to a local optimum.  In that case, an "intelligent" GA might for example select a first parent for its high fitness, and a second parent at random.  In "Generator", a GA I sold for a while about 25 years ago, mate selection worked that way, and it was often very effective.  Generator also replaced any duplicate individuals in the population with entirely random individuals.  I have also structured genetic algorithms specifically to evolve multiple separate populations of individuals, with minimum gene flow between the populations, in order to evolve multiple solutions corresponding to "regional" fitness optima.
Edit #2:
In genetic algorithms it is not common to directly seek the best combination of genes from both parents. The assumption is that higher-fitness parents are more likely to produce even higher-fitness offspring than lower-fitness parents.  Sometimes there is a local search (hill climbing) operation where the offspring of two parents is mutated in various ways and the best of the mutants is put in the next generation.
And, sometimes a crossover operation involves producing a larger number of offspring than the parent generation, followed by culling low-fitness individuals to keep the population size constant.  This is vaguely analogous to a local search via mutation, but uses random crossover instead of random mutation for its search.

thanks. So after choosing a parent with high fitness and a random parent, how we are supposed to apply crossover operator on them? I mean how we can find the best combination of both of these parents (other than "fitness based crossover" and "Boltzmann crossover")?

The idea behind it
I'll do an analogy: while classical GAs look like how humanity reproduced until now, "intelligent crossover" is more looking like designer babies. You first would need to identify which genes are responsible for certain behaviors and then you can bring them on to the new generation.
Said this, you will understand why there aren't so many algorithms around: it is a very case-tailored approach. Possibly each problem requires a different method and certainly will increase considerably the complexity of the crossover algorithms.

How to make your own
If you want to create one of these algorithms, you might want to add a step to your GA cycle: Evaluation, Identification, Selection, Crossover, Mutation, Replacement.
On the Identification step, you can run a probabilistic search on your population to discover which genes might be responsible for higher fitness results. Then you can:

proceed normally: Select two parents based on their fitness and perform crossover giving more importance to the "good" genes. This is the "safest" way.
fuse Selection and Crossover: use the data retrieved during the Identification step to create good individuals directly from the population pool. This system is dangerous as it might reduce the population diversity at alarming rates, leading to stagnation. This is how the Fitness-based Crossover are generally built  [paper]


text:
How does the weight update formula for logistic regression work?
I am trying to use Logistic Regression to make a spam filter, but I am having trouble understanding the weight update part. I have processed my email dataset, and I have an attribute vector of the top n words that are most likely to be contained within a spam.
From my understanding, during training, I will have to implement an optimization formula after each training example in order to update the weights.
$$
w_l \leftarrow w_l + \eta \cdot \sum_{i=1}^m [ y^{(i)} - P(c_+ \mid \vec{x}^{(i)} )] \cdot x_l^{(i)}
$$
How does a formula such as this work? How can it be implemented in Python?

I think i found out how that works, so i made a short article about it .
https://medium.com/@kourloskostas/python-spam-filter-86b21d7d1564
I hope it helps!

You are right ,will post the answer here as well

Actually, if you could excerpt the most salient passages from the medium article, that would really help with this answer, specifically.  (Welcome to SE:AI btw--glad to have you here!)

Thanks ! I will do my best to clear the article up!


text:
How can I compare EEG data with accelerometer data in 1 algorithm?
I have frequency EEG data from fall and non-fall events and I am trying to incorporate it with accelerometer data that was collected at the same time.  One approach is, of course, to use two separate algorithms and find the threshold for each.  Then comparing the threshold of each.  In other words, if the accelerometer algorithm predicts a fall (fall detected = 1) and the EEG algorithm detects a fall, based on the power spectrum (fall detected = 1), then the system outputs a "1" that a fall was truly detected.  This approach uses the idea of a simple AND gate between the two algorithms.
I would like to know how to correctly process the data so that I can feed both types of data into one algorithm, perhaps a CNN. Any advice is really appreciated, even a lead to some literature, articles or information would be great.

Hi and welcome to this community! Why do you want to compare the EEG and the accelerometer data? What your main goal or problem? What do you mean by compare? Furthermore, it may be useful if you describe more in detail your data. For example, the number of examples, etc.

@GeorgeWhite Thanks for trying to edit this post in order to improve it (specifically, to integrate the information given in the answer), but I've already done it (in case you didn't notice it), so I had to reject your edit. However, I really appreciate that you spent some time to improve this site :)

I must have been doing at about the same time. Thanks for straightening it out.


text:
Which deep learning models are suitable for image-to-image mapping?
I am working on a problem in which I need to train a neural network to map one or more input images to one or more output images (1 channel for image). Below I report some examples of input&output. In this case I report 1 input and 1 output image, but may need to pass to more inputs and outputs, maybe by encoding this in channels. However, the images are all of this kind, maybe rotated, tralated or changed a bit in shape. (fyi, they are fields defined by fluid dynamics simulations)
I was thinking about CNN, but the standard architecture used for image classification (convolutional layers + fully connected layers) seems not to be the best choice. Instead, I tried using the U-net architecture, composed of compression+decompression convolutional layers. This works quite fine, but maybe there is some other architecture that could be more suited to my problem.
Any suggestion would be appreciated!

did you figured out what worked well. I am kind of working on the similar problem. If will be great if you can share any ideas. Thank you

Something close to UNet has given the best performances

Since you have already tried U-Net. You may look into Siamese Networks (with CNNs for images), they are very well known for computing similarity via deep learning. This is a central idea and can be performed with both text and images. As a tip, you may be able to leverage a lot of architecture from U-Net to Siamese.
Hope it helps, Some useful links to start with :

https://medium.com/@prabhnoor0212/siamese-network-keras-31a3a8f37d04
https://www.aclweb.org/anthology/W16-1617.pdf

Hi, thank you for the tip. However I don't see how I could use the siamese architecture in my problem, I have only one image from which I should output another image, I am not comparing anything

It will be great if you can provide more information about your dataset. (train/test). Another suggestion could convert your U-NET to VAE (variational autoencoder). Using this you can generate more images given an image. Let me know if this helps I can edit the answer above.  Training setup can be of 2 types with VAE (1)unsupervised and (2)supervised. Usually, its known to be trained in unsupervised fashion but in your case supervised could be better. Again, this raises, why this is different from U-NET?- I think because you can control the variation. But we can observe results empirically.


text:
How is the DQN able to generalise the learning to unseen states with such a loss function?
I am trying to understand how deep Q learning (DQN) works. To my current understanding, each $Q(s, a)$ functions is estimated to be a function of a feature vector of its state $\phi$(s) and the weight of the network $\theta$.
The loss function to minimise is $||\delta_{t+1}||^2$ where $\delta_{t+1}$ is shown below. The loss function is from the website talking about function approximation. Even though it is not explicitly deep Q learning, the loss function to minimise is similar.
$$\delta_{\mathrm{t}+1}=\mathrm{R}_{\mathrm{t}+1}+\max _{\mathrm{a}\in\mathrm{A}} \boldsymbol{\theta}^{\top} \Phi\left(\mathrm{s}_{t+1}, \mathrm{a}\right)-\boldsymbol{\theta}^{\top} \Phi\left(\mathrm{s}_{\mathrm{t}}, \mathrm{a}\right)$$
Source: https://towardsdatascience.com/function-approximation-in-reinforcement-learning-85a4864d566.
Intuitively, I am not able to understand why the loss function is defined as such. Once the network converges to a $\theta$ using gradient descent, does that mean that the $Q_{max}(s,a)$ is found?
In essence, I am not able to grasp intuitively how the neural network is able to generalise the learning to unseen states.
The algorithm I am looking at to help me understand the deep Q networks is below.

Source: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

I've tried to rewrite the title so that it contains what I think is the actual question. Make sure that the title contains what was your actual question.

Especially in continuous space, convergence of the value function is mainly a theoretical property. Without seeing enough of the state space, as you suggest, there's no way to ensure that your Q function will generalize to the whole state space. Convergence results for Q learning with function approximation generally show that in the limit of infinite data, your value function will converge to the desired fixed point -- note that this is only true when your agent explores occasionally, for an infinite amount of time.
When your parameters have converged, this simply means that your Q function has fit the data you've collected. As you explore more, your agent may get "surprised" and your parameters may start to change again.
Also, convergence of the parameters in function approximation can never guarantee that an optimal value function was found in practice -- the only guarantee you can wish for is that the optimal value function that can be produced with your model has been found. For instance, the parameters of the linear Q function you posted can converge, even if the optimal Q function is not linear.

Well, you want your network to have a good prediction powers for the Q-values. So you compare Q-value at time t with the reward that you've got at time t after having executed action a + the prediction of the best Q-value of your neural network at time t+1. Note, that you are optimizing using a prediction and not a true value. That is called bootstrapping, look up TD-learning to have a better grasp of the concept.


text:
In deep learning, is it possible to use discontinuous activation functions?
In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)?
(My guess: for example, ReLU is non-differentiable at a single point, but it still has a well-defined derivative. If an activation function has a jump discontinuity, then its derivative is supposed to have a delta function at that point. However, the backpropagation process is incapable of considering that delta function, so the convex optimization process will have some problem?)

Even the first artificial neural network - Rosenblatt's perceptron [1] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, Michael Negnevitsky. Artificial intelligence: a guide to intelligent systems. Second Edition shows how to train such networks on pages 170-174.
Error backpropagation algorithm can be modified to accommodate discontinuous activation functions. The details are in paper [2]. That paper points out a possible application: training a neural network on micro-controllers. As the multiplication of the output of the previous layer $x_j$ by the weigth $w_{ij}$ is exspensive, the author suggested to approximate it with a left shift by $n$ bits (multiplication by $2^n$) for the corresponding $n$ in which case the activation function is discontinuous (a staircase).
An example of a neural network with discontinuous activation functions applied to a restoration of degraded images is in Ref. [3]. Applications of recurrent neural networks with discontinuous activation functions to convex optimization problems are in Ref. [4]. Probably more examples can be found in the literature.
References

Rosenblatt, F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev. 1958 Nov; 65(6):386-408. PMID: 13602029 DOI: 10.1037/h0042519
Findlay, D.A. Training networks with discontinuous activation functions. 1989 First IEE International Conference on Artificial Neural Networks, (Conf. Publ. No. 313), London, UK, 1989, pp. 361-363.
Ferreira, L. V.; Kaszkurewicz, E.; Bhaya, A. Image restoration using L1-norm regularization and a gradient-based neural network with discontinuous activation functions. 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, 2008, pp. 2512-2519.  DOI: 10.1109/IJCNN.2008.4634149
Liu, Q.; Wang, J. Recurrent Neural Networks with Discontinuous Activation Functions for Convex Optimization. Integration of Swarm Intelligence and Artificial Neural Network, pp. 95-119 (2011) DOI: 10.1142/9789814280150_0004

The problem is how do you find which way to adjust your function without some type of gradient information?

@FourierFlux This "some type of gradient" does not have to be continuous. Just estimate where the error is lower and modify the weights to move to that region. The gradient is only continuous in analytic formulas, anyway. It is discrete in a computer. Besides that, following the exact gradient is expensive, so approximations are used such as the stochastic gradient descent. This means that you don't have to know the exact gradient. Discrete approximations to derivatives will do.

I would say that it is possible, but probably not a very good idea.?Like you say, the hard requirement is that the network (and thus its components, including the activation functions) must be differentiable.  ReLU isn't, but you can cheat by defining f'(0) to be 0 (or 1).
A continuous function means that gradient descent leads to some local minimum¹, for piecewise continuous functions, it may not converge (i.e. the breakpoints themselves may not be part of the segment you descend, so you will never get get to an actual minimum).  This is not likely to be a problem in practice, though.
¹ At least for functions that are bounded from below, like cost functions are.

I think that this answer could be improved by citing any paper or, in general, a reliable source that discusses the topic (i.e. the usage of discontinuous activation functions in neural networks).

I'm not aware of any, off hand. Feel free to suggest edits!


text:
Understanding the nature of psychological defense system by artificial intelligence
Do you think psychological defense system, for example, repression, regression, reaction, formation, isolation, undoing, projection, introjection, sublimation, etc., could be created by artificial intelligence systems? Can AI also be used to better understand the psychological defense system?
If yes, what tools do we need? Maybe supervised learning algorithms, such as PSO or ANN, are better suited for these levels?
That doesn't seem to be that easy, and I think it needs to have a more general understanding of these algorithms. So I asked here.
On the other hand, what do you think is the appropriate workspace or workspace available for this job?
For example, I think the interactions between robots that use different levels of the defense system by our selection as a game theory round play, could be good and suited to the community which is equal to the psychological connection of this part of us and as a testing environment.
But the robots themselves are dealing with a real problem that makes it even more difficult, som according to this approach, by selecting a more nonlinear problem to solve we can count the using the levels of these defense mechanisms (for example, in simple problem 30% 1th level, 20% 2th level and ...).


text:
How do I determine the best neural network architecture for a problem with 3 inputs and 12 outputs?
This post continues the topic in the following post:
 Is it possible to train a neural network with 3 inputs and 12 outputs?.
I conducted several experiments in MATLAB and selected those neural networks that best approximate the data.
Here is a list of them:

Cascade-forward backpropagation
Elman backpropagation
Generalized regression
Radial basis (exact fit)

I did not notice a fundamental difference in quality, except for Elman's backpropagation, which had a higher error than the rest.
How to justify the choice of the structure of the neural network in this case?


text:
What is the difference between graph semi-supervised learning and normal semi-supervised learning?
Whenever I look for papers involving semi-supervised learning, I always find some that talk about graph semi-supervised learning (e.g. A Unified Framework for Data Poisoning Attack to Graph-based Semi-supervised Learning).
What is the difference between graph semi-supervised learning and normal semi-supervised learning?

The authors of your cited paper use the term graph-based semi-supervised learning (G-SSL) to refer to semi-supervised learning techniques which take graph structured data as their input.
Given their main example, the MNIST dataset, is not graph structured, they detail a method for converting the raw Euclidean data $X$ into said form (represented by its adjacency matrix $S$), and then compute the Laplacian $L$ of this graph:

We consider the graph-based semi-supervised learning (G-SSL) problem. The input include labeled data $X_{l} ∈ \mathbb{R}^{n_{l}×d}$ and unlabeled data $X_{u} \in \mathbb{R}^{n_{u}×d}$, we define the whole features $X = [X_{l}; X_{u}]$. Denoting the labels of $X_{l}$ as $y_{l}$, our goal is to predict the labels of test data $y_{u}$. The learner applies algorithm $A$ to predict $y_{u}$ from available data $\{X_{l}, y_{l}, X_{u}\}$. Here we restrict $A$ to label propagation method, where we first generate a graph with adjacency matrix $S$ from Gaussian kernel:
  $S_{ij} = \exp(−γ\lVert x_i − x_j\rVert ^{2})$, where the subscripts $x_{i(j)}$ represents the $i(j)$-th row of $X$. Then the graph Laplacian is calculated by $L = D − S$, where $D = \text{diag}\{\sum_{k=1}^{n} S_{ik}\}$ is the degree matrix.

A Unified Framework for Data Poisoning Attack to
Graph-based Semi-supervised Learning (2019)

This is consistent with the terminology as used in other literature:

Semi-supervised learning for node-level classification.
  Given a single network with partial nodes being labeled and others remaining unlabeled, ConvGNNs can learn a robust model that effectively identifies the class labels for the unlabeled nodes [22]. To this end, an end-to-end framework can be built by stacking a couple of graph convolutional layers followed by a softmax layer for multi-class classification.

A Comprehensive Survey on Graph Neural Networks (2019)


text:
How can I perform lane detection with reinforcement learning?
I'm quite new to reinforcement learning and my project will consist of detecting lanes with RL.
I'm using q-learning and I'm having a hard time thinking how my q table should look like, I mean - what could represent a state. My main idea is to feed the machine with a frame that contains a road picture, which the edge detection function is being applied to (and by thus getting lots of lines that exits in the frame). And train the machine which lines are the correct lane line. I already have a deterministic function that already recognizes the lanes and it will be the function that will teach the machine. I already organized some lane parameters such as (lane length, lane cords, lane color (white or yellow have a better probability to be a lane), lane diameter and the lane incline).
Now, my only issue is how should I construct the Q-table. Basically, what could represent a state and which lanes or decisions I should reward.

I'm not sure RL suits for this task, why don't you use computervision algorithms?

RL has been used for lane keeping, which may require lane detection. Maybe have a look at the literature.

A Q-table is probably not the right tool for this problem. I would look at other RL algorithms at Spinning Up.

I will agree with malioboro, maybe RL is an overkill for such a task. Even though with the trend of autonomous driving research lately, papers dealing with lane changing almost certainly exist, you should check them out for more details.
As stated in Lane Change Decision-making through DeepReinforcement Learning with Rule-basedConstraints , "When the states are discrete and finite, the Q-function canbe easily formulated in a tabular form. But in many practical applications, for example, lane change decision-making task,the state space of them is very large or even continuous,using the Q-learning algorithm will lead to dimension disaster. Therefore, tabular Q-learning algorithm does not applicable to
the learning problem of continuous state space and continuousaction space"
So, i would recommend using image processing techniques, or Deep-Q-Learning. Q-Learning is not capable of dealing with continuous problems like lane changing or lane tracking.


text:
Is it a good idea to first train a spiking neural network and then convert it to a conventional neural network?
In many papers about artificial spiking neural networks (SNNs), the performance of them is not up to par with traditional ANNs. I have read how some people have converted ANNs to SNNs using various techniques.
There has been work done on using unsupervised learning in SNN to recognise MNIST digits through spike-timing-dependent plasticity (for example, the paper Unsupervised learning of digit recognition using spike-timing-dependent plasticity, by Diehl and Cook, 2015). This form of learning is not possible in traditional ANNs due to their synchronous nature.
I was wondering would it be a good idea to first train an SNN in an unsupervised manner to learn some of the structure of the data. Then convert to a traditional ANN to take advantage of their superior performance with some more training. I can see this being useful for training a network on a sparsely labelled dataset.
I am quite a novice in this area, so I was looking for feedback on any immediate barriers as to why this would not work or if it is even worth doing.


text:
Is a neural network able to optimize itself for speed?
I am experimenting with OpenAI Gym and reinforcement learning. As far as I understood, the environment is waiting for the agent to make a decision, so it's a sequential operation like this:
decision = agent.decide(state)
state, reward, done = environment.act(decision)
agent.train(state, reward)

Doing it in this sequential way, the Markov property is fulfilled: the new state is a result of the old state and the action. However, a lot of games will not wait for the player to make a decision. The game will continue to run and perhaps, the action comes too late.
Has it been observed or is it even possible that a neuronal network adjusts its weights so that the PC computes the result faster and thus makes the "better" decision? E.g. one AI beats another because it is faster.
Before posting an answer like "there are always the same amount of calculations, so it's impossible", please be aware that there is caching (1st level cache versus RAM), branch prediction and maybe other stuff.

It's important to note that neural networks are usually trained with gradient descent or other learning algorithms (which are usually iterative/numerical algorithms that use data to find some target function, which will hopefully be represented by the neural network). So, I am not sure if it's a good idea to talk about "self-optimization" (in the title).

Yes and No.
Yes, it is possible to achieve that result. But instead of a self-optimizing Neural Network, I'd recommend another approach:
1. Don't let training time interfere.
If you are trying to train the agent DURING the environment runtime, that's probably the problem.
Training time is usually much bigger than evaluation time. And deployed models usually don't train, so it won't be a problem in production.
You can do 2 things about that:
1.1 "Pause" the game during training.
It might look like "cheating", but in your agent's point of view, they are not actually playing during train time. And once again, it's just simulating how it would behave in production.
But if you can't pause it:
1.2 Disable training during runtime.
Store all states and decisions. Wait until the game is over and then you train the whole batch.

Professional chess players don't try to learn during a blitz challenge. But they do study their own games later to learn from their mistakes.

2. Optimizing hyper-parameters for speed.
You could tweak some hyper-parameters (like the size of your NN) looking for a faster model. Keep in mind it would still be an algorithm that always runs in a fixed time, but you might find some way to make it always
2.1 Using Machine Learning for this Optimization
There are some meta-learning techniques and other methods like NEAT, that can automate your search for a simple effective topology. NEAT already rewards the simplest architectures, penalizing complexity (which is usually attached to speed), but you could also force it to consider running time specifically.
3. Another Network for Another Task
You could a make another short network for deciding if the next move should be accurate or fast. Based on this result, it will choose between precision or speed. This choice could be by flagging a parameter (like branch prediction) or even running a whole new algorithm:
NeedForSpeed = TimeEstimation(state)
#Sorry, I couldn't resist the pun!

if NeedForSpeed > 0.8
  decision = agent.instantReactionDecisionTree(state)
elif NeedForSpeed > 0.5
  decision = agent.decideStandard(state, branchPrediction=True )
elif NeedForSpeed > 0.2
  decision = agent.decideStandard(state)
else:
  decision = agent.DeepNN(state)

Bonus: Use Other ML algorithms
Some algorithms have explicit parameters that directly affect the tradeoff for time x precision.
For instance, MCST (Monte Carlos Search Tree) can keep running forever until it explores all possibilities, but it usually finishes before, giving you the best solution found so far.
So, one possibility would be trying some other method instead of Neural Networks.


text:
In machine learning, how can we overcome the restrictive nature of conjunctive space?
In machine learning, problem space can be represented through concept space, instance space version space and hypothesis space. These problem spaces used the conjunctive space and are very restrictive one and also in the above-mentioned representations of problem spaces, it is not sure that the true concept lies within conjunctive space.
So, let's say, if we have a bigger search space and want to overcome the restrictive nature of conjunctive space, then how can we represent our problem space? Secondly, in a given scenario which algorithm is used for our problem space to represent the learning problem?

Hi and welcome to AI SE! I've heard of concept space and hypothesis space, but I've never heard of version space. Can you please clarify what you mean by that? Also, what do you mean by "conjunctive space"? To make sure, are you talking about computational learning theory, right? This post is a bit unclear to me. Please, provide links to the resources that mention these concepts/terms!

@nbro These terms probably come from Tom Mitchell's "Machine Learning" book (that's the title, just "Machine Learning"). The version space with respect to a hypothesis space + a set of training examples is defined as the subset of hypotheses that are consistent with all the training examples. "Conjunctive space" is not used as exactly that phrase as far as I'm aware, but the OP probably refers to the fact that all hypotheses (as considered in these early chapters of the book) are conjunctions of predicates (e.g. "positive IF $X_1 = x$ AND $X_2 = y$ AND $\dots$")

@DennisSoemers Ok, thanks for clarifying this. I was aware of this book, but I haven't yet had the opportunity to read it.


text:
Can a neural network learn to predict a number given a binarized image of a rectangle?
Let's assume that we have a regression problem. Our input is just binarized image that contains a single rectangle and we want to predict just a float number.  Actually, this floating-point number depends on rectangle angle, rectangle size and rectangle location. Is this problem can be solved by a neural network?
I think, it can not be solved by a neural network, because rectangle angle, size and location are latent variables and without learning these latent variable, above problem can not be solved. What do you think?

It can definitely be learned, the question is the approach. It would be expensive and difficult from a modeling directive to do this Densely, so usually convolutions are the way to go. An issue with convolutions is that is generally focuses on equivariant and relative features, so if you need specific location within the approach might be worth the simple alteration of CoordConv. Regardless of approach, that type of input to output is possible, you just have to consider it when modeling.


text:
How can I formulate a fuzzy inference system to approximate the tangent function?
Consider the function $f(x)=\tan(2x)$. How can I determine a fuzzy system that approximates $f(x)$? How to choose membership functions and how to determine fuzzy rules? Any help would be appreciated.


text:
What are advantages of using meta-heuristic algorithms on optimization problems?
What are the advantages and disadvantages of using meta-heuristic algorithms on optimization problems? Simply, why do we use meta-heuristic algorithms, like PSO, over traditional mathematical techniques, such as linear, non-linear and dynamic programming?
I actually have a good understanding of meta-heuristic algorithms and I know how they work. For example, one advantage of this kind of algorithms is that they can find an optimal solution in a reasonable time.
However, my lack of knowledge about other methods and techniques brought this question to my mind.

Meta-heuristics are particularly suited for combinatorial optimization problems, given that, although they are not usually guaranteed to find the optimal global solution, they can often find a sufficiently good solution in a decent amount of time. So, they are an alternative to exhaustive search, which would take exponential time. For example, ant colony optimization algorithms have been used to approximately (or exactly, in the case of small or medium-size instances) solve the travelling salesman problem, whose decision version is an NP-complete problem (which means that, unless P=NP, there is no polynomial-time solution to solve it).
Meta-heuristics can also be easily applied to many problems, given that they are not problem-specific. For example, in the case of genetic algorithms, you just need to encode the possible solutions, but, in principle, you can apply genetic algorithms to a wide range of problems, although they may not always be the best solution to each of these problems. Moreover, as opposed to gradient-based optimisation algorithms, there's no need for the gradient of the objective function. For instance, in the case of genetic algorithms, you just need a way of evaluating the solutions (e.g. the fitness or the novelty).
Meta-heuristics often incorporate some form of randomness in order to escape from local minima. Ant-colony optimization algorithms or simulated annealing are two good examples of this approach.
If you are still interested in meta-heuristics, the book Clever Algorithms: Nature-Inspired Programming Recipes (by Jason Brownlee) is a very good resource for learning about them. There's also a Github repository with the implementation of the algorithms described in this book.

I was already preparing this answer before the last edit to the question, which indicates that the OP is already aware of an advantage of meta-heuristics that I mention in this answer.


text:
Why does the BatchNormalization layer produce different outputs during training and inference?
I modified resnet50 architecture to get a regression network. I just add batchnorm1d and ReLU layers just before the fully connected layer. During the training, the output of batchnorm1d layer is nearly equal to 3 and this gives good results for training. However, during inference, output of batchnorm1d layer is about 30 so this leads to too low accuracy for test results. In other words, outputs of batchnorm1d layers give very different normalized output during training and inference.
What is the reason for this situation, and how can I solve it? I am using PyTorch.

What's your batch size? BN is known to work poorly with small batch sizes.

batch size is 16

are you confident you are applying the same pre-processing in training and in testing?

yes, ı used exactly same data for training and testing. The only difference between them is batch size.

did you shuffle your batches properly? meaning samples of the same class aren't always in the same mini-batch


text:
Did people analyze dynamics of very simple LSTMs?
I wonder if researchers tried to understand how LSTMs work by analyzing the dynamics of simple LSTM (e.g. with 2 units)? For example, how the hidden state evolves depending on the properties of weight matrices.
It seems like a very natural thing to try (especially because it is easy to draw hidden states with 2 units on a 2D plane). However, I haven't found any papers that would play with such a toy example.
Are there such papers at all?
Or with such a simple example, it is impossible to get some understanding because of its over-simplicity? (which I doubt because even logistic maps generate very complicated behavior)

Yes they have: https://youtu.be/jvj4WefJ0S0?list=PLiJYYsK6pD-nVcROKw2YDOVaNMjST5YyL&t=1711

authors: Κώστας Μιράκας