text:
How come that the addition of features can decrease the performance of a neural network?
I have a Remaining Useful Life (RUL) prediction problem that I want to solve. When I added two or more features as inputs to my ANN, the accuracy of my ANN has been decreased. More precisely, I've added features like RMS or KURTOSIS (or both). I was expecting the system to improve, but it is getting worse.
Why might this be happening? What are the potential reasons for this degradation in performance?
I know that when we added more nodes in layers (like hidden layers), overfitting can happen. Would that be related to my problem: using more than two features?

A simple case is if the new features noisy.

Additional features can also cause overfitting if they have low or misleading information.
Consider the following problem:
$X = [1, 3, 3, 4, 5]$, $Y = [1, 3, 4, 4, 5]$.
Suppose that the real dataset was generated from the relationship:
$Y = X$, with a probability of 0.2 of adding or subtracting 1.
A reasonable model estimate is $Y = X$. Note that no model can fit this data perfectly, because the two 3 inputs map to different outputs.
Now, suppose we add a new feature: a random number between $0$ and $10$:
$W = [1 ,5, 2, 6, 3]$
It may not be obvious, but a sufficiently deep and broad neural network can learn a new function:
$g(W) = 1$ if $W = 2,4,7,8,9,$ or $0$.
$g(W) = 0$ otherwise.
and define a new prediction: $Y = X - g(W)$.
This happens to produce a perfect fit on the training data. However, it will perform extremely poorly on new data (like a test set), because it has learned a meaningless pattern out of random noise. Coincidentally, it will be wrong on about 50% of samples, while our first model will be wrong on only 20% of samples.

Maybe you mean Y = [1, 2, 3, 4, 5]?

@nbro No, I intended for there to be some error in the original set, but on second look, I need to add some extra data for this example to make sense.

authors: جاسم البهادلي


text:
What is the fundamental difference between CNN and RNN?
What is the fundamental difference between convolutional neural networks and recurrent neural networks? Where are they applied?

Better do not think about RNN/CNN as different networks, but as different network capabilities: a network can be stateless or stateful ( as RNN, LSTM, deep ); a network can/cannot have spatial operators (as 2D convolution, like CNN); ...

Recurrent neural networks (RNNs) are artificial neural networks (ANNs) that have one or more recurrent (or cyclic) connections, as opposed to just having feed-forward connections, like a feed-forward neural network (FFNN).
These cyclic connections are used to keep track of temporal relations or dependencies between the elements of a sequence. Hence, RNNs are suited for sequence prediction or related tasks.
In the picture below, you can observe an RNN on the left (that contains only one hidden unit) that is equivalent to the RNN on the right, which is its "unfolded" version. For example, we can observe that $\bf h_1$ (the hidden unit at time step $t=1$) receives both an input $\bf x_1$ and the value of the hidden unit at the previous time step, that is, $\bf h_0$.

The cyclic connections (or the weights of the cyclic edges), like the feed-forward connections, are learned using an optimisation algorithm (like gradient descent) often combined with back-propagation (which is used to compute the gradient of the loss function).
Convolutional neural networks (CNNs) are ANNs that perform one or more convolution (or cross-correlation) operations (often followed by a down-sampling operation).
The convolution is an operation that takes two functions, $\bf f$ and $\bf h$, as input and produces a third function, $\bf g = f \circledast h$, where the symbol $\circledast$ denotes the convolution operation. In the context of CNNs, the input function $\bf f$ can e.g. be an image (which can be thought of as a function from 2D coordinates to RGB or grayscale values). The other function $\bf h$ is called the "kernel" (or filter), which can be thought of as (small and square) matrix (which contains the output of the function $\bf
 h$). $\bf f$ can also be thought of as a (big) matrix (which contains, for each cell, e.g. its grayscale value).
In the context of CNNs, the convolution operation can be thought of as dot product between the kernel $\bf h$ (a matrix) and several parts of the input (a matrix).
In the picture below, we perform an element-wise multiplication between the kernel $\bf h$ and part of the input $\bf h$, then we sum the elements of the resulting matrix, and that is the value of the convolution operation for that specific part of the input.

To be more concrete, in the picture above, we are performing the following operation
\begin{align}
\sum_{ij}
\left(
\begin{bmatrix}
1 & 0 & 0\\
1 & 1 & 0\\
1 & 1 & 1
\end{bmatrix}
\otimes
\begin{bmatrix}
1 & 0 & 1\\
0 & 1 & 0\\
1 & 0 & 1
\end{bmatrix}
\right)
=
\sum_{ij}
\begin{bmatrix}
1 & 0 & 0\\
0 & 1 & 0\\
1 & 0 & 1
\end{bmatrix}
= 4
\end{align}
where $\otimes$ is the element-wise multiplication and the summation $\sum_{ij}$ is over all rows $i$ and columns $j$ (of the matrices).
To compute all elements of $\bf g$, we can think of the kernel $\bf h$ as being slided over the matrix $\bf f$.
In general, the kernel function $\bf h$ can be fixed. However, in the context of CNNs, the kernel $\bf h$ represents the learnable parameters of the CNN: in other words, during the training procedure (using e.g. gradient descent and back-propagation), this kernel $\bf h$ (which thus can be thought of as a matrix of weights) changes.
In the context of CNNs, there is often more than one kernel: in other words, it is often the case that a sequence of kernels $\bf h_1, h_2, \dots, h_k$ is applied to $\bf f$ to produce a sequence of convolutions $\bf g_1, g_2, \dots, g_k$. Each kernel $\bf h_i$ is used to "detect different features of the input", so these kernels are different from each other.
A down-sampling operation is an operation that reduces the input size while attempting to maintain as much information as possible. For example, if the input size is a $2 \times 2$ matrix $\bf f = \begin{bmatrix} 1 & 2 \\ 3 & 0 \end{bmatrix}$, a common down-sampling operation is called the max-pooling, which, in the case of $\bf f$, returns $3$ (the maximum element of $\bf f$).
CNNs are particularly suited to deal with high-dimensional inputs (e.g. images), because, compared to FFNNs, they use a smaller number of learnable parameters (which, in the context of CNNs, are the kernels). So, they are often used to e.g. classify images.
What is the fundamental difference between RNNs and CNNs? RNNs have recurrent connections while CNNs do not necessarily have them. The fundamental operation of a CNN is the convolution operation, which is not present in a standard RNN.

Basically, a CNN saves a set of weights and applies them spatially. For example, in a layer, I could have 32 sets of weights (also called feature maps). Each set of weights is a 3x3 block, meaning I have 3x3x32=288 weights for that layer. If you gave me an input image, for each 3x3 map, I slide it across all the pixels in the image, multiplying the regions together. I repeat this for all 32 feature maps, and pass the outputs on. So, I am learning a few weights that I can apply at a lot of locations.
For an RNN, it is a set of weights applied temporally (through time). An input comes in, and is multiplied by the weight. The networks saves an internal state and puts out some sort of output. Then, the next piece of data comes in, and is multiplied by the weight. However, the internal state that was created from the last piece of data also comes in and is multiplied by a different weight. Those are added and the output comes from an activation applied to the sum, times another weight. The internal state is updated, and the process repeats.
CNN's work really well for computer vision. At the low levels, you often want to find things like vertical and horizontal lines. Those kinds of things are going to be all over the images, so it makes sense to have weights that you can apply anywhere in the images.
RNN's are really good for natural language processing. You can imagine that the next word in a sentence will be highly influenced by the ones that came before it, so it makes sense to carry that internal state forward and have a small set of weights that can apply to any input.
However, there are many more applications. In addition, CNN's have performed well on NLP tasks. There are also more advanced versions of RNN's called LSTM's that you could check out.
For an explanation of CNN's, go to the Stanford CS231n course. Especially check out lecture 5. There are full class videos on YouTube.
For an explanation of RNN's, go here.

IMHO, this is a quite confusing explanation.

CNN vs RNN

A CNN will learn to recognize patterns across space while RNN is useful for solving temporal data problems.
CNNs have become the go-to method for solving any image data challenge while RNN is used for ideal for text and speech analysis.
In a very general way, a CNN will learn to recognize components of an image (e.g., lines, curves, etc.) and then learn to combine these components to recognize larger structures (e.g., faces, objects, etc.) while an RNN will similarly learn to recognize patterns across time. So a RNN that is trained to convert speech to text should learn first the low level features like characters, then higher level features like phonemes and then word detection in audio clip.

CNN
A  convolutional network (ConvNet) is made up of layers.
In a convolutional network (ConvNet), there are basically three types of layers:

Convolution layer
Pooling layer
Fully connected layer

Of these, the convolution layer applies convolution operation on the input 3D tensor. Different filters extract different kinds of features from an image. The below GIF illustrates this point really well:

Here the filter is the green 3x3 matrix while the image is the blue 7x7 matrix.
Many such layers passes through filters in CNN to give an output layer that can again be a NN Fully connected layer or a 3D tensor.

For example, in the above example, the input image passes through convolutional layer, then pooling layer, then convolutional layer, pooling layer, then the 3D tensor is flattened like a Neural Network 1D layer, then passed to a fully connected layer and finally a softmax layer. This makes a CNN.

RNN
Recurrent Neural Network(RNN) are a type of Neural Network where the output from previous step are fed as input to the current step.

Here, $x_{t-1}$, , $x_{t}$ and $x_{t+1}$  are the values of inputs data that occur at a specific time steps and are fed into the RNN that goes through the hidden layers namely $h_{t-1}$, , $h_{t}$ and $h_{t+1}$ which further produces output $o_{t-1}$, , $o_{t}$ and $o_{t+1}$ respectively.

In the case of applying both to natural language, CNN's are good at extracting local and position-invariant features but it does not capture long range semantic dependencies. It just consider local key-phrasses.
So when the result is determined by the entire sentence or a long-range semantic dependency CNN is not effective as shown in this paper where the authors compared both architechrures on NLP takss.
This can be extended for general case.

On a basic level, an RNN is a neural network whose next state depends on its past state(s), while a CNN is a neural network that does dimensionality reduction (make large data smaller while preserving information) via convolution. See this for more info on convolutions

This is misleading. CNNs are not strictly used for dimensionality reduction. Furthermore, it is the down-sampling operation that reduces the dimensions of the input (not necessarily the convolution).

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
Why are neural networks always trained "by themselves"?
In the current rush of artificial intelligence research, fueled by NN, independent of the paper I choose, the NN are always trained by themselves. Sure, there are architectures that combine CNN and RNN or LSTMs in a way that can help to solve multiple problems interacting (like labeling images with human readable text snippets), but independent of this, they always learn by themselves.

Supervised networks just get a bunch of examples to learn from.
Reinforcement algorithms run around virtual spaces falling over hundreds of times before learning to walk properly.

This might sound silly but no one helps them and no one plays with them. Children, researchers, puppies, dolphins, ... all intelligent beings we are aware of interact with each other. There are many forms of learning, one very important one being social learning which includes observational learning. We imitate, copy and learn from others all the time. There is also a new idea of looking at knowledge that suggests that we all aren't as independent as our culture might have taught us to think we are. That knowledge really lies in the connections between individuals instead of inside of each agent itself, just like the connections of neurons are what makes the brain work, not the neurons themselves.
Couldn't such interaction between learning agents be enormously valuable? If you throw 5 agents in a room (NN a Knowledge Based Agent a simple pre-programmed one etc) and let them learn from each other through observation and inspiration, what effects could one expect?
I am not talking about ensemble learning as this just throws several hypotheses together and takes the average over all results. I am talking about a complex way of agents interacting with each other during learning.
Is there any research on this idea? If so, what were the results? This is the only one I found:

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

The way children learn is in many ways supervised. It is true that certain abilities are there genetically (visual system, object recognition, to large extent voice recognition), but a lot of human experience is gained as a result of response, either from the environment or from the mentor (parent, teacher, etc).
Social interaction is possible only when a human already learned a lot (things we consider trivial, but in fact pretty complex), in this sense this is equivalent to AlphaGo learning to play Go by playing with a copy network. NNs that can only perform object or voice recognition can't interact, they are too primitive for this. And a nearly all of todays AI is on this stage right now.
However, it is true that NN learning is much less efficient than human learning and current models are far from AGI. I think RL is the closest area that studies real multi-agent environments.

What about situations in which someone is „stuck“ in a way? With reinforcement learning (I believe this would be the one benefitting the most anyways), there are often situations where iterations get wasted because the algorithm gets stuck in a certain local maxima. If several agents learned separately but then had phases of observations to copy good ideas, these local maxima can quickly be overcome.

Not at all. Basic difference is that "supervised learning  "would mean somebody intervenes human brain and trains neuron with electrick sygnals. Children learn unsupervised - they just get data from "sensors"  and create abstractions from it without understanding, or whatever else, what they need to do

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is it expected that during self-play reinforcement learning that player 1 or player 2 wins the majority of games?
I'm testing various learning rates and neural network configurations. I'm testing over 10000 games, with the first 2000 having random starting moves and general randomness throughout of about 20%, i.e. 20% of moves are random.
In all configurations, I initiate the weights to random values.
What I've found is that in all configurations, Player 1 will win the majority of games, or Player 2 will. There's no 50/50 split.
Is this expected or normal? I am training on the Connect Four game.

This is weird, how big are the majorities? I would also decrease your epsilon (probability of random move) as training progresses. Do you test with an epsilon?

@jaden What are "the majorities", I'm not familiar with this term. I do reduce the randomness over time, it starts at something like 50% and drops to 10% after the first 10% of moves. 20% was way off.

By majorities i mean by how much does player 1 beat player 2 (eg 90%) and vice versa.

It's hard to tell, it's a win or lose, there's no score or anything.

Sorry, I meant in regards to what you said about there is no 50/50 split. What is the split? Like 90/10?

@Jaden, yeah it's like 90/10 but starts to go up the more i increase the amount of random moves, but only because those moves ending randomly are getting wins for the other player. It's  also sometimes player 1 or sometimes player 2, I believe related to what the random weights maybe lean to at the start.

What Reinforcement Learning algorithm are you using? Something like Q-learning, or Policy Gradients? Can you also elaborate on how the "self-play" is set up? Does each player have a Network of their own (and if so, are they using same or different configs), or are they both using and learning parameters for the same network at the same time? I realise this request for more info is a bit late, hopefully you're still around :)

@dennis_soemers I am always around. It was using backward propagation. They were using the same network. The game was Connect4 so I figured they didn't need their own networks as the state given is always the same, just swapped out the "player" vs "opponent" values in the input.

From wikipedia article on  Connect Four:

Connect Four is a solved game. The first player can always win by
  playing the right moves.

It's pretty reasonable that even without training 1st player win more often by small margin.
With some training  1st player should mostly win.


text:
How do I create an AI for a two-players board game?
Goal
I want to create an artificial intelligence to compete against other players in a board game.
Game explanation
I have a board game similar to 'snakes and ladders'. You have to get to a final field before your opponent does. But instead of depending on luck (throwing the dices) this game uses something like 'food'. You can go as far as you'd like, but it costs food to move (the more you move the more one extra field costs) and you can only get food in some special fields. And there aren't any snakes or ladders so you have to run the whole part. There are some more rules, for example, you can go backward and are only allowed to go into the goal if you've got less than some amount of 'food' and there are some extra fields with other special effects.
For one player
If there was only one player as there isn't anything like 'luck' in this game, I theoretically could just compute every single method to find the one and only the best method. Practically, I should use an algorithm that requires less computational power.
For two or more players
The challenge comes with the other player(s). I cannot visit an already taken field. And some other fields give me bonuses depending on my relative position to the other player (I'll just talk about two-player games). For example, only if I'm behind him that special field gives me some extra food.
My question
It would be ideal if I had some kind of a neural network that knows the field bonuses and I would give my position, the opponents position, the food and so on (the state of the game) and it would compute a value between -100 and 100 (assuming fields from 0 to 100) of how many fields I should go (forward or backward).
I read a bit about Q-learning, deep reinforcement learning and deep neural networks. Is this the right approach to solving my problem? And if yes, have you got any more concrete ideas? The multiple actors and the sheer endless possibilities for moving depending on endless states make it hard for me to think of anything. Or is there a different, way better way that slipped past me?

Welcome to AI!  Excellent question.  One point I'll make, because this appears to be deterministic game, is that you could take a classical approach, and see if you can "solve" the game to identify "perfect play".  My question is: what is your intent?  Are you looking to build the strongest AI possible?  If so, Deep Learning would definitely be the way to go, barring solution of the game.  (On my own project, we're currently taking a modular, classical approach, to produce human-like play that can strengthen or weaken, depending on the skill of the human opponent.)

Assuming it is a turn-based game and, for each turn, there's an optimal choice that will lead to the winning state (zero-sum), you can basically simplify the question to "What is the optimal sequences of moves for me to win, considering the current situation that is presented on the board?". So you will need to perform your algorithm every turn as the optimal sequence will change when the board changed.
There's a relatively reliable AI algorithm that's being implemented to win games like chess, backgammon, etc. The technique is called minimax. In summary, minimax is simply a search algorithm to minimize the other player's score, while maximizing your score.
One of the problems you will encounter is that the search tree becomes so wide as we search through deeper parts of the tree, so it is essential to also implement alpha-beta pruning to cut down the amount of search. In general, alpha-beta pruning is simply eliminating branches that are very unlikely to be the optimal choice (in terms of score) thus reducing the number of searches to speed up the algorithm.

Looks like your answer bore fruit re: https://ai.stackexchange.com/questions/4975/how-do-i-write-a-good-evaluation-function   Thanks for contributing!

To make an AI opponent, you'll need to create a sub-routine that considers the current state of the board and chooses a move, just like the player would.
Now, how does this subroutine choose what move to make? You need to take the current board and calculate its value. Then consider every possible move you could make. Then consider, for each of those, every possible move your opponent could make. Iterate to maximum depth. You will have constructed a tree structure (breadth first).
Prune your tree. Any branch that guarantees you a decrease in board value can be pruned.
Then, somehow, compare remaining branches. Either you optimistically weigh possible best outcomes and choose those branches. Or continue pruning based on potential of worst outcomes.

would love to get your thoughts on mbrane We're going classical for the time being, focusing on human-like play and variation as opposed to sheer strength.


text:
Why is the access to the dynamics model unrealistic in Q-Learning?
Pieter Abbeel says that having access to the dynamics model, $P(s' \mid s,a)$, is unrealistic because it assumes we know the probability that we will reach all future states.
I don't understand how this is unreasonable? Could someone explain this to me like in a simple way?

It is an unreasonable assumption for many scenarios to know the full transition model of an environment.
In the deterministic toy scenarios used for explaining RL, such as a grid world, it seems like a trivial requirement. However, it starts to get harder very quickly as problems and state features get more complex.
Even in relatively simple environments that are possible to model in this way, the calculations start to become difficult enough that there is an argument to not bother. For instance, a common toy example is the game of Blackjack - in order to calculate the probability of different rewards after sticking you would need to resolve the remaining game tree of typically 0,1,2, but up to more, for the dealer, accounting for their starting card, and perhaps accounting for which cards have been drawn so far (if you are simulating this more realistically, without replacement).
However, that could be resolved if we wanted. To make things harder, you need to consider agents that are set to solve problems where we don't have any meaningful probabilistic model of how the state will progress.
A good example would be any real world agent that used a vision system as part of its state. Consider a flying drone set with the task of locating and tracking migrating whales over some area of the ocean. It is rewarded for capturing footage of whales. Its state is the current image plus GPS and telemetrics data.
Knowing $P(s' \mid s,a)$ in the whale-tracking example implies that you would know the probability distribution of the next video frame, somehow accounting for the chaotic movement of waves, the likely behaviour of any creatures in shot, and effects of wind turbulence on drone position etc. Even the most sophisticated physics simulations with ray-tracing rendering engines cannot rise to that challenge with much accuracy, and the computation required even for a rough render of expected image(s) based on a physics engine would be far beyond anything you could run in real time on the drone. However, it is still possible to build an agent which learns how to solve this kind of environment through gathering statistics on its task using images as state data.
Scaling that challenge back a bit, you can view many of the Atari game-playing challenges like this. The input state is a collection of still frames from the game. Knowing $P(s' \mid s,a)$ implies knowing the probability distribution of next set of frame images, given the current inputs. This is a near impossible challenge, even if you know the code for the game, providing an analytical result for $P(s' \mid s,a)$ is unrealistic - the best way to do it would be to have a memory clone of the game that you could use to look ahead then reset. Whilst this is theoretically possible in a computer game, it is not something you can consider when interacting with the real world.


text:
Find anomalies from records of categorical data
I have a data-set with $m$ observations and $p$ categorical variables (nominal), each variable $X_1, X_2,\dots, X_p$ has several different possible values.
Ultimately, I am looking for a way to find anomalies i.e. to identify rows for which the combination of values seems incorrect with respect to the data I saw so far.
So far, I was thinking about building a model to predict the value for each column and then build some metric to evaluate how different the actual row is from the predicted row.
I would greatly appreciate any help!

First of all, you mention that you have categorical data. I don't see how you can define similarity so that you can also define the distance between the predicted value and the ground truth (error). You can do that only if the data are ordinal.
If you want to just classify between normal and anomalous points (binary classification), without caring about further classification of the anomaly types themselves, one of the most common algorithms is the One-Class Support Vector Machine (OC-SVM).
Anomalies are unpredictable in nature and sometimes hard to replicate and record. Therefore, there is usually lack of anomalous data and supervised learning approaches suffer because if you sacrifice some "precious" anomalous points to train the algorithm, you cannot use them to test it.
The main advantage of OC-SVM is that it is semi-supervised learning, meaning that you train it only with normal data and then it can detect samples that deviate from the trained behaviour during testing and classify them as anomalous. Thus, you "save" all the rare anomalous points for testing purposes!
Take a look at this short Python example, it has all you need :)


text:
How to compute the output of a neural network produced by NEAT?
I used to work with layered neural networks, where, given certain inputs, the output is produced layer-by-layer.
With NEAT, a neural network may assume any topology, and they are no longer layered. So, how do we compute the output of such a neural network? I understand time-steps must be taken into account, but how?  Should I keep the inputs until all hidden neurons are processed and output is produced? Should I wait for the output to stabilize?

The networks in NEAT are still implicitly layered. There are neurons that need to be evaluated before other neurons can be evaluated and so this gives us our layers.
If you don't know the structure of your network then you can use Kahn's algorithm to find an arbitrary (by arbitrary I just mean one of the possible partially ordered sets) ordering of the nodes in the network. Then you evaluate your neurons in the order given to you by Kahn's algorithm. This works because the ordering of your network (which is a directed acyclic graph) is a partially ordered set.


text:
Game AI evaluation function and making progress towards winning
In two-player games, the exact value of the evaluation function doesn't matter, as long as it's bigger for better positions. However, for learning, it's customary when it does change when the best move gets made. This way, the learning can minimize the difference between the directly computed value $f(0, p)$ of a position $p$ and the value obtained from $n$ step minimax $f(n, p)$.
What I'm missing here is a way to direct the evaluation function to actually winning. For example, a perfect evaluation function for a won position in chess would always return $+1$ without any hint on how to progress towards a checkmate. In a chess variant without the fifty-move limit, it could play useless turns forever.
I guess this is a rather theoretical problem, as we won't ever have such a good function, but I wonder if there's a way to avoid it?

What I'm missing here is a way to direct the evaluation function to actually winning. For example, a perfect evaluation function for a won position in chess would always return +1 without any hint how to progress towards checkmate. In a chess variant without the fifty-move limit, it could play useless turns forever.
I guess, this is a rather theoretical problem as we won't ever have such a good function, but I wonder if there's a way to avoid it?

It certainly isn't just a theoretical problem, it can occur in practice too. For "large" games this problem won't occur in the early game, but it can start occurring in the later stages of a game when terminal positions can actually be reached directly through exhaustive searches. It can also occur right from the beginning in extremely small / simple games, like Tic Tac Toe.
Addressing this is not (necessarily) just a matter of defining the evaluation function alone, it can also depend on which search or learning algorithm is used. So, I'll consider a few different cases.

Case 1: Minimax / Alpha-Beta / other similar "exhaustive" searches
When using Minimax / Alpha-Beta / other search algorithms based on those, the easiest solution to the problem you describe is to use iterative deepening. As soon as you prove a win for yourself at a certain depth level d using iterative deepening, you can simply stop the search, don't search if there are any other wins to be proven at depth d + 1, just play along the line you've just proven to be a winning line. This way, you will always go for the win in the lowest number of moves.
A related advantage of this approach is that, as soon as you prove a loss at depth d for yourself, you can also cancel the search process and play the move that was best according to your evaluations at depth d - 1. This will often be your best chance to still grab a win in cases where your opponent fails to see their win at depth d.

Case 2: Monte-Carlo Tree Search / other searches with randomness
Monte-Carlo Tree Search is a well-known search algorithm that incorporates an element of randomness in its search. With these kinds of algorithms, the problem you describe tends not to be a real issue. Due to the randomness in the search, wins that can be achieved in a small number of moves tend to be evaluated better than longer-distance wins in practice. In long-distance wins, there is a greater chance that the randomness in the search process causes an incorrect move to be played somewhere along the long-distance win, which reduces the evaluation of such a line of play.

Case 3: (Reinforcement) Learning approaches
These approaches tend to involve some element of randomness due to the need for exploration in learning, which leads to similar reasoning as described for MCTS above. Also, in Reinforcement Learning, we typically use a discount factor gamma < 1.0 (e.g. gamma = 0.99) which causes distant rewards to be viewed as less important than close rewards, even if we don't do such discounting for the final evaluation of the performance of an algorithm. See, for example, a lot of the work on Atari games (DeepMind's DQN, etc.). Algorithms are evaluated according to their undiscounted scores, but learning still uses a bit of discounting because, in practice, this is found to be beneficial for learning.


text:
Which functions can't neural networks learn efficiently?
There are a lot of papers that show that neural networks can approximate a wide variety of functions. However, I can't find papers that show the limitations of NNs.
What are the limitations of neural networks? Which functions can't neural networks learn efficiently (or using gradient-descent)?
I am looking also for links to papers that describe these limitations.

I want clarify. For example lets consider feedforfard NN. Yes it can represent anything but for some tasks it could demand too many data or too many computational power or incredible count of neurons and layers. for example X1*X2. It can be represented and this function is very simple and we expect that our cool approximator can do it easy but it actualy can only inefficient proximate it by creation a banch of hypeplane spliting space.

The fact that you cannot find a paper about limitations... I don't know how that would even happen. It seems almost impossible to get something setup, and not read the warning page.

One of the important qualifications of the Universal approximation theorem is that the neural network approximation may be computationally infeasible.

"A feedforward network with a single layer is sufficient to represent any function, but the layer may be infeasibly large and may fail to learn and generalize correctly." - Ian Goodfellow, DLB

I can't think of any function that I would definitively declare as unlearnable, but neural networks have many problems. Consider adversarial examples and adversarial patches, which highlight the poor generalization going on under the hood of recent advances in computer vision.
Neural Networks are also inherently limited by the innate priors baked into their architecture and the sample density of their training data. Check out this recent discussion at Stanford's AI Salon between Yann LeCun and Christopher Manning on innate priors if that is the kind of limitation you are talking about.

For example X1 * X2, this function can be approximated by feedforward NN but can't be efficiently represented by splitting space with linear hyperplanes, also NN poor with fitures combinations like x1*x2, x1/x2, 1/x1 and so on, because neuron creates only linear hyperplane and it isn't efficient.

A random function cannot be learned efficiently by any algorithm, in particular, neural networks. However, if you are looking for a function with (exponentially) smaller description size, I do not know but any function that is conjectured to be average-case hard probably cannot be learned efficiently by neural networks, for example,

Any cryptographic hard-core predicate (related: Could Deep Learning be used to crack encryption?).
Parity learning with noise (related: Cryptographic Primitive Based on Hard Learning Problems).
Combinatorial examples: planted SAT, planted clique (related: Barak's survey).

This answer depends very much so on the type of neural network and algorithm used for training.
If you are using gradient descent on a neural network of one input layer, one output layer, and no hidden layers there are many functions that you can't learn. One simple one is the XOR function. Due to the fact that XOR is not linearly separable, it can not be represented by a neural network with no hidden layers.
If you are using NEAT to build recurrent neural networks then all functions(**) can be represented given enough time and data. This is due in part to the fact that recurrent neural networks are Turing Complete.
One of the biggest causes for limitations when using neural networks is based on the difficulty of interpretation as to what the network is doing. The network is gradually building up an understanding of the function as it goes from the input layer to the output layer, but it is very difficult for us to understand this building up process and interpret what the neural network is attempting to do. This makes it very hard if not impossible to manually tweak your neural network in a meaningful way.
Another limitation is the need for training (in large amounts) in order to have a meaningful representation of your data. Neural networks have a tendency to need large amounts of data before converging to a meaningful hypothesis space. This has resulted in clever algorithms to generate training data without needing human interaction, such as Generative Adversarial Networks, but the underlying problem remains.
** Not all functions can be computed by neural networks, however, all computable functions can be. An example of an uncomputable function is the mapping of all programs from the program to whether or not this program will halt (The Halting Problem).


text:
What is the difference between a convolutional neural network and a regular neural network?
I've seen these terms thrown around this site a lot, specifically in the tags convolutional-neural-networks and neural-networks.
I know that a neural network is a system based loosely on the human brain. But what's the difference between a convolutional neural network and a regular neural network? Is one just a lot more complicated and, ahem, convoluted than the other?

TLDR:
The convolutional-neural-network is a subclass of neural-networks which have at least one convolution layer. They are great for capturing local information (e.g. neighbor pixels in an image or surrounding words in a text) as well as reducing the complexity of the model (faster training, needs fewer samples, reduces the chance of overfitting).
See the following chart that depicts the several neural-networks architectures including deep-conventional-neural-networks: .

Neural Networks (NN), or more precisely Artificial Neural Networks (ANN), is a class of Machine Learning algorithms that recently received a lot of attention (again!) due to the availability of Big Data and fast computing facilities (most of Deep Learning algorithms are essentially different variations of ANN).
The class of ANN covers several architectures including Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN) eg LSTM and GRU, Autoencoders, and Deep Belief Networks. Therefore, CNN is just one kind of ANN.
Generally speaking, an ANN is a collection of connected and tunable units (a.k.a. nodes, neurons, and artificial neurons) which can pass a signal (usually a real-valued number) from a unit to another. The number of (layers of) units, their types, and the way they are connected to each other is called the network architecture.
A CNN, in specific, has one or more layers of convolution units. A convolution unit receives its input from multiple units from the previous layer which together create a proximity. Therefore, the input units (that form a small neighborhood) share their weights.
The convolution units (as well as pooling units) are especially beneficial as:

They reduce the number of units in the network (since they are many-to-one mappings). This means, there are fewer parameters to learn which reduces the chance of overfitting as the model would be less complex than a fully connected network.
They consider the context/shared information in the small neighborhoods. This feature is very important in many applications such as image, video, text, and speech processing/mining as the neighboring inputs (eg pixels, frames, words, etc) usually carry related information.

Read the followings for more information about (deep) CNNs:

ImageNet Classification with Deep Convolutional Neural Networks
Going Deeper with Convolutions

P.S. ANN is not "a system based loosely on the human brain" but rather a class of systems inspired by the neuron connections exist in animal brains.

Convolutional Neural Networks (CNNs) are neural networks with architectural constraints to reduce computational complexity and ensure translational invariance (the network interprets input patterns the same regardless of translation— in terms of image recognition: a banana is a banana regardless of where it is in the image). Convolutional Neural Networks have three important architectural features.
Local Connectivity: Neurons in one layer are only connected to neurons in the next layer that are spatially close to them. This design trims the vast majority of connections between consecutive layers, but keeps the ones that carry the most useful information. The assumption made here is that the input data has spatial significance, or in the example of computer vision, the relationship between two distant pixels is probably less significant than two close neighbors.
Shared Weights: This is the concept that makes CNNs "convolutional." By forcing the neurons of one layer to share weights, the forward pass (feeding data through the network) becomes the equivalent of convolving a filter over the image to produce a new image. The training of CNNs then becomes the task of learning filters (deciding what features you should look for in the data.)
Pooling and ReLU: CNNs have two non-linearities: pooling layers and ReLU functions. Pooling layers consider a block of input data and simply pass on the maximum value. Doing this reduces the size of the output and requires no added parameters to learn, so pooling layers are often used to regulate the size of the network and keep the system below a computational limit. The ReLU function takes one input, x, and returns the maximum of {0, x}. ReLU(x) = argmax(x, 0). This introduces a similar effect to tanh(x) or sigmoid(x) as non-linearities to increase the model's expressive power.

Further Reading
As another answer mentioned, Stanford's CS 231n course covers this in detail. Check out this written guide and this lecture for more information. Blog posts like this one and this one are also very helpful.
If you're still curious why CNNs have the structure that they do, I suggest reading the paper that introduced them though this is quite long, and perhaps checking out this discussion between Yann Lecun and Christopher Manning about innate priors (the assumptions we make when we design the architecture of a model).

This is a better answer to me in that it explains exactly how CNNs are a specific type of NN. Other answers don't mention that weight sharing is enforced.

A convolutional neural network is one that has convolutional layers. If a general neural network is, loosely speaking, inspired by a human brain (which isn't very much accurate), the convolutional neural network is inspired by the visual cortex system, in humans and other animals (which is closer to the truth). As the name suggests, this layer applies the convolution with a learnable filter (a.k.a. kernel), as a result the network learns the patterns in the images: edges, corners, arcs, then more complex figures. Convolutional neural network may contain other layers as well, commonly pooling and dense layers.
Highly recommend CS231n tutorial on this matter: it's very detailed and contains a lot of very nice visualizations.

The everyday definition of convolution comes from the Latin convolutus meaning 'to roll together'. Hence the meaning twisted or complicated.
The mathematical definition comes from the same root, with the interpretation of taking a "rolling average".
Hence in Machine Learning, a convolution is a sliding window across an input creating one averaged output for each stride the window takes. I.e. the values covered by the window are convoluted to create one convoluted output. This is best demonstrated with an a diagram:

The convolution can be any function of the input, but some common ones are the max value, or the mean value.
A convolutional neural network (CNN) is a neural network where one or more of the layers employs a convolution as the function applied to the output of the previous layer.
If the window is greater than size 1x1, the output will be necessarily smaller than the input (unless the input is artificially 'padded' with zeros), and hence CNN's often have a distinctive 'funnel' shape:

Although this is a good answer, I think, it does not mention anything about "regular neural networks". It seems that your answer would be more an answer to the question here. So, you may consider deleting it from here and pasting it there or edit it to include something about "regular NNs"

It's a very simplified explantion. I am just talking about the core idea.
A neural network is a combination of many layers.
A neural network (Multiple Layer Perceptron: Regular neural network ): It does a linear combination (a mathematical operation) between the previous layer's output and the current layer's weights(vectors) and then it passes data to the next layer by passing through an activation function. The picture shows a unit of a layer.

A neural network (Convolutional Neural Network):  It does convolution (In signal processing it's known as Correlation) (Its a mathematical operation) between the previous layer's output and the current layer's kernel ( a small matrix ) and then it passes data to the next layer by passing through an activation function. The picture shows a Convolution operation. Each layer may have many convolution operation


text:
In the multi-linear regression, how is the value of weight $b_2$ calculated?
In multivariate linear regression (linear regression with more than one variable) the model is $yi = b_0 + b_1x_{1i} + b_2x_{2i} + ...$ , and so on. But how is the $b_n$ value calculated iteratively? Can it be calculated non-iteratively? What is the intuition behind using that method to calculate $b_2$?

It is calculated the same way $b_1$ is calculated.
Nearly following your notation, say your multiple linear regression function is
$H(X_i) =  b_0 + b_1x_{1,i} + ...+ b_nx_{n, i}$
for data instance $X_i=x_{1,i},...,x_{n, i}$
and weights $b_0,...,b_n$.
And say your error function is $E(X,Y) = \sum_i(H(X_i)-Y_i)^2$
where $X$ is the collection of all data points $X_i, Y_i$.
From your error function $E$, for whatever weights you have (with a gradient based method), calculate the partial derivative
$\partial E /\partial b_i$ and use this to update all of your weights at once in each iteration of your optimization routine.

put your b_1, ... ,b_n coefficients into a vector b
put all your x_{ij} into a matrix X
then all components of b are calculated at the same time with this equation
b = (H(X) - Xb)^T (H(x) - Xb)
But this calculation (estimation) is only consistent (search consistent estimator) when certain assumptions are present (read here https://en.wikipedia.org/wiki/Ordinary_least_squares).


text:
Does training happen during NEAT?
When one uses NEAT to evolve the best fitting network for a task, does training take place in each epoch as well?
If I understand correctly, training is the adjustment of the weights of the neural network via backpropagation and gradient descent. During NEAT, say, a generation runs for 1000 iterations. During that time, is there any training involved, or does each genome randomly poke around and the winner takes it to the next stage?
I've used NEAT, but the fact that neural networks are not trained does not make sense to me. At the same time, I can't find any code in my framework (Neataptic.js) that would train the generation during the epoch.

NEAT uses genetic algorithms both to search for improved connection weights and for improved architectures.
Whilst it is possible to train a NEAT-generated neural network using backpropagation of error gradients, libraries implementing "original" NEAT will not implement that.
There are a couple of reasons:

There is often no training data, in a supervised learning sense. The fitness function of a NEAT system can be measured arbitrarily by performance at some task, and in the general case this consists of some environment simulation that interacts with an agent controlled by the NN, and not training data per se.
Evolved network topologies do not conform to stacked layers models preferred by ML frameworks designed to run typical deep learning architectures.

Both of these issues can be resolved with a little effort (for instance for the second issue there are frameworks which will work with arbitrary feed forward connection graphs, they are just a little more niche than e.g. Keras). However, NEAT is often used because it can solve problems without needing to frame them as supervised learning or reinforcement learning.
Other than the hard work of putting it together, there is nothing stopping you creating a train-on-data stage which could alternate between the two approaches, perhaps with controllable weighting from evolution to gradient-based training. To add the gradient-based training, then either:
a) Your original problem is one of fitting to a classifier or regression problem. In which case your fitness function and training loss function could be the same.
b) Your original problem is one of controlling an agent in an environment, and you could potentially use something like the REINFORCE algorithm based on most-recent assessments, in order to provide gradients to train the NN. Other policy gradient methods could also be a good fit, as the NEAT network typically outputs a policy as opposed to a value prediction.
I have never tried these (nor much with NEAT at all other than demos and a bit of theory). For (a) I would expect the combination to be successful, but wonder why you were bothering with NEAT in the first place. For (b) I am less sure whether you would get useful results, because REINFORCE relies on multiple runs with the same network, whilst NEAT relies on stochastic search across multiple networks. Applying REINFORCE training across a whole population could be very CPU intensive.


text:
multi vs one prediction using Regression
I was trying to build a prediction system where I have the input data arranged in multiple columns. The input data would be of the type where I have

weather,
service type (bronze, silver, gold),
size (xs, s, m, l, xl, xxl),
time,
availability,
pin code, and
the result (target).

Each of the data types is arranged in columns with a specific code. I have read this, this, this , this, and this.
They are helpful but do not give me a clear picture. I would like to achieve a multi-vs-one prediction. Most of the schemes available are one-vs-one where the data is a 1*1 entity.
Here is a sample code that I was working with:
regressionModel = linear_model.LinearRegression()
    """ 3. Processing is not necessary for current concept """
    y = pd.DataFrame(modifiedDFSet['Code'])
    print(y.shape)
    drop2 = ['Code']
    X = modifiedDFSet.drop(drop2)
    print(X.shape)
    """ 4. Data Scaling, Data Imputation is not necessary. Training and Test data is prepared using train-test-split """
    train_data, test_data = train_test_split(X, test_size=0.20, random_state=42)
    """ 5. the Regression Model """
    # h = .02  # step size in the mesh
    # logreg = linear_model.LinearRegression()
    # we create an instance of Neighbours Classifier and fit the data.
    regressionModel.fit(X, y)
    d_predictions = regressionModel.predict(y)

X.shape and y.shape would yield (500, 6) and (500, 1), respectively, which would obviously cause a dimensional error in the d_predictions, meaning the regression model does not take multiple column inputs.
I have a hypothesis that I can create a scoring scheme that will take into account the importance of each of the columns and create a scheme that creates a score and the end result would be a one-vs-one regression problem. Looking for some direction with respect to my hypothesis. Is it correct, wrong or halfway?

No. I do not want to use any tool or software. I intend to hardcode using python 3x.  I am open to exploring new libraries. I am using scikit learn at the moment.

I think the model will have no problem taking a multicolumn input. In fact, from your code, this is exactly how you trained it. It expects an input of size [k, 6], where k is k>=1.
Instead you are feeding it with [k, 1] sized data, which are the dimensions of y. So you if you run it like this it should work:
regressionModel.predict(X)


text:
Which neural networks are suitable for visual place recognition?
I am doing a project on visual place recognition in changing environments.  The CNN used here is mostly AlexNet, and a feature vector is constructed from layer 3.
Does anyone know of similar work using other CNNs, for example, VGGnet (which I am trying to use) and the corresponding layers?
I have been trying out the different layers of VGGnet-16.  I am trying to get the nearest correspondence to the query image by using the cosine difference between the query image and database images. So far no good results.

You should consider using a ResNet architecture and make use of BatchNorm layers. This is a state-of-the-art architecture that is much easier to train and will give much better results. The depth is up to you and should be as much as you need to accuratly solve your task (e.g. start with a ResNet50 or ResNet20). AlexNet and VGG are not really up-to-date anymore and much harder to optimze than ResNet architectures.

Neural networks construct increasingly complex representations of data on each of their layers , so you are free to choose any neural network architechture for this purpose . since the lower layers of neural network (layers near the input layer) mostly compute low level representations of the image (like gabor filters etc) most architechtures won't have much difference at this level. so you can use VGGnet if you want with proper fine-tuning from layer 3 itself.

Thanks for replying. AlexNet is one of the simplest CNNs, in VGGnet, there are 3 layer 3s and because VGGnet is deeper, each of the layers are derived from more preceding layers (and suceeding layers in the case of back propagation). I don't get good results from all layers of layer 3.

alexnet is simple only in terms of number of layers ,but its quite heavy on parameters , consumes more memory than vggnet , and results can be mostly due to bad finetuning , can you post the full details of model? or perhaps code?


text:
In the case of invalid actions, which output probability matrix should we use in back-propagation?
As discussed in this thread, you can handle invalid moves in reinforcement learning by re-setting the probabilities of all illegal moves to zero and renormalising the output vector.
In back-propagation, which probability matrix should we use? The raw output probabilities, or the post-processed vector?

I am also quite new to this field, but I think you should use the normalized outputs for the backpropagation. In general, you would want to backpropagate all the calculations you did in the forward pass, so why would you want to exclude the step of the normalization in your backward pass? This would essentially make the renormalization to have no effect (different loss values but no different model weight update).
For example, in policy gradients, you backpropagate through the log propability of the selected action. In the forward pass, the sampling of the probability (that determines which action is selected) is not affected by the renormalization (you might just get different loss values in the end in your loss function). But, compared to this, in the backward pass, you need the actual value of the log probability to calculate the gradient that updates the model weights.
So (I think) the normalization is mostly just done for the backpropagation to get "renormalized" gradients. And no unbalanced gradients between states with more/less allowed actions.


text:
Name of a multiarmed bandit with only some levers available
In order to model a card game, as an exercise, I was thinking of an elementary setting as a multiarmed bandit, each lever being the distribution of expected rewards of a specific card.
But, of course, the player only has some cards in the hand each round, or, equivalently, for a given round, it has available a number $n$ of arms randomly selected from the total number $N$ of levers.
Is this just a "contextual bandit" or has it some specific, narrower, name that I could use to look up in the literature?

To clarify, is there only one state of the game? Can you only select 1 card?

Yep, habitual game mechanics: you have received $n$ random cards from an stack of $N$, you can play one. In some cases, (eg bridge) perhaps even less than $n$. Seems contextual, but very particular.


text:
Does NEAT require only connection genes to be marked with a global innovation number?
Does NEAT require only connection genes to be marked with a global innovation number?
From the NEAT paper

Whenever a new gene appears (through structural mutation), a global innovation number is incremented and assigned to that gene.

It seems that any gene (both node genes and connection genes) requires an innovation number. However, I was wondering what was the node gene innovation number for. Is it to provide the same node ID across all elements of the population? Isn't the connection gene innovation number sufficient?
Besides, the NEAT paper includes the following image which doesn't show any innovation number on node genes.

It is actually the other way around: connection IDs is what is debated!
Nodes always have innovation IDs (in the image, it is just their identifying number).
Node IDs are sufficient to identify connections. If a connection links nodes 3 and 6, then it is the same as another connection linking nodes 3 and 6: no need for an extra ID. So why the extra innovation IDs then?
On the one hand, this is an implementation choice: maybe these extra IDs would allow you to create a more complex but faster code?
On the other hand, there is a debate around whether a connection between two nodes means the same thing at different times in evolution. If you have no innovation IDs, then you cannot tell apart an old connection from 3 and 6 and another that was independently created later in a different genome (imagine the old connection was removed first). Is this relevant? As said, it is an open debate. Surely, it is not crucial at a basic level!
This question (and my answers) is related to this other question on Stack Overflow.

Do you have any links to arguments on either side? I'd be interested to read some opinions.

I can't say much more than what I have written. In a conversation that I followed, I understood that removing IDs from connections simplifies code. The one drawback that was mentioned is what I already said here, about not being able to track connections between the same nodes at different times. Colin Green is responsible for SharpNEAT and knows A LOT about this. He is currently refactoring all the code and brought up this topic, so maybe you can reach him if you have more doubts! https://github.com/colgreen/sharpneat-refactor

In the original paper the innovation ID is on the connections only.
The connection is the object that is keeping the information; nodes can be discerned by the connections.

This image represents a possible crossover operator that makes a distinction between disjoint and excess genes and therefore creates children depending on these. It's part of my master thesis, I'd be glad to expand the topic, but for now I'll just use it as example.
In the image we assume that the connections that are joining two nodes have the same innovation number.
As you can see, there is no need of assigning an innovation number to the nodes: nodes are just a result of what the connection say. This also allows for a more dynamic approach that can be used to spot invalid nets even before building them (checking if there are cyclics or nodes that aren't receiving any input or giving any output) and correct them in order to obtain only valid graphs.
Nodes are added just because there is a connection that is pointing to that specific node. This is enough to grant its presence (node number 8 in child 2).
As last point, for the data normalization theory (Normalization is a process of organizing the data in database to avoid data redundancy, insertion anomaly, update anomaly & deletion anomaly) we should avoid rendundancy at any cost and this is why we should try to keep track of the smallest amount of object possibles. So if we can deduct the nodes from the connections we should do it.

I believe the correct interpretation is the oposite. Note that in your diagram only nodes have identifying numbers, and that connections are defined by the nodes they connect.

True, nodes have IDs, connections are defined by the connected IDs. Keeping only the connections, though, allows a slimmer genome representation without loosing any information as you can store nodeFrom, nodeTo, weight and innovation number. Nodes are then discerned from the list of connections.


text:
Are mathematical models sufficient to create general artificial intelligence?
Are mathematical models sufficient to create general artificial intelligence?
I am not sure if it is possible to represent e.g. emotions or intuition using mathematical model. Do we need a new approach in order to solve this problem?

Mathematical models are essentially highly formalised knowledge. When it comes to computer engineering, there is literally no other choice - anything you can write code for, or design a machine for, will have an associated mathematical model. That model may not be fully explored or comprehended analytically by theorists, it may be just too complex (and driven by calculations) or even mathematically intractable. However, that doesn't make it non-mathematical, just more driven by empirical results than theory.
We don't have complete models for how AGI definitely work, nor tight enough definitions of general intelligence to base maths from where we can say "if we implemented a framework based on this maths, we could build an AGI". Right now, exploration and experiments based on intuition of what might work are far ahead of such theory.
The theoretical work behind e.g. neural networks is chipping away at the problem, and there are more general over-arching theories about intelligent rational behaviour available e.g. the equations of AIXI. AIXI doesn't cover emotions and intuition directly, but does attempt to cover knowledge and how a rational agent would approach understanding the world in general from scratch. It is possible that an embodied system driven by a software implementing something like AIXI could exhibit intuition and emotions in an emergent fashion, but whether or not that would happen in practice is not at all clear from the theory. AIXI is just one theory/model out of many, and I am not qualified to analyse it in depth, but its creators have strong pedigree in AI research, so it is as good as any IMO if you are interested in starting to research AI from a theoretical perspective.
Despite all the unknowns, the success of deep learning and the loose analogies between artificial neural networks and biological ones, makes it look likely that neural networks or something like them will form a component of an AGI. The current state of the art for more narrow problem solving, learning from examples or experience through backpropagation of error gradients, might not be the most important or key component. However, whatever the structure, whether it is a small extension of existing systems, or involves some new  science, it will be describable mathematically.


text:
Two chatbots - One teaches another
I am seeking the information for this kind of chatbot architecture : There are two chatbots. One plays the role of teacher, and another is a student who is learning. The goal is to test the student's quality, and to improve the student's ability.
I didn't find much reference. There are :
Bottester: Testing Conversational Systems with Simulated Users
And the ParlAI, a python-based platform for enabling dialog AI research has the notion of "Teacher agent", which seems to be what I am looking for.
Of course, we also have deep reinforcement learning which might be related.
I prefer to have some classical references for this approach to chatbots.
Currently, reinforcement learning is not in my consideration.
Constructing two chatbots talking to each other, like what Facebook did, is not what I want. Because in this case, both of them are student agents.

Simple method: teacher opens a ssh connection to the student, and transfers its conceptual database to it. Use wifi if you want something wireless. 100% of quality on knowledge transference.

That is maybe not the thing in OPs mind.

@mico, exactly. This is in fact what made me to search for references. If   this approach has been seriously studied, then there must be a more non-trivial method. If not, then it is really a new approach and the benefit is not known yet.

The Hidden Agenda User Simulation Model (Schatzmann/Young) describes a chatbot training design in which a user simulator assembles and executes a conversational agenda, with the direct goal being to train the target chatbot.
Perhaps you can add specificity to this design by casting the user simulator as the teacher, and creating an agenda in which it is communicating information to the (student) chatbot. The trained behavior expected is, perhaps, correct responses to topical questioning by the teacher.

There has been some work in this area but with three chat bots.  Xing Han Lu has developed some code called Generative Adversarial Bots (GABs) which builds on the concept of Generative Adversarial Networks invented by Ian Goodfellow.  See his pioneering paper "Generative Adversarial Networks".
There is a very brief Google presentation here.
The basic idea of GABs is to "compare the performance and human-likeness of conversational chatbots by generating conversation between two bots, and evaluating the response using Turing Tests".  "Generative Adversarial Bots (GABs) are bots that are pitched up against each other, and generate a conversation that is used to train a third bot".
You might want to also check out "Adversarial Learning for Neural Dialogue Generation" by Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter and Dan Jurafsky.  From their abstract:
In this paper, drawing intuition from the Turing test, we propose using adversarial training for open-domain dialogue generation: the system is trained to produce sequences that are indistinguishable from human-generated dialogue utterances. We cast the task as a reinforcement learning (RL) problem where we jointly train two systems, a generative model to produce response sequences, and a discriminator---analagous to the human evaluator in the Turing test--- to distinguish between the human-generated dialogues and the machine-generated ones. The outputs from the discriminator are then used as rewards for the generative model, pushing the system to generate dialogues that mostly resemble human dialogues.
Jiwei Li, one of the authors, has posted the code here.


text:
In YOLO, when is $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$, and what are the ground-truth labels for $x_i$ and $y_i$?
I'm trying to implement a custom version of the YOLO neural network. Originally, it was described in the paper You Only Look Once: Unified, Real-Time Object Detection (2016). I have some problems understanding the loss function they used.
Basic information:

An input image is divided into $S \times S$ grid (that gives the total of $S^2$ cells) and each cell predicts $B$ bounding boxes and $c$ conditional class probabilities. Each bounding box predicts $5$ values: $x,y,w,h,C$ (center of bounding box, width and height and confidence score). This makes the output of YOLO an $S \times S \times (5B*c)$ tensor.

The $(x,y)$ coordinates are calculated relative to the bounds of the cell and $(w,h)$ is relative to the whole image.

I understand that the first term penalizes the wrong prediction of the center of a bounding box; the 2nd term penalizes wrong width and height prediction; the 3rd term the wrong confidence prediction; the 4th is responsible for pushing confidence to zero when there is no object in a cell; the last term penalizes wrong class prediction.

My problem:
I don't understand when $\mathbb{1}^\text{obj}_{ij}$ should be $1$ or $0$. In the paper, they write (section 2.2. Training):

$\mathbb{1}_{i j}^{\mathrm{obj}}$ denotes that the $j$th bounding box predictor in cell $i$ is "responsible" for that prediction.

and they also write

Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). It also only penalizes bounding box coordinate error if that predictor is "responsible" for the ground truth box

So, is it right that, for every object in the image, there should be exactly one pair of $ij$ such that $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$?

If this is correct, this means that the center of the ground truth bounding box should fall into $i$th cell, right?

If this is not the case, what are other possibilities when $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$, and what ground truth labels $x_i$ and $y_i$ should be in these cases?

Also, I assume that ground truth $p_i(c)$ should be $1$ if there is an object of class $c$ in the cell $i$, but what ground truth $p_i(c)$ should be equal to in case there are several objects of different classes in the cell?

Please, next time ask only 1 question per post. If you have multiple questions, even though they are about the same model or topic, split them into multiple posts, so that people can focus on 1 question/problem at a time.

The correct interpretation (based on the comment by the author of the question below):
Yep, you are right. It is actually only a single cell per object that contributes to the loss with $\mathbb{1}^\text{obj}_{ij}$ factor. That cell is identified as the one that contains the centre(oid) of the ground truth box of the corresponding object.
My original (incorrect) interpretation of the paper:
From what I read in the paper it is not a single bounding box for all cells in the grid for a particular object (your original suggestion).
Rather for every object and every cell that is assigned to this object only one bounding box contributes to the loss. The network generates $B$ bounding boxes for every cell in the grid, but we pick only one and only when that cell actually belongs to an object.
Every cell $i$, which is not a background cell, will have exactly one box $j$, such that $\mathbb{1}^\text{obj}_{ij} = 1$
This is based on the following paragraph from the paper:

Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). It also only penalizes bounding box coordinate error if that predictor is "responsible" for the ground truth box (i.e. has the highest IOU of any predictor in that grid cell).

Thank you for your reply. I've done some digging after I posted this question and found one video where Joseph Redmon(author of the paper) gives a presentation about YOLO: https://youtu.be/NM6lrxy0bxs . Somewhere about 8 min he starts to explain how they train the network and it seems like my initial assumption is rights: 1^{obj}{ij}=1 only when the center of the ground truth box falls into i cell. Also, it seems like 1^{noobj}{ij}=1 for every cell where there is no center of ground truth box.

yep, looking at the video you are totally right. Still it was nice to clarify how YOLO objective works:)


text:
How to deal with different actions for different states of the environment?
I'm new to this AI/Machine Learning and was playing around with OpenAI Gym a bit. When looking through the environments, I came across the Blackjack-v0 environment, which is a basic implementation of the game where the state is the hand count of the player and the dealer and if the player has a useable ace. The actions are only hit or stand and the possible rewards 1 if the player wins, -1 if the player loses, and 0 when it is a draw.
So, that got me thinking what a more realistic environment/model for this game would look like, taking into account the current balance and other factors and has multiple actions like betting 1-10€ and hit or stand.
This brings me to my actual question:

As far as I understand, in neural networks (and I do not very well yet, I guess) the input will be the state and the output the possible actions and how good the network thinks they are/will be. But now there are two different action spaces, which apply to different states of the game (betting or playing), so some of the actions are useless. How would be the right way to approach this scenario?

I'm guessing one answer would be to give some kind of negative reward if the network guesses a useless action, but, in this case, I think the reward should be the actual stake (negative reward) and the actual win if any. Therefore, this would cause some bias in how the game proceeds as it should start with some amount of balance and end if the balance is 0 or after a specified amount of rounds.
Limiting timesteps wouldn't be an option either, I guess, because it should be limited to rounds, so, for example, it won't end after a betting step.
Therefore, for a useless step, the reward would be 0 and the state would stay the same, but, for the neural network, it doesn't matter how many useless steps it takes because it'll make no difference to the actual outcome.
Corollary question:

Should be split up into two neural networks? One for betting and one for playing?

There are so many reinforcement learning algorithms. I recommend you this articles:

Introduction to Various Reinforcement Learning Algorithms. Part
1
Introduction to Various Reinforcement Learning Algorithms.    Part
2

There are several approaches. A NN can get state and action as an input  and expected discounted reward as an output. But you can just not pick an action if this action isn't available and there isn't necessity to give it negative or zero reward just don't learn your net to change it behavior for these actions because they are neither good nor bad.

Thanks for the links, I'm using a DQN I know there are many algos and many that are better than a DQN but I'm pretty new to this and it was easier to implement, because of more existing examples.
I guess I'll have to do a lot more research until I really get all this stuff, but again thanks for the links any resource is good at the moment :)


text:
How to deal with padded inputs in a fully connected feed forward network?
I have a fully connected network that takes in a variable-length input padded with 0.
However, the network doesn't seem to be learning and I am guessing that the high number of zeros in the input might have something to do with that.
Are there solutions for dealing with padded input in fully connected layers or should I consider a different architecture?
UPDATE (to provide more details):
The goal of the network if to clean full file paths:
i.g.:

/My document/some folder/a file name.txt > a file name
/Hard drive/book/deeplearning/1.txt > deeplearning

The constraint is that the training data labels have been generated using a regex on the file name itself so it's not very accurate.
I am hoping that by treating every word equally (without sequential information) the network would be able to generalize as to which type of word is usually kept and which is usually discarded.
Then network takes in a sequence of word embedding trained on paths data and outputs logits that correspond to probabilities of each word to be kept or not.

There are a few choices of re-representation or different architectures, but the best choice will depend on details of the data. Please describe more about your input data, to help readers understand what the options might be. For instance, is the data from a sequence (text or signal processing), does it contain a variable number of equivalent items? A few simplified examples of the the data may help.

@NeilSlater I've updated the question with additional info

If you use fully connected neural network, it will be hard to do what you want. There's no inductive bias to easily generalize from different positions of the strings which you need to extract.
I suggest you use Seq2Seq models here: http://cs224d.stanford.edu/papers/seq2seq.pdf


text:
How well can CNN for bounding box detection generalise?
Suppose a CNN is trained to detect bounding box of a certain type of object (people, cars, houses, etc.)
If each image in the training set contains just one object (and its corresponding bounding box), how well can a CNN generalize to pick up all objects if the input for prediction contains multiple objects?
Should the training images be downsampled in order for the CNN to pick out multiple objects in the prediction?
I don't have a specific one in mind. I was just curious about the general behavior.

"How well" means there should be a measure. Please include this in your question. Otherwise one could simply answer "really well" or "42%"

I suggest you to go through the r-cnn paper or go through a tutorial on it . CNNs transform the image into high dimensional vector in their last layer , in case of classification this vector is sent to a "softmax" layer , in case of bounding box regression , four values :length , breadth , location of one of the points of the bounding box , are regressed from this vector , so if you use a cnn with one regression head you end up with one bounding box irrespective of the training set.

Typically, there are mainly three steps in an object bounding box detection.
First, a model or algo is used to generate ROI(region of interest) or region proposals. These region proposals are an all large set of bounding boxes spanning the full images. (that is, an object localization component).
In the second step, visual features(Face, Person, etc... using Convolution) are extracted for each of the bounding boxes, they are evaluated. It is determined whether and which objects are present in the interested area based on visual features (i.e. an object classification component).
In the final post-processing step, overlapped boxes are combined into a single bounding box (that is, non-maximum suppression).


text:
Why is my fine-tuned YOLO model detecting other objects as a human?
I am new to deep learning and computer vision. I have a problem where I use the YOLO to detect objects.
For my problem, I just want to recognize 1 human only. So, I changed the final YOLO's layer (which contained 80 neurons) to only 1 neuron, and do the training process with transfer learning techniques. Of course, I do not use the final layer's weights, and these weights are randomly initialized for my problem. I feed only the human data to the model.
However, I realize that after longer training, the model becomes worse. It starts to recognize other objects as a human.
Should I also feed non-human data to the model?

So you have a network pretrained on 80 classes. I also assume that one of these classes are human (or else this is just not the way to go*) I suspect that the final layer contains 80 labels, correct? Then you then 'rescale' this layer to 1 label and then train on some data you possess? Then you're basically trying to teach the network that it shouldn't care about the 79 other classes, which is just nonsense I think.
What you could do, and I do not recommend this, but if you feel like you have to use this exact network, you just keep the 80 outputs and only look at the label correspondning to the human.
You shouldn't do this because the network is WAY bigger than it needs to be to only classify human/non human, which will make it slower than it needs to be.
What you rather want to do is either to train your own network (If you have lots of training data, I suspect this wouldn't be be the hardest thing to train) or obtain a CNN that is pretrained on human classification.
(*I've heard rumours that you can do pretty well on retraining a class on a pretrained network. I just don't know if the rumours are true or how to go about it.)

Yes, one way is to reuse the network with 80 classes, but it will be very expensive since i just want to care about the presence of human only. In general, if i want to detect human only, i have also to give it data of non-human right?

As I say in the answer, yes it would be expensive, but about the same as your approach. If you want to train it yourself, you need to give it human and non-human data.

What do you think about the ratio between human/non-human data?

Hard to say. Depends on the data. Use all of what you have :)


text:
How does Q-learning work in stochastic environments?
The Q function uses the (current and future) states to determine the action that gets the highest reward.
However, in a stochastic environment, the current action (at the current state) does not determine the next state.
How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?

Could you clarify: "In a stochastic environment, where the action doesn't influence state" - do you mean that the state is not 100% predictable, and the action doesn't fully determine the state? E.g. deciding to move forward, and a game then moves you 1d6 steps forward. Or do you really mean that the action choice has absolutely no influence on the resulting state - and the state will evolve by rules independent of the action choice? E.g. deciding to buy or sell, getting some reward, then the state just changes randomly. The answer you need is very different depending on which case you mean.

@NeilSlater, Thanks for your response. In this case the action has no influence at all on the next state. The state will evolve by rules independent of the action choice.

In which case, although Q learning should work without errors, you would be expending resources for the agent to learn the fact that the action does not influence state. Your situation is more commonly known as "contextual bandit" and there a variety of solvers out there. The positive news is that good old supervised learning should be fine to learn from the historical data. Then it's just a matter of how online your algorithm needs to be, and how risk averse whilst making online decisions.

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?

The learned $Q$-function is not only used during training, but also after training (in what we may call "deployment", when we expect a trained agent to behave according to what it has learned).
However, the reliance on future states is only there during training, it is no longer required for deployment.
During training, we use the following $Q$-learning update rule:
$$Q(s, a) \gets (1 - \alpha) Q(s, a) + \alpha \left( R + \gamma \color{red}{\max_{a'} Q(s', a')} \right),$$
where $s'$ is the state we reach after executing $a$ in $s$. Here, the $\color{red}{\text{red}}$ part is the part where we rely on knowledge of the future $s'$. This is available in training because we can simply pick $a$, execute it in $s$, observe $s'$, and only then trigger our update step.

Outside of training (and actually also during training), we also rely on our $Q(s, \cdot)$ function for the selection of actions. We typically select an action $a$ according to $a = \arg\max_a Q(s, a)$; we select the action $a$ that maximises $Q(s, a)$ in our current state $s$. The important thing to note here is that there is no $s'$ term in this description of how we select actions: we do not require knowledge of our future state.

Note: in my answer I decided to answer the question literally as it is written, i.e. I'm explaining how $Q$-learning can still work in the described setting where actions $a$ have no influence whatsoever on the future state reached.
In practice, I would never recommend actually using $Q$-learning in such a setting, and instead refer to Neil Slater's comment about Contextual Multi-Armed Bandit algorithms likely providing a better solution.

What if the action a has influence on the future state but in a nondeterministic way? (eg applying q learning to tic tac toe, the opponent action is non deterministic)

If the opponent is stationary (does not change its strategy / is not learning itself), then this is still fine, because the opponent can simply be viewed as being "a part of" a stochastic environment, rather than an actual intelligent agent. And stochastic environments are fine, the assumption that the environment may be stochastic is very standard in MDPs. $Q$-learning can deal with that. If the opponent is itself also a learning agent which changes while you train, both agents can get stuck cycling through suboptimal strategies with algorithms like $Q$-learning.


text:
What is the most time-consuming part of training deep networks?
Deep networks notoriously take a long time to train.
What is the most time-consuming aspect of training them? Is it the matrix multiplications? Is it the forward pass? Is it some component of the backward pass?

See also https://ai.stackexchange.com/q/7865/2444, https://ai.stackexchange.com/q/5728/2444 and https://ai.stackexchange.com/q/13612/2444.

Check out Figure 6 in this paper: PyTorch Distributed: Experiences on Accelerating Data Parallel Training
It breaks down the latency of the forward pass, the backward pass, the communication step, and the optimization step for running both ResNet50 and BERT on a NVIDIA Tesla V100 GPUs.
From measuring the pixels in the figure, I estimated the times for the forward, backward, and optimization steps as a percentage of their total time combined. (I ignored the communication step shown in the figure because that was only to show how long an unoptimized communication step would take when doing data-parallel training). Here are the estimates I got:

Forward: 23%
Backward: 74%
Optimization: 3%

So the backward pass takes about 3x as long as the forward pass, and the optimization step is relatively fast.

Forward pass
The output of a layer can be calculated given the output of the previous layer. So the GPU can parallelize this computation for every layer and over the minibatch which is done by calculating a big matrix. But it needs to be sequential from layer to layer (earlier layers to higher layers). Regarding the layer type convolutions or especially fully connected layers can result in a big matrix calculation.
Backward pass
The gradient of a layer with respect to the layer input (and layer parameters) can only be calculated given the gradient of the layer output (input gradient of a subsequent layer) and input to the layer (output of the previous layer). This again can be parallelized over a layer and minibatch but is sequential from higher layers to earlier layers. Moreover, since the backward pass relies on the outputs of the forward pass all intermediate layer outputs of the forward pass have to be cached for the backward pass which results in a high (GPU) memory usage.
Forward and backward pass take most of the time
So, these two steps take a long time for 1 training iteration, and (depending on your network) high GPU memory usage. But you should read and understand the backpropagation algorithm that basically explains everything.
Moreover, to train a network from scratch, in general, takes lots of iterations because especially in the earlier layers training the parameters is based on gradients that are affected by lots of previous layers, which can result in noisy updates, etc., that do not always push the network parameters in the right direction directly. In contrast, e.g. fine-tuning a pre-trained network on some new task can for example already be done with much less training iterations.

There is no such single hard and slow step in training neural networks ,  forward pass involves large number of matrix multiplications so does backward pass , even though there are highly optimized libraries for matrix multiplications neural networks act on very high dimensional (tensor) multiplications in both forward and backward passes which makes it difficult train . however backward pass would be even more slower or even intractable if we won't use backpropagation in case of large neural networks, since computing derivatives is time-taking.
refer training results in for exact number https://github.com/baidu-research/DeepBench#types-of-operations


text:
Why aren't there neural networks that connect the output of each layer to all next layers?
Why aren't there neural networks that connect the output of each layer to all next layers?
For example, the output of layer 1 would be fed to the input of layers 2, 3, 4, etc. Beyond computational power considerations, wouldn't this be better than only connecting layers 1 and 2, 3 and 4, etc?
Also, wouldn't this solve the vanishing gradient problem?
If computational power is the concern, perhaps you could connect layer 1 only to the next N layers.

Actually, this already exists!
I happened to make a presentation of a paper that talks about this topic. These networks are called DenseNets, which stands for densely connected convolutional networks. Just like in your question, within a dense block, the output of each layer is given as input to all subsequent layers. Put another way, in a normal feed-forward neural network the $l$th layer is a function of the previous output $x_l = H(x_{l-1})$, while in the dense net each layer is a function of all the previous outputs $x_l = H([x_0, x_1, \dots, x_{l-1}])$.

However, since it is a CNN, there is a reduction in the size of the feature maps with each pooling layer, so to keep the dimensions constant, there is an alternation between the dense block and pooling layers.
The results are clear: not only in almost all the tests the accuracy of the dense net is greater than that of the other methods, but they do so using up to 90% fewer parameters, i.e. they have a high efficiency of parameters. Moreover, as suggested by the authors themselves, the improved accuracy can be explained by the shorter connections between the layers, which allow acting during the training phase in a deep supervision fashion, solving the vanishing gradient problem. This is similar to how it was done in other methods, but with a less complicated gradient.
If you're interested you should definitely check out their paper Densely Connected Convolutional Networks (2018).

There is a similar concept called Resnet, that uses "skip connections" to achieve a similar result. The difference is that instead of direct deep connections, the skip connections are summed into the existings the next but one layer. Mathematically this creates a "short-circuit" for gradient flow to earlier layers when training, and also makes it very easy for later layers to learn the identity function, which intuitively means they can learn improvements over identity function.

@NeilSlater Although this is an old question and residual networks are older than dense nets, I think it's worth adding your answer that talks about residual networks.


text:
What is the general procedure to create an AI system that can detect fire in images?
I have no experience with any kind of AI, but I really want to develop a system that can detect fire in images. I think I will need a labelled dataset with labels "fire" or "not fire", but I am not sure how I should proceed and which steps I need to take to develop this system.
So, what is the general procedure to create an AI system that can detect fire in images?
I heard about the Keras library, which could allow us to do this. How could I do this with Keras?

What do you know about programming? How do you get an image on your comnputer?  What operating system does it run ? What image format (PNG, JPEG, ...)?  What books did you read about computer vision ? Wh See also http://refpersys.org/ ....

Many AI libraries allow the ability to feed them an Image, but others have to be modified to allow this.
Disregarding the information above, as it is just a notification, you can do this in many ways. Depending on the library you use, different methods give varying accuracy results.
The real questions are:

How complex do you want the NN to be?
Is this going to be commercial, or private?

For any of the answers above, you will have to find the best method that works for your needs. The more this matters to the world, the more complex it will have to be.