text:
Deep Q-Learning: why don't we use mini-batches during experience reply?
In examples and tutorial about DQN, I've often noticed that during the experience replay (training) phase people tend to use stochastic gradient descent / online learning. (e.g. link1, link2)
# Sample minibatch from the memory
minibatch = random.sample(self.memory, batch_size)
# Extract informations from each memory
for state, action, reward, next_state, done in minibatch:
    # if done, make our target reward
    target = reward
    if not done:
      # predict the future discounted reward
      target = reward + self.gamma * \
               np.amax(self.model.predict(next_state)[0])
    # make the agent to approximately map
    # the current state to future discounted reward
    # We'll call that target_f
    target_f = self.model.predict(state)
    target_f[0][action] = target

Why can't they use mini batches instead?
I'm new to RL, but in deep learning people tends to use mini-batches as they would result in a more stable gradient. Doesn't the same principle apply to RL problems? Is the randomness/noise introduced actually beneficial to the learning process? Am I missing something, or are these sources all wrong?

Note:
Not all the sources rely on stochastic gradient descent: e.g. keras-rl seems to rely on minibatches (https://github.com/keras-rl/keras-rl/blob/master/rl/agents/dqn.py)

DQN "library" implementations that I have seen do use mini-batches to train, and I would generally recommend this, as it usually strikes a reasonable balance between number of weight updates and accuracy of the gradients.
In your first link, and the code excerpt, the sample list is literally called minibatch. However, the developer then goes on to make a series of individual SGD steps per item within that sample.

Why can't they use mini batches instead?

They can, and it works just fine. In my DQN explorations I tend to use a single mini-batch per timestep, and the Adam optimiser to accelerate learning. I also use a mini-batch to calculate TD target values.

I'm new to RL, but in deep learning people tends to use mini-batches as they would result in a more stable gradient. Doesn't the same principle apply to RL problems?

It does, but I don't know if this is the most sample efficient that it could be. I prefer to use mini-batches and more complex optimisers because this runs faster (for a given number of replay training samples per time step), and I can run more time steps of the environment in less time.
It is possible for instance that a replay sample size of 32 per time step, with 4 mini-batch updates of 8 each would be the most sample efficient approach for a specific problem (because it makes more updates from the given samples), but it might not be the most CPU efficient.

Is the randomness/noise introduced actually beneficial to the learning process?

I don't think so, but there are plenty of other tuning issues to consider in RL.

Am I missing something, or are these sources all wrong?

They are not necessarily wrong, but may be CPU inefficient.
More likely in my opinion, your sources are explaining algorithms in principle, and did not want to add too many layers of optimisation, or too many components that may need tuning to work. There is a lot going on in a typical DQN agent, and having many performance optimisations can hide the details that the person is trying to teach.

Thanks for the detailed answer! I'm not sure I understood whether you agreed with me or not though :-)
In minibatches the gradient is estimated over N samples and then the parameters are updated. As a rule of thumb, N is the largest number of examples we can fit in the GPU memory as this provide the best stability of gradient estimates vs computational efficiency trade-off...

An N=1 technically wouldn't be wrong but would be definitely unusual in standard deep learning (personally I'd consider more "online" than "minibatches"). Moreover, the minibatch size in the examples I reported was "20" and not "1", so that would be actually wrong. Am I correct and just overthinking over simple illustrative examples or I'm missing something fundamental in the RL field?

@ent: I am basically agreeing with you. Also pointing out there are two "batch" sizes here that don't have to be the same. In Q learning, you don't necessarily want to base minibatch size on maximum possible GPU capacity, as changes to the NN feed back on itself, there is not much point trying to converge on early Q target values. So there is more tuning to consider. Even with supervised DL you might not want to use maximum batch size, if you are concerned about sample efficiency as opposed to overall training time (this is less of an issue in DL though).


text:
Chess policy network
I am interested in making a simple chess engine using neural networks. I already have a fairly good value network but I can't figure out how to train a policy network. I know that Leela chess zero outputs the probability of any of the about 1800 possible moves. But how do you train such a network? How do you calculate the loss when you  only have the 1 move that was played in the game
to work with?

It may come as a surprise, after learning all about dynamic programming, temporal difference learning, SARSA/Q-learning, to then discover that there is yet another whole dimension to reinforcement learning (on top of choices for on-policy/off-policy, model-based/model-free, bootstrap/monte carlo etc). That is value-based vs policy-based methods. Policy-based methods are often taught after value-based methods, because they are more complex.
You can learn the parameters of a policy function by training it using a policy gradient method. The archetypical policy gradient method is REINFORCE, although that is not very efficient. You may have heard of policy gradient methods developed recently: A3C, A2C, DDPG, TRPO, PPO . . . there are a few.

How do you calculate the loss when you only have the 1 move that was played in the game to work with?

You can pre-train a policy network using supervised learning (perhaps using the moves of winning players in high quality games) - that would use multi-class cross-entropy loss that you may be familiar with from supervised classification problems.
Policy Gradient methods work with a reward summation function defined as expected reward given the distribution of states. If your network parameters are $\theta$, then that may look like this:
$$J(\theta) = \sum_{s \in \mathcal{S}} \rho_{\pi}(s) \sum_{a \in \mathcal{A}} \pi(a|s,\theta)q_{\pi}(s,a)$$
where $\rho_{\pi}(s)$ is the expected proportion of time steps spent in state $s$. There is a way to take a sample gradient of this that can be used for gradient ascent - the derivation is called the Policy Gradient Theorem. It's a bit long to include in this answer, but the upshot is that you can use your sampled single step to generate an approximate gradient towards improving the policy. There are a few variations, but for instance advantage actor critic uses this:
$$\nabla J(\theta) = \hat{A}(s,a)\nabla\text{log}(\pi(a|s,\theta))$$
where $\hat{A}(s,a)$ is your current estimate of the advantage (or $Q(s,a) - V(s)$) for taking a specific action in state s.
The related loss function is
$$\mathcal{L}(\theta) = -A(s,a)\text{log}(\pi(a|s,\theta))$$
The $\text{log}$ function looks like an odd addition, but is just a consequence of adjusting $\nabla J$ to take account of ratios in which actions are taken in the current policy. In fact $\nabla\text{log}(\pi(a|s,\theta)) = \frac{\nabla\pi(a|s,\theta)}{\pi(a|s,\theta)}$ and it may help your intuition to keep it in that form (the $\text{log}$ form is concise and used elsewhere in statistics as the "score function", but it is not necessary for anything specific in RL).
The variations in policy gradients may use other functions than the advantage function, and it is not clear that there is any "best" one. The policy gradient theory basically gives us ways of estimating relative benefits of actions, and allows for any offset to estimated return from $(s,a)$ that does not further depend on the choice of action $a$. So you can use any method for getting an estimated return, and offset it with anything that you think might normalise the updates - common choices for the latter include subtracting average reward, or subtracting the state value function.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How can a genetic algorithm adapt and get better in a changing environment?
I've just started studying genetic algorithms and I'm not able to understand why a genetic algorithm can improve if, at each learning, the 'world' that the population encounters change. For example, in this demo (http://math.hws.edu/eck/js/genetic-algorithm/GA.html), it's pretty clear to me that the eating statistics will improve every year if bunches of grass grow exactly in the same place, but instead they always grow in different positions and I can't figure out how it  can be useful to evaluate (through the fitness function) the obtained eating stats given that the next environment will be different.

There are a couple of ways of dealing with this. Very often, the approach is just to design your representation and operators in a way to account for the fact that the world changes. The idea is to give the algorithm something that can be used to learn general behaviors or solutions rather than specific ones.
Take an example of learning to steer a race car around a track. You want to represent the state of the world and have the GA learn to select an appropriate action. You might choose to represent the state of the world as a vector of $[x, y, v, a]$, where $(x, y)$ is your location on the track, $v$ is your current velocity vector, and $a$ is an acceleration to apply. The fitness function could return how "good" it was to apply that acceleration. If you do this, your algorithm can probably learn to navigate this track, but a different track will be hopeless, as the locations aren't corresponding to the same turn locations on the new track.
However, what if you encode the world as $[s, v, a]$, where instead of an $(x, y)$ pair representing your current position, you have $s$ as a vector of sensor readings? Is there a wall coming up in front of you or is the track starting to bank? Now, your algorithm can learn to be more general. It doesn't need to be the exact track it learned on, because what it's learning is not to brake at a specific point, but to brake when it detects a wall coming up.
I didn't do a lot of digging into the example you linked, but if you run it for a few years, you can see evidence of this. You see agents that appear to have learned to move in lines horizontally or vertically until they encounter a green square, and then they'll stick around and eat in the patches around that square. That behavior is general, because every environment it encounters has lots of blank space with clusters of green. It's not learning "go to square $(20, 30)$". It's learning to move in a pattern until it finds green and then move around that location.
You can do this in a lot of cases where the specific environment can change, but the objective is the same. There are problems where the actual fitness function changes over time, however. For those problems, there are specific techniques to deal with dynamic fitness functions. Generally, this involves doing something to maintain diversity so that your whole population isn't getting stuck on whatever the current "best" looks like. That's a bit more advanced topic though, and I think your question was really more about the former type of problems.

You say "The fitness function could return how "good" it was to apply that acceleration", but, from your example, it's not fully clear how you can know how good it is to apply a certain acceleration. Maybe you can clarify that you can retrieve this information from e.g. the output of that action, such as if the car hits the wall or something? Maybe that's what you mean? In that case, wouldn't this just be equivalent to your second example, where the fitness depends on the observations? So, I think that's not what you mean.

Or maybe are you saying that you have a dataset of arrays $(x, y, v, a)$ for each possible position, $v$, and $a$ for a single track, and you are already given somehow the ground-truth fitness? Is that your assumption? As I said before, I don't think you would need this anyway. You would just need a way to quantify how a certain acceleration (action) is good with respect to an arbitrary $x$ and $y$ (i.e. your example of the fitness function that computes the fitness based on the observations), i.e. I don't fully understand your first example, ie why a fitness based on $x$ and $y$ is not good?

I mean, I don't specifically understand this "but a different track will be hopeless, as the locations aren't corresponding to the same turn locations on the new track.". Why would that be the case, if the fitness is given as a function of $x$, $y$ and $a$? Apart from these things, your answer seems good to me.

If you encode the (x,y) position as part of your state, then the algorithm is going to learn a specific track. If (8, 12) is the beginning of a long straight stretch, then hammering the accelerator is good. If there's a wall at (8,14) on a different track, you're going to crash, because all you've learned is to accelerate straight ahead from that location. That's why you need to use sensor data instead of absolute positions.

I understand that. What I don't understand is: why is it different to encode $(x, y)$ rather than the sensors' readings? Aren't $(x, y)$ also supposed to be sensors' readings? Maybe by sensors' readings you mean not only the location but also if the car has crashed and similar info? So, I guess that, to clarify this answer, you're essentially saying that the position alone is insufficient to act well on the environment and you need some kind of information about the obstacles around the car and the "state" of the car (i.e. has it crashed or not). Is this what you mean?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Neural network architecture for comparison
When someone wants to compare 2 inputs, the most widespread idea is to use a Siamese architecture. Siamese architecture is a very high level idea, and can be customized based on the problem we are required to solve.
Is there any other architecture type to compare 2 inputs ?

Background
I want to use a neural network for comparing 2 documents (semantic textual similarity). Siamese network is one approach, I was wondering if there is more.

A RBM (restricted Boltzmann machine) can be trained to extract document features. The same resulting machine can extract features of two or more documents.  Because documents can be just as easily processed in series using the same machine parameters and CPU (saving the feature results) as documents could be processed in parallel using separate CPUs, the Siamese idea is less of a characteristic of system architecture and more of a characteristic of the process topology. The architectural decision of how to run the process topology on available hardware should usually remain somewhat decoupled.
The comparison operation could be a MLP (multi-layer perceptron) trained to produce the comparison results based on a requisite number example document comparisons.  In such a case, the MLP should be trained with the previously trained RBMs as a front end.  You will need some human labor to produce or extract from available data sources the example document comparison results.  The example results will be need, along with a reference to the pair of RBMs feature extraction results corresponding to the comparison, to train the MLP.
Even further upstream from the RBMs, it is often useful to producing reliable and accurate document processing to pre-process the documents.  Images and text can be delineated so that features can be extracted from the images in one way and from the text in another and then from the sequence of text and images in a third way.  In such process topologies, CNNs (convolution network) typically pre-process the images and LSTM networks typically pre-process the text.  They may then directly feed the RBM or indirectly through another process component.


text:
Am I able to visualize the differentiation in backprop as follows?
I'm wondering if I can visualize the backprop process as follows (please excuse me if I have written something terrible wrong). If the loss function $L$ on a neural network represents the function has the form
$$L = f(g(h(\dots u(v(\dots))))$$
then we can visualize the derivative of $L$ wrt the $i$th function $v$ as
$$\frac{\partial L}{\partial v} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\dots\frac{\partial u}{\partial v}.$$
Am I able to view all neural networks as having a loss function of the form of $L$ given above? That is, am I correct in saying that any neural network is just a function composition and that I can write the partial derivative wrt any parameter as written above (I know I took the partial with respect to the function $v$).
Thanks

Yes, a neural network plus loss function can be viewed as a function composition as you have written, and back propagation is just the chain function repeated. Your equations $L = f(g(h(\dots u(v(\dots))))$ and $\frac{\partial L}{\partial v} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\dots\frac{\partial u}{\partial v}$ are useful for high level intuition.
At some point you need to look at the actual forms of the functions, and that introduces a bit more complexity. For instance, the loss function for a mini-batch can be expressed in terms of the output. This is the equivalent of $L = f(g)$ for a batch or mini-batch with mean squared error loss, where I am using $J$ (for cost) in place of $L$ and the output of the neural network $\hat{y}$ in place of $g$:
$$J = \frac{1}{2N}\sum_{i=1}^{N}(\hat{y}_i - y_i)^2$$
The gradient of $J$ with respect to $\hat{y}$ is equivalent to your first part $\frac{\partial f}{\partial g}$:
$$\nabla_{\hat{y}} J = \frac{1}{N}\sum_{i=1}^{N}(\hat{y}_i - y_i)$$
Many of the functions in a neural network involve sums over terms. They can be expressed as vector and matrix operations, which can make them look simpler, but you still need to have code somewhere that works through all the elements.
There is one thing that the function composition view does not show well. The gradient you want to calculate is $\nabla_{\theta} J$, where $\theta$ represents all the parameters of the neural network (weights and biases). The parameters in each layer are end points of back propagation - they are not functions of anything else, and the chain rule has to stop with them. That means you have a series of "dead ends" - or possibly another way of thinking about it would be that your $\frac{\partial L}{\partial v} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\dots\frac{\partial u}{\partial v}$ is the "trunk" of the algorithm that links layers together, and every couple of steps there is a "branch" to calculate the gradient of the parameters that you want to change.
More concretely, if you have weight parameters in each layer noted as $W^{(n)}$, and two functions for each layer (the sum over weights times inputs, and an activation function) then your example ends up looking like this progression:
$$\frac{\partial L}{\partial W^{(n)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial W^{(n)}}$$
$$\frac{\partial L}{\partial W^{(n-1)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial i}\frac{\partial i}{\partial j}  \frac{\partial j}{\partial W^{(n-1)}}$$
$$\frac{\partial L}{\partial W^{(n-2)}} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial i}\frac{\partial i}{\partial j}\frac{\partial j}{\partial k}\frac{\partial k}{\partial l}  \frac{\partial l}{\partial W^{(n-2)}}$$
. . . this is the same idea but showing the goal of calculating $\nabla_{W} L$ which doesn't fit into a single chain of gradients. Notice the last term on each line does not appear in the next line. However, you can keep all the terms prior to that and re-use them in the next line - this matches the layer-by-layer calculations in many implementations of back propagation.


text:
Neural Network Optimizers in Reinforcement Learning non-well behaved environments
https://stackoverflow.com/questions/36162180/gradient-descent-vs-adagrad-vs-momentum-in-tensorflow
Here, the nice gifs explain how different algorithms approach towards the root. Unfortunately, the environment in the gif is way too simple and real cases have much more complex environments. Also, in reinforcement learning, the solutions should change each moment in a difficult enough environment since things are dynamic.
My question is which optimizer is best for reinforcement learning in such dynamically changing environment? Adadelta should not move beyond local minima so do we have to use SGD or Adadelta with an exploration heuristic? Please let me know in detail your thoughts.

The most commonly-used optimizer in Deep Reinforcement Learning research the past few years is probably ADAM (or its AMSGrad variant, which in most frameworks like keras/tensorflow/pytorch etc. can be used by setting an amsgrad flag to True in the construction of an ADAM optimizer, and I believe is often also already set to True by default). This is a somewhat newer optimizer which isn't included in that visualization you linked to.
RMSProper is also still quite popular, and there may be some arguments that it might be better suited for non-stationary learning problems (which we typically have in RL, our learning targets tend to be non-stationary due to the agent adapting its behaviour over time).
All of these more "fancy" optimizers, typically including some form of momentum term, can in fact be especially useful for optimization in loss "landscapes" that are not smooth. Imagine that your loss landscape looks like a tall mountain, where we'd like our optimizer to "glide" all the way down. Now suppose that the overall trend of a side of the mountain is downwards, but that there are lots of little bumps along the way down, that it's not a smooth mountain. The more fancy optimizers with momentum terms are more likely to "jump" over those bumps than a regular SGD optimizer without momentum terms. This is because, when a little bump is encountered, the local gradient may point the wrong way, but a momentum term "remembers" the overall trend of gliding downwards along the mountain and still points in that same direction for a while.

Amsgrad looks good. I'll check it out. But due to search spaces being too large in neural networks, I think imagining local minimas or even saddle points would be incorrect. We will need to go mathematically and identify things. Momentum's problem is that it goes too much into wrong areas which would ruin performance in a dynamic environment.


text:
Can the same input for a plain neural network be used for a convolutional neural network?
Can the same input for a plain neural network be used for CNNs? Or does the input matrix need to be structured in a different way for CNNs compared to regular NNs?


text:
Using an 'operation ID' as a neural network input
Sorry if this is basic or covered elsewhere, I am just starting here and I wasn't able to find an answer, but I might have not been searching for the right thing. So:
I am training a neural network to predict current draw in a system. There are a number of obvious numerical inputs, like temperature, counting rate, voltage, etc.
The most predictive thing, however, is what operation the system is doing. So like, if it's doing a 'calibration' then the current profile is much different than if it's in 'standby'. I know that I can just use a different network for each operation, but in this case I have a couple hundred different macros defined and I don't want to have 200+ neural networks retrain all the time.
I also know that I can have a digital value as an input, but my understanding is that it has to be either 0/1. Also, the relationship to operation is not at all correlated - so operation 100 is not necessarily more current draw than 99 or less than 101.
So, is there a way to have an operation ID or something factor in, but not have it be in the linear combination mathematically? So, basically, tell the system to do a different training based on ID or something? I'll be using python and scikit-learn.
Thanks!

Great question.
The operational ID is actually a category, or categorical input. Each category requires its own input signal into the neural network. Each signal is actually an activation which is multiplied by a weight and added to all the other weights and inputs. This means that if you use the same input signal for all your operational IDs then 101 is more important than 50. As you have mentioned these inputs are very different, and therefore must have a binary input for each operation id. If you add more IDs then you have to retrain the neural network.
You also mentioned you will be using scikit-learn, which has built in functions for category data. Have a look at this link on the one hot encoder.


text:
Why is the Chinese Room argument such a big deal?
I've been re-reading the Wikipedia article on the Chinese Room argument and I'm... actually quite unimpressed by it. It seems to me to be largely a semantic issue involving the conflation of various meanings of the word "understand". Of course, since people have been arguing about it for 25 years, I doubt very much that I'm right. However... the argument can be thought of as consisting of several layers, each of which I can explain away (to myself, at least).

There is the assumption that being able to understand (interpret a sentence in) a language is a prerequisite to speaking in it.
Let's say that I don't speak a word of Chinese, but I have access to a big dictionary and a grammar table. I could work out what each sentence means, answer it, and then translate that answer back into Chinese, all without speaking Chinese myself. Therefore, being able to interpret (parse) a language is not a prerequisite to speaking it.
(Of course, by the theory of extended cognition I can interpret the language, but we can all agree that the books and lookup tables are simply a source of information and not an algorithm; I'm still the one using them.)
Nevertheless, this task can be removed by a dumb natural language parser and a dictionary, converting Chinese to the set of concepts and relationships encoded in it and vice versa. There is no understanding involved at this stage.
There is the assumption that being able to understand (identify and maintain a train of thought about concepts in) a language is a prerequisite to speaking in it.
We've already optimised away the language, to a set of concepts and relationships between concepts. Now all we need is another lookup table: a sort of verbose dictionary that maps concepts to other concepts and relationships between them. For example, one entry for "computer" might be "performs calculations" and another might be "allows people to play games". An entry for "person" might be "has opinions", and another might be "has possessions". Then an algorithm (yes, I'm introducing one now!) would complete a simple optimisation problem to find a relevant set of concepts and relationships between them, and turn "I like playing computer games" into "What is your favourite game to play on a computer?" or, if it had some entries on "computer games", "Which console do you own?".
The only "understanding" here, apart from the dumb optimisation algorithm, is the knowledge bank. This could conceivably be parsed from Wikipedia, but for a good result it would probably be at least somewhat hand-crafted. Following this would fall down, because this process wouldn't be able to talk about itself.
There is the assumption that being able to understand ("know" how information in affects one's self) a language is a prerequisite to speaking in it.
A set of "opinions" and such associated with the concept "self" could be implemented into the knowledge bank. All meta-cognition could be emulated by ensuring that the knowledge bank had information about cognition in it. However, this program would still just be a mapping from arbitrary inputs to outputs; even if the knowledge bank was mutable (so that it could retain the current topics from sentence to sentence and learn new information) it would still, for example, not react when a sentence is repeated to it verbatim 49 times.
There is the assumption that being able to have effective meta-cognition is a prerequisite to speaking in a language.
Except... there's not. The program described would probably pass the Turing Test. It certainly fulfils the criteria of speaking Chinese. And yet it clearly doesn't think; it's a glorified search engine. It'd probably be able to solve maths problems, would be ignorant of algebra unless somebody taught it to it (in which case, with sufficient teaching, it'd be able to carry out algebraic formulae; Haskell's type system can do this without ever touching a numerical primitive!), and would probably be Turing-complete, and yet wouldn't think. And that's OK.

So why is the Chinese Room argument such a big deal? What have I misinterpreted? This program understands the Chinese language as much as a Python interpreter understands Python, but there is no conscious being to "understand". I don't see the philosophical problem with that.

If you check the Wikipedia article on the argument that you linked, in the History section, you'll note the following statement:

Most of the discussion consists of attempts to refute it.

I think this directly answers the question,

Why is the Chinese Room argument such a big deal?

The Chinese Room Argument was a relatively early attempt in the intermingling of Philosophy and Computer Science to make as concrete an argument on the definition of a "mind", "understanding", and whether or not these can be created with a program. That is, it conjectures that a "mind" cannot.
It's famous in part because the concreteness of this argument means that a counter-argument can be made in kind. That is, the argument spawned a large quantity of refutations, attacking the argument from a large number of vectors, outlined later in the argument.
Put another way,

The Chinese Room Argument is such a big deal because of how many people in the field have found value in refuting it.

Incidentally, your post provides more evidence to this point.

The Chinese room argument is such a big deal because it takes the concept of the Turing machine and Turing's conception of the electronic digital computer (so-called) as a practical version of the universal Turing machine, and shows that the resulting concept of machine computation does not allow the creation of an internal semantics (knowledge). Explaining why this is the case is is a bit of a fraught task, though.


text:
What is the derivative function used in backpropagration?
I'm learning AI, but this confuses me. The derivative function used in backpropagation is the derivative of activation function or the derivative of loss function?
These terms are confusing: derivative of act. function, partial derivative wrt. loss function??
I'm still not getting it correct.

Overview
The derivatives of functions are used to determine what changes to input parameters correspond to what desired change in output for any given point in the forward propagation and cost, loss, or error evaluation &mdash whatever it is conceptually the learning process is attempting to minimize. This is the conceptual and algebraic inverse of maximizing valuation, yield, or accuracy.
Back-propagation estimates the next best step toward the objective quantified in the cost function in a search. The result of the search is a set of parameter matrices, each element of which represents what is sometimes called a connection weight. The improvement of the values of the elements in the pursuit of minimal cost is artificial networking's basic approach to learning.
Each step is an estimation because the cost function is a finite difference, where as the partial derivatives express the slope of a hyper-plane normal to surfaces that represent functions that comprise forward propagation. The goal is to set up circumstances so that successive approximations approach the ideal represented by minimization of the cost function.
Back-propagation Theory
Back-propagation is a scheme for distribution of a correction signal arising from cost evaluation after each sample or mini-batch of them. With a form of Einsteinian notation, the current convention for distributive, incremental parameter improvement can be expressed concisely.
$$ \Delta P = \dfrac {c(\vec{o}, \vec{\ell}) \; \alpha} {\big[ \prod^+ \! P \big] \; \big[ \prod^+ \!a'(\vec{s} \, P + \vec{z}) \big] \; \big[ c'(\vec{o}, \vec{\ell}) \big]} $$
The plus sign in $\prod^+\!$ designates that the factors multiplied must be downstream in the forward signal flow from the parameter matrix being updated.
In sentence form, $\Delta P$ at any layer shall be the quotient of cost function $c$ (given label vector $\vec{\ell}$ and network output signal $\vec{o}$), attenuated by learning rate $\alpha$, over the product of all the derivatives leading up to the cost evaluation. The multiplication of these derivatives arise through the recursive application of the chain rule.
It is because the chain rule is a core method for feedback signal evaluation that partial derivatives must be used. All variables must be bound except for one dependent and one independent variable for the chain rule to apply.
The derivatives include three types.

All layer input factors, the weights in the parameter matrix used to attenuate the signal during forward propagation, which are equal to the derivatives of those signal paths
All the derivatives of activation functions $a$ evaluated at the sum of the matrix vector product of the parameters and signal at that layer plus the bias vector
The derivative of the cost function $c$ evaluated at the current output value $\vec{o}$ with the label $\vec{\ell}$

Answer to the Question
Note that, as a consequence of the above, the derivatives of both the cost (or loss or error) function and any activation functions are necessary.
Redundant Operation Removal for an Efficient Algorithm Design
Actual back propagation algorithms save computing resources and time using three techniques.

Temporary storage of the value used for evaluation of the derivative (since it was already calculated during forward propagation)
Temporary storage of products to avoid redundant multiplication operations (a form of reverse mode automatic differentiation)
Use of reciprocals because division is more costly than multiplication at a hardware level

In addition to these practical principles of algorithm design, other algorithm features arise from extensions of basic back-propagation. Mini-batch SGD (stochastic gradient descent) applies averaging to improve convergence reliability and accuracy in most cases, provided hyper-parameters and initial parameter states are well chosen. Gradual reduction of learning rates, momentum, and various other techniques are often used to further improve outcomes in deeper artificial networks.


text:
SARSA won't work for linear function approximator for MountainCar-v0 in OpenAI environment. What are the possible causes?
I am learning Reinforcement Learning from the lectures from David Silver. I finished lecture 6 and went on to try SARSA with linear function approximator for MountainCar-v0 environment from OpenAI.
A brief explanation of the MountainCar-v0 environment. The state is denoted by two features, position, and velocity. There are three actions for each state, accelerate forwards, don't accelerate, accelerate backward. The goal of the agent is to learn how to climb a mountain. The engine of the car is not strong enough to power directly to the top. So speed has to be built up by oscillating in the cliff.
I have used a linear function approximator, written by myself. I am attaching my code here for reference :-
    class LinearFunctionApproximator:
      ''' A function approximator must have the following methods:-
          constructor with num_states and num_actions
          get_q_value
          get_action
          fit '''

      def __init__(self, num_states, num_actions):
        self.weights = np.zeros((num_states, num_actions))
        self.num_states = num_states
        self.num_actions = num_actions

      def get_q_value(self, state, action):
        return np.dot( np.transpose(self.weights), np.asarray(state) )[action]

      def get_action(self, state, eps):
        return randint(0, self.num_actions-1) if uniform(0, 1) < eps else np.argmax( np.dot(np.transpose(self.weights), np.asarray(state)) )

      def fit(self, transitions, eps, gamma, learning_rate):
        ''' Every transition in transitions should be of type (state, action, reward, next_state) '''
        gradient = np.zeros_like(self.weights)
        for (state, action, reward, next_state) in transitions:
          next_action = self.get_action(next_state, eps)
          g_target = reward + gamma * self.get_q_value(next_state, next_action)
          g_predicted = self.get_q_value(state, action)
          gradient[:, action] += learning_rate * (g_target - g_predicted) * np.asarray(state)

        gradient /= len(transitions)
        self.weights += gradient

I have tested the gradient descent, and it works as expected. After every epoch, the mean squared error between current estimate of Q and TD-target reduces as expected.
Here is my code for SARSA :-
def SARSA(env, function_approximator, num_episodes=1000, eps=0.1, gamma=0.95, learning_rate=0.1, logging=False):

  for episode in range(num_episodes):
    transitions = []

    state = env.reset()
    done = False

    while not done:
      action = function_approximator.get_action(state, eps)
      next_state, reward, done, info = env.step(action)
      transitions.append( (state, action, reward, next_state) )
      state = next_state

    for i in range(10):
      function_approximator.fit(transitions[::-1], eps, gamma, learning_rate)

    if logging:
      print('Episode', episode, ':', end=' ')
      run_episode(env, function_approximator, eps, render=False, logging=True)

Basically, for every episode, I fit the linear function approximator to the current TD-target. I have also tried running fit just once per episode, but that also does not yield any winning episode. Fitting 10 times ensures that I am actually making some progress towards the TD-target, and also not overfitting.
However, after running over 5000 episodes, I do not get a single episode where the reward is greater than -200. Eventually, the algorithm choses one action, and somehow the Q-value of other actions is always lesser than this action.
# Now, let's see how the trained model does
env = gym.make('MountainCar-v0')
num_states = 2
num_actions = env.action_space.n

function_approximator = LinearFunctionApproximator(num_states, num_actions)

num_episodes = 2000
eps = 0
SARSA(env, function_approximator, num_episodes=num_episodes, eps=eps, logging=True)

I want to be more clear about this. Say action 2 is the one which is the action which gets selected always after say 1000 episodes. Action 0 and action 1 have somehow, for all states, have their Q-values reduced to a level which is never reached by action 2. So for a particular state, action 0 and action 1 may have Q-values of -69 and -69.2. The Q-value of action 2 will never drop below -65, even after running the 5000 episodes.

On doing some research on why this problem might be occurring, I delved into some statistics of the environment. Interestingly, after a small number of episodes (~20), the agent always chooses to take only one action (this has been mentioned in the question too). Also, the Q values of the state-action pairs do not change a lot after just about 20 episodes. Same is the case for policy of environment, as may be expected.
The problem is that, although all the individual updates are being done correctly, the following scenario occurs.
The update equation used is the following :-
Q(s1, a1) <- r1 + gamma * Q(s2, a2)

Now, any update to function approximator means that the Q value changes not just for the updated (state, action) pair, but for all (state, action) pairs. How much it changes for any specific (state, action) pairs is another issue. Now, since our function approximator is altered, next time we use the previous equation for updating any other state, we use the following equation :-
Q(s3, a3) <- r3 + gamma * Q(s4, a4)

But since Q has itself been changed, the target value of the record 3 changes. This is not desirable. Over time, all the changes cancel each other and Q value remains roughly the same.
By using something known as the target function approximator(target network), we can maintain an older version of function approximator, which is used to get the Q-value of next state-action pair while the time of update. This helps avoid the problem and can be used to solve the environment.

One possible cause of SARSA with a linear approximator failing to learn a good policy for the Mountain Car problem is that you are not encoding the environment state, the position and velocity.
In David Silver's Lecture 6 that you mention in your question, he describes the state encoding, which in this case is a tile coding, here: https://youtu.be/UoPei5o4fps?t=3626.
This coarse coding is described in more detail in the Sutton and Barto book, Reinforcement Learning: An Introduction (http://incompleteideas.net/book/the-book.html), in Chapter 9, where different state encodings for linear approximators are described, and Chapter 10, where the Mountain Car problem is discussed.
In my experience, using a tile coding worked in combination with a linear function to solve the Mountain Car task. Using no encoding did not work.
The following plot shows the max q-values (the policy) when using Sarsa with no encoding. The points are colored with the corresponding action of the max q-value to more clearly show the policy the agent will follow.

This planar policy produces an average score of -200.0 per episode, averaged over 100 episodes.
Contrast that policy with the following graph of q-values produced by using Sarsa in combination with a tile encoding.

The policy of this agent scores -102.9 per episode, averaged over 100 episodes. It might be hard to see in the graph, but it has the characteristic "spiral" of the learned graph from Figure 10.1 of the Sutton and Barto book.

That is true. David Silver does mention a coarse coding approach. You mentioned that tile coding worked nicely for you. I would like to ask you if you used SARSA, or Q-Learning. I mean did you make changes in-place, or did you keep a copy separately for calculating Q-value of next state while updating?

I used Q-Learning (but I did not use a separate target network). I haven't tried SARSA yet. If you want to see all the messy details of my implementation, take a look at https://github.com/todddeluca/learning_reinforcement_learning/tree/master/exp20181220.  Cheers.


text:
Feed forward neural network using numpy for IRIS dataset
I tried to build a neural network for working on IRIS dataset using only numpy after reading an article (link: https://iamtrask.github.io/2015/07/12/basic-python-network/).
I tried to search the internet but everyone was using ml libraries and found no solution using just numpy. I tried to add different hidden layers to my feed forward neural network still it wasn't converging. I tried to use backpropagation. I used sigmoid and also relu neither of which was successful.
Can someone please give me the code which will work on IRIS dataset and built only using feed forward neural networks and numpy as the only library or if it is not possible to built such a thing with these constraints then please let me know what goes wrong with these constraints.
Also tell me will it be possible to create a neural network to predict values of a matrix multiplication i.e. if we have A * B = C with matrix A as input and C as output, can we acheive substantial amount of accuracy with feed forward neural networks here?.

You can not use the ready code directly without any manipulation. Because every piece of code is written for specific datasets. In the article that you mentioned, the writer created a small dataset and then he creates an ANN architecture for it. If you want to build an ANN based on Iris dataset, you should think and create an architecture on paper maybe before coding. You should understand the Iris dataset first, I mean you should understand what is going to be your input shapes, and also the shape of weights are related to inputs' shape. After understanding these you should decide the number of layer of your Neural Network. And that tutorial that you followed is explaining all of these clearly. Do not skip and understand everything steps by step.
If you want a build ANN architecture by yourself, you must understand every mathematics behind that. (forward and backward propagation, loss and cost, gradient descent) If you do not want to deal with all of these maths, you can use a library such as Keras, then you can create an ANN easily.
See this as a source, https://ml-cheatsheet.readthedocs.io/en/latest/
And you see my Kaggle kernel, I create this network with details and only using numpy. Also I used Keras on another section. You can check; https://www.kaggle.com/erdemuysal/gender-recognition-with-lr-and-ann


text:
Can we implement GAN (Generative adversarial neural networks) for classication problem like Fraud detecion?
Problem: Fraud detection
Task : classifying transactions as either fraud/ Non-fraud using GAN

I am working on "Fraud detection using Deep learning". The dataset for that is "Credit card frauds" from kaggle. I have to explore GAN for this problem. But unlike the algorithm of GAN, I don't want to generate data from the generator. I don't want both generator and discriminator to play a minimax game in order to achieve their objective function. I have balanced the dataset using SMOTE. Now, what if I only implement discriminator for it? Would it be a GAN ?

Generative networks was the first popular class of topologies that had compound feedback, corrective signaling at more than one layer.  In popular network designs such as MLPs (multilayer perceptrons), CNNs (convolutional neural networks), and LSTM (long short term memory) networks, the backpropagation is a single layer mechanism.  The mechanism distributes corrective a signal creating a closed loop that ideally converges to an optimal network behavior.
The speed, accuracy, and reliability of convergence of networks in an AI system depend on a number of engineering choices at a lower level of design and development.

Selection of algorithm variants and extensions
Selection of data set
Drawing of the training samples
Data normalization
Initialization
Assignment of learning rate or learning rate policy
Other hyper-parameters

Generative networks are designed to achieve a higher level balance between two networks, each of which have their own backpropagation, and thus have a compound feedback topology.
Considering the original GAN approach in the context of fraud detection, a discriminative network $D$, in a way, detects fraudulence originating from the generative network $G$, and, in a way, $G$ learns authenticity from the corrective signal provided by $D$, even if the authenticity learned is superficial.  However, in the case of credit card fraud detection from a numerical set of data, the fraudulence is not internal to the system to be designed as in the case of GAN.  Without both the $G$ and $D$ networks, the design is not a generative network.
Since the comment stated, "I have to explore GAN for this problem," the key question is this.

If this system is to benefit from a GAN, what would the GAN generate that could contribute to the solution?

If the answer is nothing, then a GAN is not the correct approach.  A discriminative network without a generative network is not a GAN.  If this question is from an assignment to train a network using a static and already existing data set, a MLP may suffice, depending on the expectations of the teaching staff.  If that is the case, the goal will be to analyze the data set using statistical tools to determine if it is already normalized well enough to lead to excellent convergence accuracy, speed, and reliability.  If normalization is indicated, find ways to present the data to the input layer of the MLP in such a way that each input is relevant and has a well distributed range of values across the data type.
Ensure there is no data loss in this process, and remove redundancy where it can easily be removed.  By redundancy removal is meant, if one of the dimensions of data have only four possible values of between 10 and 1,000 bytes, encode the four possible values in two bits because to do so removes unnecessary burden from the training.
Similarly, the output layer must be chosen to produce an encoded set of possible output values and the decoding must be written to produce usable output.  In this case, it appears that the labels are, Fraudulent and Authentic.  Which means that the last layer would be a single binary threshold cell.
For this simple MLP approach and for many of the approaches mentioned next, there are examples available in Python, Java, C++ and other languages from which implementation can be initiated.  If the design must stand up to a more realistic fraud detection scenario, either for credit cards, junk mail, banking, log-in, or communications those experienced with fraud detection know that the above approach is entirely insufficient except as a starting point for learning.  In such a case, these considerations further complicate the design of the solution.

Fraudulence and authenticity is not necessarily sufficient information for investigators.
    The likelihood of fraudulence and the reason why fraudulence is suspected is also pertinent for end users of production output,
    so that security holes can be blocked and investigation may be directed toward prosecution or some other set of remedies.
Fraudulence is usually a security breach phenomena that involves continuous improvement in the attacks until the attackers
    are located and their work is interrupted by arrest.
    What initially works as fraud detection may not work a week, a day, an hour, or a minute later.
Fraud detection, if it to be useful in a real financial context, must be part of a fraud prevention system,
    and training on a static data set may be meaningless in that larger usage environment.

In AI systems that work in real production environments, it may be worth investigating using an LSTM network in combination with a reinforcement learner.  Such may be over a beginner's level of design and implementation experience, but using either an LSTM network or a basic Q-learning approach instead of both of them together may be a reasonable starting point for dedicated students.  If the answer to the question about what a GAN could generate that would be of use must be yes, the following ideas may be helpful.

A generative network could generate transaction data with a distribution that resembles authentic transactions, and such has been successful in the past.
A generative network could generate transaction data with a distribution that resembles attacks on an account by a fraudulent user. That has been done too.
A generative network could theoretically generate hacker agents, although this may not have yet been done.
A generative network could be used to generate patterns of authentic transactions based on some authenticity measure other than the data set mentioned, and the trained discriminative network could then be employed in some way in a fraud detection system in production, although it is unclear what advantage would be gained.

The reason a GAN was suggested would need to be clarified before this answer could provide a more exact direction for AI system topology and project approach.

Yes, although the traditional technique is to use Anomaly Detection for this sort of problem. The reason for that is that the space of possible frauds is very much larger than the space of legitimate transactions and so it is difficult to model the former. The usual idea is to thoroughly explore the smaller space of legitimate transactions and flag as fraudulent anything outside of that space.
Using the GAN method could be interesting as it might identify novel fraud techniques, and those are always useful to know.

What if I don't want to implement generator and only want to implement discriminator for the above mentioned problem in GAN as I have balanced dataset through SMOTE. Is it possible and Can it be done?

Yes. The discriminator in a GAN is just a binary classifier so you can use it to check for fraud / not fraud.

Which neural network to be used as a discriminator for best possible results?

What sort of data are you working with?

Numerical data. "Credit card Fraud detection " dataset from kaggle.

@Nehasoni OK, a load of nicely normalised numerical data, I would just start with a sequential network with a single hidden layer and see how that pans out. Nothing fancy at first. You might find it useful to look through some of the other Kernels on Kaggle that others have published for that dataset. They are a great resource for improving your knowledge and skills.


text:
Beginner - Object classification data in a neural network
Imagine I wish to classify images of digits from 0-9.
Let's say I have trained the network to recognise '1'.
If I were to train the same network to recognise '2', wouldn't the backpropagation process mess up the weights and biases for '1'?
Or do programs like Tensorflow allocate a new layer of neural network for different object classification? Thanks.

The answer is yes, retraining a neural network on a new dataset will alter its internal state such that it would no longer give the same output as before (in your case 'messing it up').
There are techniques to allow you to reuse sections of trained networks for other problems (this is called 'Transfer Learning'). Transfer Learning involves freezing the weights/biases for parts of the already trained network while adding extra new layers to the end of it and training those (with Back-propagation) for your new problem. For your digit recognition problem you would normally just train the network on all the digits at the same time.
Tensorflow is lower level than the idea of neural networks and layers. It is a library that allows you to write mathematical pipelines that implement the tensor calculations required to build a neural network. To do the sort of thing that you mentioned you would use a higher level library such as Keras (https://www.tensorflow.org/guide/keras) which now comes conveniently bundled with Tensorflow.


text:
Should I apply ReLU to non negative output?
Suppose I want to predict the position of a sensor based on its reading.
I can first predict the unit vector and predict the distance to be multiplied to this vector.
And I know that distance will never be negative because all the negative parts are inside unit vector already.
Should I apply ReLU to the distance before multiplying it to the unit vector?
I'm thinking that this can be helpful to eliminate the network from needing too much training data by restricting the output ranges the network could give. But I also think that it could make the learning slower when the ReLU unit dies (value=0) so the gradient doesn't flow properly somehow.

Applying a ReLU activation function to a non-negative number is the same as applying the identity activation function.
What you may wish to do is apply $\ln(|\vec{x}|)$, $\arctan(x[0], x[1])$, ..., $\arctan(x[0], x[N-1])$, where $N$ is the number of dimensions to $\vec{x}$, and pass this directly into a parameter matrix multiplication (which involves deliberately using an identity activation in the input layer in some frameworks). This typically improves speed, accuracy, and reliability of convergence when the magnitudes exhibit a distribution close to an exponential of Gaussian and the directional components are near Gaussian when expressed in radians.


text:
Does software remain even when hardware is demolished?
For example, if I constructed a neural network and the computer running it where to be demolished, is the information/program of the neural network still an existent entity within or outside the remnants of the hardware?

Neural networks are just software. Software is just one form of data in the Von Neumann Architecture.
Most of the data is stored on the disk. There are two common types of disks:

HDD: An actual disk which can be magnetified is spinning.
SSD: Solid-state drive

If the part where the data is stored is destroyed, the data is destroyed.
Of course, the data can be loaded into memory, but the principle is the same: If the part where the data is stored is destroyed, the data is destroyed.
Data can also be partially be destroyed / corrupted. If you have systems like RAID or use error correcting codes then you can defend youself against data loss.


text:
Why do neural nets and machine learning tend to work well with MCTS, but not with regular Minimax game-playing AI?
I've often heard MCTS grouped together with neural nets and machine learning. From what I gather, MCTS uses a refined intuition (from maching learning) to evaluate positions. This allows it to better guess which moves are worth playing out more.
But I've almost never heard of using machine learning for Minimax+alpha-beta engines. Couldn't machine learning be used for the Engine to better guess which move is best, and then look at that move's subtree first? A major optimization of the minimax algorithm is move-ordering, and this seems like a good way to accomplish that.

It just does.
Take a look at this post explaining how MCTS works.

In both Alpha Go Lee and Alpha Zero the tree traversal follows the nodes that maximize the following UCT variant:
\begin{equation}
UCT(v_i,v) = \frac{Q(v_i)}{N(v_i)} + cP(v_i, v)\sqrt{\frac{N(v)}{1+N(v_i)}}
\end{equation}
where P(vi,v)
  is prior probability of the move (transition from v to vi ), its value comes from the output of deep neural network called Policy Network . Policy Network is a function that consumes game state and produce probability distribution over possible moves.

As you can see the Policy Network(it is actually just one neural network for both the value and the policy) is used to guide the search tree. Not all possible moves are explored. Also during the learning phase the policy network uses the "visit count" of the MCTS Nodes to learn. Moves that were explored more are the better ones. The state-of-the-art chess engine Stockfish evaluates about 1000 times more positions per second than Alpha Zero. It relies on exploring "most of" the possible positions. Calculating the score using heuristics is much faster than using Alpha Zero's 19 layer residual network. If Google were to use minimax then they wouldn't be able to look very far ahead. Alpha Zero explores about 70K moves per second which would be only a couple ply deep. MCTS allows it to explore only the more promising moves and simulate those positions.


text:
LSTM language model not working
I am trying to use a Keras LSTM neural network for character level language modelling. As the input, I give it the last 50 characters and it has to output the next one. It has 3 layers of 400 neurons each. For the training data, I am using 'War of The Worlds' by H.G. Wells which adds up to 269639 training samples and 67410 validation samples.
After 7 epochs the validation accuracy has reached 35.1% and the validation loss has reached 2.31. However, after being fed the first sentence of war of the worlds to start it outputs:

the the the the the the the the the the the the the the the the...

I'm not sure where I'm going wrong; I don't want it to overfit and output passages straight from the training data but I also don't want it to just output 'the' repeatedly. I'm really at a loss as to what I should do to improve it.
Any help would be greatly appreciated. Thanks!


text:
Smaller interest area for images than the size of the image in classification neural networks
I have the following binary classification problem, my labeled dataset contains images 96x96 px. Now in every image the interest area is of size 32x32 px in the center of the image, and the images are labeled based on that 32x32 px area. If whatever i am trying to detect is in the outer region of the 32x32 px area the label of that image is not affected.
The problem here is that if i use the whole image when traing, my model will not learn that the interest area is only in the center of the image but on the other hand if i crop the images to be of size 32x32 i am loosing a lot of information which can help the model to train on. I found out that i am getting the best results if i crop the images to be of size 64x64 (kind of a trade of).
Now going to the test set, for the test set it doesn't make sense to use this trade of because the model is not learning anything anyways so i would rather crop the test images to 32x32 but then the test set and train set sizes are not the same.
Has anyone came across this problem before? Can i just pad the test images to the size of the train images? is this a good way to go?

"The problem here is that if i use the whole image when training, my model will not learn that the interest area is only in the center of the image" Yes, it will, if you have enough samples and they are all classified based on this area only.
"but on the other hand if i crop the images to be of size 32x32 i am loosing a lot of information which can help the model to train on" . I don't understand what information you are loosing if you only care about the center 32x32 area.
In the end, you need to ask yourself how will the test input look like and work based on that. If you only care about an area of 32x32 which is always is (and will be) at the center of all images, ignore everything and train and evaluate based just on that area (why add extra unnecessary calculations to the model?). If the size and location of the area might change, don't crop the image.
Bottom line, the test set and the train set need to be of the nature (shape) for the model to work properly.


text:
Chatbots triggering emotions
I’m a researcher and I’m currently conducting a research project. I will conduct a study where I would like to trigger different emotions using chatbots on a smartphone (e.g. on Facebook Messenger).
Are there any existing chatbots which are able to trigger different emotions intentionally (also negative ones)?

Emotions can of course be triggered by lots of different things. I think the most rich source could well be socialbots like Mitsuku.com and Zo.ai -- Steve Worswick is the owner of Mitsuku and may be interested in helping you by doing (appropriately filtered) chat log queries. You can get him on Twitter at @Mitsuku.

Thanks a lot. I have dropped some lines to steve. I also had a look at Misuku and Zo. They are very interesting and pretty intelligent but if a participant is using them for 30 minutes I think not a wide range of emotions is triggered (most of the time only positive emotions I think).

I have played around with Mitsuku and Zo. These bots are mostly triggering positive emotions. Do you know bots which are nasty or confusing (writing garbage)?

@machinery Mitsuku will get quite aggressive if you're agressive to it. If you want a mirror, check out replika.ai.

Do you know other socialbots like Mitsuku and Zo and cleverbot? I also like cleverbot because it can get pretty confusing and surprising discussions. Unfortuantely, cleverbot is not available in skype or facebook messenger.

I was supremely unimpressed with Cleverbot unfortunately -- nothing I did with it was really coherent. But the "surprise factor" yes sometimes people like that, perhaps for entertainment.
Replika.ai is a good one too. I typically talk about Jessie too but it doesn't appear to have been kept online https://www.facebook.com/JessieHumani


text:
How do biological neurons weights get initialized?
When trying to map artificial neuronal models to biological facts it was not possible to find an answer regarding the biological justification of randomly initializing the weights.
Perhaps this is not yet known from our current understanding of biological neurons?

In short
I mentioned in another post, how the Artificial Neural Network (ANN) weights are a relatively crude abstraction of connections between neurons in the brain. Similarly, the random weight initialization step in ANNs is a simple procedure that abstracts the complexity of central nervous system development and synaptogenesis.
A bit more detail (with the most relevant parts italicized below)
The neocortex (one of its columns, more specifically) is a region of the brain that somewhat resembles an ANN. It has a laminar structure with layers that receive and send axons from other brain regions. Those layers can be viewed as "input" and "output" layers of an ANN (axons "send" signals, dendrites "receive"). Other layers are intermediate-processing layers and can be viewed as the ANN "hidden" layers.
When building an ANN, the programmer can set the number of layers and the number of units in each layer. In the neocortex, the number of layers and layer cell counts are determined mostly by genes (however, see: Human echolocation for an example of post-birth brain plasticity). Chemical cues guide the positions of the cell bodies and create the laminar structure. They also seem to guide long term axonal connections between distant brain regions. The cells then sprout dendrites in certain characteristic "tree-like" patterns (see: NeuroMorpho.org for examples). The dendrites will then form synapses with axons or other cell bodies they encounter along the way, generally based on the encountered cell type.
This last phase is probably the most analogous to the idea of random weight initialization in ANNs. Based on where the cell is positioned and its type, the encountered other neurons will be somewhat random and so will the connections to them. These connections are probably not going to be very strong initially but will have room to get stronger during learning (probably analogous to initial random weights between 0 and ~0.1, with 1 being the strongest possible connection). Furthermore, most cells are either inhibitory or excitatory (analogous to negative and positive weights).
Keep in mind this randomization process has a heavy spatial component in real brains. The neurons are small and so they will make these connections to nearby neurons that are 10-200 microns away. The long-distance connections between brain regions are mostly "programmed-in" via genes. In most ANNs, there is generally no distance-based aspect to the initialization of connection weights (although convolutional ANNs implicitly perform something like distance-based wiring by using the sliding window).
There is also the synaptic pruning phenomenon, which might be analogous to creating many low weight connections in an ANN initially (birth), training it for some number of epochs (adolescence), and then removing most low-weight connections (consolidation in adulthood).

Do you by chance also have some hints on https://ai.stackexchange.com/questions/10416/is-the-thatcher-face-illusion-only-limited-to-face-recognition I haven't received any answer yet.

I am not an DL expert but these are my short thoughts on it:
I think this is because it is believed (from an information theoretic point of view) to be the good way to avoid that the network falls into some wired state from beginning on. Remember: DNNs are nonlinear approximators for continuous functions. So they have some storage capacity to learn an amount of n function to map from input to output. When you look on topic like data leakage you will see that NNs quickly try to cheat you if they can :D. The optimization applied during training will heavily be affected by the init state. So starting with an random initialization at least avoids that your neurons do all the same at the beginning etc.
Biological reasoning:
From the viewpoint of a neurobiologist I can recommend you to read Hebbian rule and how neural systems work (eg. google how neurons find targets) in general and then to compare it to what is known about how dendrite cells in the cerebrum develop their interconnections in the first 3 years after birth. In summary there are behavioral patterns in nature which could look similar, inspiring and even reasonable. But, I would say the reason why this random init. is recommend is backed by mathematical and information theoretical assumptions rather then pure biological arguments.


text:
Is input normalization built-in into mammals sensory neurons?
The spectrum of human sensory inputs seems to fall within certain ranges suggesting normalization is built-in into biological NNs?
It also adapts to circumstantial conditions, e.g. people living in a city with certain factory smell eventually don't perceive the smell anymore, at least consciously (within working memory) / it adapts to a new baseline?

Yes, for many sensory inputs there is indeed something similar to normalization. But its not rally the same as in classical data analytics compared to what eg min/max normalization does or other technics.
Lets look on some examples and considerations:

mammals don't perceive heat or loudness in a linear way. This is because already many sensory receptors have chemical / physical limits. Double decibels will not perceived with double intensity. Inside your ear, the small hammer and abil will brace to protect you. --> its like normalization with logarithmic effects applied.
heat perception is more like a difference integration than a absolute temperature measurement. Its measured via H+ ions flow in mitochondria in the cell (if i recall correctly)
On the neuronal side gradual signals in the dendrites (analog signal) sum up gradually to later form an spike at the axon hill. where in turn a fire frequency is then encoded - the maximum frequency of this serves as a a natural maximum limit. I remember that grasshoppers increase axon fire frequency when objects started covering more ommatidial area on their "eye". The more of their "eyes" are covered by the shadow the more input on the neuron --> higher fire rate.
a lot of sensory input is post processed in higher cerebral areas. Eg. compared to what is expect able and heuristics are applied to compare a signal with former events.
when doing computational data analysis we may want go for accuracy and maximum comparability. Mostly on all data that could be available. --> eg. with respect to properties of a standard normal distribution. Hence we put some effort to be accurate and know the true parameters, remove outliers and so on --> big data comes into play here.
Nature in contrast strives often for efficiency with the means of reaching the minimal required with minimal resources.

Summary:
Compared to normalization in an analytical sense (eg. mean, min-max or other feature normalization techniques), nature is often only interested in the current difference between stimuli. And this only within some relevant range. Other information is not integrated. And normalization with the goal of making measurement points comparable only happens within this range of the mapping function provided by the sensor/neuron/receptor whatever!
So this should also answer your question about, why you are not smelling something in the city after a while any more. However, this for sure happens at higher cerebral regions (it might also be that your smell receptors saturate) its the same principle. Your consciousness just saves energy by not concentrating on something that is anyway not changing.
If you want to read more have a look here: https://en.wikipedia.org/wiki/Weber%E2%80%93Fechner_law

@DuttaA, I think within the range of the mapping its comparable to normalisation. Outside of the range its just dropping/ squashing information. There is this law by weber and fechner: "https://en.wikipedia.org/wiki/Weber–Fechner_law". Just by intuitition you will be able to say that silent and loud are equidistnace to louod and louder but you can not judge on extreme silent or loud.

@LeoGallucci the link gives 404


text:
Pixel-Level Detection of Each Object of the Same Class In an Image
I have source data that can be represented as a 2D image of many similar curves. They may oftentimes cross over one another, so regions of interest will overlap.
My goal is to implement a neural network solution to identify each instance or the curves and the pixels that are associated with each instance.
(Each image is simple in its representation of the data. A pixel in the image is either a point on one of these curves or it is empty. So the image is represented by one or zero at each pixel. For training purposes, I have labels for every pixel, and I have about 150,000 images. The information in the images can be noisy in that there may be omissions of points and point locations are quantized due to measurement limitations and preprocessing for the image preparation.)
I started looking into what semantic segmentation can do for me, but since all of the instances are of the same class, distinguished mainly by their location in the image, I don't think semantic segmentation is the type of processing I would want to perform. (Am I wrong?)
I am very interested in seeing how a neural network will work on this problem to separate each instance.
My question is this: what is the terminology that describes the process I'm looking for? (How can I effectively research for this problem?) Is this an extension of semantic segmentation or is it referred to some other way?

What you want to do is instance segmentation on a pixel level.
I can point you two different way:

MaskRCNN, which in a nutshell, uses Region Proposal Networks then segment in the proposed regions.
Some other works which don't use Region Proposals rather, regress the pixels of the image in an embeding space to be able to apply some clustering algorithm. Here is a link to a good blog post about it: https://towardsdatascience.com/instance-embedding-instance-segmentation-without-proposals-31946a7c53e1

Thanks. This looks like exactly the path I want to follow in my investigation."Instance segmentation" looks to be the right terminology to begin searching. The papers look very helpful.

Also look for YOLOv3 which is a single shot detector.


text:
How can a neural network learn when the derivative of the activation function is 0?
Imagine that I have an artificial neural network with a single hidden layer and that I am using ReLU as my activating function.
If by change I initialize my bias and my weights in such a form that:
$$
X * W + B < 0
$$
for every input x in X then the partial derivate of the Loss function with respect to W will always be 0!
In a setup like the above where the derivate is 0 is it true that an NN won´t learn anything?
If true (the NN won´t learn anything) can I also assume that once the gradient reaches the value 0 for a given weight, that weight won´t ever be updated?

In a setup like the above where the derivat[iv]e is 0 is it true that an NN won´t learn anything?

There are a couple of adjustments to gradients that might apply if you do this in a standard framework:

Momentum may cause weights to continue changing if any recent ones were non-zero. This is typically implemented as a rolling mean of recent gradients.
Weight decay (aka L2 weight regularisation) is often implemented as an additional gradient term and may adjust weights down even in the absence of signals from prediction errors.

If either of these extensions to basic gradient descent are active, or anything similar, then it is possible for the neural network to move out of the stationary zone that you have created after a few steps and then continue learning.
Otherwise, then yes it is correct that the neural networks weights would not change at all through gradient descent, and the NN would remain unchanged for any of your input values. Your careful initialisation of biases and weights will have created a system that is unable to learn from the given data. This is a known problem with ReLU activation, and can happen to some percentage of artificial neurons during training with normal start conditions. Other activation functions such as sigmoid have similar problems - although the gradient is never zero in many of these, it can be arbitrarily low, so it is possible for parameters to get into a state where learning is so slow that the NN, whilst technically learning something on each iteration, is effectively stuck. It is not always easy to tell the difference between these unwanted states of a NN and the goal of finding a useful minimum error.

@DuttA: It means that you are close to stationary point. That could be a "good" mimimum, but could also be a bad local minimum, a maximum or a saddle point. I have read articles that claim that a lot of "unwanted" stationary
and close-to-stationary points that occur during NN training are saddle points - i.e. with upward slopes in many directions but downward slopes in others.

@DuttaA I read an article about that topic too. According to the authors it is all about statistics. In order for you to find a local minimum you must find a point where all your N input variables are in a local minimum. If A of your input variables are in a local minimum and B are not (where A + B = N) then you found a saddle point. That's the reason why usually are way more saddle points than mínima.

People often place a batchnorm layer before ReLU. That effectively prevents the problem you have described.

Learning and Zero Derivatives
Artificial networks are designed so that even when the partial derivative of a single activation function is zero they can learn. They can also be designed to continue learning when the derivative of the loss1 function is zero too. This resilience to a vanishing feedback signal amplitude, by design, determines some of how calculus results are employed in the learning algorithm, hardware acceleration circuitry, or both. By learning behavior is meant the behavior of the changes to the parameters of the network as learning occurs.
For many of the activation functions used today, the derivative of the of the activation function is never the real expression 0, but there are such cases. These are examples of when the evaluation of the derivative of the activation function is effectively zero.

All the time for a binary step activation function, which is why they are usually only used for the very last layer of a network to discretize the output.
When the input of a ReLU activation function is negative, which is the case given in the question
When the granularity of the IEEE representation of the number can no longer support the smallness of the absolute value of the number upon evaluation
When the loss is zero

Nearing Zero
This last condition can easily occur if the result of the loss function output is so close to zero that the digital products of that number, during propagation, rounds to zero in the floating point hardware. Even if not zero, the number can be so small that learning slows to an untenable speed. The learning process either oscillates, in many cases chaotically, because of rounding phenomena or finds a static state and remains there. Again, this does not necessarily require a zero partial in the Jacobian.
A Familiar Analogy
The cognitive equivalent that helps intuition in understanding this but is not at all a great and across-the-board accurate analogy is the mental concept of doubt. The advantages of various directions of change or action to produce change is no longer clear. This is a rough analogy that some can connect to when considering what it means when the gradient is vanishing. When looking at gradient in historical context, where gradient is the slope of a surface in a location where gravity defines which direction is down, a vanishing gradient is a place where no direction seems to be down hill.
Flat in One Dimension by Design
In the question, where an inner layer2 is a ReLU activation function, the evaluation of the partial derivative of the loss function with respect to the parameter being adjusted will always be zero if its input is negative. However, this is by design and is one of the reasons ReLU trains fast. When the signal is negative going into the layer at that particular cell, it is thrifty to ignore it. The other cells upstream are then altered through other paths around the deactivated cell with the zero partial. A neuroscientist might smile at the oversimplification, but this is like a missing synapse between two adjacent cells in the brain.

In a setup like the above where the derivative is 0, is it true that a [network] won't learn anything?

It is false. Learning will stop if all the derivatives in a layer are zero and no other device, such as curvature, momentum, regularization, and other devices controlled by hyper-parameters, is employed. Even so, zeros across the layer would only affect the adjustment of upstream layers, layers closer to the input. Downstream, convergence activity may continue in such a case.
Zero and close to zero values (as well as near overflows) are kinds of saturation conditions, and these are studied carefully in artificial network research, but a single cell with a zero partial will not stop learning and may, in specific cases, ensure its completion and the adequacy of its result.
Some Calculus
In mathematical terms, if the Jacobian has a zero in one position, the others may remain active indicators of proper adjustment magnitude and direction for the individual parameters. If the Hessian is used or various types of momentum, regularization, or other techniques are employed, zeros across the Jacobian will probably not block upstream learning, which is part of the reason why they are used.
An Analogy for Momentum
The analogy can again be employed to clarify momentum as a principle, with the caveat that it is again an oversimplification. Beliefs have momentum, so when a belief exists and all other indications of direction for the next step is unsure, most will base their next step upon their beliefs.
This is how all organisms with a brain tend to work and why mouse traps and spider webs can catch. Without viable feedback from which learning can occur, the organism will act based on the momentum in its DNA or networks of its brain. This is usually beneficial, but can lead to loss in specific cases.
Gradients are not fool proof either. The problem of local minima can render pure gradient descent dysfunctional as well.
An Analogy for Curvature
Curvature (as when the Hessian is employed) requires a different analogy.
If a smart, blind person is on a dry, flat surface, thirsty, and needing water, the gradient may be flat, they may feel with their foot or cane for some indication of curvature. If some down curving feature of the surface is found, that may guide the person to water in more cases than a random step.
As hardware acceleration improves, the Hessian, which is to computationally heavy for CPU execution in most cases, may emerge as standard practice.
Mathematically, this is simply moving from two terms of the Taylor series expansion in multivariate space to three terms. From a mechanics perspective, it is the inclusion of acceleration to velocity.

Footnotes
[1] Loss or any of these functions that drive behavior in AI systems: Error, disparity, value, benefit, optimality, and others of similar evaluative nature.
[2] Inner layers in artificial networks are often called hidden layers, but that is not technically correct. Even though the encapsulation of a neural network may hide signals in inner layers from the programmer, it is a low level software design feature to do that, and a bad one. One can usually and definitely should be able to monitor those signals and produce statistics on them. They are not hidden in the mathematics, as in some kind of mathematical mystery, and the only difference between them and the output layer is that the output layer is intended to drive or influence something outside the network.


text:
Can analog quantum computer implement real-valued neural networks and hence do hypercomputation?
It is said, that the essence of https://www.springer.com/us/book/9780817639495 "Neural Networks and Analog Computation. Beyond the Turing Limit" is that the continuous/physical/real-valued weights for neural networks can induce super-Turing capabilities. Current digital processors can not implement real-valued neural networks, they can only approximate them. There are very little efforts to build analog classical computers. But it is quite possible that quantum computers will be analogue. So - is there research trend that investigates true real-valued neural networks on analog quantum computers?
Google is of no use for my efforts, because it does not understand the true meaning of "true real-valued neural network", it gives just real-value vs complex valued neural networks articles, which are not relevant to my question.

Digital and Analog
The question about analog computing is important.
Digital circuitry gained popularity as a replacement for analog circuitry during the four decades between 1975 to 2015 due to three compelling qualities.

Greater noise immunity
Greater drift immunity (accuracy)
No leakage of stored values

This quickly led to digital signaling standards, architecture of a general purpose computing, and central processing units on a chip. The later, combined with an array of registers to perform elementary operations is the meaning of the word microprocessor.
Quanta and Computing
Regarding quantum computing, there have been some interesting proposals to pack digital gates into much smaller volumes, but the notion that a computer can be made of transistors the size of electrons might be a bit fantastic. That's what the term quantum computing implies. That degree of miniaturization would have to defy principles of particle physics that are very strongly supported by amassed empirical evidence. Among them is Heisenberg's uncertainty principle.
All computing involves quanta, but statistically. For a transistor in a digital circuit to be statistically stable, there must be a sufficient number of Si atoms with at least 0.1% molar concentration of the atoms used to dope the Si to create a PN junction. Otherwise the transistor will not switch reliably.
The lithographic limit of most mass produced VLSI chips is around 7 nm as of this writing. Crystalline Si, nucleus to nucleus, is about .2 nm, so the miniaturization of a stable transistor is near its quantum limit already. Exceeding that limit by a considerable amount destabilizes the digital circuitry. That's a quantum physics limitation, not a lithographic limitation.
Projections, Models, and Techniques to Push Limits
Moore's law was simply an approximate model for the chip industry during the period between the invention of the integrated circuit to the physical limitation of the atomic composition of transistors, which we are now approaching.
Field effect transistors (FETs) can take the miniaturization only slightly further than the mechanics of PN junctions. 3-D circuitry has theoretical promise, but no repeatable nanotechnology mechanisms have yet been developed to form complex circuitry in the third dimension.
Returning to the Primary Question
Placing aside the magical idea that quantum computing will somehow completely revolutionize circuitry, we have a question that is both feasible and predictable.

Can an analog computer implement real-valued neural networks and hence do artificial network computation better?

If we define better in this context as cheaper and faster, while maintaining reliability and accuracy, the answer is straightforward.
It definitely takes fewer transistors to create the feed forward part of an artificial network using an analog approximation of the closed forms resulting from the calculus of artificial networks than a digital one. Both are approximations. Analog circuitry has noise, and drift and digital circuitry has rounding error. Beyond rounding, digital multiplication is much more complex in terms of circuitry than analog, and multiplication is used quite a bit in artificial network implementations.
Limitation Interplay of Gödel and Turing
The idea from the tail end of the title of the book this question referenced, "Beyond the Turing Limit," is also a little fantastic. The thought experiment of Alan Turing leading to the Turing machine and the associated computability theory (including Turing completeness) was not developed to be a limit. Quite the opposite. It was an answer to Gödel's incompleteness theory. People in Turing's time saw the work Gödel's genius as the annoying but indismissable limit threatening the centuries-old vision of using machines to automatically expand human knowledge. To summarize this work, we can state with assurance this.

The theory limiting what computers can do is not related to how the numbers are represented in electronic circuit implementations. It is a limitation of information mechanics.

These principles are important but have little to do with the limitation.

Analog computing
Miniaturization
Parallel computing
Ways that stochastic injection can help
How algorithms can be defined in programming languages

The above has to do with the feasibility of a project for which some person or corporation must pay and the intellectual capacities required to completing it, not the hard limit on what is possible.
Defining what a super-Turing capability might be would be a dismissal or a dismantling of what mathematicians consider to be well constructed theory. Dismantling or shifting the contextual frame of some computability theory is plausible. Dismissing the work that has been done would be naive and counterproductive.
Real Numbers are Now Less Real Than Integers
The compelling idea contained in the question is the reference to continuity, physicality, and the real valued nature of parameters that acquire a learned state during the training of artificial networks.
To multiply a vector of digital signals by a digital parameter matrix requires many transistors and can require a significant number of clock cycles, even when dedicated hardware is used. Only a few transistors per signal path are required to perform the analog equivalent, and the throughput potential is very high.
To say that real values cannot be represented in digital circuits is inaccurate. The IEEE standards for floating point numbers, when processed in a time series, represent real valued signals well. Analog circuits suffer from noise and drift as stated above. Both analog and digital signals only appear to be comprised of real number values. Real numbers are not real except in the world of mathematical models. What we call quantities in the laboratory are essentially measurements of means of distributions. Solidifying and integrating the probabilistic nature of reality into science and technology may be the primary triumph of the twentieth century,
For instance, when dealing in milli-Amps (mA), electric current seems to be a continuous phenomenon, but when dealing with nano-Amps (nA), the quantum nature of electric current begins to appear. This is much like what happens with the miniaturization of the transistor. Real numbers can only be represented in analog circuits through the flow of discrete electrons. The key to the advantage of an analog forward feed in artificial networks is solely that the density of network cells can be considerably higher, reducing the cost of the network in its VLSI space.
In summary, real numbers received the name for their type prior to the emergence of quantum physics. The idea that quantities formerly considered real and continuous were actually statistical averages of discrete activities at a quantum level revolutionized the field of thermodynamics and microelectronics. This is something that disturbed Einstein in his later years. In essence, mathematics using real numbers is effective in engineering because it simplifies what physicist now believe are distributions of a large numbers of quantum phenomena occurring in concert.
Summarizing the Probable Future of Analog Computing
This phrase from the question is not precisely scientific, even though it points to a strong likelihood.

It is quite possible that quantum computers will be analogue.

This modified version is more consistent with scientific fact in the way it is phrased, and is also factual.

It is possible that computers dealing with signals at a near quantum level of miniaturization will have a higher proportion of analog circuitry.

This question and its answers have many links to papers and research work regarding analog computing: If digital values are mere estimates, why not return to analog for AI?.

Many thanks!. Excellent answer and excellent pointer to the other thread. Fantastically!


text:
Could a neural network be capable to diferentiate between two boards of a game?
Let's propose, that I can define the state of a board in a board game, with 234 neurons. In theory, could I be able to train a neural network, with 468 inputs (two game boards), and 1 output, to tell me which board state is 'better'? The output should give me ~-1 if the second board is better than the first, ~0 if they are equal, and ~1 if the first board is better than the second.
If yes, what could be the number of ideal neurons on the hidden layers? What could be the ideal number of hidden layers?

one issue is that a comparison function defined by what's essentially a random oracle is not guaranteed to be transitive, antisymmetric or even reflexive. It needs to be all three.

For optimal performance, the network complexity should fit the complexity of the game. Since we do not know the latter, your question is not answerable.

I am very new to AI, and neural networks. What do you mean by the complexity of the game? If I would, for example, write down the ruleset of the game, would that help?

@AdamBaranyai: No that would not help, because there are no well-understood rules that can convert a written set of rules into a network architecture. There are general concepts and understanding e.g. a more complex function with more inflection points and high-frequency components will require a neural network with more neurons in hidden layers. Finding working architectures for new problems usually consists of a search, guided by experience from similar problems.

I would say that even if we knew the complexity of the game, we probably could not suggest the ideal number of neurons or hidden layers. With that view, then the first half of this answer is accurate, but the second half is misleading in that it implies if someone was to describe a game in detail, that they could get an answer. This is only going to be true when the game is one that has already been studied in detail.