text:
What activation function does the human brain use?
Does the human brain use a specific activation function?
I've tried doing some research, and as it's a threshold for whether the signal is sent through a neuron or not, it sounds a lot like ReLU. However, I can't find a single article confirming this. Or is it more like a step function (it sends 1 if it's above the threshold, instead of the input value)?

It is implicit in this question that an artificial neuron in a neural net is a model or equivalent to a natural neuron, but this assumption is false.

The thing you were reading about is known as the action potential. It is a mechanism that governs how information flows within a neuron.
It works like this: Neurons have an electrical potential, which is a voltage difference inside and outside the cell. They also have a default resting potential, and an activation potential. The neuron tends to move towards the resting potential if it is left alone, but incoming electric activations from dendrites can shift its electric potential.
If the neuron reaches a certain threshold in electric potential (the activation potential), the entire neuron and its connecting axons goes through a chain reaction of ionic exchange inside/outside the cell that results in a "wave of propagation" through the axon.
TL;DR: Once a neuron reaches a certain activation potential, it electrically discharges. But if the electric potential of the neuron doesn't reach that value then the neuron does not activate.

Does the human brain use a specific activation function?

IIRC neurons in different parts of the brain behave a bit differently, and the way this question is phrased sounds as if you are asking if there is a specific implementation of neuronal activation (as opposed to us modelling it).
But in general behave relatively similar to each other (Neurons communicate with each other via neurochemicals, information propagates inside a neuron via a mechanism known as the action potential...) But the details and the differences they cause could be significant.
There are various biological neuron models, but the Hodgkin-Huxley Model is the most notable.
Also note that a general description of neurons don't give you a general description of neuronal dynamics a la cognition (understanding a tree doesn't give you complete understanding of a forest)
But, the method of which information propagates inside a neuron is in general quite well understood as sodium / potassium ionic exchange.

It (activation potential) sounds a lot like ReLU...

It's only like ReLU in the sense that they require a threshold before anything happens. But ReLU can have variable output while neurons are all-or-nothing.
Also ReLU (and other activation functions in general) are differentiable with respect to input space. This is very important for backprop.
This is a ReLU function, with the X-axis being input value and Y-axis being output value.

And this is the action potential with the X-axis being time, and Y being output value.

It's a step function if the x-axis is the input, but a Delta Dirac if the x-axis is time, you mean?

You are right but the point i'm trying to get at (about it being continuous and moving back to resting potential), ill upload a picture to make myself more clear

Seems more like a sigmoid that a step function

The brains of mammals do not use an activation function.  Only machine learning designs based on the perceptron multiply the vector of outputs from a prior layer by a parameter matrix and pass the result statelessly into a mathematical function.
Although the spike aggregation behavior has been partly modeled, and in far more detail than the 1952 Hodgkin and Huxley model, all the models require statefulness to functionally approximate biological neurons.  RNNs and their derivatives are an attempt to correct that shortcoming in the perceptron design.
In addition to that distinction, although the signal strength summing into activation functions are parametrized, traditional ANNs, CNNs, and RNNs, are statically connected, something Intel claims they will correct with the Nirvana architecture in 2019 (which places into silicon that which we would call layer set up in Python or Java now.
There are at least three important biological neuron features that make the activation mechanism more than a function of a scalar input producing a scalar output, which renders questionable any algebraic comparison.

State held as neuroplastic (changing) connectivity, and this is not just how many neurons in a layer but also the direction of signal propagation in three dimensions and the topology of the network, which is organized, but chaotically so
The state held within the cytoplasm and its organelles, which is only partly understood as of 2018
The fact that there is a temporal alignment factor, that pulses through a biological circuit may arrive via synapses in such a way that they aggregate but the peaks of the pulses are not coincident in time, so the activation probability is not as high as if they were temporally aligned.

The decision about what activation function to use has largely been based on the analysis of convergence on a theoretical level combined with testing permutations to see which ones show the most desirable combinations of speed, accuracy, and reliability in ctheonvergence.  By reliability is meant that convergence on the global optimum (not some local minimum of the error function) is reached at all for the majority of input cases.
This bifurcated research between the forks of practical machine learning and biological simulations and modeling.  The two branches may rejoin at some point with the emergence of spiking - Accuracy
 - Reliability (completes) networks.  The machine learning branch may borrow inspiration from the biological, such as the case of visual and auditory pathways in brains.
They have parallels and relationships that may be exploited to aid in progress along both forks, but gaining knowledge by comparing the shapes of activation functions is confounded by the above three differences, especially the temporal alignment factor and the entire timing of brain circuits which cannot be modeled using iterations.  The brain is a true parallel computing architecture, not reliant on loops or even time sharing in the CPU and data buses.

The answer is We do not know. Odds are, we will not know for quite a while. The reason for this is we cannot understand the "code" of the human brain, nor can we simply feed it values and get results. This limits us to measuring currents of the input and output on test subjects, and we have had few such test subjects that are human. Thus, we know almost nothing about the human brain, including the activation function.

The human brain does have an activation function which is sigmoid like when considering the membrane potential at the Axon Initial Segment (AIS) as input and the rate of action potential as the output.
There is a threshold membrane potential under witch no action potential is generated, and there is a maximum output rate the neuron may produce due to the hyper polarisation after the action potential spike.
The biological neuron adjust the slope of the sigmoid and its mean (or the threshold if you like). The adjustment of the mean (and threshold) is quite fast as the AIS has a particular structure allowing to move the voltage sensitive sodium pores with an intracellular muscle (actin) and skeleton (microtubules).
.
From "The Axon Initial Segment: An Updated Viewpoint" Christophe Leterrier,
Journal of Neuroscience 28 February 2018, 38 (9) 2135-2145; https://doi.org/10.1523/JNEUROSCI.1922-17.2018
The sigmoid change is normalizing the output. When the membrane potential is artificially strongly stimulated, the average output rate is reduced, and when the stimulation is reduced, the average output rate is increased.
This normalization makes sense with recursive neural network. The system must be maintained in a stable regime.
Note that the sigmoid is not symmetric and that the biological signal processing is stochastic. The action potential spike is a dirac which thus doesn't carry any information in itself. It is the relative time of spikes that carries the information.

My interpretation of the question was 'what activation function in an artificial neural network (ANN) is closest to that found in the brain?'
Whilst I agree with the selected answer above, that a single neuron outputs a dirac, if you think of a neuron in an ANN as modelling the output firing rate, rather than the current output, then I believe ReLU might be closest?
http://jackterwilliger.com/biological-neural-networks-part-i-spiking-neurons/

The basic neuron firing is similar to a spiking function. However, there is both an immediate (enough positives charges leads to overcoming activation threshold for the axon hillock, leading to more jumping in), and feedback effect. not every neuron “fires” the same way, as transcriptors can produce more channels to increase influx of charge and strengthen an activation. Axon terminal also have vesicles with neurotransmitters. Only a few hundred neurotransmitters exist, ex dopamine, similar to an alphabet which can be used to create different words or effects in different areas of the brain.
So while the basic concept can be analogous to spiking / Hebbian activation, in reality there is modulation of states which make it more complex.


text:
What does "reimplementations of deep learning algorithms which replicate performance from the papers" mean?
In the OpenAI's Machine Learning Fellow position, it is written

We look for candidates with one or more of the following credentials:

...
Open-source reimplementations of deep learning algorithms which replicate performance from the papers

What exactly do they mean by this? Do they want us to implement the algorithms exactly as described in the papers (i.e. with the same hyper-parameters, weights, etc.)?

Accompanying ML/AI (and computer science, in general) papers with the relevant code is much desirable, both for reproducibility of the paper results as well as for faster integration of these results into applications.
Some researchers do so by themselves; but this has lately become also a community job, where a third, independent individual or team, not affiliated with the paper author(s), offers a code implementation starting from the paper.
A good, very recent case is the reproduction of the World Models paper (blog post, Github repo). Many of the implementations available at Papers with Code are indeed provided by the community.
A great write-up of such an effort, containing many lessons learned, made big raves recently: Lessons Learned Reproducing a Deep Reinforcement Learning Paper.
Understandably, individuals that have such efforts in their CV are a good recruiting option...

In addition to this, do you find the stated "statement (Open-source reimplementations of deep learning algorithms)" demanding for reimplementation of some Machine Learning Model's based on some paper or the reimplementation of the very learning techniques themselves (e.g. back-propagation)?

@ShoaibAnwar probably the former; a re-implementation of BP is not of any particular value, apart from a purely educational perspective, and such efforts abound

I believe that this:

"reimplementations of deep learning algorithms"
He is asking that they are looking for people who have made and completed an AI that perform similarly or exactly as those given in papers (IDK which papers)
"Open-source" The Fellow wants to be able to see the source code of the project.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
When to expand and when to simulate in Monte Carlo Tree Search?
In Monte Carlo Tree Search (MCTS), we start at root node $R$. Then we select some leaf node $L$. And we expand $L$ by one or more child nodes and simulate from the child to the end of the game.

When should we expand and when should we simulate in MCTS? Why not expand 2 or 3 levels, and simulate from there, then back-propagate the values to the top?  Should we just expand 1 level? Why not more?

The most common strategy is to simply expand exactly one node per iteration; you can view this as expanding the first node of the Play-Out phase ("simulation" in your image), and not expanding any other nodes of the Play-Out phase. This is also what's done in your image.
That is the most common and probably most simple strategy, but it's certainly not the only one. It is pretty much the minimum you have to expand, but you can expand more if you like. The most common approach of only expanding one node per iteration is basically there because it minimizes the risk of running out of memory; when you only add one new node to the tree per iteration, the tree grows relatively slowly, so you'd have to keep the algorithm running for a really long time before you run out of memory.
If you're not afraid of running out of memory, you can choose to expand as many nodes as you like. For example, my General Video Game AI agent expands every single node encountered during the play-out at once, because in that particular domain we only get to run relatively few iterations anyway so I'm not afraid of running out of memory, even if I do expand lots of nodes.
The backpropagation step can only store results (information) from your iterations in nodes that actually exist, nodes that have been expanded. So, the benefit of expanding more (if possible) is that you'll retain more information from early iterations, your backpropagation step immediately gets to store results in all nodes along the path instead of only the top few nodes. Generally, this is a relatively minor benefit; nodes deep in the tree / deep inside the simulations are quite unlikely to ever get visited more than once anyway, and then this doesn't matter. But expanding more can, in theory, result in slightly more accurate value estimates in nodes closer to the root.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why do we prefer ReLU over linear activation functions?
The ReLU activation function is defined as follows
$$y = \operatorname{max}(0,x)$$
And the linear activation function is defined as follows
$$y = x$$
The ReLU nonlinearity just clips the values less than 0 to 0 and passes everything else. Then why not to use a linear activation function instead, as it will pass all the gradient information during backpropagation? I do see that parametric ReLU (PReLU) does provide this possibility.
I just want to know if there is a proper explanation to using ReLU as default or it is just based on observations that it performs better on the training sets.

Use as interpretation of the activation function result the one of "error". Relu has a value 0, meaning expected optimal result is reached. Lineal has a minimum of less infinite, meaning the error can be less than zero (not applicable in most cases) and always posible to improve (near than never applicable)

A simple intuition behind this, is that an ANN with all linear activations is analogous to linear regression

A multi-layer network in which all units have linear activation functions can always be collapsed to an equivalent network with two layers of units. That is why it is essential to use nonlinear unit activation functions.
The underlying reason for using nonlinear activation functions involves a remarkable theorem of Cybenko (1989), which states that one layer of nonlinear hidden units is sufficient to approximate any mapping from input to output units. Actually, I think there is a later proof which specifies that the nonlinearity can be any non-polynomial function (e.g. sigmoidal).
This text is based on the book: Artificial Intelligence Engines: A Tutorial Introduction to the Mathematics of Deep Learning.

Links to the proof of Cybenko's theorem would be more useful to OP than links to your book, unless you can specify where in your book the topic is covered. Ideally your book outlines Cybenko's theorem?

The Cybenko's proof assumes that a sigmoid is used (if I remember correctly), but it's true that other proofs have been provided that are more general.

i would be led to believe that the proof came before the practice of using non linear activation functions


text:
Is random initialization of the weights the only choice to break the symmetry?
My knowledge
Suppose you have a layer that is fully connected, and that each neuron performs an operation like
a = g(w^T * x + b)

were a is the output of the neuron, x the input, g our generic activation function, and finally w and b our parameters.
If both w and b are initialized with all elements equal to each other, then a is equal for each unit of that layer.
This means that we have symmetry, thus at each iteration of whichever algorithm we choose to update our parameters, they will update in the same way, thus there is no need for multiple units since they all behave as a single one.
In order to break the symmetry, we could randomly initialize the matrix w and initialize b to zero (this is the setup that I've seen more often).  This way a is different for each unit so that all neurons behave differently.
Of course, randomly initializing both w and b would be also okay even if not necessary.
Question
Is randomly initializing w the only choice?  Could we randomly initialize b instead of w in order to break the symmetry? Is the answer dependent on the choice of the activation function and/or the cost function?
My thinking is that we could break the symmetry by randomly initializing b, since in this way a would be different for each unit and, since in the backward propagation the derivatives of both w and b depend on a(at least this should be true for all the activation functions that I have seen so far), each unit would behave differently. Obviously, this is only a thought, and I'm not sure that is absolutely true.

Randomising just b sort of works, but setting w to all zero causes severe problems with vanishing gradients, especially at the start of learning.
Using backpropagation, the gradient at the outputs of a layer L involves a sum multiplying the gradient of the inputs to layer L+1 by the weights (and not the biases) between the layers. This will be zero if the weights are all zero.
A gradient of zero at L's output will further cause all earlier layers(L-1, L-2 etc all the way back to layer 1) to receive zero gradients, and thus not update either weights or bias at the update step. So the first time you run an update, it will only affect the last layer. Then the next time, it will affect the two layers closest to the output (but only marginally at the penultimate layer) and so on.
A related issue is that with weights all zero, or all the same, maps all inputs, no matter how they vary, onto the same output. This also can adversely affect the gradient signal that you are using to drive learning - for a balanced data set you have a good chance of starting learning close to a local minimum in the cost function.
For deep networks especially, to fight vanishing (or exploding) gradients, you should initialise weights from a distribution that has an expected magnitude (after multiplying the inputs) and gradient magnitude that neither vanishes nor explodes. Analysis of values that work best in deep networks is how Xavier/Glorot initialisation were discovered. Without careful initialisation along these lines, deep networks take much longer to learn, or in worst cases never recover from a poor start and fail to learn effectively.
Potentially to avoid these problems you could try to find a good non-zero fixed value for weights, as an alternative to Xavier initialisation, along with a good magnitude/distribution for bias initialisation. These would both vary according to size of the layer and possibly by the activation function. However, I would suspect this could suffer from other issues such sampling bias issues - there are more weights, therefore you get a better fit to desired aggregate behaviour when setting all the weight values randomly than you would for setting biases randomly.

What has setting w to zero do with vanishing gradients? It is a problem involved with saturation of the activation function not with the initialization of the parameters (unless you initialize them to a very high or low value).

The gradient descend of a cost function is computed with respect to both the bias an the weight. Thus the actual formulation of both derivatives will depend on which cost function and activation function did we chose. In the cases that I have seen, both the derivative of b and w will be updated depending on the value of a.

@gvgramazio: take a look at the backprop formula again. To get dJ/da for layer L you sum over dJ/dz * W from layer L+1 (bias, the cost function, and activations elsewhere are not involved in that step). Intuitively this makes sense - if all weights are zero, then the influence of any activation to the next layer is also zero, so there is no gradient.  So the gradients are blocked after the first hidden layer with zero weights. There still are non-zero gradients to the weights themselves in that layer . . . so eventually (and slowly) this will fix itself.

Uh...sorry about my last comment. I checked right now my notes and did some calculus. Yeah, if the weights of all layers are all zero and the bias is not, then at the first update only the last layer will be affected, at the second iteration also the second one, at the third iteration also the third one and so on. You were right about this.

Yeah, I saw this now.

But this fact is relevant to the fact that all wights are zero. I mean that b is enough to break the symmetry  but not enough to solve the vanishing gradient problem. If I chose b randomly and weights all equal to some value (different from zero) I will get rid of both the problems, right?

@gvgramazio Yes, possibly. I have modified my answer to address that. I think there may be other more subtle problems with that fix, although I am not sure. It may just be that Xavier initialisation works well, and there is no theoretical reason to explore an alternative and slightly more complex version of it that adjusts both weights to a fixed value and biases to a distribution.

Oh, I know that. Simply the question comes to my mind since in the sources I'm following the topic was dismissed by simply saying that is necessary to randomly initialize w and not b.

Most of the explanations given for choosing something or not choosing something (like hyperparameter tuning) in deep learning are based on empirical studies, like analysing the error over a number of iterations. So, this answer is what people in deep learning side give.
Since you have asked for a mathematical explanation, I suggest you read the paper Convergence Analysis of Two-layer Neural Networks
with ReLU Activation (2017, NIPS). It talks about the convergence of SGD to global minima subject to weight initialisation being Gaussian using ReLU as an activation function. The paper considers a neural net with no hidden layer, just input, and output layers.
The very fact that analysis on such 'simple' network gets published in a very reputed and top conference itself suggests that the explanation you are seeking is not very easy and very few people work on the theoretical aspects of neural nets. IMHO, after some years as the research progresses, I might be able to edit this answer and give the necessary explanation that you sought. Till then this is the best I could do.

You surely understood what I'm looking for. The answer by Joe S is similar to the ones that were already given in previous similar posts that I linked at the beginning of my post. Look at the answer by Neil Slater, I think that he completely clear all my doubts.

@gvgramazio . Yes, but what about the theoretical explanation that you are looking for? He (Neil Slater) has provided good intuitive and qualitative explanation.

He gave my a hint in the comments below his answer and then I found it by myself. In the back-propagation, compute the formulas for the gradient with respect to w and b for the layer i. If you do all the substitutions you'll find that the gradient of the layer i is directly proportional to the weight of the following layer while this is not true for b. This mean that w=0 means that the gradient will be stuck to 0. Note that this has nothing to do with symmetry but is due to the fact that w=0. If we choose w=1, we don't have this problem (but we could have symmetry problem if b=const).

Suppose that for each neuron we have z = w^T x + b and a = g(z). I prepend d to a variable to indicate that it is the derivative of the cost function with repect to that variable. For a generic layer i, we have dw_i=dz_i x^T and db_i=dz_i but dz_i=w_{i+1}^T dz_{i+1} ∘ g'_1(z_1) where ∘ is the Hadamard product, g' is the derivative of your activation function and maybe the transpositions could be different on the base of your implementation. The important thing is that if w_{i+1} = 0 then dw_i, db_i = 0.

This mean that, with all weights equal to zero. Only the last layer will have gradient different from zero, at the second iteration, only the last two, and so on...

w should be randomized to small (nonzero) numbers so that the adjustments made by the backpropagation are more meaningful and each value in the matrix is updated a different amount. If you start with all zeros, it will still work, but take longer to get to a meaningful result. AFAIK, this was found empirically by various researchers and became common practice.
Randomizing b does not have the same effect of helping, therefore most people do not bother.
This choice is one of many that is made by the architect of the network and theoretically you could use an infinite number of w matrix initializations. The one commonly used just happens to be tested and generally works.
This video is better at explaining than I am: Lecture 8.4 — Neural Networks Representation | Model Representation-II — [Andrew Ng].

You're answer isn't a real answer because you simply stated that randomizing b doesn't have the same effect without explaining the reason. I know that somewhat isn't helpful to initialize b instead of w since nobody do that, but I wished to know why.

Fun fact: I was reading something from the professor in the link you provided when this post comes to my mind. :)

If you start with all zeros it actually won't work: you need at least some non-zero weights in every layer or else nothing will happen during training.


text:
What could happen if we wrongly assume that the POMDP is an MDP?
Consider the Breakout environment.
We know that the underlying world behaves like an MDP, because, for the evolution of the system, it just needs to know what the current state (i.e. position, speed, and speed direction of the ball, positions of the bricks, and the paddle, etc) is. But, considering only single frames as the state space, we have a POMDP, because we lack in formations about the dynamics [1], [2].
What could happen if we wrongly assume that the POMDP is an MDP and do reinforcement learning with this assumption over the MDP?
Obviously, the question is more general, not limited to Breakout and Atari games.

What could happen if we wrongly assume that the POMDP is an MDP and do reinforcement learning with this assumption over the MDP?

It depends on a few things. The theoretical basis of reinforcement learning needs the state descriptions to have the Markov property for guarantees of convergence to optimal or approximately optimal solutions. The Markov property is a requirement that the state defines 100% of the controllable variation of reward and next state (given the action) - the rest must be purely stochastic.
An MDP can be "nearly Markov", and a lot of real-world physical systems are like that. For instance, pole-balancing and acrobot tasks can be implemented as physical systems using motors, wheels, joints etc. In those real systems, there are limits to accuracy of measurement of the state, and many hidden variables, such as variable temperature (affecting length of components), friction effects, air turbulence. Those hidden variables taken strictly by formal definition would make the system a POMDP. However, their influence compared to the key state variables is low, and in some cases effectively random from the perspective of the agent. In practice RL works well in the real physical systems, despite state data being technically incomplete.
In Atari games using multiple frame images as states, there are varying degrees of to which those states are already non-Markovian. In general a computer game's state may include many features that are not displayed on the screen. Enemies may have health totals or other hidden state, there can be timers controlling appearance of hazards, and in a large number of games the screen only shows a relatively small window into the total play area. However, the Deep Mind DQN network did well on a variety of scrolling combat and platform games.
One game where DQN did notably badly - no better than a default random player - was Montezuma's Revenge. Not only does that platform puzzler game have a large map to traverse, but it includes components where state on one screen affects results on another.
It is hard to make a general statement about where an MDP with missing useful state information would benefit from being treated as a POMDP more formally. Your question is essentially the same thing expressed in reverse.
The true answer for any non-trivial environment would be to try an experiment. It is also possible to make some educated guesses. The basis for those guesses might be the question "If the agent could know hidden feature x from the state, how different would expected reward and policy be?"
For the breakout example using each single frame as a state representation, I would expect the following to hold:

Value estimates become much harder since seeing the ball next to a brick - compared to seeing a ball progressively get closer to a brick over 4 frames - gives much less confidence that it is about to hit that brick and score some points.

It should still be possible for the agent to optimise play, as one working strategy is to position the "bat" under the ball at all times. This will mean less precise control over angle of bounces, so I would expect it to perform worse than the four-frame version. However, it should still be significantly better than a default random action agent. A key driver for this observation is that seeing the ball close to the bottom of the screen, and not close to the bat, would still be a good predictor of a low expected future reward (even averaged over chances of ball going up vs going down), hence the controller should act to prevent such states occurring.

Thank you. Theoretically speaking, the RL algorithms that assume the Markov property are not guaranteed to succeed (e.g. SARSA does not converge). However, like in Breakout (and in many other cases), the reduced feature space (i.e. without ball speed) should still allow estimating state values accurately enough to succeed in playing the game. But the algorithm that assumes the Markovian property will never estimate state values exactly because he is trying to solve a problem that cannot be solved with his capabilities: Breakout without considering ball speed is not an MDP. Right?

Right. It's not strictly an MDP over the available state when velocities are hidden. But then neither are side-scrollers without considering the whole map, even when velocities are available. Yet DQN can still attempt them and often does quite well. SARSA and Q-Learning are guaranteed to converge in tabular forms (and a few other assumptions/caveats). SARSA with function approximation has a similar guarantee with some loose bounds on the maximum error at "convergence" (which is not stable/single value but should stay within error bounds).


text:
What is the importance of the endocannabinoid system for cognitive function?
The endocannabinoid system is a very important function of human biology. Unfortunately, due to the illegality of cannabis, it is a relatively new field of study. I have read a few articles about Google researching the role of dopamine in learning, and according to this article, anandamide (the neurotransmitter that closely resembles tetrahydrocannabinol):

was found to do a lot more than produce a state of heightened happiness. It’s synthesized in areas of the brain that are important in memory, motivation, higher thought processes, and movement control.

Have any neuroscientists (or any scientists) considered the importance of the endocannabinoid system for cognitive function?
If not, is there any reason this information might or might not be relevant to artificial intelligence?

The release of Adenosine, Dopamine, Endorphin, Endocannabinoids, GABA, Glutamate, Norepinephrine, Oxytocin, Serotonin, and many others into specific regions of the brain are very likely an essential part of both activation tuning of single neurons and neuroplasticity, two essential aspects of organic learning researchers have been and will continue to work to understand.
Most of those I've met in that sector of research are curious about the larger questions of what intelligence and consciousness are, and all of them appear to be interested in discovering how learning systems may be valuable in software engineering contexts.  These overarching questions are difficult to answer and the dive into the detail of learning has resulted in the expansion of ideas presented by Dr. Norbert Wiener in the mid 20th Century at MIT.
How chemical feedback in regions of the brain are secreted, how they disseminate geometrically into organic structures, how they interact with receptors, and what that does to the cell metabolism to produce change in the cell is almost definitely part of the DNA driven design of higher animal learning.  There does not appear to be anything pointless or arbitrary about it.  Adaptation to improve survival is evident and, yes, study is well underway.
Narcotics that interfere with the natural functioning of these organic signaling systems can lead to the inability to adapt in the individuals addicted to them.  That fact is a strong form of evidence that learning depends on these systems.
Oxytocin is another neurotransmitter of interest because its release is associated with what would fit into the higher levels of human thought and motivation on Abraham Maslow's famous hierarchy of needs.  Oxytocin seems to be part of reward signaling for modes of thought like authenticity, compassion, intimacy, wisdom, spirituality, and other human mental capacities and patterns of thought that transcend mere rationalism.  Why is that important?  Because the ability to lay down selfish goals for the good of the community seem to depend largely on the oxytocin system's preemptive ability over mere survival mode neural activity in mammals.
Regarding the cannabinoid receptors, there is a large enough body of media that show a correlation between a pool of successful artists and marijuana use to legitimately wonder whether there is any tie between creativity and the endocannabinoid system.  However in science (and hopefully in technology too), we are careful not to draw conclusions rashly.
It is also possible that this apparent correlation is simply a social phenomenon where the popularity of the artistic products or live performances is mainly because of potential audiences similarly stimulating their receptors with cannabinoids.  For instance, those engaging in LSD trips on stage attracted those also engaging in LSD trips into their audiences two generations ago.  Whether the work of those artists was more creative because of the impact on the serotonin receptors is largely subjective.  How can researchers come to conclusions about what is good performance?
One could analyze the audio to produce a table of notes in a performance and then develop a system to attach a numerical value to several positive quality metrics, study the price of tickets, or count downloads, but the decisions of what to measure and how to aggregate it into a final judgment of excellence is itself necessarily a subjective choice.
Nonetheless, if we place the many misconceptions of popular psychology and the drug culture aside, there is much research into the endocannabinoid system as part of the learning signaling, what machine learning researchers are currently calling reinforcement.  The more general term is, "A non-linear control system's feedback signal," already developed in detail in the 1940s (Behavior, Purpose and Teleology — A Rosenblueth, N Wiener, J Bigelow — Philosophy of Science, 1943 — U Chicago).  Some new trendy name will probably appear in the 2020s.
This article claims that the mammals under test, "exhibited enhanced learning."
Memory in Monoacylglycerol Lipase Knock-out Mice, by Bin Pan, Wei Wang, Peng Zhong, Jacqueline L. Blankman, Benjamin F. Cravatt and Qing-song Liu; Journal of Neuroscience 21 September 2011, 31 (38) 13420-13430; DOI: https://doi.org/10.1523/JNEUROSCI.2075-11.2011
To find many others: https://scholar.google.com/scholar?q=endocannabinoid+learning

Dopamine in synapses specifically has been identified as carrying a signal that appears to be very similar to the TD error used in reinforcement learning. There are even specialist dopamine-emitting neurons that have very high forward connectivity in order to "broadcast" this signal to other neurons involved in decision-making

@DouglasDaseeco: There is a difference between a control feedback signal and a learning signal. I'm pretty certain a RL algorithm based on a scalar reward signal (and thus scalar errors in predicting that reward), and using a vector feedback signal (identified in your quoted paper) as part of its state would be sufficient to learn control. Whether or not mammalian learning uses this approach is debatable (seems unlikely it would be the sole system), but there is strong evidence that a good analogy between dopamine transmission and TD error holds, which is all I am trying to point out.


text:
What is the order of the genetic operations in NEAT?
I was trying to implement NEAT, but I got stuck at the speciating of my clients/genomes.
What I got so far is:

the distance function implemented,
each genome can mutate nodes/connections,
two genomes can give birth to a new genome.

I've read a few papers, but none explicitly explains in what order what step is done. What is the order of the genetic operations in NEAT?
I know that for each generation, all the similar genomes will be put together into one species.
I have other questions related to NEAT.
Which neural networks are killed (or not) at each generation?
Who is being mutated and at what point?
I know that these are a lot of questions, but I would be very happy if someone could help me :)

What is the order of the genetic operations in NEAT?

You start by evaluating all of the initial neural networks and compute their initial fitness.
Then you speciate,
kill off the worst neural networks,
mutate and crossover to produce offspring, and
evaluate again.

The order of events is described on page 109 onwards in the original NEAT paper.

Which neural networks are killed (or not) at each generation?

The neural networks with the worst performance are killed off after speciation. None of the neural networks survive - the entire population is replaced with the offspring of the nets remaining after the culling stage. That said, you can implement elitism, where you keep some small portion of the best-performing nets and carry them over to the next generation without mutating them, but that is optional.

Who is being mutated and at what point?

At the end of each generation, after speciation and culling. To produce offspring, some of the remaining nets are subjected to mutation (think asexual reproduction - like single-celled organisms - but the offspring is mutated so that it differs from the parent). The rest are subjected to crossover in random pairs, so this would be the equivalent of sexual reproduction where you need two parents.
Hope that helps.


text:
Is there an analogy between client/server in web development and agent/environment in reinforcement learning?
I've recently come across the client-server model. From my understanding, the client requests the server, to which the server responds with a response. In this case, both the request and responses are vectors.
In reinforcement learning, the agent communicates with the environment via an "action", to which the environment sends a scalar reward signal. The "goal" is to maximize this scalar reward signal in long run.
Is there an analogy between client/server in web development and agent/environment in reinforcement learning?

Is there an analogy between client/server in web development and agent/environment in reinforcement learning?

The answer is "not really". There is no useful analogy here that allows any insight into RL from web server knowledge or vice versa.
However, you could set up an agent where the goal was to collect information, and the available actions were to make web requests. Clearly, in order to do this, you would need to make use of the client/server model for web servers, with the agent having control over the client web requests, and the environment being the network and servers of the world wide web.
There are some very hard challenges to construct an open-ended "web assistant" agent. Here are a couple that I can think of:

How to describe actions? Starting with raw web requests composed as strings would likely be very frustrating. Probably you would simplify and have a first action be a call to a search engine with some variation of the topic description, and then decisions about which links to follow, or perhaps whether to refine the search to better fetch sites related to the topic as it is being built.

How to create a model of reward for collecting information? The first major stumbling block would be to measure the amount of useful information that the agent had found on any request.

I think with current levels of Natural Language Processing, setting an agent free to discover information according to some goal from a text topic description is a very hard task, beyond cutting edge research. It would definitely be unreasonable to expect any such agent to end up with any resemblance to "understanding" subject matter from text. The agent would have very little ability to tell the difference between accurate facts or lies, or just grammatically correct gibberish.
One interesting idea for agents trying to learn unsupervised from exploring an environment is creating a reward signal from data compression. An agent will have learned something useful about its environment, if on processing new information, it is able to compress its existing world model more efficiently. This is basic concept behind ideas of general learning agents from Jürgen Schmidhuber, Marcus Hutter and others. Research into this idea could be a driver towards creating more generic AI learning systems - however, it is one idea amongst many in AI, and so far is research-only, it has not yet led to anything as practical as an AI web-searching assistant.


text:
How can a neural network learn to play sudoku?
I'm just beginning to understand neural networks and I've performed a couple of successful tests with numerical series where the NN was trained to find the odd one or a missing value. It all works pretty well.
The next test I wanted to perform was to approxmimate the solution of a Sudoku which, I thought could also be seen as a special kind of numerical series. However, the results are really confusing.
I'm using an MLP with 81 neurons in each of the three layers. All output neurons show a strong tendency to yield values that are close to either 0 or 1. I have scaled and truncated the output values. The result can be seen below:
Expected/Actual Solution:     Neural Net's Solution:

6 2 7 3 5 0 8 4 1             9 0 9 9 9 3 0 0 3
3 4 8 2 1 6 0 5 7             0 9 9 0 0 0 9 9 0
5 1 0 4 7 8 6 2 3             0 9 1 9 9 0 2 0 4
1 6 4 0 2 7 5 3 8             0 0 5 0 0 9 0 0 7
2 0 3 8 4 5 1 7 6             0 0 0 0 0 9 9 0 9
7 8 5 1 6 3 4 0 2             9 9 9 9 0 6 2 9 0
0 5 6 7 3 1 2 8 4             0 0 0 0 9 9 0 9 0
4 3 1 5 8 2 7 6 0             9 9 0 0 0 0 9 0 9
8 7 2 6 0 4 3 1 5             9 9 0 9 9 0 9 0 9

The training set size is 100000 Sudokus while the learning rate is a constant 0.5. I'm using NodeJS/Javascript with the Synaptic library.
I don't expect a perfect solution from you guys, but rather a hint if that kind of behavior is a typical symptom for a known problem, like too few/many neurons, small training set, etc.

I know, I'm not trying to actually solve it but just play around a little. My expectation is that there is a neural net that gets pretty close to a solution. But, on the contrary, my trained neural net seems to diverge from the actual solution and I'm trying to find an explanation for that.

Welcome to SE:AI!  Glad you found the answer, and hope to see you back soon!  (Not strictly related, but if interested in Sudoku decision-making, you might want to check out m,n,p-games, a form of partisan Sudoku. Here's a link to some visual references for the new mechanics: http://mbranegame.com/tutorial/)

I think it is the wrong way to frame sudoku as a regression problem in neural networks.
Firstly, you have to understand what regression is. "Regression" is when you predict a value given certain parameters, where the parameters are related to the value you have to predict. This happens because at the core neural networks are "function approximators", they model the function by adjusting their weights using lots of data. They tend to form a highly non-linear boundary to separate classes internally in a high dimensional data-space.
The sudoku doesn't fit in this scenario, the combinatorial complexity of sudoku is way too high for a neural network even if you add many layers to it, it is a totally different problem in its own right. You simple can't "regress" the right values of a perfect sudoku here, they are not numbers like "pixel " intensities in images.
However, you could apply reinforcement learning techniques to learn an optimal policy to solve sudoku.
And you have mentioned an "approximate" solution for the sudoku, what do you mean by "approximate"?  If you mean by this that only a few squares are out of place, then it is a wrong assumption, because neural networks are proven to be good image classifiers, as they are robust to translational invariance in this case, that is not what you need.
You could, however, do a small experiment to see what the neural network actually learns, replace the numbers by pixel value intensities and train a generative adversarial network on the sudoku images and see the images of sudokus produced by it, to see what actually the network can't learn.

Phew, thank you so much. This is way more information than what I hoped for. Now I kind of see how my assumptions were not right in the first place, plus I have lots of new things to read about and try out.

You can take a look at this paper that solving your problem with a neural network. You can use the pytorch implementation of the satnet layer : satnet layer API. In this supervised setup the layer also learn the boolean constraints of your model. You can find an example of a sodoku solver in the github repo.

I think DeemMind’s AlphaZero could provide an answer. Meaning, like others have said solving Sudoku is not a regression or classification problem. The net needs to learn the rules then implement them, just like in chess.
Maybe you could learn something by reading up on it: https://github.com/nikcheerla/deeplearningschool/deeplearningschool/2018/01/01/AlphaZero-Explained/

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference.  Link-only answers can become invalid if the linked page changes. - From Review


text:
Which machine learning approach should I use to estimate how many products a research group should have to improve its category?
Currently, in my country, there is a system in which certain groups of researchers upload information on products of scientific interest, such as research articles, books, patents, software, among others. Depending on the number of products, the system assigns a classification to each group, which can be A1, A, B and C, where A1 is the highest classification and C is the minimum. According to the classification of the groups, they can compete to receive monetary incentives to make their research.
At the moment, I am working on an application that takes the data of the system that I mentioned previously. I am able to say what classification the group currently has because we develop a scraper that counts the products and there is another service that is in charge of implementing all the mathematical model that the system has to calculate the category of the group.
But what I want to achieve is that my application would be able to give an estimate of how many products a research group should have to improve its category. I want to know if I can do that using neural networks.
For example, if there is a category C group, I want the application to tell the user how many articles and books it would make its category go up to B.
From what I have seen in some web resources, I could insert a training set into the neural network and have it learn to classify the groups, but I think it is unnecessary, because I can do that mathematically.
But I do not understand if it is possible for a neural network to process the current category that the group has and be able to give suggestions of how many products it needs to improve its category.
I think it must be a neural network with several outputs, so that in each one it throws the total for each one of the products, although it is not necessary to list all the products that the measurement model contemplates. But it is necessary for the network to learn which products are handled by a certain group, for example if a group does not write books, avoid suggestions that contemplate the production of books for the improvement of the category that the group has.

I believe you want a neural network that can predict future values of multiple variables given multiple inputs. This belongs to the general time series forecasting problem.
One of the best neural network architectures that can handle this problem is the LSTM, which is a type of Recurrent Neural Network. Their architecture allows them to develop a memory of what they have seen in the past and use it for future predictions. In other words, they can cross-correlate in a linear/nonlinear fashion several past steps of multiple variables to future values of multiple other variables, like a black box.
A useful tutorial for your purposes is this.


text:
What should I do when the potential value of a state is too high?
I'm working on a Reinforcement Learning task where I use reward shaping as proposed in the paper Policy invariance under reward transformations:
Theory and application to reward shaping (1999) by Andrew Y. Ng, Daishi Harada and Stuart Russell.
In short, my reward function has this form:
$$R(s, s') = \gamma P(s') - P(s)$$
where $P$ is a potential function.  When $s = s'$, then $R(s, s) = (\gamma - 1)P(s)$, which is non-positive, since $0 < \gamma <= 1$.
But considering $P(s)$ relatively high (let's say $P(s) = 1000$), $R(s, s)$ become too high as well (e.g. with $\gamma=0.99$, $R(s,s)=-10$), and if for many steps the agent stays in the same state, then the cumulative reward becomes more and more negative, which might affect the learning process.
In practice, I solved the problem by just removing the factor $P(s)$ when $s = s'$. But I have some doubts about the theoretical correctness of this "implementation trick".
Another idea could be to scale appropriately $\gamma$ in order to give a reasonable reward. Indeed, with $\gamma=1.0$, there is no problem, and, with gamma very near to $1.0$, the negative reward is tolerable. Personally, I don't like it because it means that $\gamma$ is somehow dependent on the reward.
What do you think?

Dennis Soemers provides an important point that from a theoretical standpoint, this can be seen as a non-issue. However, what you bring up is an important practical issue of potential-based reward shaping (PBRS).
The issue is actually worse than you describe---it's more general than $s = s'$. In particular, the issue presents itself differently based on the sign of your potential function. For example, in your case it looks like the potential function is positive: $P(s) > 0$ for all $s$. The issue (as you have found) is that an increase in potential (regardless of whether $s = s'$) might not be enough to overcome the multiplication by $\gamma$, and thus the PBRS term may be negative. In particular, only when the fold-change in $P$ is large enough ($\frac{P(s')}{P(s)} > \frac{1}{\gamma}$) will the PBRS term actually be positive.
The situation changes when the potential function is negative, i.e. if $P(s) < 0$ for all $s$. In this case, you can actually get a positive PBRS signal even when there is a decrease in potential! In particular, only when the fold-change in $P$ is large enough (same inequality as before) will the PBRS term actually be negative.
To summarize, when $P > 0$, a decrease in potential will always lead to a negative PBRS term, but an increase must overcome a barrier due to $\gamma$ for the term to be positive. When $P < 0$, an increase in potential will always lead to a positive PBRS term, but a decrease must overcome a barrier due to $\gamma$ for the term to be negative.
The intuition behind PBRS is that improving the potential function should be rewarded, and decreasing it should be penalized. However, it turns out that whether or not this holds true depends on things like 1) the sign of the potential function, 2) the fold-change in potential, or 3) the resolution of your environment. For #3, if the temporal resolution of your environment can be altered such that an action brings you "partway" from $s$ to $s'$, then at some environment resolution you will run into one of the two problematic circumstances above. Another issue is that PBRS is highly sensitive to, for example, adding a constant to the potential function.
Another related issue is that whether or not some constant improvement to the potential function leads to a positive/negative reward depends on how far you are from the "goal" state. Often potential functions are chosen such that they estimate how good a state is (after all, the best option for a potential function is the optimal value function). Say we choose $\gamma=0.99$ and that $P(s_{goal}) = 1000$ represents a goal state. Then increasing potential by one from $P(s) = 900$ to $P(s') = 901$ will have a negative reward of $-8.01$. In contrast, increasing potential by one from $P(s) = 90$ to $P(s') = 91$ will have a small positive reward of $+0.09$. This is another issue: the sign of the PBRS term depends on distance from the goal.
This paper has some interesting examples and outlines many of the issues above.
From my own experience, this is a large practical issue. The LunarLanderContinuous-v2 environment from OpenAI Gym includes a PBRS term, but they exclude the multiplication by $\gamma$ (i.e., $\gamma = 1$), presumably because the environment benchmark doesn't know the true discounting the RL user chooses. This environment can be solved using DDPG, for example, without significant hyperparameter tuning. However, if you use $\gamma = 0.99$ for your RL formulation, and edit the LunarLander code such that the PBRS term includes $\gamma = 0.99$, then DDPG fails to solve the environment. So, this is not a small computational issue---it has dramatic effects on training.
My solution has been to simply set $\gamma = 1$ in the PBRS term, even when using, say, $\gamma = 0.99$ in the RL formulation. This solves (or rather, circumnavigates) every issue above. While this loses out on the theoretical guarantee that adding the PBRS term does not affect the optimal policy, it can severely help training. (And there are no optimality guarantees using neural networks as function approximators anyway.)
This solution also seems to be what most benchmark environments have adopted. For example, most MuJoCo environments use PBRS terms with no $\gamma$ (equivalent to $\gamma = 1$). Alternatively, the omission of $\gamma$ could be attributed to the fact that including it would require the environment to know a priori what value of $\gamma$ the RL user chose. While feeding this into an OpenAI gym environment is easy to do, it's not typically done.
Keep in mind that while the theory guarantees that the optimal policy won't change by adding the PBRS term, adding the term doesn't necessarily help you approach the optimal policy. Yet, the whole point of using PBRS at all is to help you approach a good policy. So, it's a bit of a paradox, and I was comfortable with sacrificing the theoretical guarantee of policy invariance if it meant I could actually get to a good policy in the first place.

thank you! You confirmed my intuitions about this topic. As a reference, besides the cited paper, one might be interested in the one of the author's PhD thesis (Chapter 5): http://etheses.whiterose.ac.uk/936/

I don't think the situation you're sketching should be a problem at all.
If $P(s)$ is high (e.g. $P(s) = 1000$), this means (according to your shaping / "heuristic") that it's valuable to be in the state $s$, that you expect to be able to get high future returns from that state.
If you then continuously take actions that keep you in the same state $s$, it is essentially correct to punish these actions (with negative rewards); you were expecting to get high future returns starting from that state, but you just remain stuck in that state. This means that the actions you're taking are not providing the rewards you were expecting, so it should be "punished".
Of course, the exception to the above paragraph is the case where the transition from $s$ to $s$ (staying in the same state) does result in a positive "true reward" (actual/real reward, not shaping-reward). In that case, the true reward will offset the reduction in potential and result in a neutral or positive combined reward if it is sufficiently large.
As for the $\gamma$ parameter, a $\gamma < 1.0$ (such as $\gamma = 0.99$) intuitively means that you prioritize short-term rewards over long-term rewards (if the magnitudes of the undiscounted rewards are similar). This implies that what you wrote in the question is precisely what should happen; if you prioritize short-term rewards over long-term rewards, it's bad to "waste time" by staying in the same state, so actions that don't move you anywhere should be disincentivized through negative rewards. If $\gamma = 1.0$, you have no preference for short-term rewards over long-term rewards, so you'll be fine with "wasting time" by staying in the same state, and therefore no longer have to punish such actions.

Thank you! So do you think this "numerical issue" is not an issue at all, from a theoretical point of view? I.e. convergence is still guaranteed, provided all the assumptions and premises?

@MarcoFavorito Correct. It seems to me like it's doing exactly what it should be doing. gamma < 1.0 implies you want to punish "wasting time", and that's exactly what it's doing.

@DennisSoemers, gamma < 1.0 does not necessarily result in the potential-based term penalizing "wasting time." If the potential function is negative for all states, then "wasting time" will actually lead to a positive potential-based term. See my answer below for some intricacies/practical issues of potential-based reward shaping.

@BrendenPetersen Sure, I was specifically considering the case of the question, where potentials were said to be high positive values. In the case you propose, with negative potentials, indeed you'd get positive terms for staying in the same state. That's again not a problem by definition though. R(s, s) might be positive for a negative potential P(s), but R(s, s') for a state s' with a greater potential P(s') > P(s) will be even more positive, so that behaviour will still get reinforced more relatively.


text:
Why are GRU and LSTM better than standard RNNs?
It seems that older RNNs have a limitation for their use cases and have been outperformed by other recurrent architectures, such as the LSTM and GRU.

These newer RNNs (LSTMs and GRUs) have greater memory control, allowing previous values to persist or to be reset as necessary for many sequences of steps, avoiding "gradient decay" or eventual degradation of the values passed from step to step. LSTM and GRU networks make this memory control possible with memory blocks and structures called "gates" that pass or reset values as appropriate.

LSTMs or GRUs are computationally more effective than the standard RNNs because they explicitly attempt to address the vanishing and exploding gradient problems, which are numerical problems related to the vanishing or explosion of the values of the gradient vector (the vector that contains the partial derivatives of the loss function with respect to the parameters of the model) that arise when training recurrent neural networks with gradient descent and back-propagation through time.


text:
Was the Malaysian Airliner crash caused by a non-benign artificial intelligence system?
I watched a youtube clip of Elon Musk talking about his view on the future of AI. He gave two examples. One of the examples was a benign scenario and the other example was a nonbenign scenario where he speculated the possibilities of future AI threats and what harm a deep intelligence could do.
According to Elon, a deep intelligence in the network could create fake news and spoof email accounts. "The pen is mightier than the sword". This non-benign scenario put forth by Elon was a hypothetical, but he went into detail about how it could have been possible that an AI, with the goal of maximising the portfolio of stocks, to go long on defense and short on the consumer, and start a war.
To be more specific, this could be achieved by hacking into the Malaysians Airlines aircraft routing server, and when the aircraft is over a warzone, send an anonymous tip that there is an enemy aircraft flying overhead which in turn would cause ground to air missiles to take down what was actually a "commercial" airliner.
Although this is a plausible hypothetical nonbenign scenario of AI, I'm wondering if this actually could have been the case regarding the Malaysian Airliner crash. The Stuxnet, for example, was a malicious computer worm, first uncovered in 2010. Thought to have been in development since at least 2005 and believed to be responsible for causing substantial damage to Iran's nuclear program. The Stuxnet wasn't even an AI.
The Stuxnet blew the world's minds when it was discovered. The sheer complexity of the worm and the amount of time it took to build was impressive, to say the least.
In conclusion, was the Malaysian Airliner crash caused by a non-benign artificial intelligence system?

Stuxnet is not AI, and there is no need to posit AIs with amazing capabilities. Any reasonable definition of "hacking" for instance is beyond what any modern AI could do. This does not rule out that an event is caused by malicious intent from some "controlling" political cause, and it doesn't rule out hacking (although there's nothing that I can see here that points to it, it just seems another conspiracy theory). For now, any "smart" parts to such an event will be directed by people, and software will not be AI, but more direct, like a pre-installed back door or scanning for exploit.

I never said Stuxnet was AI. "The Stuxnet wasn't even an AI...."

This is just a conspiracy theorist's view.  There is no reason to believe any AI was involved.

Knowing the current state of AI research, I am pretty certain that there are not any meaningful AIs out there hacking into systems to enact some plan. The AI I would most be concerned about today is social profiling for targetting of "fact light, emotion heavy" political messages, or for Gerrymandering etc. That would include automated promotion of news articles and opinionss about events such as the Malaysian Airliner, in an attempt to astro-turf one or more conspiracy theories that suit a political agenda. There is evidence that both Brexit and US Election were targeted by this kind of AI.

This isn't a discussion site. Maybe post in chat when you have enough rep if you want something open-ended. By "meaningful" I mean something with capabilities implied by your question. I can know this with reasonable certainty because I follow cutting-edge published AI research, and have a good idea about what is currently possible using it. I think it highly unlikely that "weaponised" AI developed in secret is more advanced in agency and planning etc than the results you see in e.g. DARPA challenges or Boston Dynamics etc

Just because Musk is a promoter (your almost have to be in this media culture,) my sense is he's one of the "good guys".  Case and point, his major current endeavors, although for-profit, have great potential social benefit.  I am referring to his having made electric cars "cool", a great success, regardless of Tesla's future, and is leading the charge to get "our eggs out of a single basket" re: space colonization. Even the boring company was founded to alleviate a major modern stressor: traffic.  (Stress has been found to be a leading cause of health problems in general.)

@PandaSurge Please, provide a new link to the Musk video that you watched. The old one is not more accessible.

Short answer: despite the incredible advances in AI via Machine Learning (and subfields) AI is nowhere near this kind of autonomous decision-making.
I can't prove a negative, but, the level of autonomy Musk is talking about is still on the horizon.
Doesn't mean Musk is wrong about the hypothetical.  imo I'm glad he keeps bringing these kinds of issues to the fore.
Engineers and mathematicians have a bit more credibility on this subject than laypeople.  If it seems alarmist, there are many in the scientific community who feel it is warranted.
What Musk is describing is an extension of what Asimov felt compelled to warn us about via his "Three Laws of Robotics" back in 1942, when computers sucked.  For an explication of this idea re: Machine Learning, see below.

JUST FOR FUN (SORT OF:)
Author and mathematical physicist Hannu Rajaniemi just published a story in the MIT Tech Review on this very subject.
Unchained: A Story of Love, Loss, and Blockchain
Warning: the story is both wickedly funny, surprisingly moving, and very likely prescient!


text:
How to compute the action probabilities with Thompson sampling in deep Q-learning?
In some implementations of off-policy Q-learning, we need to know the action probabilities given by the behavior policy $\mu(a)$ (e.g., if we want to use importance sampling).
In my case, I am using Deep Q-Learning and selecting actions using Thompson Sampling. I implemented this following the approach in "What My Deep Model Doesn't Know...": I added dropout to my Q-network and select actions by performing a single stochastic forward pass through the Q-network (i.e., with dropout enabled) and choosing the action with the highest Q-value.
So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout?

So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout?

The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an approximation sample say 100 or 1000 actions with different dropout, to get a rough distribution.
I don't think this is feasible for practical reasons (the agent will learn so much more slowly due to these calculations, you may as well abandon Thompson Sampling and use epsilon-greedy), and you will have to avoid using importance sampling if you also want to use action-selection techniques where there is no easy way to calculate a distribution.
Many forms of Q-learning do not use importance sampling. These typically just reset eligibility traces if the selected action is different from maximising action.


text:
How is the depth of the input related to the depth of the output of a convolutional layer?
Let's suppose I have an image with 16 channels that goes to a convolutional layer, which has 3 trainable $7 \times 7$ filters, so the output of this layer has depth 3.
How does the convolutional layer go from 16 to 3 channels? What mathematical operation is applied?

Here is a related question (if not an exact duplicate).

The reason why you go from 16 to 3 channels is that, in a 2d convolution, filters span the entire depth of the input. Therefore, your filters would actually be $7 \times 7 \times 16$ in order to cover all channels of the input.
Detailed procedure
The output of the convolution automatically has a depth equal to the number of filters (so in your case this is $3$) because you have an $m \times k$ filter matrix, where $m$ is the number of filters and $k$ is the number of elements in the unrolled filter (in your case, $m = 3$ and $k = 7 \times 7 \times 16 = 784$, so the filter matrix is $3 \times 784$).
The input is usually unrolled according to the im2col procedure, where each tile corresponding to a single filter location is stretched into a column equal to the unrolled filter size. This is repeated for each filter location, so you end up with a very large matrix of size $k \times n$, where $k$ is the same as $k$ above in the filter matrix, and $n$ depends on your padding and stride.
Multiplying the $m \times k$ filter matrix with the $k \times n$ input matrix gives you an $m \times n$ output matrix, where $m$ is the number of filters.
Further reading
You can find some very nice visual explanations of the convolution procedure here and here.

Ok, that is explaining a lot of things. Thank you very much. Now I also understand why caffe2 is taking a NCVHW ordering for 3D convolutions (N=minibatch size, C=number of channels, V=Video frames, H=height, W=width).

Your input has 16 channels, each of dimension $m \times n$. There are 3 filters, namely $f_1$, $f_2$ and $f_3$ of spatial dimensions $k \times h$.
We say that a filter is applied to a channel when it is superimposed on the image, starting left-most, performing the operation of multiplying the weights of the filter with the corresponding value in the image and then summing up to a single value and moving the filter to right (then down when it reaches rightmost part) across the image according to the stride of the filter.
When a filter, e.g. $f_1$, is applied to a channel say $c$, there is a single value. Now, apply them to all channels, we get 16 values and all of them are added up to a single value. $f_1$ is moved according to the stride and the same operation is repeated to get an output with a single channel (the number of rows and columns are determined by padding, stride, dilation, and kernel size of the filers).
The aforesaid process is done by all the 3 filters giving rise to 3 channels. In this way, the convolutional layer makes the input go from 16 to 3 channels.
More detailed explanations can be found here.

It seems that this explanation is slightly wrong. The filter's depth is equal to the input's depth. Here, it seems that you're suggesting that the filter is a matrix, i.e. 2d, so it has depth 1, and that you apply this 2d matrix to each channel of the image separately, but this is not correct.


text:
Can an AI distinguish between good and bad according to people living in a restricted geographical area?
Would people go far with Artificial Intelligence and machine learning to the point where machines could learn during a long period of time to distinguish what's 'good' from 'bad' according to people living in a restricted geographical area, and then the machines take control and turn what was learned into a set of 'rules' and 'laws' (think of it as an effective machine of 'politics') that match the majority of the people's view of issues.
That should be accepted by everyone, since a contract set at the beginning says: "Everyone is ok".

Even if you could do that (which I believe is a long way off), what would be the point?
If I understand you correctly, you want an AI system to learn through observation of human beings what their 'rules of interaction' are. It sees a person killing someone else, and then that person is punished by the community, so the AI learns that killing people is not right. However, that is already codified in laws... So what it would pick up are social behaviours, which hopefully be things like "be nice to other people", "don't do anybody any harm", "be honest and truthful", etc.
The first question then is how an AI could assess events[*] as to whether they are 'good' or 'bad'. If everybody lies and steals, that would be learnt as normal behaviour, and the AI would not be able to pick up that most people would see this as 'bad'[**]. Causal relations are also hard to grasp. Somebody steals something. Then later someone else buys him a drink. So stealing stuff means other people will buy you drinks? These are really hard problems to solve. You need to know about people's motivations, and the multitude of 'threads' of interaction happening at the same time, even in a very limited area.
So, recognising events and causal links, plus a moral evaluation of them is pretty difficult. I don't think we will get there anytime soon. Unsupervised learning of behavioural is also pretty difficult, as you only have unlabelled observations and no real criteria to classify them. Plus, many actions are morally ambiguous. Killing people is generally seen as bad. What about the officers who tried to assassinate Hitler in 1944? Life is complex, and our artificial models are not anywhere near that.
So even if you were able to do all that, what would you end up with? An AI system that has picked up a lot of unwritten rules about human behaviour, and then postulates that as laws? So everyone has to behave the same way? I just don't see the point of that, even as a thought experiment.
[*] leaving aside here the question how you determine what an 'event' is in the first place
[**] Please note that even if people steal things, they steal can view theft as a bad thing.

Many thanks, I found some interesting points on your reply. But what if we think of a mechanism that builds a sort of consciousness ; for example, the act of killing might be seen as a 'good' event, the first time, but the next 10 times , it's not. Some functions could be implemented to decide as better as possible if something is good or bad. When it comes to people lives, ofc it's a bit dramatic, but in other situations, the idea might be of help.

The most interesting scientific field in terms of relevance to this question is probably "affective computing".
There are several problems with the model that you suggested. It is questionable if AGI should learn in the same way humans learn. In addition, there are several ethical problems surrounding this question, perhaps even metaethics, because the question peeks beyond human ethics.
A hardcoded ethics protocol might be possible to implement, similar to Isaac Asimov's "Three Laws of Robotics". The possibility of hacking is an issue that one would have to think proactively about - especially in this case.
If humanoid AGIs or robots are very similar to humans, they should probably be treated that way. For example, ethiologist Frans de Waal has studied empathy and social behavior in monkeys and suggested that we are very similar in behavior, so we are probably similar in terms of feelings (if it quacks like a duck and so forth). Perhaps we need an ethiology for androids too?


text:
Why does DQN require two different networks?
I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate action and the second network predicts the target Q values for finding the Bellman error.
Why can we not just make one single network that simply predicts the Q value and use it for both the cases? My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one. Is this the only reason? Am I missing something?

My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one.

It has no real impact on computation time, other than a slight increase (due to extra memory used by two networks). You could cache results of the target network I suppose, but it probably would not be worth it for most environments, and I have not seen an implementation which does that.

Am I missing something?

It is to do with stability of the Q-learning algorithm when using function approximation (i.e. the neural network). Using a separate target network, updated every so many steps with a copy of the latest learned parameters, helps keep runaway bias from bootstrapping from dominating the system numerically, causing the estimated Q values to diverge.
Imagine one of the data points (at $s, a, r, s'$) causes a currently poor over-estimate for $q(s', a')$ to get worse. Maybe $s', a'$ has not even been visited yet, or the values of $r$ seen so far is higher than average, just by chance. If a sample of $(s, a)$ cropped up multiple times in experience replay, it would get worse again each time, because the update to $q(s,a)$ is based on the TD target $r + \text{max}_{a'} q(s',a')$. Fixing the target network limits the damage that such over-estimates can do, giving the learning network time to converge and lose more of its initial bias.
In this respect, using a separate target network has a very similar purpose to experience replay. It stabilises an algorithm that otherwise has problems converging.
It is also possible to have DQN with "double learning" to address a separate issue: Maximisation bias. In that case you may see DQN implementations with 4 neural networks.

For additional reading, one can refer to the paper that introduced the Double DQN: Deep Reinforcement Learning with Double Q Learning (2016, AAAI-16) by Hado van Hasselt et al.

@amitection I don't understand why we need to separate two networks. Why can we not update Q in the original network? And for the problem of instability, do we use the same technique of soft updating of w for updating Q?


text:
What should we do when the selection step selects a terminal state?
In Monte Carlo tree search, what should we do when the selection step selects a terminal state (i.e. a won or lost state), which is, by definition, a leaf node? Expansion and simulation is not in order, as it's game over, but does the tree (score/visits) need to be updated (backpropagation)? Won't this particular node be selected continuously?

In the basic form, if you encounter a terminal leaf, you add visits and score depending on whether it is a win or loss, and backpropagate accordingly. The same as if you made a simulation step, but in this case the "simulation" is instant.
But you can improve that: If the leaf is losing, you can give it a very large negative score or even $-\infty$, so in the next selection step it surely won't be chosen, unless other moves are as bad. But if it is a winning leaf, you not only can give it a very big positive score or $\infty$, but also add a negative score for the immediate parent, so the parent won't be chosen, as it is obviously a losing state for him. This way we can save some simulations. I had encountered that situation many times in my game and Monte Carlo tree searches.
Suppose the parent has $200$ unexplored children, $10$ of which are immediate wins for the opponent. Your search may explore $100$ non-winning children and have a score of 80/100. But then it encounters the terminate child. After that, normally, it would have $80/101$, so, still, a big chance to be chosen in the next iteration. And it would take many iterations to see that this is not a good move, as it would need to get like $80/150$ or more. But if we cancel out the score or give it a negative one like $-1/101$, then we ensure it won't be chosen.
It seems in literature it's called "MCTS solver", to backpropagate proven wins and losses.


text:
How should I take into consideration the number of steps in the reward function?
I am currently implementing the paper Active Object Localization with Deep Reinforcement Learning in Python. While reading about the reward scheme I came across the following:

Finally, the proposed reward scheme implicitly considers the number of steps as a cost because of the way in which Q-learning models the discount of future rewards (positive and negative).

How would you implement this "number of steps" cost? I am keeping track of the number of steps that have been taken, therefore would it be best to use an exponential functions to discount the reward at the current time step?
If anyone has a good idea or knows the standard in regard to this I would love to hear your thoughts.

How would you implement this "Number of Steps" cost?

What the paper is referring to is the reward discounting process which is a standard way of formulating RL problems, either continuous ones, or episodic ones where the goal is to complete a task in the least time (in the episodic version, a fixed cost per time step will also achieve this).
As such, this usually is implemented in the formulation of the value function calculations. The discount factor is usually represented as gamma, $\gamma$.
For Q-learning, the factor should be in the TD target calculation:
$$G_{t:t+1} = R_{t+1} + \gamma \max_{a'}Q(S_{t+1},a')$$
For Monte Carlo control, the factor appears more like this in the calculation of a return:
$$G_t = \sum_{k=0}^{T-t} \gamma^k R_{t+k+1}$$

would it be best to use an exponential functions to discount the reward at the current time step?

Essentially that is what normal discounting is - an exponential decay of future reward. But if you have implemented "normal" Q-learning from equations like those above, it should already be there.


text:
What is the general procedure to use and train neural networks for multi-class classification?
I am very new to machine learning. I am following the course offered by Andrew Ng. I am very confused about how we train our neural network for multi-class classification.
Let's say we have $K$ classes. For $K$ classes, we will be training $K$ different neural networks.
But do we train one neural network at a time for all features, or do we train all $K$ neural networks at a time for one feature?
Please, explain the complete procedures.

Let's say we have $K$ classes. For $K$ classes, we will be training $K$ different neural networks.

No, you still train one network.
With binary classification tasks, where you have only two mutually exclusive categories, like "yes/no" or "true/false", you can get away with a single output node with a sigmoid activation. The output of the sigmoid is interpreted as indicating one category for values $> 0.5$ and the other for values $\leq 0.5$.
With multi-class classification, you have $K$ outputs (one for each category). The problem, in this case, is that if the network gets the class wrong, in general, you cannot decide in one step which one of the other $K - 1$ categories is the correct one. So, the output is actually passed through an extra softmax layer, which outputs probabilities for each class.

But do we train one neural network at a time for all features, or do we train all $K$ neural networks at a time for one feature?

You present all features for each training example to the network at the same time. So, for $N$ features you have $N$ input nodes, and you feed all of them into the neural network.

okay it means for K classes and M training examples we will be training our neural network K*M times.

One more doubt sir.Do we apply gradient descent(any varient) when using neural network? in this code they are applying advance optimization before Back Propagation. According to me in BP as we are already finding the best set of weights then applying gradient descent is irrelevant in this case.Am I correct or  missing something ?

@ReenaKandari If you have M data points, you would be using a subset (generally, between 60% and 80% of M) for training, and you do that multiple times (= epochs). However, you don't have to train the network K times for each training example - the whole point of softmax is that the correct class is "promoted" while all others are "suppressed" simultaneously, if that makes sense.

@ReenaKandari Re your other question: yes, in the overwhelming majority of cases neural networks are optimised using gradient descent. I am not proficient in Matlab, but it seems to me that in the example you provided they are minimising a cost function using gradient descent.

sir @cantordust, actually I am not aware of how Softmax works and how classes are 'promoted' and 'suppressed simultaneously '.Can you please explain it a little bit more that how these things  works in terms of neural network?

Let us suppose that you are training a neural network for classfing images of vehicles , then the input vector , image of the "vehicle" will be a 2D array of pixels. This undergoes several transformations at each layer of the neural network , the last layer of the neural network produces another vector whose dimensions are lesser than the original image vector.
So the network is mapping images to some vectors in a high dimensional space. In order to classify the images it is now sufficient to classify the vectors obtained from the network of their corresponding images, you  can do this with a simple "linear" classifier using a softmax layer.
So all the layers of the network except the last layer are transforming the image representation to a "vector". This vector is classified by a linear softmax classifier by the last layer of neural network.


text:
What's a good resource for getting familiar with reinforcement learning?
I am familiar with supervised and unsupervised learning. I did the SaaS course done by Andrew Ng on Coursera.org.
I am looking for something similar for reinforcement learning.
Can you recommend something?

To the good answers here, I would add

A brief overview of RL: Most essential concepts in one place.
Another brief overview, in presentation format.
Ben Recht's An outsider's tour of RL is pretty comprehensive and accessible.
The Bellman equations: central to the whole RL theory.
Policy gradients explained by Andrej Karpathy (mentioned in other answers as "pong from pixels", this is the link).

These barely scratch the surface of RL, but they should get you started.

There's a Youtube playlist (in the DeepMind channel) whose title is Introduction to reinforcement learning, which is a course (of 10 lessons) on reinforcement learning by David Silver.
A person who followed and finished the course wrote (as a Youtube comment):

Excellent course. Well paced, enough examples to provide a good intuition, and taught by someone who's leading the field in applying RL to games.

Before that ask yourself if you really want to learn about "reinforcement learning." Although there is much hype about reinforcement learning, the real-world applicability of reinforcement learning is almost non-existent. Most of the online courses teach you a very little about machine learning, so it is much better to get thorough with it, rather than proceeding towards reinforcement learning. Learning reinforcement learning is somewhat different from learning about unsupervised/supervised learning techniques.
Having said that, the fastest way to get a good grasp of reinforcement learning is as follow:

Read Andrej Karpathy's blog post "Pong from Pixels."
Watch Deep RL Bootcamp lectures.
To understand the math behind these techniques, refer to Sutton and Barto's Reinforcement Learning: An Introduction.
Read relevant papers (game-playing etc.).

P.S: Make sure that you are thorough with basics of neural networks, as most of the current papers in RL involve using DNNs in some or the other way as approximators.

real-world applicability of reinforcement learning is almost non-existent AlphaGo was trained with reinforcement learning.

Thanks, what I like about reinforcement learning is that it can improve itself just by doing the task over and over again. No supervision is needed. Only the model has to describe the problem correctly.

@cantordust , check https://www.alexirpan.com/2018/02/14/rl-hard.html

@thecomplexitytheorist Yes, I am aware of that post, and yes, there are difficulties associated with RL (like any other ML method). However, these difficulties do not imply that its applicability is nil. Another successful example (also mentioned in the post) is the improvement of cooling efficiency of Google's data centres by 40% - hardly insignificant.

@cantordust , you should read that post completely , there is no "reinforcement learning" learning involved in "improving cooling efficiency" , they have clearly mentioned that they made predictions based on past power consumption data.

@cantordust Supposed that they used RL, another problem with Google's claim is that they did not provide baselines for comparison. We can only see ML-ON and ML-OFF differences. Very simple heuristics may do much better, though this depends on the system, which we cannot look into.

MIT Technology review. Learning from past data is at the core of RL...and a lot of other ML algorithms. Also, all other things being equal, the ML on/off comparison is pretty convincing. At any rate, I am surprised at the negative reactions towards RL. It does have flaws and it is far from perfect, but it does have potential. That is precisely why we need more people working on it.

I recently saw a course by Microsoft on edx. It is called 'Reinforcement Learning Explained'.
Here is the link:
https://www.edx.org/course/reinforcement-learning-explained-0
This is not quite comprehensive but at least gives a good starting point.

I would say this post is a must to read:

https://rubenfiszel.github.io/posts/rl4j/2016-08-24-Reinforcement-Learning-and-DQN.html

authors: Женя Семенюк


text:
Can I build a CNN for image classification tasks just with OpenCV?
I have practiced building CNNs for image classification with TensorFlow, which is a nice library with good documentation and tutorials. However, I found that TensorFlow is too complicated and cumbersome.
Can I build a CNN for image classification tasks just with OpenCV?

OpenCV does include 2D filter convolution functions for custom separable and non-separable filters. The latter uses DFT for large filters, which may or may not be faster than the conventional method. It also includes (partial?) support for deep nets with various types of layers. Theoretically, you should be able to stitch everything together into a complete CNN. However, I have not used any of those, and I have no idea about the level of maturity of the implementation.
That said, if you are willing to implement a custom CNN from scratch, you will probably get more control over the implementation using a generic (BLAS / OpenCL / CUDA) matrix library.


text:
Is the word "pose" used correctly in the paper "Matrix Capsules with EM Routing"?
In traditional computer vision and computer graphics, the pose matrix is a $4 \times 4$ matrix of the form
$$
\begin{bmatrix}
r_{11}  & r_{12}  &  r_{12}   & t_{1} \\
r_{21}  &  r_{22}   & r_{22} &   t_{2} \\
r_{31}   & r_{32}  &  r_{32}  &  t_{3} \\
0   &   0    &  0   &   1
\end{bmatrix}
$$
and is a transformation to change viewpoints from one frame to another.
In the Matrix Capsules with EM Routing paper, they say that the "pose" of various sub-objects of an object are encoded by each capsule lower layer. But from the procedure described in the paper, I understand that the pose matrix they talk about doesn't conform to the definition of the pose matrix. There isn't any restriction on keeping the form of the pose matrix shown above.

So, is it right to use the word "pose" to describe the $4 \times 4$ matrix of each capsule?
Moreover, since the claim is that the capsules learn the pose matrices of the sub-objects of an object, does it mean they learn the viewpoint transformations of the sub-objects, since the pose matrix is actually a transformation?

Great question, and one that I think we could have done a better job of answering in the paper.
Essentially, the pose matrix of each capsule is set up so that it could learn to represent the affine transformation between the object and the viewer, but we are not restricting it to necessarily do that. So we talk about the output of a capsule as though it is an affine transformation matrix, but we can't ensure that it will be. We do things explicitly that make it more like such a matrix — like adding in the coordinates to the right-hand column — but we can't be sure. This somewhat embodies a large part of the capsule network theory — we set up scaffolding so that the network can learn to be equivalent to transformations that we think it ought to be invariant to, but we don't ensure that it is.

Don't you think having standard transformation is wrong? Principally, your proposed network in 2017 and 2018 actually learns an inverse transformation and not a direct transformation. Consequently, in the inversed matrix, it is possible that all entries have values.

I have tried to make it learn the affine transformation by giving it this as the label, and it works just fine. I'm really impressed and excited by capsule networks, and can't figure out why anyone didn't think of this before, because it's so obvious and simple. Spiking neurons also tell us that information between neurons can't be one dimensional only. It should be represented by Vectors of some kind.
UPDATE:
In the above comment I claim that it works "fine" when I make a capsule network learn the affine transformation by giving it this as the label. This is not true.. It doesn't work! - I'm sorry, I was too quick there.
I assume the reason is that the affine 4x4 matrix representation is redundant. Also it is impossible to make sensible linear interpolations between such transformations, which will affect the gradient (it will not point in the direction of the minimum).
What I have succeeded doing is to make the capsule network learn a quaternion (rotation) and a 3d vector (position) - 7 parameters in all. These can be contained in a 3x3 matrix, when fixing 2 of the parameters. But training is slow and the network cannot encode skews etc. in this 3x3 setup.
Affine transformations from images using capsule networks (matrix capsules), can also be achieved by just making the network learn its own 4x4 pose representation through the decoder part. Then a small network can be trained to transform these poses into a 7d vector (quaternion and 3d vector), from which the affine 4x4 transformation obviously can be calculated. This I have also succeeded doing. It seems like the rotation encoded in the pose has a more quaternion-like nature, which makes sense.


text:
Given a set of images that are not divided into groups, which algorithm should I use to do that?
I'm a complete newbie to NNs, and I need your advice.
I have a set of images of symbols, and my goal is to categorize and divide them into groups of symbols that look alike. Without teaching NN anything about the data.
What is the best way to do this? What type of NN suits the best? Maybe there are any ready solutions?

This is called "clustering" , If the network is already trained with data that has similar features as of the "symbols", you can use that network with its last classification layer removed , then run a clustering algorithm like "k-means" on top of the vectors obtained from the last layer of the network.


text:
Do we know what the units of neural networks will do before we train them?
I was learning about back-propagation and, looking at the algorithm, there is no particular 'partiality' given to any unit. What I mean by partiality there is that you have no particular characteristic associated with any unit, and this results in all units being equal in the eyes of the machine.
So, won't this result in the same activation values of all the units in the same layer? Won't this lack of 'partiality' render neural networks obsolete?
I was reading a bit and watching few videos about backpropagation and, in the explanation given by Geoffrey Hinton, he talks about how we're trying to train the hidden units using the error derivatives w.r.t our hidden activities rather than using desired activities. This further strengthens my point about how by not adding any difference to the units, all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal.

In reverse order to how you asked:

all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal

This actually happens if you initialise the weights equally (e.g. all zero). Gradients in that case are the same to each neuron in the same layer, and everything changes in lockstep. A neural network without random weight initialisation will simply not work.

So won't this result in the same activation values of all the units in the same layer? Won't this lack of 'partiality' render neural networks obsolete?

No, because random weight initialisation causes the gradients to be different, and the neuron activations will typically diverge to represent different "hidden" features that activate differently depending on the input.

What I mean by partiality there is that, you have no particular characteristic associated with any unit and this results in all units being equal in the eyes of the machine.

One interesting side-effect of this behaviour is that the "partiality" will often be assigned effectively randomly too, as the neural network will converge to features that somehow work. These features have no guarantee to be meaningful to a human being in the context of a problem to solve. They might be something that can be mapped to the problem, they might be some linear combination of something that can be understood, but often there is no obvious interpretation.