text:
Can we detect the emotions (or feelings) of a human through conversations with an AI?
Can we detect the emotions (or feelings) of a human through conversations with an AI?
Something like a "confessional", disregarding human possibilities to lie.
Below, I have the categories joyful, sadness, anger, fear and affection. For each category, there are several words that can be in the texts that refer to it.

Joy: ( cheerful, happy, confident, happy, satisfied, excited, interested, dazzled, optimistic, relieved, euphoric, drunk, witty, good )

Sadness: ( sad, desperate, displeased, depressed, bored, lonely, hurt, desolate, meditative, defrauded, withdrawn, pitying, concentrated, depressed, melancholic, nostalgic )

Anger: ( aggressive, critical, angry, hysterical, envious, grumpy, disappointed, shocked, exasperated, frustrated, arrogant, jealous, agonized, hostile, vengeful )

Fear: ( shy, frightened, fearful, horrified, suspicious, disbelieving, embarrassed, embarrassed, shaken, surprised, guilty, anxious, cautious, indecisive, embarrassed, modest )

Affection: ( loving, passionate, supportive, malicious, dazzled, glazed, homesick, embarrassed, indifferent, curious, tender, moved, hopeful )

Flow Example
Phrase 1: "I'm very happy! It concludes college."
Categorization 1:
 - Joy (+1)

Sadness (-1)

Phrase 2: "I'm sad, my mother passed away."
Categorization 2:
 - Sadness (+1)

Joy (-1)

Phrase 3: "I met a girl, but I was ashamed."
Categorization 3:
 - Fear (+1)
Is this a clever way to follow and / or improve, or am I completely out of the way?
I see that there is a Google product that creates parsing according to the phrases. I do not know how it works, because I like to recreate the way I think it would work.
Remembering that this would not be the only way to categorize the phrase. This would be the first phase of the analysis. I can also identify the subject of the sentence, so we would know if the sadness is from the creator of the message or from a third party, in most cases.

NLTK
Sentiment Analysis Python Example

I think a big problem with algorithms and NLP is not just context, but subtext.  The meaning is not always the literal meaning, and simple statements can often have layers of meaning.  Regardless, I think your endeavor is worthwhile, even if it doesn't produce strong results initially.  This seems like a problem that will have to be "chipped away" over many iterations.

Thank you. I found a library in Python that does a similar form (by words). But it takes human labor to add words and their values.

There are a number of papers on facial recognition techniques (and other biometrics) to gauge emotion.  As a former poet and dramatist, I can tell you that's a much less complex approach than trying to intuit the meaning of subtle text (which might turn out to require something akin to AGI--many humans seem not to get subtext!) But if you can correlate facial expression with text or spoken language, it would probably be fairly accurate in determining if the literal meaning is correct, or if sarcasm or irony is involved.

"I'm so happy because today I've found my friends" is a great, simple example of the underlying problem.  Without the melody, the interpretation is in opposition to what was almost certainly intended.

Makes sense. Maybe the text or video is not enough for a precise value, but the two together improve accuracy. Thank you.

Following the example of the repository I added, you can join forces of compound words or single words. In case "happy" with value 3 and "so happy" the value 4 and "so much happy" the maximum value 5.

Exactly.  Curt Cobain was many things, but happy was not one of them.  :)

I don't want to pour cold water over your approach, but I am very sceptical and (having worked in sentiment analysis myself) think it is way too simplistic.
Various communicative intents are encoded in language, and there is a wide range of linguistic features that are employed for that purpose. Choice of words is only one of them; it is the most obvious one, as we can easily see the words themselves. But words in isolation do not mean anything, context is important. It is of course not difficult to come up with example sentences where the sentiment effect of the words you list is reversed. The easy one being negation: I'm not happy about this. Sure, you can check if there is a not before the word, but what about I would be happy if you stopped making such a noise. -- surely here the current state would be one of unhappiness? If you think about real examples, it suddenly becomes very complicated.
Also, words usually have multiple meanings: This cup is just shy of one litre. I'm sure you'd agree that this does not express 'fear'. And The shunter moved the tender on the old steam engine. is not about affection. But solving this problem involves word sense disambiguation, which in itself is a hard problem to solve.
The problem is, initially word-based approaches look really good and transparent, as you can easily see what's going on. But language unfortunately doesn't play ball, and in real life systems don't tend to work very well. Lexical choice is only one way to encode sentiment, there are also grammatical patterns. But these are often very subtle, and not yet well-explored in linguistic research.
To end on a positive note, have a look at research in evaluation (which is kind of related to sentiment). For example, Susan Hunston's Corpus Approaches to Evaluation,  (Routledge 2011). That should give you some further pointers.

I fully understand what you're saying and agree. I'm not saying that the way I explained here is correct. I am looking for new logics, for new steps. For example, in your sentence: "I would be happy if you stopped making such a noise.". The "If you" turns the sentence into a condition. I can create scoring rules for conditions. As a programmer, I can turn logic into an algorithm. the point is, I am here trying to raise this issue because I believe there are millions of logics to be implemented in this basic that I created. I thank you for your participation, I'll study your directions!

I think you are definitely on a very sensible track. No one defines right or wrong in emotion field. It's not hard science. It's all theories.
I have recently read a paper regarding emotions in Reinforcement Learning (RL). It has explained briefly emotion from 3 perspectives: psychology, neuroscience and computer science. In particular, your way of emotion definition matches one of the categorical emotion theory in the psychology perspective. The other theories in psychology perspective and componential emotion. You can try to implement them and try out which one works well. The paper has also introduced ways to measure the level of emotions (emotion elicitation).
Here is the link for the paper I have mentioned. I am sure you will receive lots of inspiration. I have also written a summary of this paper. Take a look if the original paper is too long to read.
I don't have any concrete solution for implementing. But the general idea is always trying to categorize abstract concepts and quantify them. And try some thing, and iteratively modify and improve it. All the best!

Thanks for the material. I fully agree. Therefore, I believe that the best way to obtain real and useful data for this type of analysis needs to come from communities, polls, etc. Something like, "What does this word mean to you?" That is, a long and laborious search. Remembering that the search results vary from culture, parents, needs, etc.

It could work using supervised learning , as long as you have the required dataset.
However, a low error ratio using unsupervised learning of the human emotion spectrum would prove to be more difficult.
Ex :
How would you defined being in love to a neural network ?
Joy +1 , Sadness -1 ?
Now , How would you define being in love with , let’s say, someone you know you could never be with ?
Joy -1 , Sadness +1, but at the same time , the only fact that you are thinking about that person bring a Joy +1 .
Human emotions are quite complex .
A good start (in my humble opinion) would be to read about ‘emotion-related’ hormones , and how they affect the brain ( dopamine , serotonin, etc).
Some emotions are really a precise mix of these hormones , probably giving you a good hint on how to ‘caregorize’ your network.

Yes. Maybe with an operation screen. But the idea is to be unsupervised.


text:
Why do we need learning in unsupervised learning?
I am not clear with the concept that an unsupervised model learns. We are giving an input and output to the supervised model, so that it can generate a particular value, pattern or something out of it which can be used to categorize something in the future. By contrast, in unsupervised learning, we are clustering, so why do we need learning?
Can anyone detail me with some real-world examples?

Imagine you have a dataset of people who have cancer.
You have information about their age, physique, diagnosis, treatments, and results.
Using this data, you want to prescribe a set of treatments for a new patient, P.
Obviously, if there is someone in the dataset that has very similar traits as P and had a positive result with their treatments, you could prescribe the same set of treatments. However, this is incredibly unlikely and becomes more infeasible as more information about P is observed (e.g. Has brown hair and hates pasta).
A better option is to cluster the dataset into groups that have positive outcomes for treatment results. For example, perhaps patients with lung cancer who smoke and are given treatment A do better than patients with lung cancer who didn't smoke and are given the same treatment A. These patients should then be divided based on this outcome.
Once these different clusters are found, patient P can be evaluated against each of the clusters and a set of treatments can be prescribed (e.g. Most of the treatments from cluster A, but 1 treatment from cluster B).
Unsupervised learning is the method of finding these clusters, which helps find structure to the data to better answer questions.

Supervised Learning: This is performed with the help of a teacher. A child works on the basis of the output that he/she has to produce. Their actions are supervised by a teacher. Similarly in ANNs, each vector requires a corresponding target vector, which represents the desired output.
Unsupervised Learning: Consider the learning process of a tadpole, it learns by itself, it isn't taught by any teacher. In ANN, during the training process, the network receives the input patterns and organizes these patterns to form clusters. When a new input pattern is applied, the neural network gives an output response indicating the class to which the input pattern belongs. If for an input, a pattern class cannot be found then a new class is generated.
In this case, there is no feedback from the environment, the network must itself discover patterns, regularities, features or categories from the input data and relations for the input data over the output.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
Did Turing foresee the required capabilities to pass the Turing test?
In Section 1.1 of Artificial Intelligence: A Modern Approach, it is stated that a computer which passes the Turing Test would need 4 capabilities, and that these 4 capabilities comprise most of the field of Artificial Intelligence:

natural language processing: to enable it to communicate successfully in English
knowledge representation: to store what it knows and hears
automated reasoning: to use the stored information to answer questions and to draw new conclusions
machine learning: to adapt to new circumstances and to detect and extrapolate patterns

Did Alan Turing discern the requirements for the field of artificial intelligence (the necessary subfields) and purposefully design a test around these requirements, or did he simply design a test that is so general that the subfields which developed within artificial intelligence happen to be what is required to solve it? That is, was he prescient or lucky? Are these Turing's subdivisions, or Peter Norvig's and Stuart Russell's?
If Turing did foresee these 4 requirements, what did he base them on? What principles of intelligence allowed him to predict the requirements for the field?

I find it unlikely that you'll find a firm answer, so I will try my best to guide you towards information which may help you form an opinion either way. Turing had the controversial opinion (which remains controversial today) that:

Digital computers have often been described as
  mechanical brains. Most scientists probably
  regard this description as a mere newspaper
  stunt, but some do not. One mathematician has
  expressed the opposite point of view to me rather
  forcefully in the words “It is commonly said that
  these machines are not brains, but you and I
  know that they are.” […] I shall give most attention
  to the view which I hold myself, that it is
  not altogether unreasonable to describe digital
  computers as brains.

In essence Turing believed that the digital computers of the time had the capacity to mimic the human brain if programmed correctly. The problem for Turing then, was finding the correct procedures to mimic the brain. Remember the Imitation Game was being used as a means to explore the question "can machines think?" Turing was engaging in an intellectual debate about the nature of "thinking", he was not making an attempt to describe how machines would be taught to think.
Since, both artificial intelligence and machine learning are at their core very similar to "artificial thinking" (as Turing may have called it) it seems necessary that a test designed to decide whether machines can think would also encompass whether machines could be intelligent or whether machines could learn.
Therefore, it seems possible that Turing did not make the question with these capabilities in mind, but rather that these capabilities are what guided Turing to this question as they are so innately tied to thinking.
If you would like to look further into Turing's thought process here is his paper on the imitation game and here is a discussion on the proceedings of a BBC broadcast about "Automatic Calculating Machines" in which Turing was a speaker.

Very interesting. What precisely do you mean by "these capabilities... are so innately tied to thinking." Do we have any sources/reasons to believe that these 4 capabilities are a breakdown of the process of thinking?

When we talk about thinking we often use words like remember, learn, and reason as components of thinking.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to compute the derivative of the error with respect to the input of a convolutional layer when the stride is bigger than 1?
I read that to compute the derivative of the error with respect to the input of a convolution layer is the same to make of a convolution between deltas of the next layer and the weight matrix rotated by $180°$, i.e. something like
$$\delta^l_{ij}=\delta^{l+1}_{ij} * rot180(W^{l+1})f'(x^l_{ij})$$
with $*$ convolution operator. This is valid with $stride=1$.
However, what happens when stride is greater than $1$? Is it still a convolution with a kernel rotation or I can't make this simplification?

Backpropagation with stride > 1 involves dilation of the gradient tensor with stride-1 zeroes. I created a blog post that describes this in greater detail.

From the paper found from the post linked below:

'We find that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks'

All that means that only values are skipped (=pooling is made) to the matrix, otherwise all works like a convolution should do.
Sources:
https://arxiv.org/pdf/1412.6806.pdf
https://stackoverflow.com/questions/44666390/max-pool-layer-vs-convolution-with-stride-performance

This is not an answer to my question; you are talking about pooling, my question is about simple convolution in backpropagation algorithm.

Well, the point is that strides introduce pooling kind of phenomenom and otherwise it does not change CNN performance and if I read my source right, also correctness.

Probably more direct professional expertice you could get on datascience.SE site. I belong to both of these, thus I knew sth of the issue.

I have just the same problem, and I was trying to derive the backpropagation for the convolutional layer with stride, but it doesn't work.
When you do the striding in the forward propagation, you chose the elements next to each other to convolve with the kernel, then take a step $>1$. This results in the fact that in the backpropagation, in the reverse operation, the delta matrix elements will be multiplied by the kernel elements, (with the rotation) but not as strided, but you are picking elements that are not next to each other, something like $DY_{11} * K_{11} + DY_{13} * K_{12} + DY_{31} * K_{21} + DY_{33} * K_{22}$, which is NOT the equivalent as a convolution with a stride $>1$.
So as far as I am concerned, if I would like to implement the ConvNet by myself to get a better grasp of the concept, I have to implement a different method for the backprop, if I allow strides.


text:
In order to classify the taste of crystals, how should I make the training, validation and test datasets?
Right now, I'm planning to make a deep neural network for classifying taste of crystals, with their molecular structure, which includes information like the number of atoms or the mass of each atom.
How should I make a data set for training, testing, and validation?

You can cluster all your features in one matrix X, in which each line would be one element of the data set you want to construct, and each column would be a different feature of this element.
You construct then a Y vector containing the different target classes, where the i-th element will be the target class of the i-th X element.
For the following I suppose you use python
Create train and test sets using sklearn wrapper train_test_split on X and Y just built, in which you can shuffle your X and Y (non-avoidable in this case) :
from sklearn.model_selection import train_test_split

BTW, I recommend you to rescale your data with sklearn with for example one of :
from sklearn.preprocessing import MinMaxScaler, StandardScaler

It allows better learning and so better generalization

Well done on train and test split and scaling. However, @stain also asked about validation set. Some packages (eg most of Keras supervised learning algorithms) can consider a portion (by with absolute or relative size) of training set as the validation set. Otherwise, one can perform train_test_split twice (one on whole data and once on the train set). See https://datascience.stackexchange.com/a/15136/46505

I also thought of mentioning cross-validation using k-splits (kfolds) generalizing the concept of training. As it is mentioned in your link @O_o answer seems also relevant here.


text:
Are all fully observable environments episodic?
According to the definition of a fully observable environment in Russell & Norvig, AIMA (2nd ed), pages 41-44, an environment is only fully observable if it requires zero memory for an agent to perform optimally, that is, all relevant information is immediately available from sensing the environment.
From this definition and from the definition of an "episodic" environment in the same book, it is implied that all fully observable environments are, in fact, episodic or can be treated as episodic, which doesn't seem intuitive, but logically follows from the definitions. Also, no stochastic environment can be fully observable, even if the entire state space at a given point in time can be observed because rational action may depend on the previous observation that must be remembered.
Am I wrong?

No, not all fully observable environments are episodic. Let's take a look again at the definitions from the book:
Fully Observable Environment (section 2.3.2)

If an agent’s sensors give it access to the complete state of the environment at each point in time, then we say that the task environment is fully observable. A task environment is effectively fully observable if the sensors detect all aspects that are relevant to the choice of action

Episodic Environment (section 2.3.2)

In an episodic task environment, the agent’s experience is divided into atomic episodes. In each episode the agent receives a percept and then performs a single action. Crucially, the next episode does not depend on the actions taken in previous episodes.

Take note of the "crucial" part at the end of the definition of episodic environment. A fully observable environment that is not episodic (and therefore sequential in the book's taxonomy) is chess. Chess is fully observable because the player can view the positions of all active pieces on the chess board, and that is all the information that needs to be known in order to take the optimal action. But chess is not episodic, because the player's current move depends on all previous moves, and the current move will have downstream effects in later turns.
In fact, if you look at Figure 2.6 in the book on pg. 45, they provide three examples of fully observable sequential (i.e. not episodic) environments: crossword puzzles, chess, and backgammon. There are of course many more. Most games are sequential as that is the main appeal of them - how to best sequence my moves now in order to ensure victory over my opponent at a future time?

I'm not sure your claim that chess is not episodic is correct, even though Figure 2.6 really says that "chess with a clock" is sequential and not episodic. If you consider a single chess game, that could be viewed as a single episode. Another chess game would be another episode, which doesn't really depend on the previous episodes, so chess would be an episodic task.

I see your point, I think it depends on what you consider the agent's action space though. In a game of chess the action space is moving a piece and that move will depend on the previous moves within the game. With what you're suggesting, the action space is "play a game of chess", which could make sense if your task is to say learn a strategy that's effective against a particular opponent. But I think that's a different level of abstraction than how you characterize the game of chess itself. At that higher abstraction, nearly all games are episodic.

When someone says "episodic task", I think of RL. Are you familiar with RL? if I was to formulate chess as an RL problem, how would you define an episode? Wouldn't an episode be a full game of chess until termination? That's why I'm not sure about your conclusion (and that figure 2.6; I actually didn't fully read that section, so I don't exactly why Norvig and Russell that decided to categorize chess as non-episodic.).

Yeah based on the definition R&N give I think they don't have the RL sense of episode in mind, because it's about a single percept->action iteration, and they're differentiating whether the action sequence is Markovian or not.


text:
In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?
My understanding is that the convolutional layer of a convolutional neural network has four dimensions: input_channels, filter_height, filter_width, number_of_filters.  Furthermore, it is my understanding that each new filter just gets convoluted over ALL of the input_channels (or feature/activation maps from the previous layer).
HOWEVER, the graphic below from CS231 shows each filter (in red) being applied to a SINGLE CHANNEL, rather than the same filter being used across channels.  This seems to indicate that there is a separate filter for EACH channel (in this case I'm assuming they're the three color channels of an input image, but the same would apply for all input channels).
This is confusing - is there a different unique filter for each input channel?

This is the source.
The above image seems contradictory to an excerpt from O'reilly's "Fundamentals of Deep Learning":

...filters don't just operate on a single feature map. They operate on the entire volume of feature maps that have been generated at a particular layer...As a result, feature maps must be able to operate over volumes, not just areas

...Also, it is my understanding that these images below are indicating a THE SAME filter is just convolved over all three input channels (contradictory to what's shown in the CS231 graphic above):

https://arxiv.org/pdf/1707.09725 chapter 2

Great question and answers below. The thing that threw me was that you actually need the same amount of filters as the amount of output channels. So a 2dConv on an RGB image that outputs an RGB image will require 3 filters of shape [3,W,H]. So the actual volume of weights is [3,3,W,H]. I wrote an implementation in pytorch here. https://stackoverflow.com/a/62678843/5484902

The following picture that you used in your question, very accurately describes what is happening. Remember that each element of the 3D filter (grey cube) is made up of a different value (3x3x3=27 values). So, three different 2D filters of size 3x3 can be concatenated to form this one 3D filter of size 3x3x3.

The 3x3x3 RGB chunk from the picture is multiplied elementwise by a 3D filter (shown as grey). In this case, the filter has 3x3x3=27 weights. When these weights are multiplied element-wise and then summed, it gives one value.
So, is there a separate filter for each input channel?
YES, there are as many 2D filters as the number of input channels in the image. However, it helps if you think that for input matrices with more than one channel, there is only one 3D filter (as shown in the image above).
Then why is this called 2D convolution (if the filter is 3D and the input matrix is 3D)?
This is 2D convolution because the strides of the filter are along the height and width dimensions only (NOT depth) and therefore, the output produced by this convolution is also a 2D matrix. The number of movement directions of the filter determines the dimensions of convolution.
Note: If you build up your understanding by visualizing a single 3D filter instead of multiple 2D filters (one for each layer), then you will have an easy time understanding advanced CNN architectures like Resnet, InceptionV3, etc.

this is a good explanation, but more specifically the question I'm trying to understand is whether the filters that operate on each input channel are copies of the same weights, or completely different weights.  This isn't actually shown in the image and in fact to me that image kind of suggests that it's the same weights applied to each channel (since their the same color)... Per @neil slater 's answer, it sounds like each filter actually has number of input_channels versions with different weights.  If this is also your understanding, is there an "official" source  confirming this?

Yes, indeed, that's also my understanding. For me, that was clear when I tried to think of that grey cube to be composed of 27 different weight values. This means that there are 3 different 2D filters rather the same 2D filter applied to each input layer.

I could not find any official source for confirming this. However, when I was trying to wrap my head around this same concept, I created a dummy input and weight filter in Tensorflow and observed the output. I was content with that. If I find any official explanation. I will edit my answer above.

If you follow the Tensorflow path. You can print your weight filter after showing your dummy CNN layer an input sample.

@Moshsin Bukhari I will definitely try to explore the filters within TensorFlow.  Would you be willing to share your code for how you went about about exploring what's contained in the filters?  Are you able to print the values of the filter at each step in the network for example?

@MohsinBukhari Is there any resources illustrating what you say ?

@pentanol see the original CNN paper and understand the mathematics. Until then the answer above is the resource itself. Good Luck!

In a convolutional neural network, is there a unique filter for each input channel or are the same new filters used across all input channels?

The former. In fact there is a separate kernel defined for each input channel / output channel combination.
Typically for a CNN architecture, in a single filter as described by your number_of_filters parameter, there is one 2D kernel per input channel. There are input_channels * number_of_filters sets of weights, each of which describe a convolution kernel. So the diagrams showing one set of weights per input channel for each filter are correct. The first diagram also shows clearly that the results of applying those kernels are combined by summing them up and adding bias for each output channel.
This can also be viewed as using a 3D convolution for each output channel, that happens to have the same depth as the input. Which is what your second diagram is showing, and also what many libraries will do internally. Mathematically this is the same result (provided the depths match exactly), although the layer type is typically labelled as "Conv2D" or similar. Similarly if your input type is inherently 3D, such as voxels or a video, then you might use a "Conv3D" layer, but internally it could well be implemented as a 4D convolution.

thanks for this explanation.  It sounds like each filter actually has number of input_channels versions with different weights.  Do you have an "official" source that confirms this understanding?

@RyanChase: Yes that is correct. I would just point you at Andrew Ng's course on CNNs - starting here with how colour image would be processed:  https://www.coursera.org/learn/convolutional-neural-networks/lecture/ctQZz/convolutions-over-volume

I  would like to note that, in that source (http://cs231n.github.io/convolutional-networks/), filters (weights or kernesl) are volumes (i.e. 3-dimensional), and they have the same 3rd dimension has the one of the input volume. Furthermore, as it is (at least) now stated in that source, the volumes have been sliced across the 3rd dimension in order to visualize better the application of the filter to the input volume. I don't think that, in general, "there is a separate kernel defined for each input channel / output channel combination." is correct.

Note that the filters (or kernels) are the weights that need to be learned (i.e. they are not fixed, but they are actually the parameters of the CNN). It might be that they are (i.e. the slices of the filter), at the end, the same across the 3rd dimension.

@nbro: Yes you can implement a 2D convolution across multiple 2D slices as a single 3D convolution with the kernel depth same as number of channels. Mathematically this is identical to my description. You can also view it as a truncated fully-connected feed forward network with shared weights (many of which are zero). This answer focuses on what the view of 2D filters is, because the OP is asking about how the 2D filters are arranged. They may in fact be arranged into a larger 3D kernel, but they are still applied as 2D kernels using the "trick" that the 3D convolution is equivalent.

I'm following up on the answers above with a concrete example in the hope to further clarify how the convolution works with respect to the input and output channels and the weights, respectively:
Let the example be as follows (wrt to 1 convolutional layer):

the input tensor is 9x9x5, i.e. 5 input channels, so input_channels=5
the filter/kernel size is 4x4 and the stride is 1
the output tensor is 6x6x56, i.e. 56 output channels, so output_channels=56
the padding type is 'VALID' (i.e. no padding)

We note that:

since the input has 5 channels, the filter dimension becomes 4x4x5, i.e. there are 5 separate, unique 2D filters of size 4x4 (i.e. each has 16 weights); in order to convolve over the input of size 9x9x5 the filter becomes 3D and must be of size 4x4x5
therefore: for each input channel, there exists a distinct 2D filter with 16 different weights each. In other words, the number of 2D filters matches the number of input channels
since there are 56 output channels, there must be 56 3-dimensional filters W0, W1, ..., W55 of size 4x4x5 (cf. in the CS231 graphic there are 2 3-dimensional filters W0, W1 to account for the 2 output channels), where the 3rd dimension of size 5 represents the link to the 5 input channels (cf. in the CS231 graphic each 3D filter W0, W1 has the 3rd dimension 3, which matches the 3 input channels)
therefore: the number of 3D filters equals the number of output channels

That convolutional layer thus contains:
56 3-dimensional filters of size 4x4x5 (= 80 different weights each) to account for the 56 output channels where each has a value for the 3rd dimension of 5 to match the 5 input channels. In total there are
number_of_filters=input_channel*output_channels=5*56=280
2D filters of size 4x4 (i.e. 280x16 different weights in total).

I recommend chapter 2.2.1 of my masters thesis as an answer. To add to the remaining answers:
Keras is your friend to understand what happens:
from keras.models import Sequential
from keras.layers import Conv2D

model = Sequential()
model.add(Conv2D(32, input_shape=(28, 28, 3),
          kernel_size=(5, 5),
          padding='same',
          use_bias=False))
model.add(Conv2D(17, (3, 3), padding='same', use_bias=False))
model.add(Conv2D(13, (3, 3), padding='same', use_bias=False))
model.add(Conv2D(7, (3, 3), padding='same', use_bias=False))
model.compile(loss='categorical_crossentropy', optimizer='adam')

print(model.summary())

gives
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 28, 28, 32)        2400
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 28, 28, 17)        4896
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 28, 28, 13)        1989
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 28, 28, 7)         819
=================================================================
Total params: 10,104

Try to formulate your options. What would that mean for the parameters if something else would be the case?
Hint: $2400 = 32 \cdot (3 \cdot 5 \cdot 5)$
This approach also helps you with other layer types, not only convolutional layers.
Please also note that you are free to implement different solutions, that might have other numbers of parameters.

One addition to Mohsin's answer: try the awesome CNN explainer. There you can explore CNN's graphically, which makes things "click" very fast.
If you click on conv_1_1, you will see how three different kernels are used to calculate one convolution. That means you have one 2D filter per channel, that could together be interpreted as one 3D filter.

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference.  Link-only answers can become invalid if the linked page changes. - From Review

Other answers claim that you have a different 2-dimensional kernel (i.e. a matrix) for each channel. This is not wrong, but it's just a conceptual interpretation of the 2d convolution, which emphasizes that different channels may provide different information during the convolution: this is the only advantage that I see of this interpretation!
Another interpretation (which I think has more advantages, as I will explain below) is to think of a kernel as a 3d multi-dimensional array that has the same depth as the input. So, for example, if you have an input (an image or feature map) of shape $H \times W \times D$, then a single kernel needs to have the shape $K \times L \times D$, where $D$ is the depth of both the input and kernel. If you have more than one kernel, they will all have the depth $D$. This interpretation is consistent with the excerpt that you are quoting.
This second interpretation is conceptually useful when you need to deal with 3d convolutions: if you follow this interpretation, a 3d convolution is just a convolution where kernels don't necessarily have the same depth as the input (and that would be the only difference between 2d and 3d convolutions). The other advantage of this interpretation is that your confusion would not have arisen in the first place: your confusion arouse because you thought that kernels are 2d arrays. Another advantage is that, if you have $N > 1$ kernels (which is the usual case), then you know immediately that $N$ is the depth of the output volume and you don't need to think about multiple channels of the input volume: you can simply think in terms of multi-dimensional arrays.
So, to clarify, in case it wasn't already super clear: no, you don't apply the same 2d matrix to multiple channels of the input!

My understanding from this paper https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/TASLP2339736-proof.pdf was that the filters are used on each input channels (i.e input feature map in the paper) separately and the result is summed, as described in eq. 8. Here they use different filters for each channel, but you could totally use the same filter.
As to knowing the software implementation in a particular ML library such as Tensorflow or PyTorch, this requires inspection as suggested by @Mohsin Bukhari

The Conv2d filter is defined as a 3D-represented tensor, but it's indeed a collection of num_input_channels kernels applied laterally. The different kernels' layers' values are distinct and learned distinctly too. Theoretically it's necessary for it to be that way; otherwise it wouldn't have the degrees of freedom to distinguish well across the input channels, e.g. find color features in the case of the first layer of a color image input!

For anyone trying to understand how convolutions are calculated, here is a useful code snippet in Pytorch:
batch_size = 1
height = 3
width = 3
conv1_in_channels = 2
conv1_out_channels = 2
conv2_out_channels = 2
kernel_size = 2
# (N, C_in, H, W) is shape of all tensors. (batch_size, channels, height, width)
input = torch.Tensor(np.arange(0, batch_size*height*width*in_channels).reshape(batch_size, in_channels, height, width))
conv1 = nn.Conv2d(in_channels, conv1_out_channels, kernel_size, bias=False) # no bias to make calculations easier
# set the weights of the convolutions to make the convolutions easier to follow
nn.init.constant_(conv1.weight[0][0], 0.25)
nn.init.constant_(conv1.weight[0][1], 0.5)
nn.init.constant_(conv1.weight[1][0], 1)
nn.init.constant_(conv1.weight[1][1], 2)
out1 = conv1(input) # compute the convolution

conv2 = nn.Conv2d(conv1_out_channels, conv2_out_channels, kernel_size, bias=False)
nn.init.constant_(conv2.weight[0][0], 0.25)
nn.init.constant_(conv2.weight[0][1], 0.5)
nn.init.constant_(conv2.weight[1][0], 1)
nn.init.constant_(conv2.weight[1][1], 2)
out2 = conv2(out1) # compute the convolution

for tensor, name in zip([input, conv1.weight, out1, conv2.weight, out2], ['input', 'conv1', 'out1', 'conv2', 'out2']):
    print('{}: {}'.format(name, tensor))
    print('{} shape: {}'.format(name, tensor.shape))

Running this gives the following output:
input: tensor([[[[ 0.,  1.,  2.],
          [ 3.,  4.,  5.],
          [ 6.,  7.,  8.]],

         [[ 9., 10., 11.],
          [12., 13., 14.],
          [15., 16., 17.]]]])
input shape: torch.Size([1, 2, 3, 3])
conv1: Parameter containing:
tensor([[[[0.2500, 0.2500],
          [0.2500, 0.2500]],

         [[0.5000, 0.5000],
          [0.5000, 0.5000]]],

        [[[1.0000, 1.0000],
          [1.0000, 1.0000]],

         [[2.0000, 2.0000],
          [2.0000, 2.0000]]]], requires_grad=True)
conv1 shape: torch.Size([2, 2, 2, 2])
out1: tensor([[[[ 24.,  27.],
          [ 33.,  36.]],

         [[ 96., 108.],
          [132., 144.]]]], grad_fn=<MkldnnConvolutionBackward>)
out1 shape: torch.Size([1, 2, 2, 2])
conv2: Parameter containing:
tensor([[[[0.2500, 0.2500],
          [0.2500, 0.2500]],

         [[0.5000, 0.5000],
          [0.5000, 0.5000]]],

        [[[1.0000, 1.0000],
          [1.0000, 1.0000]],

         [[2.0000, 2.0000],
          [2.0000, 2.0000]]]], requires_grad=True)
conv2 shape: torch.Size([2, 2, 2, 2])
out2: tensor([[[[ 270.]],

         [[1080.]]]], grad_fn=<MkldnnConvolutionBackward>)
out2 shape: torch.Size([1, 2, 1, 1])

Notice how the each channel of the convolution sums over all previous channels outputs.

Just to make two details absolutely clear:
Say you have $N$ 2D input channels going to $N$ 2D output channels.  The total number of 2D $3\times3$ filter weights is actually $N^2$.  But how is the 3D convolution affected, i.e., if every input channel contributes one 2D layer to every output channel, then each output channel is composed initially of $N$ 2D layers, how are they combined?
This tends to be glossed over in almost every publication I've seen, but the key concept is the $N^2$ 2D output channels are interleaved with each other to form the $N$ output channels, like shuffled card decks, before being summed together.  This is all logical when you realize that along the channel dimensions of a convolution (which is never illustrated), you actually have a fully connected layer!  Every input 2D channel, multiplied by a unique $3\times 3$ filter, yields a 2D output layer contribution to a single output channel.  Once combined, every output layer is a combination of every input layer $\times$ a unique filter.  It's an all to all contribution.
The easiest way to convince yourself of this is to imagine what happens in other scenarios and see that the computation becomes degenerate - that is, if you don't interleave and recombine the results, then the different outputs wouldn't actually do anything - they'd have the same effect as a single output with combined weights.

Refer to "Local Connectivity" section in here and slide 7-18.
"Receptive Field" hyperparameter of filter is defined by height & width only, as depth is fixed by preceding layer's depth.
NOTE that "The extent of the connectivity along the depth axis is always equal to the DEPTH of the input volume" -or- DEPTH of activation map (in case of later layers).
Intuitively, this must be due to the fact that image channels data are interleaved, not planar. This way, applying filter can be achieved simply by column vectors multiplication.
NOTE that Convolutional Network learns all the filter parameters (including the depth dimension) and they are total "hwinput_layer_depth + 1 (bias)".

There are only restriction in 2D. Why?
Imagine a fully connected layer.
It'd be awfully huge, each neuron would be connected to maybe 1000x1000x3 inputs neurons. But we know that processing nearby pixel makes sense, therefore we limit ourselves to a small 2D-neighborhood, so each neuron is connected to only a 3x3 near neurons in 2D. We don't know such a thing about channels, so we connect to all channels.
Still, there would be too many weights. But because of the translation invariance, a filter working well in one area is most probably useful in a different area. So we use the same set of weights across 2D. Again, there's no such translation invariance between channels, so there's no such restriction there.


text:
What are some interesting recent papers that synthesize symbolic AI with Deep Learning?
A lot of people seem to be under the impression that combining GOFAI and contemporary AI will make models more general. I'm particularly interested in reasoning through analogy or case-based reasoning.

Here is a very similar but older question, so yours seems to be a duplicate. Am I right?

Read some of Jacques Pitrat's books

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Why is no activation function used at the final layer of super-resolution models?
I'm trying to implement some image super-resolution models on medical images. After reading a set of papers, I found that none of the existing models use any activation function for the last layer.
What's the rationale behind that?

I am not into the field of super-resolution, but I think this question applies to general neural network construction.
Usually, you try to solve a classification problem or a regression problem with your neural network.

For classification, you try to predict probabilities that a specific output corresponds to a specific class. Therefore, every output value should be a probability and therefore have a range between 0 and 1. To achieve this, you usually use a softmax or sigmoid function as your last layer to squash the output between 0 and 1. In addition to this (which is wanted in classification tasks), these functions raise the probability output of likely classes while decreasing the probability of all other unlikely classes (therefore enforcing the network to choose for one specific class over the others).

For the regression task, you are not looking for probability values as your output values but instead for real-valued numbers. In such a case, no activation function is wanted, since you want to be able to approximate any possible real value and not probabilities.

So, in the case of super-resolution, I think the generated output is a map where each value corresponds to a pixel value of the super-resolution image. In that case, your pixels are real number values and no probabilities. So, you are solving a regression problem.
But you could also go with a classification approach, where you have 256 output maps that give probability to each possible pixel value between $0$ and $255$.

As discussed here:
https://www.researchgate.net/post/What_should_be_my_activation_function_for_last_layer_of_neural_network
Linear is the preferred activation function.
But then linear activation function is equivalent to no activation function at all:
https://datascience.stackexchange.com/questions/13696/lack-of-activation-function-in-output-layer-at-regression


text:
What is the relation between the definition of learnability of Vapnik and Gold and learnability of neural networks?
Gold showed that a language can be learned only if it contains a finite set of sentences.
We know that deep neural networks can implement any function. Does this contradict the Gold's result?
What is the relation or difference between the definition of learnability of Vapnik and Gold and the definition of learnability of neural networks?

That's a problem about expressivity v.s. learnability.

This is an old question, but I suggest that you provide references where we can find the definitions that you were talking about.


text:
What is an intuitive explanation of how Google's AutoML works?
I recently read that Google has developed a new AI that anyone can upload data to and it will instantly generate models, i.e. an image recognition model based on that data.
Can someone explain to me in a detailed and intuitive manner how this AI works?

Google's AutoML is really a good idea in terms of autonomous model design. You can find the details in this blog. Let me explain briefly.
We, data scientists, design new networks by following existing models, trying and failing and trying again and again by analyzing weaknesses and strengths of the created models. However, we, as humans, have limited capabilities of designing/analyzing such networks. That's why Google created an AI which analyzes the strengths and weaknesses of each node while making a prediction. This AI analyzes each node and tries to improve the results by adding/removing/modifying connections of each node/layer. I guess AutoML AI takes state-of-art network as a base and starts modifying network according to your data to create a customized model.
While doing that, two technologies are being used: Transfer learning and Reinforcement learning.
Transfer learning is being used to start training from the most accurate point possible.
Reinforcement learning is being used to modify network to achieve a better success. This is the key part of this technology.
So, for users, it is more like upload your data, let AI modifies the network for you and give you a custom model which is specific to your data.


text:
Is there any website that allows you to choose an algorithm, code it and visualise how it works?
I would like to do some practical implementation of a planning algorithm (of course, something a bit simple and easy).
Is there any website where I can pick an algorithm (e.g. A* or hill climbing), code it, and visualize how it works/executes?
The site doesn't necessarily need to be restricted to planning or search algorithms.  For example, in the context of machine learning, I would also like to be able to pick the learning algorithm and model (e.g. linear regression), code it, and visualize how it works.

Not planning, but this is a visual in-browser neural network for your interest:
http://playground.tensorflow.org/

https://www.cs.waikato.ac.nz/ml/weka/downloading.html
Great little tool to experiment with various algorithms and compare their efficiencies.


text:
How could a chat bot learn synonyms?
I have started to make a chatbot. It has a list of greetings that it understands and responds to with its own list of greetings.
How could a bot learn a new greeting or a synonym for a word it already knows?

This answer describes the "word vector" toolkit in NLP. The result of analyzing a large corpus to find words that occur in similar context provides dense vectors for each word that can then be used for similarity. For bots, the goal is generally a similarity and not exact synonyms. Synonyms can be hard-coded using WordNet if needed. For your greeting question, following blog post can help: Do-it-yourself NLP for bot developers.

There is pretty simple way:
Write a program that analyzes large amounts of texts. Find sentences that contain our greeting. Then find exact same sentences except that instead of our word there is another word. The more such examples you find the higher is probability that it is synonim and not word from same category with different meaning.

You could train a model to classify sentences into user intents. For example, an intent could be "greeting". Another intent could be "help", or any other capability that your bot is able to talk about.
To train you model, you should provide several examples for the same intent. For example, for "greeting", you could provide "Hi", "Hello", "What's up", etc...
You should also apply some preprocessing before feeding sentences into your model, such as word embeddings or semantic similarity  with WordNet. These techniques allow to transform strings into representations that capture the similary of word meanings.  The ability of your model to detect synonyms without being retrained will highly depend on this preprocessing.


text:
What else can boost iterative deepening with alpha-beta pruning?
I read about minimax, then alpha-beta pruning, and then about iterative deepening. Iterative deepening coupled with alpha-beta pruning proves to quite efficient as compared to alpha-beta alone.
I have implemented a game agent that uses iterative deepening with alpha-beta pruning. Now, I want to beat myself. What can I do to go deeper? Like alpha-beta pruning cut the moves, what other small change could be implemented that can beat my older AI?
My aim to go deeper than my current AI. If you want to know about the game, here is a brief summary:
There are 2 players, 4 game pieces, and a 7-by-7 grid of squares. At the beginning of the game, the first player places both the pieces on any two different squares. From that point on, the players alternate turns moving both the pieces like a queen in chess (any number of open squares vertically, horizontally, or diagonally). When the piece is moved, the square that was previously occupied is blocked. That square can not be used for the remainder of the game. The piece can not move through blocked squares. The first player who is unable to move any one of the queens loses.
So my aim is to cut the unwanted nodes and search deeper.

Welcome to AI!  Sounds like an interesting problem.  (I've take the liberty of adding combinatoric and combinatorial-games tags.)  Can I ask, do you have a sense of the size of the gametree?  What are your restriction on number of plies the algorithm can search in a reasonable time?  I mention it because, if the game tree expands sufficiently, depth search may not be fruitful until the game tree starts to become tractable.

First thing you're going to want to add is probably a Transposition Table, as also suggested by SmallChess.
Afterwards, I'd look into Aspiration Search and/or Principal Variation Search (also see this page).
Then I'd look into things like the Killer Move Heuristic, and maybe also see if you can simply implement existing parts of your engine more efficiently (e.g. use bitboards for your state representation).
Other than all of that, the chess programming wiki probably has lots of other interesting pages as well.

You can try move ordering where we store the values till depth d, sort them and use them in particular order before we go for depth d+1 ...

Try cache or transposition table. Without it, your search tree might explode.

To make boost iterative deepening with alpha-beta pruning you can use the
SSS* Search algorithm, its a best first strategy algorithm. The SSS* Algorithm can improve the time efficiency of the overall algorithm but it increases the space complexity.
I am linking the wiki to it https://en.wikipedia.org/wiki/SSS*
I will update the answer as soon as i get a better solution.


text:
Algorithms can be greedy. What are some other algorithmic vices?
Greedy algorithms are well known, and although useful in a local context for certain problems, and even potentially find general, global optimal solutions, they nonetheless trade optimality for shorter-term payoffs.
This seems to me a good analogue for human greed, although there is also the grey goo type of greed that is senseless acquisition of material (think plutocrats who talk about wealth as merely a way of "keeping score".)
Technical debt is an extension of development practices that fall under the algorithmic definition of greed (short-term payoff leads to trouble down the road.)  This may be further extended to any non-optimized code in terms of energy waste (flipping of unnecessary bits) which will only increase as everything becomes more computerized.
So my question is:

What are other vices that can arise in algorithms?

Algorithms can be racist, sexist, and otherwise bigoted. When we feed them data produced by systems that are biased against groups of people, the algorithm will learn to behave that way. We're used to garbage in garbage out, now we have to worry about racism in racism out.
See:

Facial Recognition Is Accurate, if You’re a White Guy

Rise of the Racist Robots – how AI is Learning all our Worst Impulses

You may want to include more information from your links in this answer; it's always better to include information in the answer itself rather than depending on links.

Algorithms can learn to lie:
See:

Robots Evolve to Deceive (MIT Tech Review, 2007)
Robots 'Evolve' the Ability to Deceive (MIT Tech Review, 2009)
Evolving Robots Learn To Lie To Each Other (Popsci, 2009)

Deception as a strategy has been observed in animal populations:

Do Animals "Lie"? Yes, Even to Their Own Kind, Biologist Says (Rochester University, 1995

Here are the Best Liars in the Animal Kingdom (Nat Geo, 2017)

Can Animals Lie? (Springer)

Why animals lie: how dishonesty and belief can coexist in a signaling system.

The Philosophy of Deception (Martin, 2009)

Algorithms can learn to cheat:

"A machine learning agent intended to transform aerial images into street maps and back was found to be cheating by hiding information it would need later in “a nearly imperceptible, high-frequency signal.”
"...a computer creating its own steganographic method to evade having to actually learn to perform the task at hand is rather new."
Source: This clever AI hid data from its creators to cheat at its appointed task (TechCrunch)


text:
Can prior knowledge be encoded in deep neural networks?
I was reading Gary Marcus's a Critical Appraisal of Deep Learning. One of his criticisms is that neural networks don't incorporate prior knowledge in tackling a problem. My question is: have there been any attempts at encoding prior knowledge in deep neural networks?

Learning in a NN is iterative, thus, based in prior knowledge. In particular, initial value of NN parameters is obviously a prior knowledge.

@pasabaporaqui thanks a lot for the feedback. What my question is driving at is, in most real life problems we have additional information about the domain. Is there a way to inject this prior domain knowledge to improve the success of the model?

Strictly talking, a neural net doesn't learn. The process of learning is done outside the net (i.e. backpropagation algorithm). This is one of the reason is not correct to say that a NN simulates brain (in the brain, learning is done inside the net). If it doesn't learn, it doesn't use prior knowledge. Prior knowledge can be used to optimize the learning algorithm, including the initial net parameters, not the net behavior itself. If one rule is included in the net (as output and using it in the error function), it is not prior knowledge but a problem constrain that the net will try to fulfill

Lookup LSTM in relation to neural networks-- it's a method to take advantage of prior knowledge

Neural nets incorporate prior knowledge. This can be done in two ways: the first (most frequent and more robust) is in data augmentation. For example in convolutional networks, if we know that the "value" (whatever that is, class/regression) of the object we are looking is rotational/translational invariant (our prior knowledge), then we augment the data with random rotations/shifts. The second is in the loss function with some additional term.

Yes, we can do it in a deep learner.
For example, suppose we have an input vector likes $(a, b)$ and from prior knowledge, we know $a^2 + b^2$ is important too. Hence, we can add this value to the vectors likes $(a, b, a^2 + b^2)$.
As another example, suppose date time is important in your data, but not encoded in the input vector. We can add this to the input vector as a third dimension.
In summary, it depends on the structure of the prior knowledge, we can encode it into the input vector.

I think the questioner was looking for something like LSTM rather than ways the network designer could incorporate their own prior knowledge.

To add to the Foivos's answer,
Convolutional Neural Networks are shift-invariant.
Fukushima introduced this to his Neocognitron.
There is a trail to introduce scale-invariance to CNN.
https://arxiv.org/abs/1411.6369
Also, CNN uses structural characteristic for the prior knowledge.
And neural networks are locally smooth.
It is not perfect, but neural networks are incorporating a lot of prior knowledge.

It kinda depends on how exactly you define knowledge, and what you believe about what the weights in a trained NN model really represent. But to answer this question in the most straightforward possible way (hopefully without sounding glib), then yes, a NN can be pre-trained, and then you can take that model and apply additional training to it, so in a sense, it is using "prior knowledge".
If, OTOH, knowledge means something a little different to you, and you're thinking about the kind of knowledge that's encoded in a semantic network, or a conceptual graph, or something of that nature, then I don't know - offhand - of any direct way to integrate that into an ANN. What you might be able to do is combine the NN with a different kind of reasoner that reasons of the semantic network / conceptual graph, and then integrate the results.  AFAIK, the best way to do that is an unsolved research problem.

A simple example of this is token embeddings. If "prior knowledge" just means anything known prior to creation of the graph, then using pretrained vector embeddings meets this criteria. This is simply a way to provide a fixed method for projecting tokens into higher-dimensional space instead of training it at the same time as the rest of the model. Given that vector embeddings are somewhat interpretable and that the same embedding can be reused across tasks and models, I'd consider pretrained embeddings to be prior knowledge being incorporated.
The embeddings could also technically be handcrafted, but I'm not aware of any work like that and am skeptical of its usefulness in deep models.


text:
Understanding a few terms in Andrew Ng's definition of the cost function for linear regression
I have completed week 1 of Andrew Ng's course. I understand that the cost function for linear regression is defined as $J (\theta_0, \theta_1) = 1/2m*\sum (h(x)-y)^2$ and the $h$ is defined as $h(x) = \theta_0 + \theta_1(x)$. But I don't understand what $\theta_0$ and $\theta_1$ represent in the equation. Is someone able to explain this?

Specifically, theta0 is the bias and theta1 is 'slope' of the regression line.

The prediction made by linear regression can simply be thought of as a vector dot product.
$$\overrightarrow{x}^T \cdot \overrightarrow{y}$$
One of those two vectors is the "data" for one case (like a row in your data matrix), the other is a vector of the model's parameters, which is usually called $\overrightarrow{\theta}$ or $\overrightarrow{\beta}$.
So in the case shown by yourself, we have:
$$h(x) = \theta_0 + \theta_1 \cdot x$$
Often we add a row of ones to the beginning of the data matrix, that way we are consistent in the sense that the $\theta_0 = 1 \cdot \theta_0$
This way we arrive at:
$$h(x) = \overrightarrow{\theta}^T \cdot  \overrightarrow{x}$$

As said above, they are weights to your hypothesis function that are changed during training to minimize your Error function. You can think of them like slope and y intercept in basic algebra. However, a linear regression hypothesis function can be parameterized by many more weight terms than just theta_0, and theta_1.
I detail this process more in this post: How does an activation function's derivative measure error rate in a neural network?


text:
What are the current theories on the development of a conscious AI?
What are the current theories on the development of a conscious AI? Is anyone even trying to develop a conscious AI?
Is it possible that consciousness is an emergent phenomenon, that is, once we put enough complexity into our system, it will become self-aware?

To answer this question, first we need to know why developing conscious AI is hard.
The main reason is that there is no mathematically or otherwise rigorous definition of consciousness.
Sure you have an idea of consciousness as you experience it and we can talk about philosophical zombies but it isn’t a tangible concept that can be broken down and worked on.
Moreover, the majority of current research in AI is primarily a pragmatic approach in that one is trying to construct a model that can perform well according to some desired cost function. This is a very very big and exciting field and encompasses many research problems and every new finding is based either on mathematical theory or empirical evidence of a new algorithm/model construction/etc.
Because of this, progress is based on and compared against previous progress as it is the scientific method.
So to answer your question, no one is trying to actually make a “conscious” AI because we don’t know what that word means yet, however that doesn’t stop people talking about it.

What is consciousness? There are some real challenges in setting up consciousness as a goal, because we don't have that much scientific understanding yet of how the brain does it or what balance there needs to be between long-term memory, short-term memory, the implicit work of interpretation, the contrasting conscious modes of automatic processing and deliberate processing (Khanemann's S1 and S2). John Kihlstrom (psychology emeritus at Berkeley) has a lecture set on Consciousness available in iTunesU that you might check out. Carnegie-Mellon Uni has a model called ACT-R which directly models conscious behaviours like attention-paying.
What might bound our understanding of it? Philosophy has been considering the question of consciousness for a long time. Personally I like Hegel and Heidegger (philosophers). Both are very difficult to read, but Heidegger (interpreted by Hubert Dreyfus) usefully critiqued the 'Good Old-Fashioned AI' projects of the seventies and pointed out how much work there is just interpreting a visual input. Hegel is often maligned, but to see him well interpreted, check out Robert Brandom's talks to LMU on the logic of consciousness and Hegel as an early Sellers-ian pragmatist. If consciousness is to take hold of the truth and the certainty, it undertakes 'a path of doubt, or more properly a highway of despair', along which it never sets itself above correction. There is something about Hegel's treatment of consciousness in recursive terms, without succumbing to a vicious regress, that I think is going to be borne out before the end.
Recent developments. The Deep Learning approaches and pragmatic successes of the present are exciting, but it will be interesting to see how far they can go in integrating and generalising from necessarily the small information sets actual human minds are exposed to.  While Deep Learning and data mining are hugely visible, symbolic approaches are also out there still getting better and more varied. But there is a lack of overarching theoretical interpretation that would allow generalisations.
Two big-theory toe-holds. If I had to pick a project I thought worth attending to, Giulio Tononi (et al) have set up a very nice modernisation of the problem in 'Integrated Information Theory' But you might want to extend that with something like Rolf Pfeifer's 'How the body shapes the way we think', because some of the 'integrated information' is implicit in having arms and legs, eyes and nose (put there by the information accumulating work of evolution.) But there's so much good work that has been done - the pros are writing papers faster than I can read them.
More specific to your question, there are attempts to simulate human brains hoping that overall aim will help fund research and produce answers to each para above.

I read an interesting and honest article by Douglas Hofstadter in The Atlantic (https://www.theatlantic.com/technology/archive/2018/01/the-shallowness-of-google-translate/551570/) critiquing the use of the term "Deep Learning" to refer to Google's Deep Learning translation program, and using entertaining examples to show the programs  lack of true understanding.  He made a convincing argument that good language translation will not be possible without first developing a conscious AI.

Yes, I don't think anyone conscious would make quite this mistake below, although I can imagine a translator sometimes being unable to make out any better with arcane terms of art.  Its the lack of self-awareness in the output: https://www.reddit.com/r/softwaregore/comments/6bxh2m/when_google_translate_meets_the_german_language/

About the first part: My old AI prof used to say that asking if computers can think is like asking if submarines swim. The answer depends on how you want to define "swim" more than it does on what the machine is actually doing.

In addition to Jaden's excellent answer "no one is trying to actually make a “conscious” AI because we don’t know what that word means yet" I'd like to add that the word "yet" there is highly optimistic.
It's highly problematic and likely impossible to distinguish between a conscious being and a being that behaves exactly as if it was conscious. Philosophers have been struggling with that for centuries; some even espoused solipsism, which is a "I live in the Matrix" philosophy. In particular, how can you tell whether your childhood friend or your spouse or anybody else is a conscious being rather than an embodiment of AI that acts exactly as a conscious being would?
It's possible, of course, to go "if it walks like a duck and quacks as a duck then it's a duck" way. In that case a Turing Test passing AI would be automatically considered conscious. However, most people wouldn't accept the duck criteria of consciousness; otherwise they would very soon have to call their Alexa operated household appliances conscious.
My two cents are basically the same as Jaden's, except that I'm more pessimistic about ever understanding what consciousness is.

I think the Duck Test is sufficient if an AI passes a full Turing Test, administered by intelligent adults.  (My problem with the Chinese Room is that it is based on qualia, and I suspect the only way to validate the qualia of another entity is to become that entity.)

I might also argue that any percept fulfills the most basic definition of consciousness, and the ability to make any decision based on input satisfies the most basic definition of intelligence.  For all we know, consciousness may just be some form of meta-cognition, and entirely a function of the complexity of the system.

Consciousness is the ability to be aware of your own thoughts, your immediate environs, feelings and nothing more. It is the mechanism of our brain to control our lower kind of thoughts, the one based on associations and emotions.
Consciousness is observing our thoughts and feelings just like we observe real world with our eyes. It is not complicated.
The real question is not whether machines are capable of consciousness but whether they are capable of emotions.

There are plenty of species apart from modern humans exhibiting strong emotions.  These emotions are instinctive - they are very important in defining social behavior which enhance the survivability and fitness of the "group".  I can't see how human emotion is qualitatively any different from that of other mammals.  In contrast, human consciousness is several orders of magnitude more developed in humans than the nearest other species.  Where human emotion does seem to differ from other species, is only the ability to control emotion through consciousness.

This answer makes no sense without a definition of 'your'.


text:
Calculating tangent vector of curve s(P,$\alpha$) at given point $\alpha$ = 0
I am reading the paper "Transformation Invariance in Pattern Recognition – Tangent Distance and Tangent Propagation", where the tangent vector is calculated for the given curve $s(P,\alpha)$ at $\alpha=0$ by differentiating with respect to $\alpha$, that is, $\frac{\partial s(P,\alpha)}{\partial\alpha}$. For the curve, I have taken one $2D$ image and I am rotating it with matrix $R=\left[\matrix{cos(\alpha)\space -sin(\alpha)\\sin(\alpha) \space\space\space\space\space cos(\alpha)} \right]$.
As my image is fixed, the curve is just a function of $\alpha$. Therefore, to find the tangent vector, what I am doing is as follows:

I am rotating the image by the matrix $R^{'}$ which is $R^{'}=\left[\matrix{-sin(\alpha)\space -cos(\alpha)\\cos(\alpha) \space\space\space\space\space -sin(\alpha)} \right]$
This rotates the image by $90$ degree, which is not the expected result.

I have done the same exercise by differentiating numerically and I am getting the expected answer which is as follows:$\alpha$=0">
$\alpha$=0">
Please, help me to understand my mistake in taking derivative of the matrix and multiplying it with image.

What does the rotation have to do with the differentiation of the curve function?

You can see 2D image as a point in 16 X 16 dimensional space. So if you keep rotating you get a curve in 256 dimensional space wrt rotation angle. So I want to differentiate this curve wrt to rotational angle.


text:
How do you efficiently choose the hyper-parameters of a neural network?
How do you efficiently choose the hyper-parameters of a neural network (e.g. the learning rate, number of layer, weights, etc.)?

Are you asking which techniques exist to search for the best combination of hyper-parameters?

In deep learning era, there are two possible choices. Caveat approach and Panda approach.
Caviar Approach
In this approach, it is supposed that you have a very powerful cluster system that enables you to run different models simultaneously on different nodes. In this manner, you construct a d-dimensional space which each corresponds to a special hyperparameter. Then you can have a grid approach to partition the space and for each intersection of the grids, you can have a possible initialisation although there are some better ways in order to find better initialisations.
Panda Approach
This approach is widely used among students due to using desktop computers. In this approach, you usually try to find an initialisation of hyperparameters based on your experience or the other architectures' which are available and you try to refine them step by step by using cross-validation.

To answer your question, the efficiency depends on your computing power. For more details, you can take a look at the contents of the third week.

I have never heard of these terms/expressions. Where did you hear them? In general, you don't need a cluster. It depends on your NN model. A GPU (or even a CPU) will suffice if you have a simple NN model. It also depends on your time requirements. Furthermore, note that there are other hyper-parameter techniques other than grid search.

I added the link. You can't train a YOLO model on a single GPU.

The question is not clear. Maybe the OP is asking for techniques to look for combinations of hyper-parameters. In that case, your answer is not valid. I would first ask for clarifications.

Combination of hyperparameters? Isn't it exactly what I've tried to convey?

No, I'm talking about something different...


text:
Why exactly do neural networks require i.i.d. data?
In reinforcement learning, successive states (actions and rewards) can be correlated. An experience replay buffer was used, in the DQN architecture, to avoid training the neural network (NN), which represents the $Q$ function, with correlated (or non-independent) data. In statistics, the i.i.d. (independently and identically distributed) assumption is often made. See e.g. this question. This is another related question. In the case of humans, if consecutive data points are correlated, we may learn slowly (because the differences between those consecutive data points are not sufficient to infer more about the associated distribution).
Mathematically, why exactly do (feed-forward) neural networks (or multi-layer perceptrons) require i.i.d. data (when being trained)? Is this only because we use back-propagation to train NNs? If yes, why would back-propagation require i.i.d. data? Or is actually the optimisation algorithm (like gradient-descent) which requires i.i.d. data? Back-propagation is just the algorithm used to compute the gradients (which is e.g. used by GD to update the weights), so I think that back-propagation isn't really the problem.
When using recurrent neural networks (RNNs), we apparently do not make this assumption, given that we expect consecutive data points to be highly correlated. So, why do feed-forward NNs required the i.i.d. assumption but not RNNs?
I'm looking for a rigorous answer (ideally, a proof) and not just the intuition behind it. If there is a paper that answers this question, you can simply link us to it.

There is an assumption behind the theory training a neural network, that also applies to many other supervised learning methods, that a training sample is representative of the data set as a whole - that it has been sampled fairly from the population that the learning algorithm has been set up to approximate.
The term i.i.d. stands for "independent and identically distributed". If you pick trajectories from RL then the sampling (per individual* record as used to train with) is not independent. Even if you choose the start of the trajectory randomly, you will have made one random choice then your remaining choices are made according to the trajectory - technically they are chosen by the policy and environment dynamics over a single step, which is usually not enough to make the second, third etc steps from a trajectory fully independent random samples from the training population. To make the sampled observations independent, you should make a new random choice, and to be identically distributed that choice has to be made fairly over the whole dataset.
If a subset of samples drawn for a mini-batch (or as consecutive individual samples) are correlated for an online algorithm like gradient descent, it causes a problem. For the algorithm to converge towards a globally optimal solution, it needs errors and the gradients they generate to be unbiased samples of the "true" function gradients across the loss function. Whilst correlated data from a trajectory does not do this at all - it exhibits a strong sampling bias, causing weight updates to move consistently in the wrong direction.
You can demonstrate this effect trying to learn any simple function like $y = x^2$ with $x$ from $-1$ to $+1$. Training a NN with randomly selected $x,y$ pairs results in far better results than training it serially with ~2000 records starting $(-1.000, 1.000) (-0.999, 0.998) (-0.998, 0.996)$ etc

So, why do feed-forward NNs required the i.i.d. assumption but not RNNs?

RNNs do need i.i.d. data, but the "unit" of sampling here is each sequence.

* By "individually" I mean in terms of how they are used in the training process. A "unit" here is the smallest set of records that leads to a complete measure of loss in your optimisation.
It is ok to draw longer trajectories at random if your smallest unit of measurable error is from a trajectory, as in Monte Carlo or TD($\lambda$) for RL, or for RNNs. There can be subtle problems with this - the trajectories still have to be i.i.d. as in they represent fair samples from the assumed population of trajectories.
If you are working at the level of longer trajectories, typically this reduces systemic bias: In RL it reduces bias from initial conditions due to bootstrap process. For RNNs I am less sure what this would mean, but suspect there is some equivalent. Typically, it also increases variance, meaning you may need more training samples. There is a bias/variance trade-off, which is why setting $\lambda$ somewhere between 0 and 1 in TD($\lambda$) is often the optimal choice. Note in RL this is a different source of bias and variance than discussed when considering number of parameters and regularisation in supervised learning.

Comments are not for extended discussion; this conversation has been moved to chat.

What do you mean by piecewise learning method? ReLU network?

@nn0p: I mean that it learns from a finite set of samples of a target function that have been provided to it. It does appear that the term is used elsewhere for othe rmeanings, so I will change that sentence.

Suppose that we have some optimization criterion $J(x)$, which we aim to optimize (maybe maximize, maybe minimize), which we can compute for a single example $x$.
In an "ideal world", where we have no restrictions on computation time and memory, we would generally want to run training algorithms on the complete "ground truth" population. For example, if we're training a model (may be a DNN, but may also be some other kind of Machine Learning model), we'd ideally train it on the complete population of all "real-world" images that have ever been produced, ever will be produced, or ever could be produced (and, of course, all with accurate labels). We could write our full optimization criterion as:
$$\sum_{x \in \mathcal{P}} J(x),$$
where $\mathcal{P}$ denotes the complete population. If we have a model that we would like to train using gradient descent (such as a Neural Network), this means we have to compute the gradient with respect to some trainable parameters $\theta$:
$$\nabla_{\theta} \sum_{x \in \mathcal{P}} J(x).$$

In practice, we do not have this ideal scenario, we do not have access to the complete population $\mathcal{P}$. Often, we find ourselves approximating the population with a (hopefully rather large) dataset $\mathcal{D}$. This could be a collection of images like ImageNet, or in (Deep) Reinforcement Learning it could be a large experience replay buffer. If the dataset $\mathcal{D}$ is an accurate representation of the complete population's distribution, we can estimate the gradient of the objective we ultimately care about (computed over the complete population) by the gradient computed over the dataset $\mathcal{D}$:
$$\nabla_{\theta} \sum_{x \in \mathcal{P}} J(x) \approx \nabla_{\theta} \sum_{x \in \mathcal{D}} J(x).$$
Collecting such a dataset rather than the complete population is often actually feasible in practice. If we can afford to compute the objective/gradient over such a complete dataset, that's great. Note that in such a case, the data can already be viewed as being i.i.d.: it's the best approximation we have of a single, complete distribution (the population distribution).

However, computing the objective / gradient over a large complete dataset is often still prohibitively expensive in terms of computation time. This is one of the reasons (not the only one) why we often use minibatches $B$. Then, we approximate the gradient of the objective of the dataset (which is itself an approximation of the gradient for the complete population) by computing it only over a minibatch $B$:
$$\nabla_{\theta} \sum_{x \in \mathcal{P}} J(x) \approx \nabla_{\theta} \sum_{x \in \mathcal{D}} J(x) \approx \nabla_{\theta} \sum_{x \in \mathcal{B}} J(x).$$
If we're repeatedly (over many, many training iterations) going to use such an approximation to take gradient descent steps, it is crucial that the gradient computed over this minibatch is actually an accurate approximation of the "true" gradient over the complete population; if it's not an accurate approximation, we're optimizing the wrong objective! In my opinion, this means that we have an even stronger requirement for our minibatch than just wanting it to be identically distributed. It's not sufficient for our minibatch to be sampled from any arbitrary identical distribution. We want the instances in our minibatch to be sampled from one very specific identical distribution; the dataset/population distribution! If they're not all sampled from that particular distribution, they're an unreliable approximation of the objective we truly care about.

Now, you may wonder if it wouldn't be possible for multiple biased, "unrepresentative" minibatches with "opposite" biases to "cancel each other out" if they're used in different, subsequent gradient descent steps. Couldn't we first run a bunch of gradient descent steps on minibatches that only contain images of dogs, and afterwards "cancel out" any errors by also running a bunch of updates on minibatches containing only images of cats?
If you're lucky, it might work sometimes, but it's unreliable. One problem is that the first iterations using only dog images may cause you to end up in a poor area of the "parameter space", which you would never have ended up in if had used a correct mix of cat and dog images all along. Escaping that poor area may be much more difficult than simply ensuring you never reach it. Another problem is that it would be extremely difficult to find the "correct" number of gradient descent updates to run with cat images. Run too few, you're still stuck recognising only dogs. Run too many, and you may forget how to recognise dogs at all and only start recognising cats. Things like momentum in more sophisticated optimizers will exacerbate this issue.

Note that I don't think the requirement for i.i.d. batches is necessarily unique to gradient descent. Other learning techniques may have the same requirement (maybe for similar reasons, maybe for different reasons).

Comments are not for extended discussion; this conversation has been moved to chat.

Short answer
One reason why we assume/require i.i.d. data is that it simplifies the computations. More specifically, if we assume the samples to be i.i.d., their joint probability is then simplified to a product of marginal probabilities.
Long answer
In a dataset $D$, suppose we have $n$ samples. We define their joint probability (i.e. the probability of these samples occurring at the same time) as follows
$$P(z_1, z_2, \dots, z_n) \tag{1}\label{1}$$
For instance, if each $z_i$ is binary (i.e. can take one of two values, e.g. $0$ or $1$). Then, to define the probability distribution over all possible values of all $z_i$, we need to compute $2^n$ probabilities (which corresponds to all combinations of the values of all $z_i$s).
More importantly, if the samples are correlated, this probability must be calculated as a product of conditional probabilities (by definition).
However, if the samples are i.i.d., then the join probability in equation (\ref{1}) can be computed as a product of marginal probabilities
$$P(z_1, \dots, z_n)=\prod_{i}P(z_i) \tag{2}\label{2}$$
which may be simpler than calculating with conditional probabilities because marginal probabilities may be simpler to compute.
Example: binary cross-entropy
In the case of a binary classification problem, we assume to have a labelled dataset $D = \{(x_i, y_i) \}$, where $y_i$ would be the binary label ($0$ or $1$) for the corresponding input $x_i$. So, we could define our likelihood function parametrized by the parameters $w$ as follows
$$\ell(w) = P(y_1, y_2, \dots, y_n \mid x_1, x_2, \dots, x_n; w) \tag{3}\label{3}$$
If we assume $(x_i, y_i)$ to be independent of $(x_i, y_j)$, for all $i \neq j$, then this joint probability (\ref{3}) of labels given the inputs can also be written as a product of the marginals of the labels given the inputs.
$$\ell(w) = \prod_i P(y_i \mid x_i; w) \tag{4}\label{4}$$
For numerical stability (because sums are more stable than products of small numbers), rather than considering the likelihood, we can consider the log-likelihood, which is just the logarithm of $\ell$. However, this transforms the product in (\ref{4}) into a sum (note that this is just a rule of logarithms!).
$$\log \ell(w) = \sum_i \log P(y_i \mid x_i; w) \tag{5}\label{5}$$
We can also do this because the logarithm is a strictly increasing function, so the maxima/minima of $\ell$ are attained at the same parameters as the maxima/minima of $\log \ell$.
So, now, the goal is to find the parameters $w$ of the log-likelihood such that the probability of the samples is maximized. Equivalently, rather than maximizing the log-likelihood, we can minimize its negative (this is what is usually done in practice!), which leads to what people call the cross-entropy function (which is just the negative log-likelihood).
$$\text{CE}(w) = - \log \ell(w)$$
So, minimizing the cross-entropy $\text{CE}(w)$ is exactly the same thing as maximizing the $\log \ell(w)$.
Given that the labels $y_i$ are binary, we can assume that $P(y_i \mid x_i; w)$ in (\ref{5}) is a Bernoulli distribution, so we could define the probability that the label is equal to $1$ (or $0$, it's the equivalent) as follows
$$P(y_i=1 \mid x_i; w)=\hat{p}^{y_i} (1-\hat{p})^{(1-{y_i})},$$
where $\hat{p}$ is the output of the neural network $f(x_i; w) = \hat{p}$ when fed with $x_i$, and $\hat{p}$ is just an estimate of the parameter $p$ of the Bernoulli distribution we try to learn.
So, now, our cross-entropy $\text{CE}(w)$ can be written as follows
$$\text{CE}(w) = \sum_i \log \left( \hat{p}^{y_i} (1-\hat{p})^{(1-y_i)} \right) $$
So, now, we just need to estimate $w$ to produce $\hat{p}$. So, basically, we have avoided the computation of conditional probabilities of the labels.


text:
How can I train a neural network to grade the user's answers to a questionnaire?
I have a questionnaire consisting of over 10 questions. The questionnaire is being answered by a lot of people, which I have manually graded. Each question can give the user up to 10 points depending on how they have answered.
Let's say that my dataset is big enough, how would I go about using a neural network to automatically grade these questions for me?
I have used a convolutional neural network before for image classification, but, when dealing with text classification, where should I start? Is there some sort of tutorial out there that covers this with a similar example?

use the embedding layers which are mainly used for text.
input the question number and the text that the student wrote to the algorithm. Make the problem into a regression of a number from 0 to 10 ( or a classification of class 10, see which one gives better performance).  So the NN will look at text given a question number and try to figure out which 'points' should it get. It will be a supervised training problem since you already have examples of text rated from 0 to 10.


text:
How the actor use the output from the critic to make action in actor-critic network?
I am reading about the actor-critic architecture. I am confused about how the actor determines the action using the value (or future reward) from the critic network.
Below you have the most popular picture of actor-critic network.

It looks like the input of the actor network is only the "state" variable ($s_t$), it has nothing to do with the critic network.
However, from the equation below

the actor seems to be related to critic network.
I have a few questions

Does actor network has two inputs, state variable and future reward (output from critic network), or only the state variable?
If the actor network does take future reward as input, how does it use it? Only during the training stage, or in the action making stage?
Is there a "policy iteration" procedure happens during decision making stage, i.e. for every state $s_t$, policy network will make several attempts with critic network, and output the best policy?

"It looks like the input of the actor network is only the "state" variable (), it has nothing to do with the critic network.". Note that the picture contains an arrow from the critic which crosses the the actor.

Where did you find that equation? Can you please links us to the source of that equation?

@nbro I thought that is the TD error feed back for retrain the policy network and critic network.

it is from https://www.slideshare.net/zhihua98/actor-critic-algorithm in page 11


text:
Why are the nodes (or neurons) in neural networks depicted as circles?
Why are the nodes (or neurons) in neural networks depicted as circles?
What is the difference between a circle and a box in diagrams of neural networks?

There is no fully standardised representation of neural network architecture. It will depend on where you have seen this. Some writers might want to show a key difference by changing the shape of the node. Others will not care and will just show circles or boxes for everything. Could you please show an example of where you have seen this?

I guess the reason is that we call them nodes and nodes are usually depicted using circles in graph theory. We model them as graphs for both forward and backward paths.


text:
Is there any grid world dataset or generator for reinforcement learning?
I would like to start programming a multi task reinforcement learning model. For this, I need not just one maze or grid world (or just model-based), but many with different reward functions. So, I am wondering if exists a dataset or a generator for such thing, or do I need to code everything by my self?

Depending on your needs and the size of the project, you might be better off making a custom set of environments. If you'd rather not do that, though, you should take a look at OpenAI's CoinRun environment. A high-level description can be found in their blog post.
The "RandomMazes" version of this environment might be useful to you. And if you want to make the mazes smaller, you can redefine MAX_MAZE_DIFFICULTY in coinrun.cpp.
Note that, although the levels are procedurally generated, reward is only ever given when the agent picks up the single coin. So, this might not be as much variety in reward function as you wanted.


text:
How to deal with a huge action space, where, at every step, there is a variable number of legal actions?
I am working on creating an RL-based AI for a certain board game. Just as a general overview of the game so that you understand what it's all about: It's a discrete turn-based game with a board of size $n \times n$ ($n$ depending on the number of players). Each player gets an $m$ number of pieces, which the player must place on the board. In the end, the one who has the least number of pieces wins. There are of course rules as to how the pieces can be placed so that not all placements are legal at every move.
I have the game working in an OpenAI's gym environment (i.e. control by step function), have the board representation as the observation, and I have defined the reward function.
The thing I am struggling with right now is to meaningfully represent the action space.
I looked into how AlphaZero tackles chess. The action space there is $8*8*73 = 4672$: for every possible tile on the board, there are 73 movement-related modalities. So, for every move, the algorithm comes up with 4672 values, the illegal ones are set to zero and non-zero ones are re-normalized.
Now, I am not sure if such an approach would be feasible for my use-case, as my calculations show that I have a theoretical cap of ~30k possible actions ($n * n * m$) if using the same way of calculation. I am not sure if this would still work on, especially considering that I don't have the DeepMind computing resources at hand.
Therefore, my question: Is there any other way of doing it apart from selecting the legal actions from all theoretically possible ones?
The legal actions would be just a fraction of the ~30k possible ones. However, at every step, the legal actions would change because every new piece determines the new placement possibilities (also, the already placed pieces are not available anymore, i.e. action space generally gets smaller with every step).
I am thinking of games, like Starcraft 2, where action space must be larger still and they demonstrate good results, not only by DeepMind but also by private enthusiasts with for example DQN.
I would appreciate any ideas, hints, or readings!

A paper by Google for you: Deep Reinforcement Learning in Large Discrete Action Spaces

One way I can think of is to redefine "actions" in a game to make them more fragmented, in such a way that a player has multiple actions per turn. In chess, for example, we can define an action as choosing a tile from which to move, or choosing the motion from the chosen tile, as 2 separate actions.
As an example a turn might consist of the following two actions:

Choose E4
Move forward 2 spaces

That way there are 64 + 73, rather than 64 * 73 possible actions. The transition model would indicate that it's still the same player's turn after a "tile selection" action is done.
Of course, this would require increasing the state space in such a way that you can determine which action is legal. So there's a difference between a board state where nothing is "selected" and the same board state where one tile is selected by one player. In the chess example, this would require 2 more boolean CNN layers, one for each player, indicating which tile (if any) is "selected".
I never tried this myself, and I imagine that this might make the learning slower and more difficult, since it requires a deeper tree in the MCTS for the same set of actions.


text:
What is the purpose of the new neurons in the constrained neural network?
I would like to train a constrained neural network. I found a paper on this: https://papers.nips.cc/paper/4-constrained-differential-optimization.pdf.
However, I don't really understand how to change my feedforward neural network. I don't understand exactly the role of the new output neurons, which serve as lagrangian multipliers. Can someone explain this to me in more detail?
Which steps in backpropagation do I have to change?


text:
What is the difference between backpropagation and predictive coding?
Reading the high-level descriptions of backpropagation and predictive coding, they don't sound so drastically different. What is the key difference between these techniques?
I am currently reading the following paper if that helps ground the explanation:
Predictive Coding-based Deep Dynamic Neural Network for Visuomotor
Learning

In this paper, the author proved that backpropagation is a special case of predictive coding.

I would say that these concepts are quite different, even though they might have a few things in common (or might be vaguely related).
Back-propagation is an algorithm used (in machine learning) to compute the gradient of a function with respect to its parameters. This gradient is then used by algorithms, like gradient descent, to update the parameters of the model (e.g. a neural network).
Roughly, predictive coding is a general (neuroscience) theory of how the brain builds an internal model of the external world, of how it continuously predicts the sensory inputs (from the world) using its current model of the external world, and of how it updates this internal model once the sensory inputs are actually received.
You could think of the output of an ML model, while it is being trained (using gradient descent with back-propagation), as a prediction associated with the inputs. However, note that the output of the model is often not a prediction of the actual input, but e.g. a label (which is, nonetheless, associated with the input). Furthermore, it is the model that produces an output and not the back-propagation algorithm (even though the back-propagation algorithm is often used to train such models, e.g. neural networks). We could think of back-propagation as the way of updating these predictions associated with the inputs, but, anyway, it is well known (and accepted) that our brain does not perform back-propagation, but we learn in an associative fashion (Hebbian learning).