text:
What effect does a negative output of a neuron have on neighbouring neurons?
Artificial neural networks are composed of multiple neurons that are connected to each other. When the output of an artificial neuron is zero, it does not have any effect on neighboring neurons. When the output is positive, the neuron has an effect on neighboring neurons.
What does it mean when the output of a neuron is negative (which can e.g. occur when the activation function of a neuron is the hyperbolic tangent)? What effect would this output have on neighboring neurons?
Do biological neural networks also have this property?

I mean both of them

I think you might be misinterpreting what a neural network is doing with its outputs. The most an output of a neuron can generally mean, is how sensitive it is to a certain feature of the input. If you were to look at a 100 layer neural network, and saw an individual neuron was negative, there is very little you can infer from this. It doesn't really mean anything tangible. If you were to research this particular frozen state of the network, you could eventually interpret exactly what the network is doing, but for all intents and purposes, a negative output from a neuron doesnt tell you much

In the case of artificial neural networks, your question can be (partially) answered by looking at the definition of the operation that an artificial neuron performs. An artificial neuron is usually defined as a linear combination of its inputs, followed by the application of a non-linear activation function (e.g. the hyperbolic tangent or ReLU). More formally, a neuron $i$ in layer $l$ performs the following operation
\begin{align}
o_i^l = \sigma \left(\sum_{j=1}^N w_j o_j^{l-1} \right) \tag{1}\label{1},
\end{align}
where $o_j^{l-1}$ is the output from neuron $j$ in layer $l-1$ (the previous layer), $w_j$ the corresponding weight, $\sigma$ an activation function and $N$ the number of neurons from layer $l-1$ connected to neuron $i$ in layer $l$.
Let's assume that $\sigma$ is the ReLU, which is defined as follows
$$
\sigma(x)=\max(0, x)
$$
which means that all negative numbers become $0$ and all non-negative numbers become themselves.
In equation \ref{1}, if $w_j$ and $o_j^{l-1}$ have the same sign, then the product $w_j o_j^{l-1}$ is non-negative (positive or zero), else it is negative (or zero). Therefore, the sign of the output of neuron $j$ in layer $l-1$ alone does not fully determine the effect on $o_i^l$, but the sign of the $w_j$ is also required.
Let's suppose that the product $w_j o_j^{l-1}$ is negative, then, of course, this will negatively contribute to the sum in equation \ref{1}. In any case, even if the sum $\sum_{j=1}^N w_j o_j^{l-1}$ is negative, if $\sigma$ is the ReLU, no matter the magnitude of the negative number, $o_i^l$ will always be zero. However, if the activation function is hyperbolic tangent, the magnitude of a negative $\sum_{j=1}^N w_j o_j^{l-1}$ affects the magnitude of $o_i^l$. More precisely, the more negative the sum is, the closest $o_i^l$ is to $-1$.
To conclude, in general, the effect of the sign of an output of an artificial neuron on neighboring neurons depends on the activation function and the learned weights, which depend on the error the neural network is making (assuming the neural network is trained with gradient descent combined with back-propagation), which in turn depends on the training dataset, the loss function, the architecture of the neural network, etc.
Biological neurons and synapses are more complex than artificial ones. Nevertheless, biological synapses are usually classified as either excitatory or inhibitory, so they can have an excitatory or inhibitory effect on connected neurons.

So what means when we receive a negative in next neuron? Could you give me an example

@MehdiAmrollahi I am exactly explaining this in my answer. Please, read it again until you understand it.

Maybe I do not understand the question - isn't it the case the the artificial neurons in layer X are only affected by those in layer X-1 (providing inputs), not other neurons in layer X?

@GeorgeWhite I think this is a nice question you should ask on the main website. I'll answer it (if nobody does it).


text:
How does reinforcement learning with video data work?
My goal is to train an agent to play MarioKart on the Nintendo DS. My first approach (in theory) was to setup an emulator on my pc and let the agent play for ages. But then a colleague suggested to train the agent first on pre recorded humanly played video data, to achieve some sort of base level. And then for further perfection let the agent play for its own with the emulator.
But I have no clue how training with video data works. E.g. I wonder how to calculate a loss since there is no reward. Or am I getting the intuition wrong?
I would appreciate it if someone could explain this technique to me.

In reinforcement learning, to learn off policy control, you need data on the states, actions and rewards at each time step. If, in addition to a recorded video, you had a recording of controller inputs, and could add reward data by hand, then you could use a standard reinforcement learning method, e.g. DQN. Simply run the DQN training loop as normal, but skip the parts where it acts in the environment, and only train on batches of recorded experience.
With only video data, your options are limited. However, it might still be useful, because a significant part of the challenge is a machine vision task. For a DQN agent, it will need to convert frames from the video (e.g. last 4 frames) into a prediction of the different rewards that it could get depending on which controller buttons are pressed. If you can teach a separate neural network to perform a vision task on relevant video data, it may may help. You could use the learned weights from the first layers of this network as the starting point for your Q values network, and it will likely speed up a DQN figuring out the relationship to its predictions. This sort of task switch following learning is called transfer learning, and is often used in computer vision tasks.
A possibly useful starting task if you have a video, but no controller or reward data, would be to predict the next frame(s) of the video, given say four starting frames (you need more than one so that the neural network can use velocity information). It should be possible to generate the training data using opencv or ffmpeg from your recordings.

So first I train a network to identify important objects? (Can I use tho YOLO network here?) And with the trained identifier I train a second network to predict the next frame from four predecessor frames.
But then? And will this really give me an advantage? I can still let the agent play for itselfs. I think I will have enougth work setting up the enviroment and rewards and so on...

@bruh: If you have information about the objects available that might be useful as a transfer learning starter and you could use e.g. YOLO. However, my suggestion in the answer is simply to predict the next frame, because you already have that in the video data, so it requires a lot less data preparation. Yes you can start directly with the RL, that is what the DQN authors did for all the Atari games, and it worked well for most of them. However, you asked whether it was possible to learn from just the video, and this answer addresses the original question.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How can I prove that all the a-cuts of any fuzzy set A defined on $R^n$ are convex?
How can I prove that all the a-cuts of any fuzzy set A defined on $R^n$
are convex if and only if
$$\mu_A(\lambda r + (1-\lambda)s) \geq min \{\mu_A(r), \mu_A(s)\}$$
such that $r, s \in R^n$, $\lambda \in [0, 1]$ ?
That's a fuzzy question on my assignment. Any idea on how to start with?

A fuzzy set A in $R^n$ is said to be a convex fuzzy set if its
  $\alpha$-cuts $A_\alpha$ are (crisp) convex sets for all $A \in (0,1]$
  .

Let A be a convex fuzzy set if and only if for all $r, s \in$ $R^n$, $\lambda \in [0, 1]$ .
Let $\alpha=\mu_A\leq\mu_B$
Then
\begin{equation}
    r\in A_{\alpha}, s\in A_{\alpha}
\end{equation}
and also
\begin{equation}
    \lambda r + (1-\lambda)s \geq \alpha = min \{\mu_A(r), \mu_A(s)\}
\end{equation}
Conversely, if the membership funciton $\mu_A$ of the fuzzy set A satisfies the inequality of Theorem 13.1 Convex fuzzy set, then taking $\alpha=\mu_A(r), A_\alpha$ may be regarded as set of all points $s$ for which $\mu_A(s)\geq\alpha=\mu_A(r)$. Therefore for all $r,s \in A_\alpha$,
\begin{equation}
    \mu_A(\lambda r + (1-\lambda)s) \geq min \{\mu_A(r), \mu_A(s)\} = \mu_A(r)=\alpha
\end{equation}
which inplies that $\lambda r + (1-\lambda)s \in A_\alpha$. Hence $A_\alpha$ is a convex set for every $\alpha \in [0,1]$

We can assume without loss of generality that
\begin{equation}
\min\{\mu_A(r), \mu_A(s)\} = \mu_A(r) = \alpha.
\end{equation}
$\implies$
a-cut of fuzzy set $A$ is on $R^n$ is convex. A-cut can be defined as
\begin{equation}
A = \{x \in R^n| \mu_A(x) \geq \alpha\}
\end{equation}
If we take two elements $r$ and $s$, by the definition of convex set, number $\lambda r + (1 + \lambda)s$ is also an element of that set. Since it's an element of that set that means
\begin{equation}
\mu_A(\lambda r + (1 + \lambda)s) \geq \alpha
\end{equation}
$\impliedby$
$\mu_A(\lambda r + (1 + \lambda)s) \geq \alpha$.
We know from $\min\{\mu_A(r), \mu_A(s)\} = \mu_A(r) = \alpha$ that $\mu_A(s) > \alpha$. We have an affine combination $\lambda r + (1 + \lambda)s$ for which also $\mu_A(\lambda r + (1 + \lambda)s) \geq \alpha$ so we know that all numbers $\lambda r + (1 + \lambda)s$ satisfy inequality $\mu_A(\cdot) \geq \alpha$ (belong to the same set as $r$ and $s$) which means this is a convex set again by the definition of a convex set.

I posted an answer , it would be nice if you could tell me if it's true . In addition, since you know about Fuzzy Logic do you have any idea on this ?

@estamos sorry, I actually don't know about fuzzy logic, I googled a-cuts of fuzzy sets and I know some facts about convex sets which was enough to derive a proof. I can't really help you with other question, and I don't know if your answer is correct or not since it introduces some new things that I'm not familiar with. Sorry for misleading you that i know about fuzzy logic. You should probably post your answer as a separate question and if noone responds here consider posting your questions on Math StackExchange which is more math oriented website.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is there a probabilistic version of minimax?
How would a probabilistic version of minimax work?
For example, we may choose a move that could result in a very bad outcome, but that outcome might just be extremely unlikely so we might think it would be worth the risk.

You might want to look up 'Markov decision process' and how it is used in reinforcement learning. This is probabilistic minimax in essence.

Yes, there is at least one probabilistic version of minimax, which is called expectiminimax. In expectiminimax, in addition to min and max nodes, there are also chance nodes, which perform a weighted sum of the successors, so the probabilities associated with chance nodes must be known. Given that expectiminimax assumes the existence of random events (represented by the chance nodes), the decisions are thus based on expected values.
Section 5.5 of the book Artificial Intelligence: A Modern Approach provides a description of the expectiminimax algorithm, which was introduced by Donald Michie in Game-playing and game-learning automata (1966). The paper Optimal strategy in games with chance nodes (2007) also gives a decent description of the expectiminimax algorithm.


text:
Why does the adversarial search minimax algorithm use Depth-First Search (DFS) instead of Breadth-First Search (BFS)?
I understand that the actual algorithm calls for using Depth-First Search, but is there a functionality reason for using it over another search algorithm like Breadth-First Search?

The primary reason is that Breadth-First Search requires much more memory (and this probably also makes it a little bit slower in practice, due to time required to allocate memory, jumping around in memory rather than working with what's still in the CPU's caches, etc.). Breadth-First Search needs memory to remember "where it was" in all the different branches, whereas Depth-First Search completes an entire path first before recursing back -- which doesn't really require any memory other than the stack trace. This is assuming we're using a recursive implementation for DFS -- which we normally do in the case of minimax.
You can clearly see this if you look at pseudocode for the two approaches (ignoring the minimax details here, just presenting pseudocode for straightforward searches):
BreadthFirstSearch(start):
    Q = new queue()
    Q.append(start)
    while Q is not empty:
        node = Q.pop()
        if node is leaf:
            do something with leaf
        else:
            for each child of node:
                Q.append(child)

DepthFirstSearch(start):
    if start is leaf:
        do something with leaf
    for each child of start:
        DepthFirstSearch(child)
        // probably do something with return value from the recursive DFS call

You see that the BFS requires a queue object that explicitly stores a bunch of stuff in memory, whereas DFS doesn't.

There's more to the story once you get to extensions of Minimax, like Alpha-Beta pruning and Iterative Deepening... but since the question is just about Minimax, I'll leave it at that for now.


text:
Are neurons in layer $l$ only affected by neurons in the previous layer?
Are artificial neurons in layer $l$ only affected by those in layer $l-1$ (providing inputs) or are they also affected by neurons in layer $l$ (and maybe by neurons in other layers)?

It depends on the architecture of the neural network. However, in general, no, neurons at layer $l$ are not only affected by neurons at layer $l-1$.
In the case of a multi-layer perceptron (or feed-forward neural network), only neurons at layer $l-1$ directly affect the neurons at layer $l$. However, neurons at layers $l-i$, for $i=2, \dots, l$, also indirectly affect the neurons at layer $l$.
In the case of recurrent neural networks, the output of neuron $j$ at level $l$ can also affect the same neuron but at a different time step.
In the case of residual networks, the output of a neuron at a layer $l-i$, for $i=2, \dots, l$, can directly affect the neurons at layer $l$. These non-neighboring connections are called skip connections because they skip layers.
There are probably other combinations of connections between neurons at different layers or the same layer.


text:
Would AI be appropriate for converting unstructured text into an XML?
I need to understand whether it is better to use AI algorithms (ML, DL, etc.) instead of the classic parser (based onto grammars with regular expression and automaton) for the following task: structuring in XML an unstructured plain text.
The text is a legal document, so the structure is well defined and a classic parser could do a good job.
In the case AI could be a viable way, what would be an appropriate approach for the task?

The question confuses ML (including DL) with AI. AI is a bigger field than ML and includes rule-based systems
You probably need to extract entities (spans of text) from the unstructured text and embed them into an XML. ML (and DL) are good when the problem is fuzzy (you need very many rules to solve the problem) so it could be a valid option if you have a variety of document structures that each needs its own set of rules. You would need enough data to train your models in this case. Otherwise if you have limited document structures, very limited data (maybe none) and 100% accuracy is expected then going with rules is the obvious choice.

100% accuracy it's a must for my task. Yes you are right, rule based system is in the AI field. What do yo mean for "very limited data (maybe none)"?

data to train ML or DL models on. The size of data available plays a big role in how to approach the problem

I have a lot of data to train the model, but 100% accuracy maybe is the problem. Do you have any suggestion what to use?

100% accuracy could be observed in some simple use cases of ML but even then it can't be guaranteed. it makes sense to start with a rule based system and see where the complexity and limitations are. This is usually good even if you would use ML in the end (and need to communicate to business people that 100% can't be guaranteed in this case)

Yes I know that 100% accuracy it's impossibile and your advice it's a confirmation to what I thought about my needs and ML. Thanks a lot.

A rule-based approach will guarantee a correct result, and it works perfectly fine. On the other hand, an AI based approach will introduce errors as AI cannot produce result with 100% accuracy, and also decreasing speed. As the document you are processing is a legal document, it would be better to use a parser as AI is only adding unnecessary wasted time and non-accurate results.
Hope I can help you.

what kind of parser do you advise? Because it's natural language I do not think that CFG parsers are goods. Maybe some regexs fit the task.


text:
Why is dropout favoured compared to reducing the number of units in hidden layers?
Why is dropout favored compared to reducing the number of units in hidden layers for the convolutional networks?
If a large set of units leads to overfitting and dropping out "averages" the response units, why not just suppress units?
I have read different questions and answers on the dropout topic including these interesting ones, What is the "dropout" technique? and this other Should I remove the units of a neural network or increase dropout?, but did not get the proper answer to my question.
By the way, it is weird that this publication A Simple Way to Prevent Neural Networks from Overfitting (2014), Nitish Srivastava et al., is cited as being the first on the subject. I have just read one that is from 2012:
Improving neural networks by preventing co-adaptation of feature detectors.

Dropout only ignores a portion of units during a single training batch update.  Each training batch will use a different combination of units which gives them the best chance of that portion of them working together to generalize. Note the weights for each unit are kept and will be updated during the next batch in which that unit is selected. During inference, yes, all units are used (with a factor applied to activation...the same factor that defines the fraction used)...this becomes essentially an ensemble of all the different combinations of units that were used.
Contrasted with fewer units, the fewer units approach will only learn what those fewer units can be optimized for.  Think of dropout as an ensemble of layers of fewer units.(with the exception that there are partial weight sharing between the layers)

I believe the comment about "each training batch" using a "different combination of units" is incorrect, at least as per the original paper on dropout. The dropout is applied per observation, not per mini batch. I believe this also means that weights of units that are dropped for only some of the observations in a mini batch are indeed updated during back propagation because they contribute to the loss function, which is an average across all observations in a mini batch. See: https://stats.stackexchange.com/questions/335690/understanding-dropout-method-one-mask-per-batch-or-more.

The idea of dropout is that, at training time, with a certain probability $p_i \in [0, 1]$, the unit (or neuron) $i$ is dropped, $\forall i$, that is, the output of unit $i$ is set to zero so that $i$ does not affect the other units it is connected to, both during the forward and backward (or back-propagation) passes (or steps). At every mini-batch, you randomly drop usually different units, so, across different mini-batches (and consequently epochs), you do not always or necessarily drop the same units.
The title of the paper Improving neural networks by preventing co-adaptation of feature detectors emphasizes that dropout prevents the co-adaptation of the units (the feature detectors), so units attempt to detect certain features independently of other units, which reduces overfitting, that is, it improves the generalization ability of the neural network.
At test time, no unit is usually dropped. However, there is an approximation of a deep Gaussian process and Bayesian neural network that is based on the application of dropout at training and test times. This is called Monte Carlo dropout or, in short, MC dropout, for reasons you can understand if you read the paper Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.
There's also the possibility to drop the connections between the neurons, which is called DropConnect, rather than the neurons themselves. These two approaches are slightly different, even though DropConnect can be seen as a generalization of dropout. In DropConnect, you do not switch off completely the units, but only the contributions of certain units to the output of other units. In dropout, you completely switch off certain units.
If you decided to deterministically (and manually) reduce the number of units before training, essentially, you would train a fixed smaller network, but this will not necessarily reduce overfitting or, more precisely, co-adaptation of the units. In dropout, you randomly select the units to drop, so, at every mini-batch (or epoch, depending on the implementation of dropout), you effectively train a random subset of the units of the original neural network and, because of this, it can be thought of as an ensemble of smaller neural networks.
The two papers Improving neural networks by preventing co-adaptation of feature detectors (2012) and Dropout: A Simple Way to Prevent Neural Networks from Overfitting (2014) have exactly the same authors, but the latter paper was published in the Journal of Machine Learning Research, while the former wasn't apparently published in any journal. In fact, Dropout: A Simple Way to Prevent Neural Networks from Overfitting does not even cite Improving neural networks by preventing co-adaptation of feature detectors, but it cites the master's thesis Improving Neural Networks with Dropout (2013) by Nitish Srivastava, who is one of the authors of dropout.

(+1) But not that it's not necessarily true that different units are dropped at each training iteration. Vanilla dropout is essentially a Bernoulli(p) binary mask, so the same unit can be dropped during 2 successive iterations with probability $p^2$. Likewise, the entire dropout mask could, by chance, be the same between two iterations with some small probability.

@SycoraxsaysReinstateMonica Thanks for the feedback. I've started this answer with "The idea of dropout", because I wanted more to give the intuition. However, you're right, the same units could be dropped during two successive training iterations. I've edited my answer to try accommodating the information in your comment.


text:
Why is reinforcement learning not the answer to AGI?
I previously asked a question about How can an AI freely make decisions?. I got a great answer about how current algorithms lack agency.
The first thing I thought of was reinforcement learning, since the entire concept is oriented around an agent getting rewarded for performing a correct action in an environment. It seems to me that reinforcement learning is the path to AGI.
I'm also thinking: what if an agent was proactive instead of reactive? That would seem like a logical first step towards AGI. What if an agent could figure out what questions to ask based on their environment? For example, it experiences an apple falling from a tree and asks "What made the Apple fall?". But it's similar to us not knowing what questions to ask about say the universe.

Some AI researchers do think RL is a path to AGI, and your intuition about how an agent would need to be proactive in selecting actions to learn about is exactly the area these researchers are now focused on.
Much of the work in this area is focused on the idea of curiosity, and since 2014 this idea has gained a lot of traction in the research community.
So, maybe RL can lead to AGI. We don't know for sure yet.
However, many of the classic arguments against AGI aren't addressed by the RL approach. For instance, if like Searle, you think computers just don't have the right kind of hardware to do thinking, then running an RL algorithm on that hardware isn't going to yield AGI, just ever increasingly robust narrow AI. Ultimately Searle's arguments get into issues of metaphysics, so it isn't clear that there exists any argument that would convince someone like Searle that a particular computer-based technique is AGI-capable.
There are also other arguments. For example, the cognitivist school of thought thinks that statistical learning approaches to AI, and, in particular, the black-box approaches of statistically-driven RL, are unlikely to lead to general intelligence because they do not engage in the kind of systematic reasoning process that proponents of cognitivism assume is necessary for general intelligence. Some more extreme proponents of this school might say that a logical planning algorithm like STRIPS is innately more intelligent than any approach based on deep learning, because it involves sound logical deduction rather than mere statistical calculation. In particular, STRIPS can correctly generalize to any new domain, as long as it is fed the correct sense data, while an RL approach will need to learn how to act there.
So, while there are definitely reasons to be optimistic about RL as a direction for achieving AGI, it's definitely not yet settled.

A relatively recent but interesting paper that discusses this topic in more detail is Reward is enough (Artificial Intelligence, 2021) by David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton (so by some of the godfathers of RL, who are all at DeepMind).
Their reward-is-enough hypothesis (RIEH) (page 4) is

Hypothesis (Reward-is-Enough). Intelligence, and its associated abilities, can be understood as subserving the maximisation of reward by an agent acting in its environment.

This hypothesis is slightly different from the reward hypothesis (RH), which states that all goals can be represented by rewards and the achievement of those goals can be viewed or formulated as the maximization of rewards, because the RIEH also states that the abilities needed to achieve the main goal in the environment arise from the maximization of the reward, so the RIEH is a stronger hypothesis than the RH.
The authors give examples to explain the RIEH (emphasis mine).

Sophisticated abilities may arise from the maximisation of simple rewards in complex environments. For example, the minimisation of hunger in a squirrel’s natural environment demands a skilful ability to manipulate nuts that arises from the interplay between (among other factors) the squirrel’s musculoskeletal dynamics; objects such as leaves, branches, or soil that the squirrel or nut may be resting upon, connected to, or obstructed by; variations in the size and shape of nuts; environmental factors such as wind, rain, or snow; and changes due to ageing, disease or injury. Similarly, the pursuit of cleanliness in a kitchen robot demands a sophisticated ability to perceive utensils in an enormous range of states that includes clutter, occlusion, glare, encrustation, damage, and so on.

They also try to argue why language, perceptron, social intelligence and general intelligence could all arise from the maximization of a single reward signal (e.g. survival).
Moreover, they also say that similar sophisticated abilities associated with intelligence could arise from the maximization of different reward signals, i.e. the emergence of these abilities is robust to the choice of reward objective.
Additionally, they also talk about prior knowledge and learning, but, in my view, they should have emphasized/noted that, for example, perception, without the suitable sensors (inductive bias) cannot emerge: this is not a limitation of the RIEH, as it says nothing about how these abilities actually arise, or the nature of the agent needed for them to arise, or which specific reward signal should be maximized.
In the end, they also conjecture that RL is the main framework that could be used to find out whether these conjectures/speculations are true or not.
They do not go into philosophical arguments, such as the Chinese-Room argument or the problem consciousness: their argument to address these issues would probably be that any ability (even consciousness, if it's an ability) required to achieve the ultimate goal would arise in the process of maximization of the reward.


text:
How can we compute the gradient of max pooling with overlapping regions?
While studying backpropagation in CNNs, I can't understand how can we compute the gradient of max pooling with overlapping regions.
That's also a question from this quiz and can be also found on this book.

When gradients in a neural network can follow multiple paths to same parameter, the different gradient values from the sources can often be added together, because the operations in the forward direction are also sums and $\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$.
That is the case already with gradients of kernels (which are sums over the image area), and is equally the case for overlapping aggregation, including maximums, minimums or averages.
So in the 1d case, if you have a max pool over the input params $[a_0, a_1, a_2, a_3, a_4]$ a max function $m_0 = max(a_0, a_1, a_2)$, $m_1 = max(a_2, a_3, a_4)$ which overlap at $a_2$, and gradients $\nabla_{\mathbf{m}} J = [\frac{\partial J}{\partial m_0}, \frac{\partial J}{\partial m_1}]$, then you would allocate those gradients to vector $\mathbf{a}$ according to which items in each group was the max of that group, adding them when they overlapped.
Examples:
If $\mathbf{a} = [3,0,1,2,0]$ and  $\nabla_{\mathbf{m}}J = [0.7, 0.9]$, then $\nabla_{\mathbf{a}}J = [0.7, 0, 0, 0.9, 0]$
If $\mathbf{a} = [3,0,4,2,0]$ and  $\nabla_{\mathbf{m}}J = [0.7, 0.9]$, then $\nabla_{\mathbf{a}}J = [0, 0, 1.6, 0, 0]$

Denote x(h,w) input to max-pooling, and y(h,w) - output.
Then $\frac{dL}{dx}(h,w)= \sum \frac{dL}{dy}(h',w')$
over all y(h',w') which have been obtained  from x(h.w) such that y(h',w') = x(h,w).
related to this p 11

So this is the same answer as mine (add the gradients when they overlap), but for the 2D case.


text:
How to compute the number of weights of a CNN?
How can we theoretically compute the number of weights considering a convolutional neural network that is used to classify images into two classes:

INPUT: 100x100 gray-scale images.
LAYER 1: Convolutional layer with 60 7x7 convolutional filters (stride=1, valid
padding).
LAYER 2: Convolutional layer with 100 5x5 convolutional filters (stride=1, valid
padding).
LAYER 3: A max pooling layer that down-samples Layer 2 by a factor of 4 (e.g., from 500x500 to 250x250)
LAYER 4: Dense layer with 250 units
LAYER 5: Dense layer with 200 units
LAYER 6: Single output unit

Assume the existence of biases in each layer. Moreover, the pooling layer has a weight (similar to AlexNet)
How many weights does this network have?
Here would be the corresponding model in Keras, but note that I am asking for how to calculate this with a formula, not in Keras.
import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Conv2D, MaxPooling2D

model = Sequential()
model.add(Conv2D(60, (7, 7), input_shape = (100, 100, 1), padding="same", activation="relu")) # Layer 1
model.add(Conv2D(100, (5, 5), padding="same", activation="relu")) # Layer 2
model.add(MaxPooling2D(pool_size=(2, 2))) # Layer 3
model.add(Dense(250)) # Layer 4
model.add(Dense(200)) # Layer 5

model.summary()

Here is a related (if not duplicate) question.

Calculating the number of parameters in a CNN is very straightforward.
A CNN is composed of different filters, which are essentially 3d tensors. CNN weights are shared, meaning they are used multiple times and reused in different locations. Each layer has $n$ tensors, each with dimension $w \times h \times c$, where $w$ = width, $h$ = height, $c$ = channels (the input channel size). Therefore, the number of parameters of a convolutional layer is $w * h * c * n$. There is also a bias for each output channel, so the number of biases is $n$. At the end the parameter number is calculated with: $n * w * h * c + n$. See more about this in hear: Article
The pooling layer does not have weights, it only has hyperparameters. You may have confused the two. There are hyperparameters for the stride, the factor and etc. These are predefined and not trainable.
For Keras, you can use the solution here.


text:
How to define the "Pre-Processing" in machine learning?
Is every process (such as data acquisition, splitting the data for validation, data cleaning, or feature engineering) that is done on the data before we train the model always called the pre-processing part? Or are there some processes that are not included?

Despite "data acquisition" which is debatable, I think the other steps are generally considered "preprocessing".

I never considered the acquisition or splitting of data to be pre-processing. The reasons being acquiring the data is just the physical act of getting it and not doing anything to it.  Similarly, splitting for train/valid/test can be done without any modifications to the data.  I accept that there are likely many ways of viewing the different processes and I don't feel strong enough to put it in an answer, so a comment it is.  Being quite general, perhaps your question could be reworded or redone to focus on one particular task in the modeling pipeline.

Data preprocessing consists of all those techniques used to generate the final datasets (with an appropriate size, structure, and format) for the machine learning algorithms or models. Data acquisition should not be part of data preprocessing, but the step preceding it, which gathers the raw data (which may e.g. be noisy).
The book Data Preprocessing in Data Mining (2014), by Salvador García et al., which provides a good overview of the data preprocessing techniques and their connection with data mining and machine learning algorithms and models, defines data preprocessing as follows.

Data preprocessing includes data preparation, compounded by integration,
  cleaning, normalization and transformation of data; and data reduction tasks; such as feature selection, instance selection, discretization, etc. The result expected after a reliable chaining of data preprocessing tasks is a final dataset, which can be considered correct and useful for further data mining algorithms.

From page 10 onwards, there is a description and categorization of the main data preprocessing techniques. I will just list them, so refer to the book for a definition and explanation of each of these techniques.

Data Preparation

Data Cleaning
Data Transformation
Data Integration
Data Normalization
Missing Data Imputation
Noise Identification

Data Reduction

Feature Selection
Instance Selection
Discretization
Feature Extraction/Instance Generation

Here are two screenshots (from the cited book) that illustrate some of the data preparation

and data reduction techniques.

Thank you for the reference!.. but from your first statement, what do you think about data augmentation?

I think data augmentation could be considered part of data pre-processing. It should correspond to the data integration part I mentioned above (maybe combined with other steps of the process).


text:
How should I weight the factors that affect the choice of an action in a strategy board game with multiple actions?
I have written an AI that plays a strategy board game. There are lots of different types of moves (e.g. attack, defend, help ally colony, etc.).
I calculate the best moves to do depending on a variety of factors, such as the value of nearby enemy colonies, the number of armies the colony currently has, etc (each of these has separate weightings). I'm trying to find the optimal weighting for each of the different factors.
Currently, I decide the best configuration of parameters in a King of the Hill style tournament. I choose random values between a suitable range for each of the different parameters and then play two of these AI against each other 20 times. I have a total of 100 AI that play against the king, and then take the final king as the best AI.
The problem is that this is quite slow and I feel it's very inefficient, as a lot of the AI don't play well at all (probably due to the randomness of parameter values).
I'm wondering if there's a more efficient way to determine the optimal value of parameters?

You could use a genetic algorithm to optimise the parameter settings. Here you don't choose random parameters all the time, but only at the beginning. Each AI (which is a vector of parameter settings) plays each other one for a ranking (you can probably reduce the number of total games by using a ladder-style ranking where only neighbours play against each other). Also, you don't need 100 to start with, as you might come across better combinations than those present initially throughout the processing.
Then you discard the worst AIs (ie parameter vectors), and recombine the best ones, adding some random mutations into the mix. In theory this should converge faster, as you preserve good parameter settings (depending on how interdependent they are) and remove bad ones from the 'pool'.
You can also have a higher rate of mutations initially, which slowly goes down as you progress through the generations.


text:
What is the difference between artificial intelligence and artificial neural networks?
I have made several neural networks by using Brain.Js and TensorFlow.js.
What is the difference between artificial intelligence and artificial neural networks?

From wikipedia:

Artificial neural networks (ANN) or connectionist systems are computing systems that are inspired by, but not identical to, biological neural networks that constitute animal brains. Such systems "learn" to perform tasks by considering examples, generally without being programmed with task-specific rules.

Artifical Intelligence on the other hand refers to the broad term of

intelligence demonstrated by machines

This obviously doesn't clear much up, so the next logical question is: "What is intelligence?"
This, however, is one of the most debated questions in computer science and many other fields, so there isn't a straight answer for this. The most you can do is decide yourself what you think intelligence refers to, because as far as we know, there is no agreed upon way of quantifying intelligence, and so the definition of such will remain ambiguous.

Artificial intelligence is a broad field, ANN is a specific technique with in that field.

Artificial intelligence can refer to a broad range of techniques by which machines (algorithms) demonstrate utility (fitness in an environment, where the environment may be either virtual or physical.)
This can include symbolic AI, which utilizes logic and search exclusively.  (Symbolic AI is sometimes referred to as "good old fashioned AI" aka gofai, or "Classical AI".)

A key distinction is that Neural Networks constitute a form of "statistical AI", which renders them capable of learning by trial/error & analysis.

The recent strength & applicability of statistically driven AI methods has been facilitated by advances in processing power and memory.

I would explain it as Artificial Intelligence is a huge topic concerning many fields as: robotics, computer vision, machine learning, etc. It focuses on any "inteligent" task that a computer can do.
Artificial Neural Networks are a sub-topic of Machine learning, and probably as you've seen, as you said you have some experience with them, deals with a specific way of solving problems using a set of 'neurons' that try to imitate actual biological neurons. Explaining it in a really simplistic way, it is a method of fitting a function to your specific data in such a way that it still stands and gives good predictions on test data. By 'training' the network, you're basically trying to find better values for the weights(analogous to synapses in an actual brain, connections between the neurons) between the neurons in order to give better outputs in general on that specific type of data instead of just one case.


text:
Does each filter in each convolution layer create a new image?
Say I have a CNN with this structure:

input = 1 image (say, 30x30 RGB pixels)
first convolution layer = 10 5x5 convolution filters
second convolution layer = 5 3x3 convolution filters
one dense layer with 1 output

So a graph of the network will look like this:

Am I correct in thinking that the first convolution layer will create 10 new images, i.e. each filter creates a new intermediary 30x30 image (or 26x26 if I crop the border pixels that cannot be fully convoluted).
Then the second convolution layer, is that supposed to apply the 5 filters on all of the 10 images from the previous layer? So that would result in a total of 50 images after the second convolution layer.
And then finally the last FC layer will take all data from these 50 images and somehow combine it into one output value (e.g. the probability that the original input image was a cat).
Or am I mistaken in how convolution layers are supposed to operate?
Also, how to deal with channels, in this case RGB? Can I consider this entire operation to be separate for all red, green and blue data? I.e. for one full RGB image, I essentially run the entire network three times, once for each color channel? Which would mean I'm also getting 3 output values.

You are partially correct.  On CNNs the output shape per layer is defined by the amount of filters used, and the application of the filters (dilation, stride, padding, etc.).
CNNs shapes
In your example, your input is  30 x 30 x 3.  Assuming stride of 1, no padding, and no dilation on the filter, you will get a spatial shape equal to your input, that is 30 x 30.  Regarding the depth if you have 10 filters (of shape 5 x 5 x 3) you will end up with a 30 x 30 x 10 output at your first layer.  Similarly, on the second layer with 5 filters (of shape 3 x 3 x 10, note the depth to work on the previous layer) you have 30 x 30 x 5 output. The FC layer has the same amount of weights as the input (that is 4500 weights) to create a linear combination of them.
CNN vs Convolution
Note that the CNNs operate differently from the traditional signal processing convolution.  In the former, the convolution operation performs a dot product with the filter and the input to output a single value (and even add bias if you want to).  While the latter outputs the same amount of channels.
The CNNs borrow the idea of a shifting kernel and a kernel response.  But they do not apply a convolution operation per se.
Operation over the RGB
The CNN is not operating on each channel separately.  It is merging the responses of the three channels and mixing them further.  The deeper you get the more mix you get over your previous results.
The output of your FC is just one value.  If you want more, you need to add more FC neurons to get more linear combinations of your inputs.

About the images inside the CNN layers: I really recommend this article since there is no one short answer to this question and it probably will be better to experiment with it.
About the RGB input images: When needed to train on RGB pictures it is not advised to split the RGB channels, you can think of it by trying to identify a fictional cat with red ears, green body and blue tail. Each separated channel don't represent a cat, most certainly not with high confidence. I would recommend to transform you RGB images to gray scale and measure the network performance. If the performance are not sufficient you can make a 3D convolution layer. For example: If 30x30x3 is the input image, the filter has to be NxNx3.

Thanks for the link  I understand what you mean about losing information by processing the channels separately (independently). But I'm having difficulty finding a universal way to combine the them into one channel. Converting to grayscale (or "luminosity" using YCbCr conversion) may work in some cases, but not if for example there is a green gat on a red background. However a 3D convolution seems like overkill? Or is it reasonable to ignore these exceptions?

For a 3 channel image (RGB), each filter in a convolutional layer computes a feature map which is essentially a single channel image.  Typically, 2D convolutional filters are used for multichannel images. This can be a single filter applied to each layer or a seperate filter per layer. These filters are looking for features which are independent of the color, i.e. edges (if you are looking for color there are far easier ways than CNNs). The filter is applied to each channel and the results are combined into a single output, the feature map. Since all channels are used by the filter to compute a single feature map, the number of channels in the input does not affect the structure of the network beyond the first layer. The size of a feature map is determined by the filter size, stride, padding and dilation(not commonly used - see here if you are interested.).
In your example, a 30 x 30 x 3 input convolved with 10 5 x 5 filters will yield a volume of 30 x 30 x 10 if the filters have a stride of 1 and same padding (or, 26 x 26 x 10 with valid padding / 34 x 34 x 10 with full padding).
Same padding buffers the edge of the input with filter_size/2 (integer division) to yield an output of equal size (assuming stride is 1) while valid padding would result in a smaller output. Valid padding doesn't crop the image as you said, it's more of a dilution of the signal at the edges, however the results is essentially the same. Note that even with same padding the edge pixels are used in fewer convolutions - a 5 x 5 filter with same padding will use a central pixel 25 times (every position on the filter) but only 9 times for a corner pixel. To use all pixels evenly full padding must be used which buffers the edge of the input with filter_size - 1.

Each feature map becomes a channel in the output volume. Therefore, the number of channels in the output volume is always equal to the number of filters in the convolutional layer. So, the second layer would output a volume of size 30 x 30 x 5 (stride 1, same padding).
The last layer in your example (fully connected) multiplies the value of each pixel in each feature map by a learned weight and sums the result. If the network is a binary classifier, the summed value results in a 1 or 0 output if a threshold is reached or as a decimal value for a regression model. This is determined by the FC neurons' activation function.
If visualizing this helps you as much as it helps me, I highly recommend having a look at the interactive examples here. Note that what is shown by this tool is the signal propagating through the network, i.e. the output from each layer, not the filters/weights themselves.
If you are interested in a bit more depth about ANNs and convolutional layers, I cover all the basics in my thesis(this is where the image is from) - p.9-16 ANNs & p.16-23 CNNs.


text:
How can an AI freely make decisions?
Suppose a deep neural network is created using Keras or Tensorflow. Usually, when you want to make a prediction, the user would invoke model.predict. However, how would the actual AI system proactively invoke their own actions (i.e. without the need for me to call model.predict)?

Neural networks, deep learning and other supervised learning algorithms do not "take actions" by themselves, they lack agency.
However, it is relatively easy to give a machine agency, as far as taking actions is concerned. That is achieved by connecting inputs to some meaningful data source in the environment (such as a camera, or the internet), and connecting outputs to something that can act in that environment (such as a motor, or the API to manage an internet browser). In essence this is no different from any other automation that you might write to script useful behaviour. If you could write a series of tests, if/then statements or mathematical statements that made useful decisions for any machine set up this way, then in theory a neural network or similar machine learning algorithm could learn to approximate, or even improve upon the same kind of function.
If your neural network has already been trained on example inputs and the correct actions to take to achieve some goal given those inputs, then that is all that is required.
However, training a network to the point where it could achieve this in an unconstrained environment ("letting it loose on the internet") is a tough challenge.
There are ways to train neural networks (and learning functions in general) so that they learn useful mappings between observations and actions that progress towards achieving goals. You can use genetic algorithms or other search techniques for instance, and the NEAT approach can be successful training controllers for agents in simple environments.
Reinforcement learning is another popular method that can also scale up to quite challenging control environments. It can cope with complex game environments such as Defense of the Ancients, Starcraft, Go. The purpose of demonstrating AI prowess on these complex games is in part showing progress towards a longer-term goal of optimal behaviour in the even more complex and open-ended real world.
State of the art agents are still quite a long way from general intelligent behaviour, but the problem of using neural networks in a system that learns how to act as an agent has much research and many examples available online.

"However, training a network to the point where it could achieve this in an unconstrained environment ("letting it loose on the internet") is a tough challenge." And if you let it learn based on what they see in unconstrained environments, you can easily get things like Microsoft's Tay turning into a Nazi after spending less than 24 hours on Twitter.

The short answer, i think, is that it cannot.
The AI system will only do, and it will only be good at the task that the programmer made it for. Of course you could have an AI that, for example, can trigger a prediction on the input with different models depending on some other variables, but that will still be based on what the programmer wrote, it will never be able to do or learn new unintended things. Like having the model.predict() for an image classification NN in a loop and only stop when it detects a dog and then use another model to predict the breed for example.
What you mentioned about "letting the AI lose on the network" usually is part of some concerns about AI that it could evolve, learn new actions and start acting on its own.
But those people unknowingly are actually talking about a general AI or strong AI, an AI system that could be as smart as a human so it could act in its own too. But as far is a know at least, we are not even close to creating such a system.
Hope I actually answered your question and didn't deviated too much from what you actually asked. Please tell me if so.

There have been plenty of examples of AIs doing unexpected things, but they are still constrained to the domain they are designed for. A Twitter bot may learn to be racist, but it will still only be able to post tweets; it won't be able to stab people.

You invoke it in a loop. Imagine a digital assistant responding to voice queries. It might look something like this:
for(;;) {
   var audio = RecordSomeAudio();
   var response = model.predict(audio);
   if(response.action == "SAYSOMETHING") {
      PlaySomeAudio(response.output);
   }
}

Note that the model gets invoked repeatedly and can decide in a given situation whether to respond or not. In a digital assistant context, part of the model would be to check for if the user raised a query (e.g. "Hey Google" etc.).


text:
What is a cascaded convolutional neural network?
For a project I am doing, I found the paper Face Alignment in Full Pose Range: A 3D Total Solution.
It is using a cascaded convolutional neural network, but I wasn't able to find the original paper explaining what that is.
In layman's terms and intuitively, how does a cascaded CNN work? What does it solve?

The paper you are citing is the paper that introduced the cascaded convolution neural network. In fact, in this paper, the authors say

To realize 3DDFA, we propose to combine two achievements in recent years, namely, Cascaded Regression and the Convolutional Neural Network (CNN). This combination requires the introduction of a new input feature which fulfills the "cascade manner" and "convolution manner" simultaneously (see Sec. 3.2) and a new cost function which can model the priority of 3DMM parameters (see Sec. 3.4)

where 3DDFA stands for 3D Dense Face Alignment, the framework proposed in this paper for face alignment, in which a dense 3D Morphable Model (3DMM) is fitted to the image via cascaded CNNs (the regressor), where the term dense refers to the number of points of the face that will be modeled. See figure 1 of this paper, which should provide some intuition behind the purpose of this framework.
In section 3 (page 3), they also say

In this section, we introduce how to combine Cascaded Regression and CNNs to realize 3DDFA. By applying a CNN as the regressor in Eqn. 1, Cascaded CNN can be formulated as:

\begin{align}
\mathbf{p}^{k+1} = \mathbf{p}^{k} + \text{Net}^{k}
(\text{Fea}(\mathbf{I}, \mathbf{p}^k)) \tag{1}\label{1}
\end{align}
where

$k$ is the iteration number
$\mathbf{p}$ is the regression objective
$\text{Net}$ is the CNN structure
$\text{Fea}$ contains the two constructed image features

Pose Adaptive Feature (PAF) (section 3.2.1)
Projected Normalized Coordinate Code (PNCC) (section 3.2.2)

$\mathbf{I}$ is the image

The expression cascaded CNN apparently refers to the fact that equation \ref{1} is used iteratively, so there will be multiple CNNs, one for each iteration $k$. In fact, in the paper, they say

Unlike existing CNN methods that apply different network structures for different fitting stages, 3DDFA employs a unified network structure across the cascade. In general, at iteration $k$ ($k = 0, 1, \dots, K$), given an initial parameter $\mathbf{p}^k$, we construct PNCC and PAF with $\mathbf{p}^k$ and train a two-stream CNN $\text{Net}^k$ to conduct fitting. The output features from two streams are merged to predict the parameter update $\Delta \mathbf{p}^k$
$$
\Delta \mathbf{p}^k = \text{Net}^k(\text{PAF}(\mathbf{p}^k, \mathbf{I}), \text{PNCC}(\mathbf{p}^k, \mathbf{I}))
$$
Afterwards, a better intermediate parameter $\mathbf{p}^{k+1} = \mathbf{p}^k + \Delta \mathbf{p}^k$ becomes the input of the next network $\text{Net}^k$ which has the same structure but different weights with $\text{Net}^k$.

In figure 2 of the paper (page 4), the structure of this two-stream CNN, $\text{Net}^k$, at iteration $k$, is shown.

I wasn't familiar with this cascade CNN. I only quickly read certain parts of the paper and I tried to provide some intuition behind it. Maybe later I will refine this answer to provide even more intuition.


text:
What is the neuron-level math behind backpropagation for a neural network?
I am quite new in the AI field. I am trying to create a neural network, in a language (Dart) where I couldn't find examples or premade libraries or tutorials. I've tried looking online for a strictly "vanilla" python implementation (without third-party libraries), but I couldn't find any.
I've found a single layer implementation, but it's done only with matrices and it's quite cryptic for a beginner.
I've understood the idea between the feed forwarding, a neuron calculates the sum of its inputs, adds a bias and activates it.
But I couldn't find anything a neuron-level explanation of the math behind backpropagation. (By neuron-level I think of the math down to the single neuron as a sequence of operations instead of multiple neurons treated as matrices).
What is the math behind it? Are there any resources to learn it that are suitable as a beginner?

backpropagation is a bit harder. You basically calculate a loss function between the values you predict and the labels, and then propagate it with some form of gradient descend to the weights in different layers. Also implementing it without matrices might be very inefficient. If you search online for "backpropagation simple implementation" or "backpropagation math step by step" or something else I'm sure you will find plenty of examples.

You can think of the matrix math as just a notation for the several individual multiplications, etc. not magic.

Here is an example with values which may help to explain what is happening. https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/

Check Michael Nielsen's book Neural Networks and Deep Learning. Math of backpropagation is discussed at length and he does not use any libraries - just Python.

you begin by getting the loss of the output then you propagate and apply gradients back through the layers using the chain rule, to do a few layers of this  statically isnt hard just verbose, to do so for arbitrary numbers of layers means your looking at implementing something close to autodiff

Backpropagation is actually a lot easier than it is made out to be - if you have a basic understanding of calculus and the chain rule, and the single multi-variable calculus rule that to combine 2 gradient vectors, you simply add them.
This is hands down the best walk through of back prop I've found on the internet. If you are still confused after that, feel free to ask me any further questions. Here is also a quick forward and backward pass example I made for a simple CNN (only a few layers though, and the gradient only goes back to channel 1 of filter 1)

The Stanford university video you liked (ep 4 and 6) are indeed the clearest and simplest explanation on the topic I've found online! Thanks again!


text:
Effect of batch size and number of GPUs on model accuracy
I have a data set that was split using a fixed random seed and I am going to use 80% of the data for training and the rest for validation.
Here are my GPU and batch size configurations

use 64 batch size with one GTX 1080Ti
use 128 batch size with two GTX 1080Ti
use 256 batch size with four GTX 1080Ti

All other hyper-parameters such as lr, opt, loss, etc., are fixed. Notice the linearity between the batch size and the number of GPUs.
Will I get the same accuracy for those three experiments? Why and why not?

You might find the following somewhat useful to understand the effect of batch size: https://deeplearning.ai/ai-notes/optimization/?utm_campaign=BlogAINotesOptimizationNovember122019&utm_content=105388620&utm_medium=social&utm_source=linkedin&hss_channel=lcp-18246783

This should make a difference, but how big is the difference heavily depends on your task. However, generally speaking, a smaller batch size will have a lower speed if counted in sample/minutes, but have a higher speed in batch/minutes. If the batch size is too small, the batch/minute will be very low and therefore decreasing training speed severely. However, a batch size too small (for example 1) will make the model hard to generalize and also slower to converge.
This slide (source) is a great demonstration of how batch size affects training.

As you can see from the diagram, when you have a small batch size, the route to convergence will be ragged and not direct. This is because the model may train on an outlier and have its performance decrease before fitting again. Of course, this is an edge case and you would never train a model with 1 batch size.
On the other hand, with a batch size too large, your model will take too long per iteration. With at least a decent batch size (like 16+) the number of iterations needed to train the model is similar, so a larger batch size is not going to help a lot. The performance is not going to vary a lot.
In your case, the accuracy will make a difference but only minimally. Whilst writing this answer, I have run a few tests on batch size effect on performance and time, and here are the results. (Results to be added for 1 batch size)
Batch size 256 Time required 98.50849771499634s : 0.9414
Batch size 128 Time required 108.53689193725586s : 0.9668
Batch size 64 Time required 129.92272853851318s : 0.9776
Batch size 32 Time required 162.13709354400635s : 0.9844
Batch size 16 Time required 224.82269191741943s : 0.9854
Batch size 8 Time required 351.2729814052582s : 0.9861
Batch size 4 Time required 514.2667407989502s : 0.9862
Batch size 2 Time required 829.1623721122742s : 0.9869

You can test out yourself in this Google Colab.

As you can see, the accuracy increases while the batch size decreases. This is because a higher batch size means it will be trained on fewer iterations. 2x batch size = half the iterations, so this is expected. The time required has risen exponentially, but the batch size of 32 or below doesn't seem to make a large difference in the time taken. The accuracy seems to be normal as half the iterations are trained with double the batch size.
In your case, I would actually recommend you stick with 64 batch size even for 4 GPU. In the case of multiple GPUs, the rule of thumb will be using at least 16 (or so) batch size per GPU, given that, if you are using 4 or 8 batch size, the GPU cannot be completely utilized to train the model.
For multiple GPU, there might be a slight difference due to precision error. Please, see here.
Conclusion
The batch size doesn't matter to performance too much, as long as you set a reasonable batch size (16+) and keep the iterations not epochs the same. However, training time will be affected. For multi-GPU, you should use the minimum batch size for each GPU that will utilize 100% of the GPU to train. 16 per GPU is quite good.

@ClementHui And each model saved from above three will have almost same inference accuracy, right?

They will have similar accuracy, give or take maybe 3%

You don't need a batch size that large probably. @voo_doo

No.  Different batch sizes mean different gradients (check stochastic gradient descent concept you will get how loss calculated) are calculated in each step, and thus the gradient descent will likely end up in different places in parameter space.
In addition, how this is actually parallelized might make a difference, including the order of operations and converting between FP precision.
Additional check resources:issue of multi gpus


text:
Is it possible to train a neural network with 3 inputs and 12 outputs?
The selection of experimental data includes a set of vectors of different dimensions. The input is a 3-dimensional vector, and the output is a 12-dimensional vector. The sample size is 120 pairs of input 3-dimensional and output 12-dimensional vectors.
Is it possible to train such a neural network (in MATLAB)? Which structure of the neural network is best suited for this?

Hello, could you provide an overview of the data and/or problem you're trying to solve? Without such information I can not give you a full answer.

Hello, I do direct kinematics of the robot. The input 3-dimensional vector consists of the rotation angles of the corresponding drive. The working body of the robot is a triangular platform. As the output, I record the coordinates of the vertices of the triangle in the base coordinate system (we get three 3-dimensional vectors = only 9 coordinates) and three angular velocities along the x, y and z axes. In total, a 12-dimensional output vector is obtained.

I need to approximate data using a neural network, i.e. train it when applying the appropriate 3-dimensional vector to produce the corresponding 12-dimensional vector. I have a training database.

Please see my adjusted starting communication.

There is nothing stopping you, you can setup Dense Neural Networks to have any size inputs or outputs (simple proof is to imagine a single layer NN with no activation is just a linear transform and given input dim $n$ and output dim $m$, it's just a matrix of $n$ x $m$, trivially this works though with any number of hidden layers)
The better question is should you?. In all honesty, it depends on the data that you have, but, usually, with only 120 examples you'll either overfit completely or do relatively well if the true solution is a very simple function, but, in general, in the common situations where that isn't the case I find myself more likely or not using Bayesian approaches, so I can actually consider confidence (with little data, this is really nice)

Are there any recommendations for choosing the number of layers and neurons in them, as well as activation functions?

@AndrewSol For the most part no, usually grid searches or previous empirical results are used.

@ mshlis - Re: Please see my adjusted starting message.

The perceptron convergence theorem states that any architecture will lead to a correlation between the data.
Yes, you can!


text:
How is back-propagation useful in neural networks?
I am reading about backpropagation and I wonder why I have to backpropagate.
For example, I would update the network by randomly choosing a weight to change, $w$. I would have $X$ and $y$. Then, I would choose $dw$, a random number from $-0.1$ to $0.1$, for example. Then, I would do two predictions of the neural network and get their losses with the original neural network and one with $w$ changed by $dw$ to get the respective losses $L_{\text{original}}$ and $L_{\text{updated}}$. $L_{\text{updated}} - L_{\text{original}}$ is $dL$. I would update $w$ by $\gamma \frac{d L}{dw}$, where $\gamma$ is the learning rate and $L$ is the loss.
This does not need a gradient backpropagation throughout the system, and must have somehow a disadvantage because no one uses it. What is this disadvantage?

Since there's already a good answer I'd like to point out that BP only requires one forward pass (per update) whereas this method requires 2.

You might find this interesting/related: https://arxiv.org/abs/1707.04585
Reversible residual layers are another approach at optimizing backpropagation. Google recently announced Reformer, a transformer model that somewhat leverages this: https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html

The method you propose is already known, its basically a numerical approximation to the gradient. It is not used to train neural networks because its well... an approximation. You still need to do two forward passes to get an approximation, which introduces noise and might make the training process fail.
Using backpropagation to compute the gradient is an exact solution, so why would you use an approximation if the exact computation is equally efficient?
Numeric approximations of the gradient only make sense if exact computation is not possible.

Isn't it much more computationally intensive to compute the gradient every single time in contrast to two forward propagation?

@Aphrodite No, the gradient computed by backpropagation is very efficient, and exact, your approximation is not really faster.


text:
What are the differences between TensorFlow and PyTorch?
What are the differences between TensorFlow and PyTorch, both in terms of performance and functionality?

TensorFlow was developed by Google and is based on Theano (Python library), while Facebook developed PyTorch using the Torch library. Both frames are useful and have a great community behind them. Both provide machine learning libraries to accomplish various tasks and do the job. TensorFlow is a powerful and deep learning tool with active visualization and debugging capabilities. TensorFlow also offers serialization benefits since the entire graphic is saved as a protocol buffer. It also has support for mobile platforms and offers a production-ready implementation. PyTorch, on the other hand, is still gaining momentum and attracting Python developers, since it is more Python friendly. In summary, TensorFlow is used to speed things up and create AI-related products, while research-oriented developers prefer PyTorch.


text:
In which cases is the categorical cross-entropy better than the mean squared error?
In my code, I usually use the mean squared error (MSE), but the TensorFlow tutorials always use the categorical cross-entropy (CCE). Is the CCE loss function better than MSE? Or is it better only in certain cases?

As a rule of thumb, mean squared error (MSE) is more appropriate for regression problems, that is, problems where the output is a numerical value (i.e. a floating-point number or, in general, a real number).  However, in principle, you can use the MSE for classification problems too (even though that may not be a good idea). MSE can be preceded by the sigmoid function, which outputs a number $p \in [0, 1]$, which can be interpreted as the probability of the input belonging to one of the classes, so the probability of the input belonging to the other class is $1 - p$.
Similarly, cross-entropy (CE) is mainly used for classification problems, that is, problems where the output can belong to one of a discrete set of classes. The CE loss function is usually separately implemented for binary and multi-class classification problems. In the first case, it is called the binary cross-entropy (BCE), and, in the second case, it is called categorical cross-entropy (CCE). The CE requires its inputs to be distributions, so the CCE is usually preceded by a softmax function (so that the resulting vector represents a probability distribution), while the BCE is usually preceded by a sigmoid.
See also Why is mean squared error the cross-entropy between the empirical distribution and a Gaussian model? for more details about the relationship between the MSE and the cross-entropy. In case you use TensorFlow (TF) or Keras, see also How to choose cross-entropy loss in TensorFlow?, which gives you some guidelines for how to choose the appropriate TF implementation of the cross-entropy function for your (classification) problem. See also Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions? and Does the cross-entropy cost make sense in the context of regression?.

tks, i have my thumb rule now, commonly, MSE for regression, CCE for classification

We sometimes see that binary cross-entropy (BCE) loss is used for regression problems.  This post is my opinion on using BCE for regression problems.
The figure below is the plots of BCE, $-t*\log(x) - (1-t)*\log(1-x)$, for several target values $t = 0.0, 0.1, ..., 0.5$. (The plots for $t>0.5$ are mirror images of those for $t<0.5$, so I omitted them.)
As you can see, when the target value $t$ is closer to the medium ($t=0.5$), BCE is flatter around its minimum ($x \sim t$). This means that BCE is less 'focal' when the target value is intermediate value.
So, BCE suits your purpose when the edge values ($t=0$ and $t=1$) are of special importance for you but the difference between intermediate values ($t=0.4$ and $t=0.5$, for example) is not very important for you.
On the other hand, when any value of target is equally important for you, then BCE will not be a good choice.  Another loss function, MSE for example, is better for you.
Note added:  If you use BCE for regression problems, it will be better to subtract $-x*\log(x) - (1-x)*\log(1-x)$ from the original BCE expression, so that the loss becomes zero when the prediction value coincides with the target one, $x=t$.  This will not matter for backpropagation, but will be convenient to monitor the value.
Note added:  After I submit this post, I came to think that we can tune how 'focal' the loss function is around its minimum, by simply multiplying an arbitrary factor of target values.  For example, we can tune BCE loss $L_{\rm BCE}(x,t)$ by making it $f(t)*L_{\rm BCE}(x,t)$ where $f(t)$ is whatever factor you want.  This factor tunes how focal the loss is around its minimum for each target value $t$.

In a classification problem it's better to get higher error and higher error slope when we predict the label wrong.
As you see in the graph by using cross-entropy you get high error when the algorithm predict a label wrong and small error when the prediacted label is close enough, so it helps us to separate the predicted classes better.

Zero is one and one is zero


text:
How can I determine the mathematical relation between the input and output variables?
I would like to take in some input values for $n$ variables, say $R$, $B$, and $G$. Let $Y$ denote the response variable of these $n$ inputs (in this example, we have $3$ inputs). Other than these, I would like to use a reference/target value to compare the results.
Now, suppose the relation between the inputs ($R$, $B$ and $G$) with the output $Y$ is (let's say):
$$Y = R + B + G$$
But the system/machine has no knowledge of this relation. It can only read its inputs, $R$, $B$ and $G$, and the output, $Y$. Also, the system is provided with the reference value, say, $\text{REF} = 30$ (suppose).
The aim of the machine is to find this relation between its inputs and output(s). For this, I have come across some quite useful material online like this forum query and Approximation by Superpositions of a Sigmoidal Function by G. Cybenko and felt that it were possible. Also, I doubt that Polynomial Regression may be helpful as suggested Here.
One vague approach that comes to my mind is to use a truth table like approach to somehow deduce the effect of the inputs on the output and hence, get a function for it. But neither am I sure how to proceed with it, nor do I trust its credibility.
Is there any alternative/already existing method to accomplish this?

I would like a direction for that. I know it is related to machine learning and artificial intelligence. Had I had the liberty to start from the very basic of ML, I would love it. But time bounds me.

@DuttaA Actually this isn't a standard ML problem. If $Y$ can be any arbitrary function of the inputs, ML is usually used to find a black box model to predict $Y$ and not care about the question "what is the actual function?" The OP chose an unfortunate example where the question "what is the function?" can be answered exactly using linear regression. But what if it was something like $Y = \sin(x)/x$? A neural net can approximate it well within a certain range. But how can we actually learn that $Y = \sin(x)/x$? This is not a standard ML task.

@Bridgeburners As you say that the problem quoted here can be solved using linear regression, you would be surprised to know that my immediate need is exactly  solving this exact equation.i.e. R,G,B and Y. But inevitably, I will need to solve more complicated equations like the Sine function etc.

I found a solution to this some time back. I have been studying function approximation (within linear regression) for some time.
Here's how I did it:
Neural Networks have been proved to be universal function approximators. So, even a single hidden layer would be sufficient to approximate a function simple as addition (Even somewhat complex functions like the Sine and any random CONTINUOUS wiggly function have been approximated)
First, I used a high level API like TensorFlow and Keras and implemented it here
The model was trained on the data (input-output pairs)
R    = np.array([-4, -10,  -2,  8, 5, 22,  3],  dtype=float)
B    = np.array([4, -10,  0,  0, 15, 5,  1],  dtype=float)
G    = np.array([0, 10,  5,  8, 1, 2,  38],  dtype=float)

Y    = np.array([0, -10, 3, 16, 21, 29, 42],  dtype=float)

And trained as follows:
#Create a hidden layer with 2 neurons
hidden = tf.keras.layers.Dense(units=2, input_shape=[3])

#Create the output (final) layer which symbolises value of **Y**
output = tf.keras.layers.Dense(units=1)

#Combine layers to form the neural network and compile it
model = tf.keras.Sequential([hidden, output])
model.compile(loss='mean_squared_error', optimizer=tf.keras.optimizers.Adam(0.1))
history = model.fit(RBG,Y, epochs=500, verbose=False)

The model converges in about 50 epochs

Also, I have implemented the same using only C/C++ and used GNU plot to visualize the results.

There are always a large number of possible functions that can produce a given set of input-output values. The challenge is to find a simplest function (according to whatever criteria you choose) to produce those values.
One approach is to write a general function of the input variables, comprising terms of all order in R, G, and B, with a coefficient for each term, then search for values of the coefficients that A) reproduce the known input-output values accurately and B) leave the largest number of coefficients equal to zero.
Several different algorithms can be used to do the search efficiently.  My choice would be a genetic algorithm to seek the minimum of the RMS difference between the produced and known I-O values, summed with the product of a gradually increasing parameter and the number of nonzero coefficients.

I have brushed over Genetic Algorithm and found it to be really novel for me and very useful. But, I need to deploy the trained model onto a MCU and hence it will be too bulky for the device.

As I see your explanation, what you're looking for can be stated via another way saying
Y = aX + b

where Y is output vector, X input vector and a & b are the coefficients you want to find.
Why so? And what is happening?
First thing first: I recommend to look the video [4] about how matrices and vectors work together and form after multiplication very familiar equations:
Y = a1 * x1 + a2 * x2 + a3 * x3

Now you see, you do not only get R + G + B but also some constants supplied for each of the variables.
About polynomial solutions I found [2] and [3] but reading through [3] you'll soon notice it is about completely different approach:
Y = a1 * x + a2 * x^2 + a3 * x^3

which you don't want.
So, you find something called linear regression and a so called Deep Neural Network for example to solve it [1].
I would summarize the source [1] in these steps:

You have to find some training data. That is examples of correct Y values, when X values are known.
Then you build in Python note book some code that has: a Neural Network with hidden layer(s), Activation Function, Back Propagation and Objective Function.
Many many iterations with the data, called Training.

There are plenty of samples and courses online about these in detail, but with correct tools and tutorials some dozen lines and no more are needed.
Process is ended with validation phase with some more known data items and results. It can tell you how well the estimated model works.

Final Notes:
As you may observe, the solution includes quite a many funny terms you have to learn somehow before mastering the task. For example Udemy has great online courses on this topic, also free tutorials are available on another sites. Your plans sound quite ambitious compared to the knowledge you have so far, so I really do recommend you learn little bit more to be able to fine tune the already given examples online. For example tutorial [5] includes one. It is at first quite complicated code, you'd need quite a lot practice to master it line by line.

In short:
Find your favorite tutorial, study neural networks a little bit (basics) and pick code sample to start experimenting. It is a long way but it is worth it.

Source:
[1] https://lightsapplications.wordpress.com/linear-regression-and-deep-learning/
[2] https://www.ritchieng.com/machine-learning-polynomial-regression/
[3] https://arachnoid.com/polysolve/
[4] https://www.youtube.com/watch?v=F2lJ7oSwcyY
[5] https://missinglink.ai/guides/neural-network-concepts/backpropagation-neural-networks-process-examples-code-minus-math/

Thank Your for the detailed reply. I have begun learning Neural Networks and have also found some papers related to my application. The links provided by you are awesome.


text:
Why does the bias need to be a vector in a neural network?
I am learning to use tensorflow.js. I am also using the tfvis library to print information about the neural net to the web browser. When I create a create a dense neural net with a layer with 5 neurons and another layer with 2 neurons, each layer has a bias vector of length 5 and 2 respectively. I checked the docs (https://js.tensorflow.org/api/0.6.1/#layers.dense), and it says that there is indeed a bias vector for each dense layer. Isn't a vector redundant? Doesn't each layer only need a single number for the bias? See the code below:
//Create tensorflow neural net
this.model = tf.sequential();

this.model.add(tf.layers.dense({units: 5, inputShape: [1]}))
this.model.add(tf.layers.dense({units: 2}))

const surface = { name: 'Layer Summary', tab: 'Model Inspection'};
tfvis.show.layer(surface, this.model.getLayer(undefined, 0))

In a simple feed-forward network, each artificial neuron has a separate bias value. This allows for greater flexibility for the output layer function than if each neuron had to use a single whole-layer bias. Although not an absolute requirement, without this arrangement it may become very hard to approximate some functions. Moving from a bias vector to a single scalar bias value per layer will most of the time reduce the effectiveness of a neural network through lost flexibility in how it fits to the target function.
Once you have $N$ output neurons in a layer leading to needing $N$ values for bias, then it is fairly straightforward to model this collection of bias values as a vector.
Often you will see a neural network layer function written in this form or similar:
$$\mathbf{y} = f(\mathbf{W}\mathbf{x} + \mathbf{b})$$
Where $f()$ is the activation function (applied element-wise), $\mathbf{W}$ the weights matrix for the layer and $\mathbf{b}$ is the bias. When written in this form, it is easy to see that $\mathbf{y}$, $\mathbf{W}\mathbf{x}$ and $\mathbf{b}$ must all be vectors of the same size.
This layer design has become so standard that it is possible to forget that other designs and implementations are possible for neural network parameters, and can sometimes be useful. Frameworks like TensorFlow also make it easier to take the standard approach, which is why you need a vector for bias on the example you are using. Whilst you are learning, and probably 99% of the time after that, it will be best to go with what the framework is doing here.

Makes sense, thanks! Not sure why I thought otherwise, in hindsight. Maybe one of the original tutorials I watched had it the more inflexible way.

To emphasize (and this is not emphasized in this answer), in the case of neural networks, the biases or, more precisely, the connections (or weights) between biases and other neurons are also learnable parameters, so the back-propagation algorithm calculates a gradient of the loss function that contains the partial derivatives with respect to the connections between the biases and other neurons too and, in the gradient descent step, these connections can also be updated.
Each neuron usually has its own bias. For example, in Keras, this is the case, as you can easily verify. However, in principle, you could also have a layer with a single scalar bias that is shared across all neurons of that layer, but this would probably have a different effect. The role of the bias is discussed in several places on the web. For example, in this Stack Overflow post or in this Stats SE post.


text:
What is the intuition behind TD($\lambda$)?
I'd like to better understand temporal-difference learning. In particular, I'm wondering if it is prudent to think about TD($\lambda$) as a type of "truncated" Monte Carlo learning?

TD($\lambda$) can be thought of as a combination of TD and MC learning, so as to avoid to choose one method or the other and to take advantage of both approaches.
More precisely, TD($\lambda$) is temporal-difference learning with a $\lambda$-return, which is defined as an average of all $n$-step returns, for all $n$, where an $n$-step return is the target used to update the estimate of the value function that contains $n$ future rewards (plus an estimate of the value function of the state $n$ steps in the future). For example, TD(0) (e.g. Q-learning is usually presented as a TD(0) method) uses a $1$-step return, that is, it uses one future reward (plus an estimate of the value of the next state) to compute the target. The letter $\lambda$ actually refers to a parameter used in this context to weigh the combination of TD and MC methods. There are actually two different perspectives of TD($\lambda$), the forward view and the backward view (eligibility traces).
The blog post Reinforcement Learning: Eligibility Traces and TD(lambda) gives a quite intuitive overview of TD($\lambda$), and, for more details, read the related chapter of the book Reinforcement Learning: An Introduction.

I am a novice in Reinforcement Learning and I have been struggling for several monthes about the TD()'s logic. Initially it seemed to me that it was a successfull purely heuristic formula without any theoretical foundation. But nowadays,  I understand it simply as a mean's calculation, using the recurrent formula that states that when you a have a mean and a new value arrives, it modifies the mean by an amount equal to its difference with it (the mean) divided by the new values number.
To summarize, the exposed mean calculation is an instance of a general formula of recurrent mean calculation that uses as increasing factor for the difference between the new value and the actual mean multiplied by any number between 0 and 1. By the way, this number - usually called  step size parameter - can be dynamic, and in the first paragraph (usual mean calculation) its amount is the inverse of the number of values considered in the mean's calculation.
Intuitively, we can understand that it is an accurate estimation procedure independently the initial (guessed or not) value. With a high number of estimates (new values arriving), the initial values fades its importance, and that it can be extended to treat many (lambda) new values simultaneously.
Until now I have not found this explanation nowhere, even if it is very simple, and I am not so sure that it is sound. I will appreciate if someone let me know if this intuition is correct and if it has already been exposed somewhere.

I would argue that maybe the word "incremental" could be better.
Instead of directly comparing the Monte Carlo and the TD($\lambda$), I would like to compare the $\lambda$-return and the TD($\lambda$) algorithm, since Monte Carlo is an extreme case of $\lambda$-return method.

The $\lambda$-return algorithm is a forward view algorithm, which means that it requires waiting up to $n$ steps after the current step to gather all the necessary information for updating the current state. This can be unsatisfactory because updates are not made in real time. An extreme case of this scenario is the Monte Carlo method, in which updates for all states are delayed until the end of an episode.

TD($\lambda$) is a backward view algorithm that can alleviate this issue. Unlike the $\lambda$-return, which gathers all information at once and then performs a "complete update" of the current state via n-step bootstrapping, TD($\lambda$) performs "incremental updates" through eligibility traces. As it encounters a new state, it uses the new information to update those previous states that need this piece of information. Since those previous states might need more information than this single piece, they are "incrementally updated" or "partially updated" by this new information. As the process proceeds, the information that a previous state needs to complete its update will eventually be gathered completely, allowing for a complete update of that state.

Chapter 12 - Eligibility Traces of Sutton and Barto's book provides more accurate information about this topic.


text:
Given the precision and recall of this model, what can I say about it?
The following table shows the precision and recall values I obtained for three object detection models.

The goal is to find the best object detection model for that particular data set.
I evaluate the first two models as the following.

Model 1 has a high recall and precision values. High precision relates to a low false-positive rate, and high recall relates to a low false-negative rate. High scores for both show that the model is returning accurate results.

Model 2 has high precision but low recall. This means it returns very few results, but most of its identified objects are correct.

How can I evaluate the third one?

The second model has the same precision, but worse recall, than model 1. Therefore we would rather have model 1 than model 2.
The third model has worse recall than model 1, and worse precision than model 1, therefore we would rather have model 1 than model 3.
Thus, model 1 is the best object detection model.

@ John Doucette: Oh! no this is not a home work problem. This is the analysis of my research output. I want to evaluate it in the similar manner as I did for the first and second. Thanks a lot.

@NilaniAlgiriyage The way you analyzed the other two doesn't seem to make a lot of sense. You can draw comparisons (as in my answer), or you can use domain-specific valuations (e.g. this translates to a certain expected cost savings in operation), but concepts like "accurate", "high", and "low" don't have objective meanings in analysis. High precision in spam detection might be 0.9999. High precision in stock market prediction might only be 75%.

Thanks. My actual problem is this : https://ai.stackexchange.com/questions/17264/accuracy-scores-in-a-deep-learning-project

@NilaniAlgiriyage I think without knowing what the best or worst possible performance is for problems of this kind, it's still not a good idea to make statements about "high" or "low" performance. What we can say definitively is that model 1 has higher precision and recall than the other two models, so it is clearly the best choice. Model 3 has higher recall than model 2, but lower precision than model 2. If you care a lot about recall for your application you would prefer 3 to 2. If you care more about precision, you would prefer 2 to 3.

Thanks a lot for helping me to think in a different way.


text:
How can I make the kernels non-learnable and set them manually?
I'm a newbie in Convolutional Neural Networks. I have found out that kernels in convolutional layers are usually learned while training.
Suppose I have a kernel that is very good to extract the features that I want to extract. In that case, I don't want the kernels to be learnable. So, how can I make the kernels non-learnable and set them manually?
Maybe, in that case, I have to use something different from a CNN.

In most modern neural network frameworks, the update rules for training can be selectively applied to some parameters and not others.
How to do that is dependent on the framework. Some will have the concept of "freezing" a layer, preventing parameters in it being updated. Keras does this for example. Others will do the opposite and expect you to provide a list of trainable parameters - these typically come with helpers that will list all parameters in a neural network, so you would need to add some kind of filter after collecting that data to exclude your pre-trained layer. PyTorch does this (although the linked example is slightly more complex in that it stops calculating gradients too).
If your framework of choice does not allow you to select and isolate layers in the training process, then you still have a couple of options:

You could store a copy of layer parameters that you want to keep and force your learning network to re-load these parameters after each mini-batch. This does depend on you having a method that can selectively set parameters.
If your pre-trained layers are the first ones, immediately next to the input, then instead of including them in your learning network model, you can pre-process all your training data with just the fixed layers (build a model using only those layers), save the output and use that as an alternative input for the learning layers (build a second model with only the learning layers). Later, once training is complete, you can build a combined neural network out of the fixed layers and the learning layers.


text:
Why are traditional ML models still used over deep neural networks?
I'm still on my first steps in the Data Science field. I played with some DL frameworks, like TensorFlow (pure) and Keras (on top) before, and know a little bit of some "classic machine learning" algorithms like decision trees, k-nearest neighbors, etc.
For example, image classification problems can be solved with deep learning, but some people also use the SVM.
Why are traditional ML models still used over neural networks, if neural networks seem to be superior to traditional ML models? Keras is rather simple to use, so why don't people just use deep neural networks with Keras? What are the pros and cons of each approach (considering the same problem)?

Why are still traditional machine learning (ML) models used over neural networks if neural networks seem to be superior to traditional ML models?

Of course, the model that achieves state-of-the-art performance depends on the problem, available datasets, etc., so a comprehensive comparison between traditional ML models and deep neural networks is not appropriate for this website, because it requires a lot of time and space. However, there are certain disadvantages of deep neural networks compared to traditional machine learning models, such as k-nearest neighbors, linear regression, logistic regression, naive Bayes, Gaussian processes, support vector machines, hidden Markov models and decision trees.

Often, traditional ML models are conceptually simpler (for example, k-NN or linear regression are much simpler than deep neural networks, such as LSTMs).
Personally, I've noticed that traditional ML models can be used more easily compared to deep neural networks, given the existence of libraries, like scikit-learn, which really have a simple and intuitive API (even though you apparently do not agree with this).
Deep neural networks usually require more data than traditional ML models in order not to overfit. Empirically, I've once observed that certain traditional ML models can achieve comparable performance to deep neural networks in the case of small training datasets.
Even though there's already a new and promising area of study called Bayesian deep learning, most deep neural networks do not really provide any uncertainty guarantees, they only provide you a point estimate. This is a big limitation, because, in areas like healthcare, uncertainty measures are required. In those cases, Gaussian processes may be more appropriate.

This question is very broad, so let me attempt to answer it using my own background in time series analysis.
As an example, why would I continue using ARIMA to forecast a time series? Why not simply use an LSTM model by default, since this is a type of recurrent neural network that takes time-related dependencies into account?
Well, an LSTM model is not good at modelling all time series. It is effective when it comes to modelling volatile data, but ARIMA still outperforms when it comes to forecasting trend data - LSTM tends to overemphasise volatile patterns in future predictions.
Let's take an example of forecasting weekly hotel cancellations by potential customers. The second time series shows much more variability in the number of weekly hotel cancellations than the first:
H1 Time Series

H2 Time Series

Based on MDA (mean directional accuracy), RMSE (root mean squared error), and MFE (mean forecast error) - ARIMA demonstrates superior performance overall for the first time series, while LSTM shows better performance for the second:

On the basis of this example - which is quite specific given the broadness of your question - deep learning techniques are not always used because simpler models can perform better under certain circumstances. It is all about understanding the data you are working with and then choosing the model - not the other way around.

SVM is generally considered objectively better than deep learning for standard machine learning tasks.
SVM or decision trees.
Deep Learning is beneficial when there is structure in the data that can't be easily represented by some type of kernel.
I'm actually really interested in why decision trees haven't been used for computer vision in conjunction with deep learning feature extraction.


text:
Why does PyTorch use a different formula for the cross-entropy?
In my understanding, the formula to calculate the cross-entropy is
$$
H(p,q) = - \sum p_i \log(q_i)
$$
But in PyTorch nn.CrossEntropyLoss is calculated using this formula:
$$
loss = -\log\left( \frac{\exp(x[class])}{\sum_j \exp(x_j)} \right)
$$
that I think it only addresses the $\log(q_i)$ part in the first formula.
Why does PyTorch use a different formula for the cross-entropy?

When you one-hot-encode your labels with $p_i \in \{0,1\}$ you get $p_i = 0$ iff $i$ is not correct and, equivalently, $p_i =1$ iff $i$ is correct.
Hence, $p_i \log(q_i) = 0 \log(q_i) = 0 $ for all classes except the "truth" and $p_i \log(q_i) = 1 \log(q_i) = \log(q_i) $ for the correct prediction.
Therefore, your loss reduces to:
$$
H(p,q) = - \sum p_i \log(q_i) = - \log(q_{truth})
$$

@DuttaA have a look at the last part of the question: "that I think it only addresses the $\log(q_i)$ part in the first formula. So is that means Pytorch using different Cross entropy formula?". Reads to me that it actually is exactly about this. .

OMG! yes, this is what I meant, I accidentally mix language modeling cross-entropy (that have continuous values of $p_i$) and PyTorch nn.CrossEntropyLoss. I forgot in classification $p_i$ will be 0 or 1 so the PyTorch function is valid, but of course, I can't use it directly for Language Modeling case. Thank You!

@Sammy This answer partially answers the question, but I think you should also explain the function inside the log (in the PyTorch version). What function is that? Note that I know what function that is, but, for completeness, I think you should also explain it. Maybe you should also refer to the documentation. Afterward, I will upvote your answer ;)