text:
How to obtain a formula for loss, when given an iterative update rule in gradient descent?
From the reinforcement learning book section 13.3:

Using pytorch, I need to calculate a loss, and then the gradient is calculated internally.
How to obtain the loss from equations which are stated in the form of an iterative update with respect to the gradient?
In this case:
$\theta \leftarrow \theta + \alpha\gamma^tG\nabla_{\theta}ln\pi(A_t|S_t,\theta)$
What would be the loss?
And in general, what would be the loss if the update rule were
$\theta \leftarrow \theta + \alpha C\nabla_{\theta}g(x|\theta)$
for some general (derivable) function $g$ parameterized by theta?

Why do you need to calculate any loss here? You need to compute the derivate with respect to $\theta$ of $\ln \pi (A_t \mid S_t, \theta)$. I suppose that $\pi (A_t \mid S_t, \theta)$ will be represented as a neural network. You're not performing supervised learning.

If I were to implement this with pytorch, I would need to calculate some kind of loss in order to update the policy $\pi$. I simply don't quite understand how to transform the iterative notation to a loss notation (which would then be fed into the optimizer)

Have you had a look at the examples here: https://github.com/pytorch/examples/tree/master/reinforcement_learning?

@nbro this is my only question here in which I haven't linked to one of the code examples there :). My question is more general, because I want to go on to the next algorithms in the book, and then onwards for example algorithm 13.5, for which there is no code in the linked pytorch example.

your "loss" would be $-ln(\pi(a\mid s, \theta))G\gamma^t$, I somewhat answered your question here. In this case $A$ that i wrote there would be $G\gamma^t$

The minus loss will be expectation of accumulated reward or accumulated discounted reward over parametrized policy. It's explained usually in "log derivative trick" chapters, for example here:  http://www.1-4-5.net/~dmm/ml/log_derivative_trick.pdf

You can find an implementation of the REINFORCE algorithm (as defined in your question) in PyTorch at the following URL: https://github.com/JamesChuanggg/pytorch-REINFORCE/. First of all, I would like to note that a policy can be represented or implemented as a neural network, where the input is the state (you are currently in) and the output is a "probability distribution over the actions you can take from that state received as input".
In the Python module https://github.com/JamesChuanggg/pytorch-REINFORCE/blob/master/reinforce_discrete.py, the policy is defined as a neural network with 2 linear layers, where the first linear layer is followed by a ReLU activation function, whereas the second is followed by a soft-max. In that same Python module, the author also defines another class called REINFORCE, which creates a Policy object (in the __init__ method) and defines it as property of that class. The class REINFORCE also defines two methods select_action and update_parameters. These two methods are called from the main.py module, where the main loop of the REINFORCE algorithm is implemented. In that same main loop, the author declares lists entropies, log_probs and rewards. Note that these lists are re-initialized at ever episode. A "log_prob" and an "entropy" is returned from the select_action method, whereas a "reward" is returned from the environment after having executed one environment step. The environment is provided by the OpenAI's Gym library. The lists entropies, log_probs and rewards are then used to update the parameters, i.e. they are used by the method update_parameters defined in the class REINFORCE.
Let's see better now what these methods, select_action and update_parameters, actually do.
select_action first calls the forward method of the class Policy, which returns the output of the forward pass of the NN (i.e. the output of the soft-max layer), so it returns the probabilities of selecting each of the available actions (from the state given as input). It then selects the probability associated with the first action (I guess, it picks the probabilities associated with the action with the highest probabilities), denoted by prob (in the source code). Essentially, what I've described so far regarding this select_action method is the computation of $\pi(A_t \mid S_t, \theta)$ (as shown in the pseudocode of your question). Afterwards, in the same method select_action, the author also computes the log of that probability I've just mentioned above (i.e. the one associated with the  action with the highest probability, i.e. the log of prob), denoted by log_prob. In that same method, the entropy (as defined in this answer) is calculated. In reality, the author calculates the entropy using only one distribution (instead of two): more specifically, the entropy is calculated as follows entropy = -(probs*probs.log()).sum(). In fact, the entropy loss function usually requires the ground-truth labels (as explained in the answer I linked you to above), but, in this case, we do not have ground-truth labels (given that we are performing RL and not supervised learning). Nonetheless, I can't really tell you why the entropy is calculated like this, in this case. Finally, the method select_action then return action[0], log_prob, entropy.
First of all, I would like to note that the method update_parameters is called only at the end of each episode (in the main.py module). In that same method, a variable called loss is first initialized to zero. In that method, we then iterate the list of rewards for the current episode. Inside that loop of the update_parameters method, the return, R is calculated. R is also multiplied by $\gamma$. On each time step, the loss is then calculated as follows
loss = loss - (log_probs[i]*(Variable(R).expand_as(log_probs[i])).cuda()).sum() - (0.0001*entropies[i].cuda()).sum()

The loss is calculated by subtracting the previous loss with
(log_probs[i]*(Variable(R).expand_as(log_probs[i])).cuda()).sum() - (0.0001*entropies[i].cuda()).sum()

where log_probs are the log probabilities calculated in the select_action method. log_probs is the part $\log \pi(A_t \mid S_t, \theta)$ of the update rule of your pseudocode. log_probs are then multiplied by the return R. We then sum the result of this multiplication over all elements of the vector. We then subtract this just obtained result by the entropies multiplied by 0.0001. I can't really tell you why the author decided to implement the loss in this way. I would need to think about it a little more.
The following article may also be useful: https://pytorch.org/docs/stable/distributions.html.

Thanks for all the details! However, my question was mostly theoretical. In most papers i've seen, the update rules are formulated like the pseudocode equations I linked in the question. I wanted to know how IN THE GENERAL CASE I would convert such equations to a loss function. Pseudocode would be enough for this purpose.
I'm sorry, but I still can't figure out the equation which loss=... came from.

@Gulzar If you have figured out more details behind my explanations and the linked source code, feel free to add them as an answer to your own question!

Sorry about that, i had to put that project aside for too many weeks and. I hope i will get back to it asap

@nbro the entropy function you describe is the information theory entropy function found here. The entropy function you refer to with two distributions is the Cross-entropy function, which essentially compares how similar two distributions are. The entropy function used here is the average rate at which information is produced by a stochastic source of data.

The only part of the loss function I can’t discern is why the author subtracts (0.0001*entropies[i]) at each step of calculating the loss.

Chiming in because I had the same question and stumbled across your post. It seems like the general version of your question still has not been answered.
In general, a well-formed gradient update rule is all you need to be able to train the network. We are thinking of converting to a "loss function" because that is the typical flow in the structure that pytorch provides for training networks via their autograd framework.
But the pytorch loss function is really just meant to be the last in a list of nested operations over which we are going to compute gradients. So if you already know your gradient update rule, the easiest way to write it down as a pytorch "loss function" is as the integral of your gradient update rule with respect to the network's output -- then pytorch will compute your gradient update rule for you automatically.
So in your example, the loss function would be:
$-\alpha^tG \ln\pi(A_t | S_t, \theta_t)$
as already noted by @Brale and others. The minus sign is there because pytorch optimizers will typically carry out minimization, but your gradient update rule is intended to maximize.
Hopefully this explanation is helpful to others that are still learning pytorch! Also, I would highly recommend that anyone just getting started with pytorch read their latest autograd tutorial here


text:
Does fp32 & fp64 performance of GPU affect deep learning model training?
I am purchasing Titan RTX GPU. Everything seems fine with that except float32 & float64 performance which seems lower vis-a-vis some of its counter parts. I wanted to understand if single precision and double precision performance of GPU affect deep learning training or efficiency ? We work mostly with images, however not limited to that.

Deep models are very tolerant to arithmetic underflow. You can hope for neglectable differences in prediction accuracy between FP32 and FP16 models. Check this paper for concrete results.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How do self-driving cars construct paths?
I wonder how self-driving cars determine the path to follow. Yes, there's GPS, but GPS can have hiccups and a precision larger than expected. Suppose the car is supposed to turn right at an intersection on the inner lane, how is the exact path determined? How does it determine the trajectory of the inner lane?

As you say, GPS is not precise enough for the purpose (until recently it was only accurate within 5m or so, since 2018 there are receivers that have an accuracy of about 30cm). Instead, autonomous vehicles have a multitude of sensors, mostly cameras and radar, which record the surrounding area and monitor the road ahead. Due to them being flat, mostly one colour, and often with lines or other markers on them, roads are usually fairly easy to spot, which is why most success has been made driving on roads as opposed to off-road. Once you know exactly where you are and where you want to go, computing the correct trajectory is then just a matter of maths and physics.
For an academic paper on the subject of trajectory planning see Local Trajectory Planning and Tracking of Autonomous Vehicles, Using Clothoid Tentacles Method.
It quickly becomes more complex when other road users and obstacles are taken into account; here machine learning is used to identify stationary and movable objects at high speed from the sensor input. Reacting to the input is a further problem, and one reason why there aren't any self-driving cars on the roads today.
This is all on driving automation level 2 and above; on the lower levels things are somewhat easier. For example, the latest model Nissan LEAF has an automatic parking mode, where the car self-steers, guided by camera images and sonar, but still requires the driver to indicate the final position of the vehicle. Apart from that, it is fully automatic.

When a car identifies a stationary object, how does it generate a new path to go around the object? Is it just simple geometry or is there some fancy machine learning behind it?

Have a look at https://www.archives-ouvertes.fr/hal-01139316/document

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
how to normalize the state space for articulated robot environments?
What is the common representation used for the state in articulated robot environments? My first guess is that it's a set of the angles of every joint. Is that correct? My question is motivated by the fact that one common trick that helps training neural nets in general is to normalize the inputs, like setting mean = 0 and std dev = 1, or scaling all the input values to $[0, 1]$, which could be easily done in this case too if all the inputs are angles in $[0, 2 \pi]$. But, what about distances? Is it common to, for example, use as input some distance of the agent to the ground, or a distance to some target position? In that case, the scale of the distances can be arbitrary and vary a lot. What are some common ways to deal with that?

and by articulated I mean robots with joints, as opposed to, say, a wheeled robot.

Welcome to SE:AI.  We sometimes have issues with overlap, since the general field of AI is so broad.  We do like robotics questions, so long as they involve the decision-making aspects!  :)

This paper might provide some answers https://arxiv.org/pdf/1810.05762.pdf
For the observations / states they used not only angles, but also velocities, heights and positions (Table 2).
In 4.2 Learning algorithm you can see that they mention this, which is related to your question about normalization:

Additionally, for stability we whiten the current observations by
  maintaining online statistics of mean and standard deviation from the
  history of past observations.


text:
How does backpropagation with unbounded activation functions such as ReLU work?
I am in the process of writing my own basic machine learning library in Python as an exercise to gain a good conceptual understanding. I have successfully implemented backpropagation for activation functions such as $\tanh$ and the sigmoid function. However, these are normalised in their outputs. A function like ReLU is unbounded so its outputs can blow up really fast. In my understanding, a classification layer, usually using the SoftMax function, is added at the end to squash the outputs between 0 and 1.
How does backpropagation work with this? Do I just treat the SoftMax function as another activation function and compute its gradient? If so, what is that gradient and how would I implement it? If not, how does the training process work? If possible, a pseudocode answer is preferred.

Backprop through ReLU is easier than backprop through sigmoid activations. For positive activations, you just pass through the input gradients as they were. For negative activations you just set the gradients to 0.
Regarding softmax, the easiest approach is to consider it a part of the negative log-likelihood loss. In other words, I am suggesting to directly derive gradients of that loss with respect to the softmax input. The result is very elegant and extremely easy to implement. Try to derive that yourself!

Thanks so much! I went through the calculus and I got the SoftMax minus 1, is that correct? Also for implementation, do I simply use this gradient in backpropagation and feed it back through the network like normal?

You are on the right path! Note however that the negative log-likelihood depends on correct labels, while there are no labels in your expression for the NLL-softmax gradient. We have a contradiction: a supervised learning which does not use supervision. Try to correct that!

You say "For positive activations, you just pass through the input gradients as they were. For negative activations you just set the gradients to 0.", but you don't explain why that must be the case, from a calculus perspective. I think this answer would improve if you explain that more in detail.

@nbro I assume that if OP is able to write a backprop for tanh, then he should know what is the derivative of ReLU.

@ssegvic How does the derivative of ReLU come from the derivative of tanh? It's just a convention that the derivative of ReLU at zero is 0, given that the left and right derivatives at zero are different. You don't explain this in your answer.

If you know how to differentiate tanh, then you know how to differentiate ReLU.
My answer is agnostic with repect to the derivative of ReLU in zero.


text:
Which AI algorithm to use to identify a subject from many unknown factors
Say I have a set of data generated by someone. It could be either bytes from a photo, or readings from bio-sensors, and I have a huge amount of said information, from many people or subjects. Which AI algorithms could be used to learn that a set of data belongs to a subject. I would have the information map that a huge set of data belongs to  Bob, and another belongs to Alice to train the system.

You have a set of different types of data available for each of your subjects, and given one set you'd like to classify which subject it belongs to. This looks like a supervised classificiation problem.
The most popular classifiers for supervised learning are neural networks. Now given the heterogenous nature of your data types, a simple approach would be to use separate classifiers for each type of data. For example, a convolution neural net for the image data, and a simple feed forward net for the biosensor data. Another thing you try is a multi channel approach, where towards the input side you have multiple channels for the different types of data, and the final few layers are fully connected.

The image has only CNNs for the multi channel part but you could have one channel as a simple feed forward net while another one as having conv layers.
Also, if you wish to classify the data as belonging to a subject on the basis of just one of the data types from the set, then you should have separate classifiers for all types. In that case it might be worthwhile to look into classifiers other than neural nets like multi class logistic regression which might be simpler to work with for a particular data type.

Thanks for your answer. It helps me look in the right category of algorithms.


text:
Does IBM Watson use machine learning?
I was reading an article on Medium and wanted to make it clear whether a bot created on IBM Watson is an intelligent one or unintelligent.

Simply put, there are 2 types of chatbots — unintelligent ones that act using predefined conversation flows (algorithms) written by developers building them and intelligent ones that use machine learning to interact with users.

I take exception to their use of intelligence, which, in AI terms, is a relative measure of problem solving strength.  If their meaning is that heuristic methods have less utility, that seems sound, but implying that ML is not algorithmic is wildly inaccurate.  Still in all, a good question about Watson!

Yes, it does and at many parts of the solution. For one of the core components - intent detection - Intento did a benchmark comparing IBM Watson and other similar products.
Outside of intent detection, there are other areas where AI techniques help - e.g. disambiguation, bootstrapping a bot from chat logs etc. Specifically for IBM Watson, you can learn more here.

A quick glance at the IBM Watson wikipedia page reveals that it does indeed use machine learning. Watson is a complex computing system that uses a variety of cutting edge techniques and concepts such as natural language processing and machine learning.


text:
transform Location data into int which will be used as input to ML model
I have features like state, city and location. Currently i am inserting this in respective tables in DB and tranform it using its primary key.
eg. country,state,city = IN,Maharastra,Pune = 2,5,10 (primary key in DB)
transforming it as 002005010
Is this approach correct? If not suggest me correct one

In order to accurately input location data into a machine learning model it really depends on what your goal is and what type of algorithm you are working with.  If you are working with a strictly numerical algorithm and your data seems to be spread far apart, it might be easier to convert your country-state-city location to a longitude, latitude feature where the exact value is the centroid location of the given city.  This kaggle post has a good writeup of how to run build features for geo-spatial data.
However, if you have only a small number of different locations per identifier (country, city, state) you could just represent them as seperate location classes and subclasses like you would for machine learning detection model.  You could think "vehicle" -> "bike" -> "road-bike" similarly to "country" -> "state" -> "city".  In this case, you would want to look at hierachical methods for machine learning and see some ways they represent their data.  Although I would say this method is more frowned upon for larger datasets, for smaller datasets it might be a better option.

The location data i have is spread far apart. So converting it to lat and lon will be better right?

If it's widely spread apart and you have a lot of different data points then it might be a good idea to start there if you're trying to learn some sort of a regression on your data.

I am trying to create a anomaly detection model (https://ai.stackexchange.com/q/11218/23174).


text:
How can we use linear programming to solve an MDP?
Apparently, we can solve an MDP (that is, we can find the optimal policy for a given MDP) using a linear programming formulation. What's the basic idea behind this approach? I think you should start by explaining the basic idea behind a linear programming formulation and which algorithms can be used to solve such constrained optimisation problems.

This question seems to be addressed directly in these slides.
The basic idea is:

Assume you have a complete model of the MDP (transitions, rewards, etc.).
For any given state, we have the assumption that the state's true value is reflected by:

$$V^*(s) = r + \gamma \max_{a \in A}\sum_{s' \in S} P(s' | s,a) \cdot V^*(s')$$
That is, the true value of the state is the reward we accrue for being in it, plus the expected future rewards of acting optimally from now until infinitely far into the future, discounted by the factor $\gamma$, which captures the idea that reward in the future is less good than reward now.

In Linear Programming, we find the minimum or maximum value of some function, subject to a set of constraints. We can do this efficiently if the function can take on continuous values, but the problem becomes NP-Hard if the values are discrete. You would usually do this using something like the Branch & Bound algorithm. These are widely available in fast implementations. GLPK is a decent free library. IBM's CPLEX is faster, but expensive.

We can represent the problem of finding the value of a given state as:
$$\text{minimize}_V \ V(s)$$
subject to the constraints:
$$V(s) \geq r + \gamma\sum_{s' \in S} P(s' | s,a)*V(s'),\; \forall a\in A, s \in S$$
It should be apparent that if we find the smallest value of $V(s)$ that matches this requirement, then that value would make exactly one of the constraints tight.

If you formulate your linear program by writing a program like the one above for every state and then minimize $\sum_{s\in S} V(s)$, subject to the union of all the constraints from all these sub-problems you have reduced the problem of learning a value function to solving the LP.


text:
Why don't we decorrelate transitions for policy-based data?
I'm implementing PPO myself strictly follow the steps:

sample transitions
randomly shuffle the sampled transitions
compute gradients and update networks using the sampled transitions
drop transitions and repeat the above steps

I observe a strange phenomenon that randomly shuffling transitions makes the algorithm perform significantly worse than keeping it as it is. This is very strange to me. To my best understanding, neural networks perform badly when the input data are correlated. To decorrelate transitions, algorithms like DQN introduce replay buffer and randomly sample from it. But this seems not the same story to policy-based methods. I'm wondering why policy-based methods do not require to decorrelate the input data?

We do decorrelate training experience, even for policy gradient methods. This is because decorrelation helps training data be more like IID data, which helps with the convergence of SGD-like optimizers.
The shuffling is done on line 151 of OpenAI's "baselines" implementation of PPO.
I'm going to guess that there's a bug somewhere in your implementation. If you're using the same truncated advantage estimation as in the PPO paper,
$$\begin{align}
&\hat{A}_t = \delta_t + (\gamma\lambda)\delta_{t+1}+\dots+(\gamma\lambda)^{T-t+1}\delta_{T-1}\\
&\text{where}\quad\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)
\end{align}$$
make sure you're not shuffling experience before computing your advantage estimates $\hat{A}_1,\dots,\hat{A}_T$. Each of these estimates is a function of several unbroken steps of experience. After you compute your advantage estimates, then create the tuples $(s_t, a_t, \hat{A}_t)$ and shuffle and sample from those. I think that's all the information you need  per time step for constructing their objective function.


text:
To what does the number of hidden layers in a neural network correspond?
In a neural network, the number of neurons in the hidden layer corresponds to the complexity of the model generated to map the inputs to output(s).  More neurons creates a more complex function (and thus the ability to model more nuanced decision barriers) than a hidden layer with less nodes.
But what of the hidden layers?  What do more hidden layers correspond to in terms of the model generated?

This is a very interesting question that you ask. I believe, this post and this post (written by me) well address your question. However, it deserves an explanation here.
1. Fully connected networks
The more layers you add, the more "nonlinear" your network becomes. For instance, in the case of two spirals problem, which requires a "highly nonlinear separation", the first known architecture to solve the problem was pretty advanced for that time: it had 3 hidden layers and also it had skip-connections (very early ResNet in 1988). Back then, computing was a way less powerful and training methods with momentum were not known. Nevertheless, thanks to the multilayer architecture, the problem was solved. Here, however, I was able to train a single-hidden layer network to solve the spirals problem using Adam.
2. Convolutional nets (CNNs)
An interesting partial case of neural networks are CNNs. They restrict the architecture of the first layers, known as convolutional layers, so that there is a much smaller number of trainable parameters due to the weights sharing. What we have learned from computer vision, moving towards the end of CNNs layers, their receptive fields become larger. That means that the subsequent CNN layers "see" more than their predecessors. Conceptually, first CNN layers can recognize simpler features such as edges and textures, whereas final CNN layers contain information about more abstract objects such as trees or faces.
3. Recurrent nets (RNNs)
RNNs are networks with layers which receive some of their outputs as inputs. Technically, a single recurrent layer is equivalent to an infinite (or at least large) number of ordinary layers. Thanks to that recurrence, RNNs retain an internal state (memory). Therefore, it is much more difficult to answer your question in the case of recurrent nets. What is known, due to their memory, RNNs are more like programs, and thus are in principle more complex than other neural networks. Please let me know if you find an answer to your question in the last case.
To conclude, the higher number of hidden layers may help to structure a neural network. Thanks to the recent developments such as ResNets and backpropagation through time, it is possible to train neural networks with a large number of hidden layers.

More hidden layers will just escalate the possibilities amount the neurons, including the solutions from the previous hidden layers. (I will edit this once I am at home and provide you with a good link I found some time ago)
Meanwhile maybe this will help you https://stats.stackexchange.com/questions/63152/what-does-the-hidden-layer-in-a-neural-network-compute

It was proven a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions (see Universal approximation theorem).
More layers can't improve something that can already do "everything". But adding more layers reduces the number of necessary neurons, and reduces computing power needed for the network as well.


text:
Can Deep Learning be applied to Computational Fluid Dynamics
Can Deep Learning be applied to Computational Fluid Dynamics (CFD) to develop turbulence models that are less computationally expensive compared to traditional CFD modeling?

Read the paper Deep learning in fluid dynamics (by J. Nathan Kutz), and you will find your answer.

Thank you Muhammad. This is exactly what I was looking for, especially the cited article by Ling et al (2016) - http://bit.ly/2UQlznV. The article seems to be a pivotal one applying Deep Neural Networks to a model Reynolds Averaged Navier-Stokes system.


text:
How agent's reasoning skills can improve its reinforcement learning?
Agent can have reasoning skills (prediction, taking calculated guesses, etc.) and those skills can help reinforcement learning of this agent. Of course, reinforcement learning itself can help to develop reasoning skills. Are there research that explores this impact of reasoning and consciousness on the effectivenes of reinforcement learning. Or maybe people just sit and wait such skills to emerge during reinforcement learning?

It sounds like you are describing a synthesis of two competing ways to solve the MDP problem.
In reinforcement learning, we solve the MDP problem by having the agent move around its environment, observe rewards and transitions in response to the actions it takes, and build a model of the relationship between actions and rewards that allows it to maximize rewards.
An older approach is to give the agent facts about the world that can be encoded as logical rules. The agent then uses unification to reason about the consequences of actions within that framework of rules. The agent then takes actions that maximize the rewards it can expect, given the rules and the information at hand. A problem with this approach was that it did not work well in problem domains with probabilistic rules (i.e. X usually happens when action Y is taken).
A hybrid approach, somewhere between these two is the use of Value Iteration or Policy Iteration methods. These are so-called "model-based" reinforcement learning algorithms (although, I would tend to say that makes them something other than true reinforcement learning...). Like the older logic-based approaches, they start by writing down a series of rules that fully describe how things happen in the world, and then derive the logical consequences of those rules to compute the best actions the agent could take. Like reinforcement learning however, they are able to account for probabilistic rules and probabilistic rewards. If you had an exact description of a game of chance, you could write it down as an MDP, and then solve it exactly using these techniques.
Importantly, value & policy iteration methods are not feasible if the state and action spaces are very large (as they often are), and are really not feasible if the MDP is not known exactly (i.e. if you don't know all the rules of the game in advance). That's where reinforcement learning shines.


text:
What does "Wide" vs. "Deep" mean in the context of Neural Networks?
From this article, I read that "to accurately classify data with neural networks, wide layers are sometimes necessary."
However, I have seen many implementations and discussions on deep-learning, such as this, mention the concept of depth.
What is the difference in the context of neural networks?  How does width vs depth impact a neural network's performance?

The width refers to the number of neurons in a layer. The depth refers to the number of layers.
Have a look at the following question regarding the impact of these hyper-parameters on the performance of the neural network: https://stats.stackexchange.com/q/214360/82135.


text:
Detecting abnormalities in x-rays while taking into account demographics of a patient -automated
This is my first post so please forgive me for any mistakes.
I am working on an object detection algorithm that can detect abnormalities in an x-ray. As a prototype, I will be using yolov3 (more about yolo here: 'https://pjreddie.com/darknet/yolo/')
However, one radiologist mentioned that in order to produce a good result you need to take into account the demographics of the patient.
In order to do that, my neural network must take into account both text an an image. Some suggestions have been made by other people for this question. For example, someone recommended taking the result of a convolution neural network and a seperate text neural network.
Here is an image for clarification:

Image Credits: This image (https://cdn-images-1.medium.com/max/1600/1*oiLg3C3-7Ocklg9_xubRRw.jpeg) from Christopher Bonnett's article (https://blog.insightdatascience.com/classifying-e-commerce-products-based-on-images-and-text-14b3f98f899e)
For more details, please refer to above-mentioned article. It has explained how e-commerce products can be classified into various category hierarchies using both image and text data.
However, a when convolution  neural network is mention it usssualy means it is used for classification instead of detection
https://www.quora.com/What-is-the-difference-between-detection-and-classification-in-computer-vision (Link for comparison between detection and classification)
In my case, when I am using yolov3, how would it work. Would I be using yolov3 output vector which would be like this format
class, center_x, center_y, width and height
My main question is how would the overall structure of my neural network be like if I have both image and text as input while using yolov3.
Thank you for taking the time to read this.

First of all you don't need "text" input as in Christopher Bonnett's blog. Your case is more easy - demographic is table data, which can be expressed as vector of numeric values. This data should be processed - pushed through one or two fully connected layers. The trick is where, to what part of yolo to concatenate results of processing of this vector. Because it's vector data it should concatenated to fully connected layer. Where exactly should be found by experiments, but as starting point it could be concatenated to before-last (before output) fully connected layer (I think for yolo it's 4096-size layer).
Overall I'd say that is not a trivial task. It require some experience with deep learning, good understanding of yolo design, yolo algorithm and a lot of experimentation, both with architecture and hyperparameters. Probably worth solid paper. Good luck.

So basically you are saying that for testing I should concatenate the processed vector with the vector produced by the second to last fully connected layer for yolo.

Essentially yes,  but as the first stage of experiment. Final version may differ

Do you know how I can access the Yolo structure or where I can learn to do this? Thank you.

Read Yolo paper and github code, for example: https://github.com/ayooshkathuria/pytorch-yolo-v3 ;You have to know python and pytorch or tensorflow

you need to take into account the demographics of the patient

How, exactly?

Is it a difference of, say, threshold? In this case you can do this serially (as @mirror2image mentions): process the image and then conclude by comparing the size of what you saw to, say, an age-dependent threshold.
Or has the whole processing to be different? In the extreme, you would not wait until the very end before asking whether the patient is a man if you are looking for prostate cancer.

To design the model, you need enough medical understanding to make such choices. The model can handle the parameters, but you have to choose the architecture.

Hi. This seems to be more of a comment than an answer to the question.

@PhilipRaeisghasem Perhaps, but one too lengthy to fit as a comment.

Thank you for the response, but I am confused by what you mean when you say ‘difference of threshold’. Also what is an age-dependent threshold.

You say the output would be: class, center_x, center_y, width and height. There may be a minimum size (threshold) for what you see to be meaningful, and this value may be different for an old man compared to a young woman: a different threshold.

So one suggestion (I think) you made is maybe that I merge the two vectors earlier than the place mirror2image suggested. I know this is just an example but I just want to make sure I understand what you are saying.

It depends on what makes sense: To design the model, you need enough medical understanding to make such choices.


text:
How to get good results with GAN and some thousands of images?
I'm trying to generate images at minimum of size 128 x 128 with a Generative Adversarial Network. I already tried a SAGAN pytorch implementation, but I'm not very happy with results. The images look cool but and I see some correct shape but without explanation you wouldn't know what the images are about. I have a dataset of 4000 images. Lightness, colors and shapes vary a lot, but they are similar in style and on what they portray.

With a Google Cloud V100 GPU the GAN would run a week to two with default parameters. Does this sound realistic time for this kind of dataset? It's definitely not feasible for me.
Is 4000 images enough to train a GAN from scratch?
Is there any implementation with pytorch/keras that would be good to get nice results with?

With a Google Cloud V100 GPU the GAN would run a week to two with
  default parameters. Does this sound realistic time for this kind of
  dataset? It's definitely not feasible for me.

Yes, V100s are quite beefy. You shouldn't even need a week. Obviously this is based on my experience with various problems, rather than a concrete calculation.

Is 4000 images enough to train a GAN from scratch?

For the size you want to generate, it is still on the edge of what would constitute a decent training set. You will get some results(depending on architecture) but will probably want to grab more data if at all possible.

Is there any implementation with pytorch/keras that would be good to
  get nice results with?

I would check out this link: https://github.com/eriklindernoren/Keras-GAN. It has some nice implementations in Keras. As far as the particular GAN to use, I would start out with a vanilla GAN that fits your purposes and focus on toying with hyperparameters, and if that doesn't work, look into one of the other variations that correlate well with your problem.


text:
DQN Q-values are static
I am working on a DDQN with 5 LSTM layers and 3 actions as output and state space of 21 features. I am dividing the dataset into episodes of 720 timesteps, for each episode the agent acts greedily for the first 480 steps without training, collecting a replay memory, and then update the parameters each step for the subsequent 240 steps using a window size (of 96 steps) randomly sampled from the replay memory (that always saves the last 480).
My problem is that so far the agent learned the optimal policy just once, and it looks like this
(on the test set, training is off) where, as you can see, the agent dynamically changes its evaluation of the state and acts greedily accordingly. All works fine and the performances are optimal, however, I have to slightly change the normalization of the database and rerun the training to get new parameters fitted to the new database.
Trying to get to the same result has been proven impossible so far, (even keeping all settings the same!) because most of the time the agent learns to keep its q-values static such as .
Note: this is an extract of the end of an episode and the beginnig of a new one, the noisy behaviour at the extreme is due to the model being trained for each step, in the middle the training is off and the agent acts greedily (as I explained above). The problem is that the learned parameters give rise to static q values that do not change while the state does and inevitably make the agent stuck in suboptimal strategy, as they never changes actions, even on longer sequences on the test set. The middle part of the second picture, where trainig is off, should look like the first picture, however, I am unable to get back to that optimal behaviour, even keeping all the parameters as they were.
Any idea on what can be the cause of this anomalous behaviour?

Are you initializing the agent in a different way each time? I.e a different initial state

@hisairnessag3 No, in this case everything is kept the same. Hyperparameters, normalization of the database, optimizer (Nadam). The resulting graphs are obtained after a few hours of training (10-15 at least). I simply cannot reach the same optimal policy in the first place, even keeping all constant, and even less if I   change the normalization of the data.

It seems your weight initialization is likely dynamic in some way then, if not an environment based issue

Oh, you meant the weight initialization. Yes, I actually initialize the LSTM layers with he_normal initialization (without keeping track of the seed unfortunately) and the final dense layer with normal distribution. Do you think that this stocasticity can be that detrimental in most cases? I thought that on the long run training would "flatten" any or most initial differences. If not, what do you suggest me to do?

It is likely converging to a far worse local optimum from which it can't recover, so yes I would guess if all else is the same, that is where the issue would be. I would first try to adjust hyperparameters from a static seed for weight initialization.  ε and α, are likely good places to start.
If that fails, adjust some architecture params like the number of units or layers in the network as well as the starting seed value.
You seemingly have run into a core problem at the heart of ML research which is reproducibility, which is often predicated by initialization.

Thanks for the suggestion. Let me recap if I got it right. 1- I do not care about reproducibility "per se" but I care in view of having more insights about the dynamics of learning (that I didnt saved), so that for the new runs with the different normalization method, I would have a comparison in terms of hours of training, evolution of q-values, loss etc. So, given the weights issue in order to reproduce the good experiment I would have to run a bunch of new simulations and, hopefully, get there again. But at this point shouldn't I rather start experimenting with the new normalization method?

2- About the hyperparameters: should I keep the seed constant and then compare different values of alpha (I dont need epsilon as I can get rewards for every actions at each step), or should I keep alpha (and the other hyperparameters) the same as in the good run and just change different seeds till I get a new optimal result?

Yea, you can do it either way. I would intuitively just adjust hyperparams as if initialization is an issue, it likely means your model is also not optimal to solve the problem. However, you could just as easily play around with seed values to also reconverge.

3- A bit off-topic: I normalized the database by dividing it into blocks of 96 steps and normalizing each block (independently) with its own Zscore. This is good for testing on another database but not viable for an online real application, where I get one element of the timeseries at the time, thus I have to change the normalization in a way that each element is normalized only according to its past data points. I am afraid that this might be detrimental to the performances as I would not be "leaking" any bit of information about the future. Do you think this can be a big issue?

For your 3rd point are you only doing this via experience replay or in the normal training loop?

I'm not sure what you mean. I divide the database into episodes and then I save an experience replay keeping track of the last 480 steps (inside each episode). Learning si done by updating the gradient each step from when the memory is full till the end of the episode (so 240 times per episode). For the training it samples a different window of 96 steps randomly from the memory of 480 each gradient update. So in the case of normalizing batches of 96, when it trains it can get the window at any point, not necessarily where the first element of the normalized block was. I hope it was clear.

Ah ok, so you aren't doing the 96 steps merge during environmental training. In that case there should be no issue. By that I mean, in TD learning for example, we bootstrap during an episode of training. However, it only seems like you are using that nuiance during replay(which is good).

I hope so, the merge is done before on the whole dataset and then I sample twice from the data, once for the episode and once for each gradient update. The agent should indeed get mixed enough data, thus not being able to understand when one block started and exploit this knowledge. Thanks a lot for your help, I will get back soon with, hopefully, good results!

Cool, go ahead and accept my answer if you deem it conclusive. Ill go ahead and answer anymore questions you have in the comments here when I see them


text:
Can this tic tac toe program be considered AI?
I coded a tic tac toe program, but I don't know if I can call it artificial intelligence.
Here's what I did.
There is a random player, which always makes random valid moves.
And then there is the AI player, which will receive input before every move, that input is the state of the board, and all the posibilities.
The AI, will try any move that it hasn't tried before. But if it knows every possibility, it will select the one that has the higher value.
This value is assigned by the outcome of the match, if the match was won ,+1, 0 for draw, -1 for lose.
Every move, will be stored in a database, or updated if it's known.
Eventually it will know every possible move.
I also added a threshold to compare the best moves, so it really select the best move. Example, two moves with a value of 100, the AI will keep trying them both, randomly until one has surpased the other by the threshold, say 50.
It takes about 20.000 games to make the AI perfect, it never loses a game, just draws and wins.
I'm new to AI, and I'm wondering, could this be really considered Artificial Intelligence?
And how is this different from a Neural Network approach? (I'm been reading about it, but I still don't quite get it.)

This is basically reinforcement learning. The state space contains your moves, and the value function are the value you store at the end. And your rewards are the end results. And you have episodic game. It is an AI method. Consider looking at value iteration, policy iteration, SARSA, Q-learning. The difference between neural network method and yours is you are not doing function approximation with neural network for the value function, you are doing tabular method.


text:
Why do DQNs use linear activations on cartpole?
I've been reading a lot of tutorials on DQNs for cartpole. In many of them, they have the funnel layer of the neural net be a linear activation.
Why is this?
Is it just a choice made by the implementer?
Is this Choice specific to cartpole, or do most control task dqns use it?
Thanks.

Q learning predicts the action value, $q(s, a)$ for taking action $a$ in state $s$. The action value is usually the discounted sum of all future rewards. In general it can take any scalar value.
DQN uses a neural network to approximate $q(s, a)$. Although you might use this to select an action (thus think of the problem as a classification), the NN has to perform regression to predict the action values.
It is most common to use a linear final layer, and mean squared error loss in DQN, to match this regression task. So yes, you will find most control DQNs will make the same choice as the cartpole example that you are looking at.


text:
Can I calculate the training performance of GPUs by comparing their specification?
I am currently using Nvidia GTX1050 with 640 CUDA cores and 2GB GDDR5 for Deep Neural Network training. I want to buy a new GPU for training, but I am not sure how much performance improvement I can get.
I wonder if there is a way to roughly calculate the training performance improvement by just comparing GPUs' specification?
Assuming all training parameters are the same. I wonder if I can roughly assume the training performance improvement is X times because the CUDA core number and memory size increased X times?
For example, Is RTX2070 with 2304 CUDA cores and 8GB GDDR6 roughly 4 times faster than GTX1050? And is RTX2080Ti with 4352 CUDA cores and 11GB GDDR6 roughly 7 times faster than GTX1050?
Thanks.


text:
When is bias values updated in back propagation?
I am new to deep learning. I have doubts on modifying bias values during back propagation. My doubts are

Does the back propagation algorithm modifies the weigh values and bias values in the same pass?
How does the algorithm decide whether it has to change the weight value or bias value to reduce the error in a pass?
Will the learning rate same for bias and weights?

Thanks!

Does the back propagation algorithm modifies the weigh values and bias values in the same pass?

Yes.

How does the algorithm decide whether it has to change the weight value or bias value to reduce the error in a pass?

It differentiates the loss function (like MSE) with respect to the weights and biases, that is, it finds the partial derivative of the loss function with respect to each of the parameters. You can think of the partial derivative of the loss function with respect to one of the parameters of the model as representing the "contribution" of that parameter to the loss of the model.
The partial derivatives of the loss function with respect to each of the parameters (weights and biases) is collectively called the gradient, which is thus a vector of $N$ partial derivatives, where $N$ is the number of parameters of the model.

Will the learning rate same for bias and weights?

It is usually the same, but, in theory, nobody prevents you from updating the biases using a different learning rate, so you could use a different learning rate to update the biases and weights.


text:
Why are not validation accuracy and loss as smooth as train accuracy and loss?
I am training a modified VGG16 network for classification (adding 0.5 dropout after each of the last FC layers). In the following plot I am training for a small number of epochs as an example, and it shows the accuracy and loss curves of training process on both training and validation datasets. My training set size is $1725$, and $429$ for validation. Also I am training with weights=None

My question is about the validation curves, why do not they appear to be as smooth as the training ones? Is this normal during the training stage?


text:
Is it possible to make a 'forked path' neural network?
I want to make a network, specifically a CNN for image recognition, that takes an input, processes it the same way for several layers, and then at some point splits before coming to two different outputs. Is it possible to create a network such as this? It would look something like this:
Input -> Conv -> Pool -> Conv -> Pool ---------> Dense -> Output 1
                                  ||

                                     ----> Dense -> Output 2

I.E. it splits off after the second pooling layer into separate fully connected layers. Of course, it has to train to both outputs, so that it is producing minimal error on both separate outputs using these common convolutional layers. Also, I am using Python Keras, and it would help if there was some way to do this using Keras in some way. Thank you!

Keras Functional APIs can help you define complex models.
You can find the documentation here :
https://keras.io/getting-started/functional-api-guide/.
For example:
# prev_layer is the layer you want to be forked
fork1 = Dense(32, activation='relu')(prev_layer)
fork2 = Dense(32, activation='relu')(prev_layer)

# you do some operations on fork1 to get output1
# and on fork2 to get output2

model = Model(inputs=input_layer, outputs=[output1, output2])


text:
How can alpha zero learn if the tree search stops and restarts before finishing a game?
I am trying to understand how alpha zero works, but there is one point that I have problems understanding, even after reading several different explanations. As I understand it (see for example https://applied-data.science/static/main/res/alpha_go_zero_cheat_sheet.png), alpha zero does not perform rollouts. So instead of finishing a game, it stops when it hits an unknown state, uses the neural network to compute probabilities for different actions as well as the value of this state ("probability of winning"), and then propagates the new value up the tree.
The reasoning is that this is much cheaper, since actually completing the game would take more time then just letting the neural network guess the value of a state.
However, this requires that the neural network is decent at predicting the value of a state. But in the beginning of training, obviously it will be bad at this. Moreover, since the monte carlo tree search stops as soon as it hits a new state, and the number of different game states is very large, it seems to me that the simulation will rarely manage to complete a game. And for sure, the neural network can not improve unless it actually completes a significant number of games, because that is only real feedback that tells the agent if it is doing good or bad moves.
What am I missing here?
The only plausible explanation I can come up with is:  If the neural network would be essentially random in the beginning, well then for sure the large number of game states would prevent the tree search from ever finishing if it restarts as soon as it hits a previously unknown game state, so this can not be the case. So perhaps, maybe even if the neural network is bad in the beginning, it will not be very "random", but still be quite biased towards some paths. This would mean that the search would be biased to some smaller set of states among the vast number of different game states, and thus it would tend to take the same path more than once and be able to complete some games and get feedback. Is this "resolution" correct?
One problem I have though with the above "resolution", is that according to the algorithm, it should favor exploration in the beginning, so it seems that in the beginning it will be biased towards choosing previously not taken actions. This makes it even more seem like the tree search will never be able to complete a game and thus the neural net would not learn.

You're right that AlphaGo Zero doesn't perform rollouts in its MCTS. It does complete many, many games, though.
Realize that AlphaGo Zero only iterates MCTS 1,600 times before taking an action. The next state resulting from that action becomes the root of future search trees. Since a typical game of Go only lasts a few hundred moves, the board will very quickly reach a terminal state.
None of this is dependent on the initial performance of the neural network. The neural net can be incredibly bad; actions/moves will still be taken at the same frequency. Of course, since AlphaGo Zero trains with self-play, one of its two selves in a game will usually be the winner (ties are possible). So the neural net will improve over time.
I recommend going over the "Self-Play Training Pipeline" section of the paper.

Sorry this is more of a comment than an answer. I'm wondering if you have found a definitive answer to your question, because I've a very related question.
I'm also confused by the AlphaZero algorithm - my explanation for my confusion is specified here: How does AlphaZero use its value and policy heads in conjunction?.
The thing is that I think the AlphaZero algorithm is also different from the Alphago Zero algorithm. A lot of sources that I've tried referred to really mix the two together.
In particular, there's this function in the official pseudocode, which really confused me:
def run_mcts(config: AlphaZeroConfig, game: Game, network: Network):
  root = Node(0)
  evaluate(root, game, network)
  add_exploration_noise(config, root)

  for _ in range(config.num_simulations):
    node = root
    scratch_game = game.clone()
    search_path = [node]

    while node.expanded():
      action, node = select_child(config, node)
      scratch_game.apply(action)
      search_path.append(node)

    value = evaluate(node, scratch_game, network)
    backpropagate(search_path, value, scratch_game.to_play())
  return select_action(config, game, root), root

def select_action(config: AlphaZeroConfig, game: Game, root: Node):
  visit_counts = [(child.visit_count, action)
                  for action, child in root.children.iteritems()]
  if len(game.history) < config.num_sampling_moves:
    _, action = softmax_sample(visit_counts)
  else:
    _, action = max(visit_counts)
  return action

Mainly because, if you look at the definition of expanded...
  def expanded(self):
    return len(self.children) > 0

I.e. when there are no more moves, and we are at the end of the game. I wonder what I'm missing here.


text:
How much the dialects recognition and speech recognition are relevant?
In this tutorial, they build a speech recognition model to classify a one-second audio clip as one of ten predefined words. Suppose that we modified this problem as the following: Given an Arabic dataset, we aim to build a dialects recognition model to classify a two-second audio clip as one of $n$ local dialects using ten predefined sentences. I.e. for each of these ten sentences, there are $x$ different phrases and idioms which refer to the same meaning$^*$. Now how can I take advantage of the mentioned tutorial to solve the modified problem?
$*$ The $x$ different phrases and idioms for each sentence are not predefined.

Is my modified problem similar to this one?

The tutorials you link are not much relevant, there are already existing implementations of your exact problem.
You can use https://github.com/swshon/dialectID_e2e, there are many other similar implementations on github.

Can you explain more how they are not relevant? Thank you.

You can use those things too, they are just not as related as the project I linked you.


text:
What are possible functions assigned on decision nodes for decision tree prediction?
In Decision Tree or Random Forest, each tree has a collection of decision nodes (in which each node has a threshold value) and a class labels (or regression values).
I know that threshold values are used for comparison with a corresponding feature value. As far as I know, the comparison is performed either "<", ">"  or "==" predicate.
Anything else for the functions taking threshold value and a feature value as inputs??

For a binary split, there are only three possible operations (or arguably only two if you consider one-hot encoding). Any other kind of split would simply not be binary. Almost every tree-based model is restricted to binary splits, due to the combinatorial explosion when considering ternary or even more complex splits.
Of course you could write your own algorithm that uses recursive non-binary splits. Beware though that you'll be facing the same difficulty that has led the vast majority of algorithms to be limited to strictly binary splits.
Have a look at this related question.


text:
Are successive actions independent?
The proof of the consistency of the per-decision importance sampling estimator assumes the independence of
$$\frac{\pi(A_t|S_t)}{b(A_t|S_t)}R_{t+1}\quad\text{ and }\quad \prod_{k=t+1}^{T-1}\frac{\pi(A_k|S_k)}{b(A_k|S_k)}.$$
See the proof of Theorem 1 in "Eligibility Traces for Off-Policy Policy Evaluation".
The result is also stated in Equation (5.14) of Sutton and Barto's RL book.
I'm guessing that this is itself a consequence of an assumption of independence between
$$\frac{\pi(A_t|S_t)}{b(A_t|S_t)}\quad\text{ and }\quad \frac{\pi(A_{t+1}|S_{t+1})}{b(A_{t+1}|S_{t+1})}.$$
I don't understand how this assumption can be justified. Consider the extreme case of a nearly deterministic policy $\pi$ and deterministic MDP dynamics. It would seem to me that the two values above are then surely not independent.
Am I missing something?

This is a consequence of the Markovian assumption, which underpins all of RL.
The Markovian assumption says that it doesn't matter how we reached a given state, only that we reached it, when deciding how likely it is that we move to subsequent states. This naturally implies that our choice of actions must also depend only on the current state.
You are correct that the assumption is somewhat unrealistic. However, it usually can be made to yield a reasonable approximation of the true dynamics without involving too many state variables. By solving this approximation, we hope to find a policy that also works well in the real problem.
Here's an example. In robot navigation, the real dynamics of the robot do depend on where the robot has been in the past, because as the robot's battery drains, the voltage levels it outputs will change slightly, and its wheels may become more prone to slippage. Thus, logically, our choice of action should change based on both the current state and our choice of actions in previous states (which drained the battery to a greater or lesser degree). However, if we try to incorporate this into the model, we'll end up having a combinatorial explosion in size of the dynamics function $P(S_{t+1} | S_t, a_{t}, S_{t-1}, a_{t-1} ... S_0, a_0)$ that actually captures this process (in particular, it will now be a function of 2t inputs). This, in turn, will necessitate a combinatorial blowup in the complexity of our policy (it will increase in complexity by an exponent of 2t). To keep things tractable, we accept that the dynamics will become lossy, or we can add some extra detail to the local state (e.g. a battery level) to capture the more complex dynamics in a Markovian way. Either way though, we'll be back to a world where future actions won't depend on past ones.
To be clearer, if we assume that $P(S_{t+1} | S_t) = P(S_{t+1} | S_{t}, S_{t-i})$ for any $i$, then the other relationships you mention should not seem surprising. That assumption is the Markovian assumption. It converts the state transition function into a Matrix that represents a Markov Chain. If we don't make that assumption, then most RL algorithms do not apply to our problem.


text:
Variational Autoencoder task for better feature extraction
I have a CNN with the regression task of a single scalar.
I was wondering if an additional task of reconstructing the image (used for learning visual concepts), seen in a DeepMind presentation with the loss and re-parametrization trick of Variational Autoencoder, might help the principal task of regression.
So you can imagine some convolutions with the role of feature extraction with some output X (let's say a vector of 256 values), that X goes into the VAE which computes Z and then the reconstructed image. And then the original regression task would take either X or Z in order to compute that scalar value.
Has anyone tried such an approach, is it worth the work?
Thank you


text:
Maximum Q value for new state in Q-Learning never exists
I'm working on implementing a Q-Learning algorithm for a 2 player board game.

I encountered what I think may be a problem. When it comes time to update the Q value with the Bellman equation (above), the last part states that for the maximum expected reward, one must find the highest q value in the new state reached, s', after making action a.
However, it seems like the I never have q values for state s'. I suspect s' can only be reached from P2 making a move. It may be impossible for this state to be reached as a result of an action from P1. Therefore, the board state s' is never evaluated by P2, thus its Q values are never being computed.
I will try to paint a picture of what I mean. Assume P1 is a random player, and P2 is the learning agent.

P1 makes a random move, resulting in state s.
P2 evaluates board s, finds the best action and takes it, resulting in state s'. In the process of updating the Q value for the pair (s,a), it finds maxQ'(s', a) = 0, since the state hasn't been encountered yet.
From s', P1 again makes a random move.

As you can see, state s' is never encountered by P2, since it is a board state that appears only as a result of P2 making a move. Thus the last part of the equation will always result in 0 - current Q value.
Am I seeing this correctly? Does this affect the learning process? Any input would be appreciated.
Thanks.

Your problem is with how you have defined $s'$.
The next state for an agent is not the state that the agent's action immediately puts the environment into. It is the state when it next takes an action. For some, more passive, environments, these are the same things. But for many environments they are not. For instance, a robot that is navigating a maze may take an action to move forward. The next state does not happen immediately at the point that it starts to take the action (when it would still be in the same position), but at a later time, after the action has been processed by the environment for a while (and the robot is in a new position), and the robot is ready to take another action.
So in your 2-player game example using regular Q learning, the next state $s'$ for P2 is not the state immediately after P2's move, but the state after P1 has also played its move in reaction. From P2's perspective, P1 is part of the environment and the situation is no different to having a stochastic environment.
Once you take this perspective on what $s'$ is, then Q learning will work as normal.
However, you should note that optimal behaviour against a specific opponent - such as a random opponent - is not the same as optimal play in a game. There are other ways to apply Reinforcement Learning ideas to 2-player games. Some of them can use the same approach as above - e.g. train two agents, one for P1 and one for P2, with each treating the other as if it were part of the environment. Others use different ways of reversing the view of the agent so that it can play versus itself more directly - in those cases you can treat each player's immediate output as $s'$, but you need to modify the learning agent. A simple modification to Q learning is to alternate between taking $\text{max}_{a'}$ and $\text{min}_{a'}$ depending on whose turn you are evaluating (and assuming P1's goal is to maximise their score while P2's goal is to minimise P1's score - and by extension maximise their own score in any zero-sum game)

Thanks so much for the detailed answer. So if I understand what you are saying correctly, the q table should only be updated after P1 has had a chance to reply with a move?
I think I get it. The state right after P2 takes an action is only used to find the immediate reward but plays no part whatsoever in finding the future expected reward. In order to get this future reward, one must wait until the p1 agent responds to the move, and consider this state as the one that yields the future reward.
You will probably find this facetious, but I'd like to make sure this makes sense.

P1 is in state s1 and makes move a1 resulting in s2. P2 evaluates s2 and responds with an action that leads to state s3 (but doesn't update the q-table yet). From here P1 makes another move, resulting in state s4.
It is at this point that P2 updates the q table using the Bellman equation. In this equation, the immediate reward is the reward from state s3 (the one that resulted from P2 making a move) and the future reward is the reward that resulted from P1 responding to s3, i.e., the reward for s4.
Is this correct?
Many thanks!

Your first comment is mostly correct. However, the immediate reward is assessed at same time as $s'$ and you should think of it the same say as $s'$ here - it is returned by the environment. There is a minor complication that the game may terminate on either P1 or P2's turn.

From the second comment, you should be careful about your labelling. The time steps don't occur like that as far as the agent is concerned. However, in your terms, you update the estimate for $Q(s_1, a_1)$ when P1 receives control of the game back (when receiving the immediate reward and next state) i.e. once it knows that the next state is $s_3$. The agent should look ahead using its existing  Q table about what the best action is. That is not the same as "the reward for $s_4$" - it is the expected discounted return for $s_3, a_3$.

If P1 (the random AI) makes action a1 resulting in state s1 and P2 (learning agent) makes action a2 onto state s1, resulting in state s2, wouldn't I be updating the estimate for Q(s1, a2), since I am estimating how good P2's move (a2) is given the state reached by P1 (s1) using the reward gained s2 as the immediate reward?

@Pete: Yes I missed that you started with the non-learning agent. The numbering scheme you are using for turns is not helping.

Yes, sorry about that, it is definitely confusing. Okay, I just wanted to make sure. And just to clarify, when it is P2's first move, the Q Table cannot be updated, since it has not received P1's reply to the move and thus cannot calculate the expected discounted reward. I.e. the q table should only be updated starting from P2's second move. Correct?

@Pete: Correct, until P2 knows $s, a, r, s'$ then it cannot calculate $NewQ(s,a)$. Technically this update can be made just before P2 makes its second move (because in Q learning you don't care what $a'$ is or the actual results of making that second move, you use the existing estimates), but P2 must be ready to make such a move.

Right. Thanks for being patient and expansive in your answers. You helped me a ton!


text:
Is it mostly the case to train with available models
I quite often find projects using pre-trained model and using them as a starting point for their new model that learns something novel from thier dataset or on-live learning process - e.g. using a webcam or live audio.
Is this quite usual and recommended to speed up training a model? For example using a model trained on ImageNet as a first layer to your model that will categorise faces specifically.

Yes, it is recommended to start with pre-trained model if you don't have high-end hardware. You can use a pre-trained model for fine-tuning (their trained weight as your initial weight) or use it as feature extractor (you remove few last layers, and then train it).
Why we need a pre-trained network?

Because training a good deep model takes a lot of time and needs a lot of hardware. Some good models, like DenseNet, ResNet, even VGG16 need days of training. You can read from VGG original paper:

On a system equipped with four NVIDIA Titan Black GPUs, training a single net took 2–3 weeks depending on the architecture.

Sometimes our case is similar to the dataset of pre-trained models out there. For example, if we need to classify some images of flower type, even our case has different class, we can use some first Convolutional layers of pre-trained model and use it as a feature extractor.

You can read more about transfer learning from this paper or this page.

Extremely useful thank you. So I guess part of the challenge is knowing what models are available and would suit your domain?

that's true, you can't just pick model randomly and then use it in your case. There is a simple rule to choose the right pre-trained model, if I remember correctly it's available on the page I mentioned


text:
Binary vector expected value
Raul Rojas' Neural Networks A Systematic Introduction, section 8.2.1 calculates the variance of the output of a hidden neuron.
Raul Rojas says that "for binary vectors we have $E[x_i^2] = \frac{1}{3}$" where $x_i$ is the input value transported through each edge to a node.
I don't quite get how he reaches this result.
Thank you for your time :)

The integral from 0 to 1 of x^2dx is 1/3, but isn't this related to a continuous random variable, instead of a binary one?

Some lines above the author says

By the law of large numbers we can also assume that the total input to the node has a Gaussian distribution

hence we can assume $X \sim \mathcal{N}(0,1)$ with the $X$ domain being continuous
Then he says the input vector is assumed to be binary which changes the domain from continuous to discrete so we can discretize it assuming $-1 \le X \le 1$ is mapped into zero and $X< -1 $ and $X > 1$ are mapped to 1
Finally according to the 68-95-99.7 Rule we can compute
$$ E(X^2) = P(X=1) \cdot 1^2 + P(X=0) \cdot 0^2 = 0.32 $$
Finally probably the author rounds this up to $\frac{1}{3} \simeq 0.33$


text:
Alphazero policy head loss not decreasing
I am now working on training an alphazero player for a board game. The implementation of board game is mine, MCTS for alphazero was taken elsewhere. Due to complexity of the game, it takes a much longer time to self-play than to train.
As you know, alphazero has 2 heads: value and policy. In my loss logging I see that with time, the value loss is decreasing pretty significantly. However, the policy loss only demonstrates fluctuation around its initial values.
Maybe someone here has run into similar problems? I would like to know if its my implementation problem (but then the value loss is decreasing) or just a matter of not enough data.
Also, perhaps importantly, the game has ~17k theoretically possible moves, but only 80 at max are legal at any single state (think chess - a lot of possibles but very few are actually legal at any given time). Also, if MCTS has 20 simulations, then the improved probabilities vector (against which we train our policy loss) will have at most 20 non-zero entries. My idea was that it might be hard for the network to learn such sparse vectors.
Thank you for any ideas!

Loss is tricky in RL. Loss non-decrease could be caused by increased rate of exploration, and that would be good thing. AlphaZero class of algorithms converging slow anyway and it seems you have huge action space, which is not helping. May take a lot of time/oscillations until you will see improvement.

Just out of interest, what game are you training it on.  (Also, any update on how it went:)

The loss of the policy head here is really quite different from losses in, for instance, more "conventional" Supervised Learning approaches (where we typically expect/hope to see a relatively steady decrease in loss function).
In this AlphaZero setup, the target that we're updating the policy head towards is itself changing during the training process. When we improve our policy, we expect the MCTS "expert" to also be improved, which may lead to a different distribution of MCTS visit counts, which in turn may lead to a different update target for the policy head from previous update targets. So it's perfectly fine if our "loss" increases sometimes, we may still actually be performing better. The loss is useful for the computation of our gradient, but otherwise it doesn't have much use -- it certainly cannot be used as an accurate indicator of performance / learning progress.

but only 80 at max are legal at any single state (think chess - a lot of possibles but very few are actually legal at any given time). Also, if MCTS has 20 simulations, then the improved probabilities vector (against which we train our policy loss) will have at most 20 non-zero entries.

This can be a problem yes. The fact that the majority of moves are not legal at any point in time is not a problem, but if you only have 20 MCTS simulations for a branching factor of 80... that's certainly a problem. The easiest fix would be to simply keep MCTS running for longer, but obviously that's going to take more computation time. If you cannot afford to do this for every turn of self-play, you could try:

using only a low MCTS iteration count for some moves, not adding these distributions to the training data for the policy head
using a larger MCTS iteration count for some other moves, and only using the distributions of these moves as training data for the policy head

This idea is described in more detail in Subsection 6.1 of Accelerating Self-Play Learning in Go.


text:
Does reinforcing correct predictions increase model accuracy further?
Let's say I've trained a CNN that is predicting/inferring live samples that it hasn't seen before. In the event the network makes a correct prediction, would including this as a new sample in its training set increase the model accuracy even further when re-training the network?
I'm unsure about this since it seems as though the network has already learnt the necessary features for making the correct prediction, so adding it as a new training sample might be redundant. On the other hand it might also reinforce to the network that it's on the right track, perhaps giving it further confidence to generalize with whatever features its learnt in regards to that class, that it might be able to apply to the same class in other images it might otherwise make an incorrect prediction with?
The reason I'm thinking of this is that manually labeling each image is a time-consuming process, however if a simple "Correct/Incorrect" popup box was presented after the network made a live prediction, then it's simply a matter of clicking a single button to generate a new labelled training sample, which would be a far easier labeling task.
So how useful would it be to do something like this?

You ask at the top and in the title whether adding correct samples only would help, but your plan includes collecting data for both true and false samples. Is there a reason why you are asking about focusing only on correct samples? Is this a multi-class classifier and you don't want to bother users with assigning correct label just yes/no?

yes thats correct, but Im the only user. I just want to label the model as quickly as possible since its the bottleneck for me rather than getting the data. ie: if I could collect 1000 images easily that my model can predict with 90% accuracy, its much faster for me to click yes/no on each of them and get 900 correctly labelled images, than manually labelling them all. The thing is if I do that, will my accuracy increase past 90%? Ie: after doing all that, will the "no" images have a higher chance of being correct after re-training?

Your suggestion is risky. It might make improvements to your classifier, but it may also reduce generalisation.
The two conflicting factors in play are:

Adding data points to a model which has capacity to learn more detail can improve its performance.
Training from a different distribution of data points than your target population can reduce its performance.

From the first point, you need your model to be able to accept the new data. This can be harder to achieve than you might think at first - if you have tuned some regularisation parameters by using cross-validation, then you may have at least in part saturated the model's capacity in order to prevent over-fitting. That means fitting to new data could require re-tuning your hyper-parameters as well - but you probably won't need to start from scratch, just search nearby.
It is hard to tell the impact from the second point. There is not going to be a general answer here, too much depends on how the data is arranged for your problem. My gut feeling is that you will notice an improvement to the model initially, but that there will be diminishing returns due to the self-selection of already-correct data points.
You should definitely keep back a cross-validation set of data distributed as it is in production, that has been properly and fully labelled in order to assess this work. This will be your way to assess whether generalisation is improving using your approach.
Worst case: You may need to go back and re-label the mistakes in order to significantly improve performance of your model. So I would suggest part of your auto-labelling process should store the misclassified items somewhere so you can re-visit them and spend the extra effort.
Sadly, even collecting more properly labelled data is not a guaranteed fix - some models already have enough data. You can check whether that might applies in your case by training with different sizes of training set (from the available training data you have so far), and seeing what the trend in performance is when you increase the training set size. In fact, you should probably do this first, before you invest significant effort in collecting more data.

Good answer and thank you - I completely missed the fact that removing some samples conditionally would of course change the distribution.A question though - would this be a bad thing in the reverse case? Ie: If I specifically sought out to find incorrect predictions and include those in the training set? This would still be a different distribution I'd imagine but would your answer still apply in this scenario?

@user4779: Yes my answer would still apply. There would be a difference in behaviour - what the NN got better at, and worse at due to that selection. I have tried something similar in the past - re-training progressively by selecting error cases - and in my case it performed very badly. Of course, as always, actual results depend on your problem.

Thanks again Neil. One final question if you have the time. If it performs badly on some samples, would the correct strategy be to find something about the error samples that might be unique, and then manually capture more training samples that are similar to those error cases, labeling and including all of them regardless of whether the network predicts them correctly or not?

@user4779: Again it will depend. With that description, you are heading towards error analysis, which usually requires some domain knowledge. E.g. http://mlwiki.org/index.php/Error_Analysis


text:
Picking a random move in exploitation in Q-Learning
I've been unsure about a principle of Q-Learning, I was hoping someone could clear it up.
When a new state is encountered, and thus there are no existing Q values, and that the algorithm decides to exploit, and not explore, how is the move chosen, since all the values are 0?
Is it chosen randomly? This intuitively would make sense, since after this, the state-move pair would have a value  and thus the matrix would get filled up throughout the iterations. But I just want to make sure I understand this correctly...
Thanks

It depends on the implementation of the software package that you are using. If you call a function that returns the maximum value and all values are the same then it might return the value at first index or some other one. The point is it doesn't matter which action is chosen since all of them are the best at the same time. So it's basically random but you should treat it as if you are trying to pick the best action.

That's precisely the problem I'm having. Switching from an implementation where an random Q value is selected to one where the code selects the first one when they are all 0 drastically alters my agent's performance. Racking my brain trying to understand where this could come from...

this is such a small thing to consider it's basically irrelevant since it only happens the first time you enter the state, it shouldn't affect the final performance. There must be a deeper reason why your performance differs so much.

Yes I think there a few issues here. The main one is the size of the state space ($10^{12}$), although the number of states that are likely to come about is much smaller. Another issue that the agent is trained against a Minimax AI, which always follows the same path, and although it is able to win against it most times, a human player is not guaranteed to play the way a Minimax algorithm responds to changes in state. Perhaps unsuitable hyperparameters? Not sure... Thanks for the input though.

Oh, so you are training on minimax and testing against human ? If that's the case it is possible that the agent overfits minimax player since MM player will usually choose the same move in the same position. When you play against human it is possible that agent visits states never met before (since there are so many states). If that happens behaviour is undefined since the agent didn't learn anything about those states before, you might get lucky or not. Have you considered "pretraining" it on a random agent ? Also $10^{12}$ states is quite a lot for tabular Q-learning to solve.

Precisely. Perhaps I should not be testing against human player. I have tried training the agent against a random player, with around 500 000 games, reaching a win rate of about 70% against the random player. However, winning against a human player seems to be the real challenge. Perhaps I will limit my thesis to studying the correlations between MM agent and learning agent without taking into account human players. Since the result vs AI and random are quite good. The state space is indeed quite big. I used a different data structure, a dictionary with dynamically generated entries though.


text:
What are the various methods for speeding up neural network for inference?
One way to speed up a neural network is to prune the network and reducing number of neurons in each layer. What are the other methods to speed up inference?

These are some ways to speed up inference:

Reduction of float precision: This is done post-training. According to work in this segment, a very little accuracy is sacrificed for a huge benefit in memory usage reduction. Also it speeds up inference. float32 -> float8.
Reference paper: https://arxiv.org/pdf/1502.02551
Using ReLU or similar small compute power activations: Benefits are obvious when you don't need to compute heavy exponents, like in tanh or sigmoid.
Binary Neural Architecture: This is new. This taps upon the ability to use binary valued weights and activations (1 bit) compared to 32 bit float counterparts. Estimation and learning is done through POPCNT and XNOR operations (for matrix products) and STE (straight through estimator) for backpropagation. You do have to sacrifice for a large number of neurons to learn the same features, but, the speed is on average 7x faster. Reference paper: https://pjreddie.com/media/files/papers/xnor.pdf

https://software.intel.com/en-us/articles/binary-neural-networks

Hardware standpoint: Using specialised hardware to compute matrix products, pretty standard examples are GPUs and TPUs.

Yeah, but they do seem promising


text:
Is it possible for a NN to reach the same results as CNNs?
Can a normal neural network work as good as a convolutional network? If yes, how much more time and neurons would it need compared to a CNN?

NNs won't be able to reach the performance of CNNs, in general.
By a Neural Network, I am assuming you're pointing to 'vanilla' vector based neural networks.
Let's take the MNIST dataset, it performs almost similar in both NNs and CNNs; the reason being digits of almost same size, and similar spatial drawings. To put it easy, all the digits roughly take up the same area as the other 60k. If you were to zoom in our zoom out these digits, the trained NN may not perform well.
Why?
Plain NNs lack the ability to extract 'position independent' features. A cat is a cat to your eye, no matter whether you saw it in the center of an image, or left corner. CNNs use 'filters' to extract 'templates' of cat. Hence CNNs can 'localize' what it is searching for.
You could say, NNs look at the 'whole' data to make a sense, CNNs look at the 'special' features at certain parts to make a sense.
This is also a reason why CNNs are popular. NNs are suited for their own applications. Every input node in a vectorized Neural Networks represent a feature, and that feature is tied to a purpose throughout training, i.e. its position is fixed throughout the training.

Yes. In theory, a single layer neural network can compute any function. In practice, such a network would have to be much larger than a CNN with equivalent functionality and would therefore be much harder to train.


text:
Adding input features - is complete re-training required?
I've never worked with very large models that require weeks or months of training, but in such a situation, what happens if you want to add extra features inputs, do you need to re-train the entire model again from scratch, or how is it handled?  E.g. if Tesla added an extra sensor to its cars, would it need to re-train its network again from scratch to include this extra sensor input?

I'll try to explain how I would do it and the intuition behind it, feel free to correct me if something doesn't make sense.
Lets consider you have an input of shape F where F is the number of features. If you were to construct a simple feed forward neural network you'll need to multiply the input with a weight matrix of shape (F, hidden_dim). Now if we want to add one more feature, the input will be of shape F+1 and the multiplication with the first layer will not work. What we could do to overcome this problem is to pad the weight matrix with an extra 0. In theory the new network should be able to reproduce the results of the first because it ignores the new feature. Now if we want to learn the "importance" of the new feature we could train the model to do so. I assume that the training time should be significantly lower since some of the weights might not need to be updated.


text:
What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?
Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem to be meant for either pure text or pure numerical data.
There are those that can handle data with the dual data types, but those are generally not flexible for wrangling. For example, for pickle you have to load the entire thing into memory if you want to wrangle anything. You can just append directly to the disk like you can with hdf5, which can be very helpful for huge datasets which can not be all loaded into memory?
Also, any alternatives to Pandas for wrangling Huge datasets? Sometimes you can't load all the data into Pandas without causing a memory crash.

There are different possible ways to handle huge datasets:

If the data is too big to be fully uploaded to RAM, you can iterate over it in Pandas. You can find a brief explanation in the article Why and How to Use Pandas with Large Data, section 1. Read CSV file data in chunk size. Or add more RAM (or use powerful server hardware), if you want to continue using single machine.

If the data is really big, probably it's better to store and process it by multiple computers using special software, A specific tool depends on what you want to do with the data:

2.1. You can even keep using Pandas: there is an extension named Dask which wraps interfaces of Python iterables, NumPy, and Pandas to run on multiple machines. Also, there are other similar tools.
2.2. A more independent yes simple approach is to use some analytical, distributed DBMS like Google BigQuery or Cassandra.
2.3. And if you need something more powerful and complex, MapReduce systems like Apache Hadoop, Spark can be utilised. Moreover, there are many articles and scientific papers that describes usage of MapReduce for the machine learning.

So, as always, it is a matter of your opportunities and your aims of dealing with the data.


text:
How to do speech recognition on a single word
I would provide a sound signal of about 2-3 seconds to my neural network. I have trained my network with a single word, like if I speak "Hello" the network may tell if "Hello" is spoken or not, but some other word like "World" is spoken, it will say "Hello" is not spoken. I just want classification of sound if its a specific command or word.
What is the best way to do this, I am not a that much advanced in DNN, I only know about NN and CNN, I want to know if there is some research paper or tutorial, or need some explanation about the work.

If you have fixed length speech data you can detect the content using only CNN. You can see that problem as a binary classification (1 if the spoken word is correct, 0 otherwise).
But first, you need to make the input length is fixed. For example, you use 2 seconds as the fixed length. If the recorded speech is more than 2 seconds, you need to crop it, and if the recorded speech is less than 2 seconds you can pad it with 0 values.
Next, You can either use raw data (time-domain) or transform your data using some features extractors method (FFT, MFCC, or MFSC). Then, use CNN as you use it to classify the image. You can assume the graphic of the sound wave as a 2D image.
But, If your data have a variety of length, you can combine CNN to detect each phoneme then combine it as a sequence using RNN or HMM. You can read this method also in the mentioned papers.

Yes, I need to crop, I have thought the sound data in form of a dequeue, I would have a window of 2s, pass this 2 seconds to model, pop from the front of the window a sample, and push the sample recorded at the back and repeat this algorithm

One more thing I want to ask, say for "hello" word recognition, how much dataset size would provide a considerable good accuracy, if you know this. Otherwise i understood the logic, I would read the papers to implement it. Thanks

yes, you can use your "window" method. if you use small "shift", It would give some variation of your training data. I can't give an exact number, try with a smaller size (~100 samples for each class) and increase it if the accuracy is still small. To improve accuracy you still need to use some data that have similar sound with "hello".


text:
Why is it so common to initialize weights with a Guassian distribution divided by the square root of number of neurons in a layer?
I have seen in several jupyter notebooks people initializing the NN weights using:
np.random.randn(D, M) / np.sqrt(D)

Other times they just do:
np.random.randn(D, M)

What is the advantage of dividing the Gaussian distribution by the squared root of the number of neurons in the layer?
Thanks

I think they use the Xavier/Glorot's initialization method. You can read from the original paper:

We initialized the biases to be 0 and the weights $W_{ij}$ at each layer with the following commonly used heuristic:
$W_{ij} \sim U [ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}}] $
where $U[−a, a]$ is the uniform distribution in the interval $(−a, a)$ and $n$ is the size of the previous layer (the number of columns of $W$)

Some people use this as some reports said this initialization method lead to better result