text:
A mathematical explanation of Attention Mechanism
I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in recurrent networks), should eventually lead to the network focusing on certain parts of the input/source. So what is attention mechanism really adding to the network?
A potential answer in the case of Encoder-Decoder RNNs is:

The most important distinguishing feature of this approach from the
  basic encoder–decoder is that it does not attempt to encode a whole
  input sentence into a single fixed-length vector. Instead, it encodes
  the input sentence into a sequence of vectors and chooses a subset of
  these vectors adaptively while decoding the translation. This frees a
  neural translation model from having to squash all the information of
  a source sentence, regardless of its length, into a fixed-length
  vector. We show this allows a model to cope better with long
  sentences.
  - Neural Machine Translation by Jointly Learning to Align and Translate

which made sense and the paper says that it worked better for NMT.
A previous study indicated that breaking down the sentence into phrases could lead to better results:

In this paper, we propose a way to address this issue by automatically
  segmenting an input sentence into phrases that can be easily
  translated by the neural network translation model. Once each segment
  has been independently translated by the neural machine translation
  model, the translated clauses are concatenated to form a final
  translation. Empirical results show a significant improvement in
  translation quality for long sentences.
   - Overcoming the Curse of Sentence Length for Neural Machine
  Translation using Automatic Segmentation

which paved the way for further research resulting in attention models.
I was also going through an article on Attention is not quite all you need where it said something similar:

An LSTM has to learn to sequentially retain past values together in a
  single internal state across multiple RNN iterations, whereas
  attention can recall past sequence values at any point in a single
  forward pass.

and a more curated blog on the family of attention mechanism gives insight on how different ways have been formulated for implementing the concept: Attention? Attention!
Specifically, I want to know how attention mechanism is formulated for this task (aforementioned) or in general. A detailed mathematical insight would be helpful, probably somewhat on these lines: Understanding Attention in NN mathematically

There's plenty, but keep in mind that these articles do not describe the same approach. They simply have attention shifting automation as part of their approaches and therefore must detect a need for shift and execute it in a way that improves speed, accuracy, reliability or some combination of them.
There is no one dominant attention approach and probably will not be. In fact, the earliest attention mechanism in common use in machines was likely the electromechanical fire alarm. In digital systems, it would be a vacuum tube electric eye driving an intruder alert followed by the first hardware interrupts in transistor microprocessor boards.
The sophistication of hardware interrupts in contemporary computer systems is probably higher than attention mechanisms in neural nets as of this writing, but that may change. Currently the dictionary definition of attention is the only constraint we can place on these newer approaches in artificial networks.

The four working links in the question (one is broken) lead to articles with some mathematical description, so the question half answers itself.
This article, Dynamic relevance: vision-based focus of attention using artificial neural networks,
Shumeet Baluja, Dean Pomerleau, 1995; revised 1996,
(abstract),
has only a few mathematical expressions but is filled with system
theory that could be described mathematically with a little reading
and thought.
Same with Self-Organization of an Artificial Neural Network
Subjected to Attention Shift Impairments and Familiarity
Preference, Characteristics Studied in Autism, Lennart Gustafsson and Andrew P. Papliński, 2004.
This one has clear and comprehensive mathematical treatment: Backpropagation Generalized Delta Rule for the Selective Attention Sigma-if Artificial Neural Network, Maciej Huk, 2012
This one too: Small and Dim Target Detection via Lateral Inhibition Filtering and Artificial Bee Colony Based Selective Visual Attention, Haibin Duan, Yimin Deng, Xiaohua Wang, Chunfang Xu, 2013
There is some here, but the visualizations and concepts are so theoretically relevant that one might not miss the more extensive mathematics: Using chaotic artificial neural networks to model memory in the brain, Zainab Aram, Sajad Jafari, Jun Ma, Julien C. Sprott, Sareh Zendehrouh, Viet-Thanh Pham
Solid math coverage here: Parallel Implementation of Artificial Neural Network Training, Stefano Scanzio, Sandro Cumani, Roberto Gemello, Franco Mana, P. Laface, 2010, (abstract)
Also here: Multiple Object Recognition with Visual Attention, Jimmy Lei Ba, Volodymyr Mnih, Koray Kavukcuoglu, 2015 ICLR
The mathematics for the very popular DiSAN is here: DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding, Tao Shen, Jing Jiang, Tianyi Zhou, Shirui Pan, Guodong Long, Chengqi Zhang, 2018
Mathematics and the algorithm is provided for this unique approach too: Graph Classification using Structural Attention, John Boaz Lee, Ryan Rossi, Xiangnan Kong, 2018

It would be interesting to develop a taxonomy of attention approaches in AI, as that has probably not yet been done. It would take quite a study to see if any of the above bullet items match up with either of the two articles referenced in the question.


text:
Why is it called Latent Vector?
I just learned about GAN and I'm a little bit confused about the naming of Latent Vector.

First, In my understanding, a definition of a latent variable is a random variable that can't be measured directly (we needs some calculation from other variables to get its value). For example, knowledge is a latent variable. Is it correct?
And then, in GAN, a latent vector $z$ is a random variable which is an input of the generator network. I read in some tutorials, it's generated using only a simple random function:
z = np.random.uniform(-1, 1, size=(batch_size, z_size))

then how are the two things related? why don't we use the term "a vector with random values between -1 and 1" when referring $z$ (generator's input) in GAN?

I will answer here your specific question (because you are asking too many questions in one single post and because I don't want to clutter my answer below). z = np.random.uniform(-1, 1, size=(batch_size, z_size)) is a sampling operation. You're sampling $z$ from a uniform distribution. In the context e.g. of VAEs, a latent vector is sampled from some distribution. This is a "latent" distribution because this distribution outputs a compact (and hidden) representation of the inputs (e.g. images). This latent distribution is trained to learn such compact representation.

I am actually not very well familiar with GANs, but the concrete sampled latent vector $z$ is not a random variable. Mathematically, a hidden variable can be a random variable. The concept of a random variable is more related to the concept of a distribution than to the concept of a vector. Each random variable has an associated distribution. A vector is a sample from a distribution (defined over vectors).

@nbro "a hidden variable can be a random variable", based on your answer, is that means "a latent variable can be a random variable"?

@nbro so, do you think, can I call $z$ as a vector with random value instead of latent vector?

Latent is a synonym for hidden.
Why is it called a hidden (or latent) variable? For example, suppose that you observe the behaviour of a person or animal. You can only observe the behaviour. You cannot observe the internal state (e.g. the mood) of this person or animal. The mood is a hidden variable because it cannot be observed directly (but only indirectly through its consequences).
A good example of statistical model that is highly based on the notion of latent variables is the hidden Markov model (HMM). If you understand the HMM, you will understand the concept of a hidden variable.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Reward does not increase for a maze escaping problem with DQN
I am using deep reinforcement learning to solve a classic maze escaping task, similar to the implementation provided here, except the following three key differences:

instead of using a numpy array as the input of a standard maze escaping task, I am feeding the model with an image at each step; the image is a 1300 * 900 RGB image, so it is not too small.
reward:

each valid move has a small negative reward (penalize long move)
each invalid move has a big negative reward (run into other objects or boundaries)
Each blocked move has the minimal reward (not common)
Find the remote detectors’ defect has a positive reward (5)

I tweaked the parameters of replay memory, reduced the size of the replay memory buffer.

Regarding the implementation, I basically do not change the agent setup except the above items, and I implemented my env to wrap my customized maze.
But the problem is that, the accumulated reward (first 200 rounds of successful escaping) does not increase:

And the number of steps it takes to escape one maze is also stable somewhat:

Here are my question, on which aspect I could start to look at to optimize my problem? Or is it still too early and I will need to train more time?

You should use an algorithm to try doing a solve for the maze optimally, maybe A* algorithm. If the optimal steps is also in the range of your network, your network may have reached it's best. If the optimal step is much less, you can try increasing the step penalty and increasing the reward for reaching the end. Hope you can succeed in this problem.

Hello, thank you for the answer. I am thinking to use a DQN because 1) I am expecting to learn from, say, 1000 images, and has a model which can help to quickly solve the maze on other 5000 images. 2) my input are images of raw pixels. Therefore, I do need certain convolutional layers anyway, right?

if you use images, you should use CNN.

Yes, that's right. The DQN is stacked with three convolutional layers followed by two fully connected layer.

If the pathways are 1 pixels apart, the network is optimized already. But if it is wider apart, you may have to change the rewards and penalties.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Calculating Parameter value Using Gradient Descent for Linear Regression Model

Consider the following data with one input (x) and one output (y):
  (x=1, y=2)
  (x=2, y=1)
  (x=3, y=2)
  Apply linear regression on this data, using the hypothesis $h_Θ(x) = Θ_0 + Θ_1 x$, where $Θ_0$ and $Θ_1$ represent the parameters to be learned. Considering the initial values $Θ_0$= 1.0, and $Θ_1$ = 0.0, and learning rate 0.1, what will be the values of $Θ_0$ and $Θ_1$ after the first three iterations of Gradient Descent

From least squares method I took the derivative with respect to $Θ_0$ and $Θ_1$ and plugged in the initial values to get the slope/intercept and multiplied it by the learning rate 0.1 to get the step size.The step size was used to calculate the new $Θ_0$ and $Θ_1$ values.
I am getting $Θ_0$ as 1.7821 when following the above. Please let me know if the approach followed and the solution correct or there is a better way to solve

X = np.array([1,2,3])
Y = np.array([2,1,2])

params = np.array([1, 0])

def loss(y, yhat):
    return ((y - yhat)**2).mean()

def model(x):
    return params[0] + params[1]*x

def loss_grad(y, yhat, x):
    return np.array([(2*(yhat-y)).mean(), (2*(yhat-y)*x).mean()])

lr = .1
for _ in range(3):
    yhat = model(X)
    l = loss(Y, yhat)
    g = loss_grad(Y, yhat, X)
    params = params - lr*g
    print(f'thetas are now {params} with new loss of {loss(Y, yhat)}')

outputs
thetas are now [1.13333333 0.26666667] with new loss of 0.6666666666666666
thetas are now [1.13333333 0.23111111] with new loss of 0.2696296296296296
thetas are now [1.14755556 0.22874074] with new loss of 0.262887242798354

I double checked this with keras too, but in numpy i explicitly wrote the gradients, i reccomend double checking your gradient or arithmetic


text:
How Does AlphaGo Zero Implement Reinforcement Learning?
AlphaGo Zero (https://deepmind.com/blog/alphago-zero-learning-scratch/) has several key components that contribute to it's success:

A Monte Carlo Tree Search Algorithm that allows it to better search and learn from the state space of Go
A Deep Neural Network architecture that learns the value and policies of given states, to better inform the MCTS.

My question is, how is this Reinforcement Learning?  Or rather, what aspects of this algorithm specifically make it a Reinforcement Learning problem?  Couldn't this just be considered a Supervised Learning problem?

If you learn a policy or a value function from experience (that is, interaction with an environment), that's RL. In the case of AlphaGo, the MCTS is used to acquire the experience.
RL could in fact be considered supervised learning (SL) or, more specifically, self-supervised learning, where the experience corresponds to the labels in SL, especially nowadays with techniques like experience replay.


text:
Are absence of labels for classes of interest in a vision dataset a big problem?
I wish to be able to detect: pedestrians, cars, traffic lights
I have two large datasets:
 - One contains instances and labels of all three classes.
 - The other contains instances of all three but only labels for pedestrians and cars. ie. there are many unlabelled traffic lights.
I want to combine the two datasets and train Yolov3 on it. Will the unlabelled presence of objects of interest significantly affect detection performance of that category?

So depends on sizes— if the second dataset has a lot of traffic lights without labels and you in a basic setup train on it, it will cause large performance decreases on that class. If it’s small, then it may just be “noise” to the model and it’ll filter it out, and still perform well. My recommendation is to adjust the network slightly and add a flag input saying which dataset the input was from: if from first, train normally. if from the second then ignore all losses pertain to boxes/classes that the network thinks a traffic light was in: that way your labels don’t try to tell the mode l “there’s no traffic light there”

authors: Ευάγγελος Γρηγορόπουλος


text:
beautify an image with reinforcement learning
I am trying to formulate and solve the following problem of image mutation. Suppose I am trying to insert an object image into a "background" image of several objects, and I will need to look for a "sweet spot" to insert the image:

I am tentatively trying to formulate the problem into a reinforcement learning process, with the following elements:
0. initial stage:

a background image where the location of objects within the image has been marked (let's suppose we have a perfect object detector)
another image of a new object, let's say, a human

1. action space:

location (x, y) for the object image to be inserted; in that sense the action space is quite large.

2. environment:

each step I will have a new image to "learn from".
An oracle function F returns 1 or 0 (roughly one computation of F takes 30 seconds).
 This function tells me the latest synthesized image hits the "sweet spot" or not (1 means hit). If so, I will stop the search and return the image.

3. constraint:
the newly inserted object shouldn't overlap with the original objects in the figure.
While my gut feeling is that this problem is somehow similar to the classic "maze escape" problem which can be solved well with reinforcement learning, the action space seems quite large in this problem.
So here are my questions:

In case I would like to formulate this "beautify" image problem into a "deep" reinforcement learning problem, how can I learn from such large action space? Or is it really suitable for a reinforcement learning process?
Can I somehow subsume the "non-overlapping" constraint into the oracle function F? If so, how should I decide the reward score? Any principled or empirical way of deciding so?

The purpose of Reinforcement Learning is to maximize some notion of cumulative reward, leading me to the point (1) :
as far as I understand, there is no timesteps in your problem and the "reward" is immediate.
Thus, I don't think reinforcement is suitable here.
On an other hand, in supervised learning, linear regression is the task of approximating a mapping function (f) from input variables (X) to a continuous output variable (y).
It has much in common with your case. If I am not wrong, you are trying to approximate a function that maps image data to (x,y) coordinates
(the "sweet spot"). So, I think regression would be a better way to go for you.
You could either generate a dataset at first, with images data and associated (x,y) coordinates for only spots validated by your function F, and then train a regression predictive model. Or you could train your model with online learning, by generating batch of images and sweet spot coordinates at each step.
Concerning the point (2), it will highly depend on how is made your F function. Since overlapping spots cannot be sweet spots, the simplest would
be to make your F function return 0 for those spots.


text:
How to add some data input in a CNN?
There is this problem I have encountered, I was trying to classify the pixels from input image into classes, sort of like segmentation, using a encoder-decoder CNN. The “interested” pixels usually locate in the top right corner of the input image, but the input images are too big, which I have to slice them in patches, by doing this, each input patch loses its “which region of the whole picture it’s from” information.
I'm using pytorch, I thought of manually add this patch location info into the input, but then it will be convoluted, which does make sense to me since it's not a part of an image.
I'm new to this, not sure if I'm thinking the whole thing right, how should I manually add this info into the input correctly or if there is some keywords I can do some researches, in order to let the CNN taking position into account? Thank you.

Is your problem that the tiling scheme is losing positional information (where it is in the image) or losing contextual information (whats in the image that wasn't in the tile)?

@mshlis The first one, where it is in the image, sorry I didn't express it clearly.

If your interest is positional information, encode it!
This could include learning an embedding for each position and leveraging that in your model. You could also use an approach to hard-encode rather than learn it (kinda like adding sinusoids in the transformer paper Attention is All You Need
an example of a paper that encodes the 2D positional info: Attention Augmented Convolutional Networks


text:
Feasibility of a team-based FPS AI?
We have seen advances in top down, RTS team games like Dota 2 and Starcraft II from companies like OpenAI who developed agents to beat real pro players most of the time. How would similar learning techniques compare to games like Overwatch that require faster reaction times and complex understanding of 3d space and effect?
Or have we not developed solutions that could be tasked with this problem?

Apologies if this is a bad question or one that isn't for this forum, I just signed up.

Hi and welcome to this community! I think that this is a very interesting question, so don't worry! Maybe I will answer it later.

To get a better understanding, let's compare a couple of games DOTA2 and Overwatch. At first, both seen similar in terms of multiplayer and strategy, but there is a significant difference.
You have mentioned it in the question. reaction time
Even though both the games require strategy and understanding of the surroundings and team play, FPS games are heavily dependent on reaction time and accuracy. As much as you try, it's next to impossible to reach the reaction times and accuracy of computer agent.
In games such as DOTA2, the game is more dependent towards strategy and how the characters interact with each other (Strengths and Weakness).
Finally, it doesn't make sense to train an AI based agent to play FPS games, because the optimal solution is already present and there is no way that AI can reach those levels. It's called AIM BOT. As good as you can get at strategy and planning, team work and reaction times, you can almost never beat it. It's next to 100% accurate and the opponent is basically dead as soon as he's in sight.

Thanks for this answer! As for aimbot, I'm sure you could win against a team of badly coordinated aim-botted with a well coordinated team (perhaps not in other fps games). I was interested in how agents could learn to work with each other in these type of enviroments, but considering how well the dota agents performed in teams, I doubt this will be the least of its concerns. If someone or some people decide to teach a bot to play overwatch I think the main issue will be the visual perception of the game and the quick response times.


text:
How does ARKit's Facial Tracking work?
iPhone X allows you to look at the TrueDepth camera and reports 52 facial blendshapes like how much your eye is opened, how much your jaw is opened, etc.
If I want to do something similar with other cameras (not TrueDepth), what are my alternative methods? Currently, I just use a simple ConvNet which takes in an image and predict 52 sigmoid values.
What do you think could be the underlying technology behind ARKit Face Tracking?

Facial classification and tracking is easier compared to tracking any other motion. This is due to the fact that the face have a large number of easily identifiable features.
Facial tracking is an additional layer on top of facial detection. Facial detection works by finding characteristics such as the cheekbones, chin, nose, eyes etc. These features are easy to detect as they have very specific properties. These points are found with the help of shadows and brightness, for example, the nose and cheekbones are highlighted, whereas the the eyes and lips have a shadows.
Using these feature points it is possible to create a mesh to construct the face. This has been a classical way to detect and classify faces. The TrueDepth camera helps to make a better structure by providing depth with the help of an IR sensor.

Facial tracking makes use of this mesh to study and track changes in the facial structure. For example when you smile, it will detect the cheekbone points changing, similarly when you open your mouth, your lips will move apart.


text:
Analysis of Training Loss and Validation Loss Graph
Here I am Showing Two Loss graphs of an Artificial Neural Network.
Model 1

Model 2

Blue -training loss
Red  -val training loss
Can you help me to analyse these graphs? I read some articles and post but doesn't give me any sense.

Simply model 1 is a better fit compared to model 2.

Graph for model 1
We notice that the training loss and validation loss aren't correlated. This means the as the training loss is decreasing, the validation loss remains the same of increases over the iterations. This means that the model is not exactly improving, but is instead overfitting the training data. This isn't what we are looking for.
Graph for model 2
In this case, there is clearly a health correlation between training loss and the validation loss. They both seem to reduce and stay at a constant value. This means that the model is well trained and is equally good on the training data as well as the hidden data.

You should stick with model 2. In case you're going ahead with model 1, make sure to use the chechpoint where both the losses are at a similar value (at around 100 -150 epochs)

In graph 2, is there any chance of overfitting?


text:
Describing the order of a tensor
When describing tensors of higher order I feel like there is an overloading of the term dimension as it may be used to describe the order of the tensor but also the dimensionality of the... "orders"?
Assume one describes the third-order tensor produced by a convolutional layer and wants to refer to its width and height. Do you say spatial dimensions? Would you write about the channel dimension? Or rather the direction? Saying "spatial order" feels really weird. But staying with dimensions makes sentences like "The spatial dimensions are of equal dimensionality." (Disclaimer: Obviously you can avoid the issue here by restructuring, but doing this at every occasion does not feel like a satisfactory solution.).

just declare and continue in accordance. Most papers just say h,w dimension for example and stick to it

Ambiguity in Terms
You are correct that there is something like overloading occurring in tensor terminology in posts and in software libraries. Confusing jargon often appears when those without the mathematical background use mathematical terms. You rarely find this confusion when reading NASA, Cambridge, MIT, or Cal-tech materials.
Tensors, Rank, Dimensions, and Channels
A tensor is a grouping of dimensions. The grouping typically represents a relation between quantities describing a system. The order (or rank) of the tensor is the number of dimensions in the grouping.
When this mathematical idea is applied formally to electrical engineering, chemical systems, biological systems, or computing systems, we can say a tensor is the grouping of the inputs and outputs of that system, abstracting the relation between them. The order (or rank) of the tensor is the total number of signal channels going to or from the system. The rank is simply the number of inputs plus the number of outputs.
That original conception was lost a little when the term tensor began to be applied to grouping inputs in one tensor and grouping outputs in another tensor, without any connection with their relationship, which is not formally correct. Those are simply vectors in mathematics, represented by arrays in computer programs.
A labelled data set prepared for machine learning is a discrete tensor formally, however it does not have a mathematical model applied to it yet. When convergence occurs in an artificial network, the result of training is a set of parameters that, if later applied to new inputs, will predict the outputs based on the net and its trained parameters. The configuration of the network cells had become the model and the trained network had become a tensor expression approximating the phenomenon that the training data represents.
The Specific Example in the Question
A third order tensor cannot have only two dimensions, width and height. It must have three dimensions, possibly width, height, and instantaneous value. Width and height are the independent variables, and instantaneous value is the dependent variable. So we have a rank of $1 + 2 = 3$.
There is no clear and formal definition of what dimensionality means, so it would be useful to discontinue its use until academic textbooks contain a consistent definition, but that's unlikely to occur. It will likely continue to be used by non-mathematicians somewhat arbitrarily. Therefore, stating that two tensors are of equal dimensionality remains ambiguous.
Domain and Range
The correct terms are domain and range. These terms unambiguously describe the variability of the independent and dependent variables of a relation respectively. In the case of an array of samples describing a tensor relationship between horizontal position, vertical position, and brightness, the horizontal domain corresponds to the width and the vertical domain corresponds to the height.
Consequently, one would not say, "Width and height are the spacial dimensions," or, "width and height are the channel dimensions."
It could unambiguously be stated, "Width in this article shall refer to the maximum horizontal pixel position minus its minimum plus one." Without saying something that complex, albeit accurate, this would be a clear statement.

The fourth convolution layer in the network has a domain of 1280, 768 and a range of IEEE 32 bit floats.

It is easily understood and close enough to technically correct to not cause readers with mathematical training to dismiss the writer as uneducated.
The equality question is now solved. The equality of domains could be stated clearly and unambiguously. Whether it sounds unusual is not particularly relevant.

The horizontal and vertical domains of network layers two and three are equal, both being [1280, 768].

(The reason [1280, 768] is often seen is because compilers of many languages, starting with FORTRAN or maybe even earlier, interpret that as a two dimensional domain for a two dimensional array. So that can be used among software people, although [768, 1280] is more likely used in programs since it is a video tradition to traverse left to right and then top to bottom.)
Misuse of the Term Tensor
It is incorrect to say that adding two arrays is an example of tensors manipulated in software. Tensors are more abstract than arrays. If one were to ...

Make symbolic manipulations to reduce a complex tensor equation to a simple one,
Show that two tensor expressions are equal for given domains, or
Determine the correlation of a set of discrete points to a model represented as a tensor equation,

... then one would be performing tensor mathematics.
There are many pieces of software that perform such symbolic manipulations, but assigning a tensor to an input sample in machine learning is not tensor math. Tensor object classes that are not part of a system of symbolic manipulation are simply an abstraction of the typical programming language's array structure, parameterizing rank in the constructor. Such a trivial abstraction is not nearly as deep as symbolic manipulation of tensor expressions.

Thank you for this elaborate discussion! Despite lacking the mathematical background, I hope to now at least use the correct terminology in my future texts :)

By definition, tensors can be of any order (usually named differently if the order is less than three). So, I use $d_i$ to indicate the dimensionality of the $i$th facet.
Unless you have three or four-order tensors which each facet has a very specific meaning, naming the orders by terms such as special or time would be limiting.


text:
How to add variation in the results of a neural networks?
I would like to create a neural network that converts text into handwriting for use with a pen plotter. Before I start on this project, I'd like to be sure that artificial intelligence is the best way to do this. A problem that I foresee with this approach is a lack of human like variation in the results. For example, the word "dog", when inputted into the network, would be the same every time, assuming I'm not missing something. I am interested if there is any way to vary the output of the network in a realistic way, even when the input is exactly the same. Could I use a second network to make the results more random, but also still look human-like? Any thoughts/ideas would be greatly appreciated.

I would suggest starting with Generative Adversarial Networks (GAN). They usually are capable of adding some randomness to the output to produce different variants. Moreover, Conditional GANs can generate outcomes regarding the observed condition. Therefore, as you change the condition (as an input to the network) you can get different results.
Some examples of NN for synthetic handwriting generators:

https://github.com/sjvasquez/handwriting-synthesis
http://blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/
https://distill.pub/2016/handwriting/


text:
From what aspect to measure the performance of an object detector?
I am on the hook to measure the prediction results of an object detector. I learned from some tutorials that when testing a trained object detector, for each object in the test image, the following information is provided:
    <object>
    <name>date</name>
    <pose>Unspecified</pose>
    <truncated>0</truncated>
    <difficult>0</difficult>
    <bndbox>
        <xmin>451</xmin>
        <ymin>182</ymin>
        <xmax>695</xmax>
        <ymax>359</ymax>
    </bndbox>
</object>

However, it is still unclear to me 1) how does these information is taken by the object detector to measure the accuracy, and 2) how does the "loss" is computed for this case. Is it something like a strict comparison? For instance, if for the object "date", I got the following outputs:
    <object>
    <name>date</name>
    <pose>Unspecified</pose>
    <truncated>0</truncated>
    <difficult>0</difficult>
    <bndbox>
        <xmin>461</xmin>  <---- different
        <ymin>182</ymin>
        <xmax>695</xmax>
        <ymax>359</ymax>
    </bndbox>
</object>

Then I will believe that my object detector made something wrong? Or they tolerant some small delta such that if the bounding box has a small drifting, then it's acceptable. But if the "label" is totally wrong, then that's wrong for sure?
This is like a "blackbox" to me and it would be great if someone can shed some lights on this. Thank you.

This is a high level explanation: So most object detectors are generally deep neural networks that return a set of boxes:
box1 : coordinates : confidence
 box2 : coordinates : confidence
 etc...
and loss is computed different model to model. But i think your specifically curious how most do box comparison and loss. Generally they first check if the box is even relevant to the target. To do this often they calculate the Intersection over Union (IOU) as known as the Jaccard index wikipedia link. This checks how overlapped they are biased on the box sizes. Now if its above some threshold (example: boxes have IOU over .5 caluclate a loss, otherwise loss is just 0) they will compute some objective. Objectives can differ, but ideally its something differentiable. Examples include vectorizing and approximiating the IOU, dice loss, etc..
After a quick google search, heres a good tutorial explaining some components commonly used in the field (region proposals, feature extraction, etc) [guide]


text:
In NN, as iterations of Gradient descent increases, the accuracy of Test/CV set decreases. how can i resolve this?
As mentioned in the title I'm using 300 Dataset example with 500 feature as an input.
As I'm training the dataset, I found something peculiar.  Please look at the data shown below.

Iteration 5000 | Cost: 2.084241e-01
Training Set Accuracy: 100.000000
CV Set Accuracy: 85.000000
Test Set Accuracy: 97.500000
Iteration 3000 | Cost: 2.084241e-01
Training Set Accuracy: 98.958333
CV Set Accuracy: 85.000000
Test Set Accuracy: 97.500000
Iteration 1000 | Cost: 4.017322e-01
Training Set Accuracy: 96.875000
CV Set Accuracy: 85.000000
Test Set Accuracy: 97.500000
Iteration 500 | Cost: 5.515852e-01
Training Set Accuracy: 95.486111
CV Set Accuracy: 90.000000
Test Set Accuracy: 97.500000
Iteration 100 | Cost: 8.413299e-01
Training Set Accuracy: 90.625000
CV Set Accuracy: 95.000000
Test Set Accuracy: 97.500000
Iteration 50 | Cost: 8.483802e-01
Training Set Accuracy: 90.277778
CV Set Accuracy: 95.000000
Test Set Accuracy: 97.500000

The trend is that as the Iteration(cost) increases(cost decreases), the training set accuracy increases as expected, but the CV Set/Test Set Accuracy decreases. My initial thought is that this has to do with precision/bias issue, but I really can't buy it.
Anyone know what this entails? Or any reference?

Training scores improving (loss decreasing and accuracy increasing) whilst the opposite happens with cross validation and test data is a sign of overfitting to the training data. Your neural network is getting worse at generalising and no amount of further training will improve it - in fact the situation will get worse the more you train.
This is the main reason you have CV data sets, to show you when this happens before you try to use your model against the test set or real world data. So it is not "peculiar" at all, but the CV set doing its job for you, allowing you notice something has gone wrong.
To improve on the situation, you need to use some form of regularisation. The simplest approach here would be to take your model from around 100 epochs (because it has best CV score that I can see). This is early stopping, and is a simple valid regularisation approach.
Alternatives for neural networks include L2 weight regularisation (also called weight decay) and dropout.
In addition to this, your question states that you only have 300 examples, and more features than examples. This is really tricky to generalise (and hard to tell if you have - for instance it looks like you only have 20 CV examples and 40 test examples, these numbers are very low and prone to giving you inaccurate estimates simply due to chance of what entries are in the data sets). I recommend you look into K-fold cross-validation in order to get more accurate scores, and help you choose the best model for generalisation.

Thanks a lot. I've been suspecting the issue of overfitting, but had no idea why it really happen. Do you have any recommended resource on NN overfiting(AKA precision and bias problem)? I'm having a hard time analysing the result of the model..

@JoshuaLee: I don't have any specific resource. Andrew Ng's course https://www.coursera.org/learn/machine-learning has some very useful guidance on best practices including the basic theory and techniques behind handling overfitting

Thanks for your help. I do truly appreciate it.


text:
How to create neural network that predicates result of exam?
Actually, I am "fresh-water", and I've never known what is neural network. Now I am trying understand how to design simple neuronetwork for this problem:
I'd like to make up such neural network that after learning it could predicate a mark of passed exam (for example, math). There is such factors that influence on a mark:

Chosen topic (integral, derivative, series)
Perfomance (low, medium, high)
Does a student work? (Yes, No, flexible schedule)
Have a student ever gotten through a add course? (Yes, No)

The output is a mark (A,B,C,D,E,F)
I don't know should I add few layers between inputs and output

Moreover, I have few results from past years:

(integral, low, Yes, No, E)
(integral, medium, Yes, Yes, B)
(series, high, No, Yes, A)
and so on. What do I need to know else for designing this NN?

Have you tried searching anywhere else on the internet first? Assuming you have and knowing the multitude of examples there are I'm not sure what is holding you back. If you've run up against a particular problem perhaps you could add that to the question?

I'd like to know should I add some layers between Xi and Y, and when do I have to add layers?

@Ayrat i think you should change your question to what you just asked. it is more general, and while the context in the problem is nice it confiscuates that your curious in how to design the architecture rather than the domain specificity

You will need one of two things or both.

A theoretical model of grade probability distribution already verified against data that contains the features listed
A data set mapping the features listed to grades (labels) for training an artificial network

The features can be specified more formally

Chosen topic --- from a finite list of topics, encoded by integer (numbered strings)
Performance --- low, medium, or high --- it is not clear from where this judgment comes, but that would need to be clarified and also included in either the verified theoretical model or among the features of the training data
Does a student work? --- hours per week --- note that the flexibility of the schedule is not as critical as the hours of the 168 hours of each week not available for sleep, self-care, class time, or study
Have a student ever gotten through a add course? --- binary --- this question is unclear in its current phrasing
Grade --- converted to numeric value from 0.0 to 4.0, which can be represented as an integer from 00 to 40

The difficult part will be finding either the verified theoretical model or the data set for training or both.
If a verified theoretical model is found, apply a gradient descent implementation to tune that model, which is a statistics and conversion problem only partly related to AI and machine learning. Otherwise using a model free artificial network will likely allow you to reach your objective. A simple feed forward network with gradient descent and back-propagation and three layers should suffice.
There are many examples in Python, Java, and other languages available online, usually in conjunction with a library that the examples leverage for computation and instructions on how to install the software. There are other questions and answers in this forum that explain what books to buy to get started. In fact, there is a tag specifically for that purpose: https://ai.stackexchange.com/questions/tagged/getting-started.

You can add as many layers (with any arbitrary number of nodes) as you want.
Please note that as you add more learning parameters (layers and nodes), your model complexity increases. This means the model can potentially learn a more complex input-output relationship. However, it also increases the risk of overfitting. Overfitting generally happens when the model you build is more complex than the data you have. In such a scenario, the model memorizes the data instead of learning from it. In other words, it can produce a very good result for the same data as it was trained on but cannot generalize well. So, it performs poorly when the inputs are slightly different from what is used to be fed at the training stage.
In practice, you may try different architectures and parameter configurations, and measure the generalization capacity of the models (via cross-validation for example) to choose the best model. In English, a generalizable model is the one that performs (almost) equally good on both training/validation and testing sets.


text:
Reinforcement learning to play snake - network seems to not get trained at all
I am trying to build a network able to play snake game. This is my very first attempt to do such stuff. Unfortunately, I've stuck and even have no idea how to reason about the problem.
I use reinforcement neural network approach (q-leaning). My network is built on top of Keras. I use 6 input neurons for my snake:

1 - is any collision directly behind
2 - is any collision directly on the right
3 - is any collision directly on the left
4 - is snack up front (no matter how far)
5 - is a snack on the right side (no matter how far)
6 - is a snack on the left side (no matter how far)

the output has 3 neurons:

1 - do nothing (go ahead)
2 - turn right
3 - turn left

I believe this is a sufficient set of information to make proper decisions. But the snake seems to not even grasp the concept of not hitting the wall - which results with instant death.
I use the following rewards table:

100 for getting the snack
-100 for hitting wall/tail
1 for staying alive (each step)

Snake tends to run randomly no matter how many training iterations it gets.
The code is available on my github: https://github.com/ayeo/snake/blob/master/main.py

I cannot comment much on your setup for inputs and outputs. It seems adequate to get some control, but does not cover the fully Markov state for the game, so I would expect that will limit the agent from ever being truly optimal. I would expect it to learn to play the game though, if you were implementing Q learning with a neural network correctly.
In your code, you are implementing a basic Q learning loop. It seems correct. However, this combination of Q learning and neural networks is known not to work - or more accurately, it rarely works this simply. The problem is mainly to do with the network receiving its own initially biased outputs back as new targets, plus receiving updates in correlated form (data on each time step is strongly correlated with data from previous time step). These biases are too large for the Q learning process to overcome, and typically the result is an agent that fixates on a single default action, because it has learned an inflated action value for it.
The problem is well known in RL research and called "The Deadly Triad" by Sutton & Barto.
The usual solution to this with Q learing is called DQN or "Deep" Q Learning ("Deep" is in quotes because this should be applied even if you just have a single hidden layer).
In basic DQN, you need to add the following features:

An experience replay table. Instead of training directly on experience as it is received, instead store $s, a, r, s'$ in memory. When it is time later in the loop to train the NN for a step, take a random sample of some M items (e.g. 32 items) as a mini-batch, calculate a latest target for them, and train once on the mini batch. You will need logic to only start this training process once you have some minimal amount of experience from behaving randomly (e.g. 500 random steps).
A "target network". When generating target Q values, use a cloned copy of the learning network, and only update this clone every N steps (with N typically set at 1000 or 10000).

These two additions are not really optional, even for really basic environments. You will need to add them to your script.


text:
Is it possible with stochastic gradient descent for the error to increase?
As simple as that. Is there any scenario where the error might increase, if only by a tiny amount, when using SGD (no momentum)?

Yes. Not only that, but error is highly noisy, prone to big spikes and sometimes quite long period of increase before decrease again or stabilize. Often it's even impossible to understand error plot without passing it through smoothing filter, so noisy it is. Specific depend on the problem of cause. It's not only for SGD but for any optimizer.


text:
How to enforce covariance-matrix output as part of the last layer of a Policy Network?
I have a continuous state space, and a continuous action space. The way I understand it, I can build a policy network which takes as input a continuous state vector and outputs both mean vector and covariance matrix of the action-distribution. To get a valid action I then sample from that distribution.
However, when trying to implement such a network, I get the error message that the parts of my output layer which I want to be the covariance matrix are singular/not positive-semi-definite. How can I fix this? I tried different activation-functions and initializations for the last layer, but once in a while I run into the same problem again.
How can I enforce that my network outputs a valid covariance matrix?

Usually it is assumed that there is no correlation between different actions, so the covariance matrix will be zero everywhere except on the main diagonal. Diagonal will represent variances of actions. Diagonal covariance matrix will be positive semidefinite if all values on diagonal are $\geq$ 0 so you need to insure that output of final layer is $\geq$ 0, which can be done with ReLU activation for example.

Now I have the problem that the matrix becomes singular, i.e. the network learns to set the diagonal zero. Any idea how to fix this?

diagonal is zero probably because the output of network is < 0 so ReLU sets it to zero. Only one of the elements needs to be zero for diagonal matrix to be singular. You can try to evade this by adding a small constant to the diagonal like 1e-6

@Brale_ 's answer is correct, it is common practice in a multitude of models to learn a representation of an independent multivariate normal, but don't let that stop you from pushing the envelope for your needs.
You can actually learn a dependent form as well. Normally the independent form is done by learning the means and standard deviations, because sampling a standard normal can achieve your draw by $z \sim N(\mu, Diag(\sigma^2))$ by $z = \mu + \sigma \epsilon$ where $\epsilon$ is drawn from a unit normal.
But you can actually achieve a similar trick for a generalized multivariate normal distribution: lets assume your trying to learn $N(\mu, \Sigma)$ where $\Sigma$ is the covariance matrix. So what you would do is learn the Cholesky decomposition because where $\Sigma = AA^T$ you can now draw it through the parametrization trick: $z = \mu + A\epsilon$.

@mshlis, if $\sigma$ is covariance matrix, then exist $A$ s.t. $\sigma=AA^T$.
But if we generate $A$ by model, and calculate $\sigma = AA^T$, then Cholesky decomposition will not successful


text:
Better to learn the same small set for multiple epochs then go to the next or learn from each one time repeatedly for multiple times?
I don't know if I worded the title correctly.
I have a big dataset (300000 of images after augumentation) and I've splitted it into 10 parts, because I can't convert the images into a numpy and save it, the file would be too large.
Now, I have a neural network (Using keras with tf). My question is, is it better to train each file individually for X epochs (File 1 for 5 epochs, then File 2 for 5 epochs, etc.), or should I do an epoch for each, repeatedly (File 1 for an epoch, File 2 for an epoch, etc., and repeat for 5 times).
I've used the first and I get an accuracy of about 88%. Whould I get an improvement by doing the latter?

Small set for multiple epochs causes overfit, so expose entire data in one epoch, then augment (change a littel bit), and train again. Use small amount of dropout also.
I have a big dataset (300000 of images after augumentation) and I've splitted it into 10 parts, because I can't convert the images into a numpy and save it, the file would be too large.

Do not store augmented data, just do augment with function right after next epoch and overwrite the dataset array


text:
Query regarding the minmax loss function formulation of the training of a Generative Adversarial Network (GAN)
Just needed a clarification on the training procedure for a standard GAN.
Of my understanding the loss function to optimize is a min max (max min causing mode collapse due to focus on one class generation) problem where the loss function

needs to maximized for the discriminator and minimized for the generator networks -
1.) In this equation are the $E_{z~p_{z(z)}}$ and $E_{x~p_{data(x)}}$ the means of the distributions of the mini batch samples? Also is the optimal case for the discriminator a maximum value of 0? and the optimal case for the generator a minimum value of $log(small~value)$ {basically a massive negative value}? If so, what happens to the first term during the training of the generator - is it taken as a constant or is the discriminator performing badly considered optimal for the generator?
2.) While putting this in code, for every step is the discriminator trained first for one step, keeping the generator constant, followed by the generator being trained for the same step, with the discriminator kept constant?
3.) In every step of training are there multiple latent vectors sampled to produce multiple generator outputs for each step? If so is the loss function an average or sum of all $V(D, G)$ for that step

Hi. Please, ask one question per post next time!!

I'll answer your questions one by one:

In this equation are the $E_{z \sim p_z(z)}$ and $E_{x \sim p_{data}(x)}$ the means of the distributions of the mini batch samples?

So let's take the first part $E_{x \sim p_{data}(x)}[log \,D(x)]$. This is read as the "expected value of $log \, D(x)$, where $x$ is sampled from $p_{data}(x)$". So, in simpler terms this means that:

You have a distribution $p_{data}(x)$.
You sample a batch of samples $x$ from this distribution.
You feed the batch to the discriminator and get its output $D(x)$.
You compute the log for the batch of predictions.
You average over the samples in the batch.

Also is the optimal case for the discriminator a maximum value of 0?

Yes, the goal of the discriminator is to maximize $V(D, G)$. This is achieved when $D(x) = 1$ and $D(z) = 0$, where $V(D, G)=0$.

the optimal case for the generator a minimum value of log(small value) {basically a massive negative value}?

Yes, the generator wants to minimize $V(D, G)$, so it wants $D(x)$ and $D(z)$ to be as small as possible (though it can affect only the second term). The minimum value for $D(z)$ is $1 / M$, where $M$ is the number of classes in the dataset. So the minimum value for $V(D, G)$ is $\log{[(M-1) / M^2]}$ (I think).

If so, what happens to the first term during the training of the generator - is it taken as a constant or is the discriminator performing badly considered optimal for the generator?

The first term is of no consequence to the Generator, because the generator can't affect it somehow (i.e. like you say it is taken as a constant). Consider the generator $G_{\theta}$ has parameters $\theta$ and the discriminator $D_{\phi}$ has parameters $\phi$. When training $G$:
$$
\frac{\log{D_{\phi}(x)}}{\partial \theta} = 0
$$
So the generator is trained of $V$.

While putting this in code, for every step is the discriminator trained first for one step, keeping the generator constant, followed by the generator being trained for the same step, with the discriminator kept constant?

In theory, yes. In practice for every step the Generator takes, we train the Discriminator for more steps (e.g. $5$ steps).

In every step of training are there multiple latent vectors sampled to produce multiple generator outputs for each step? If so is the loss function an average or sum of all V(D,G) for that step.?

Yes, at every step we sample a batch of latent vectors. The equation indicates that we average over these (i.e. the expectation) but in practice it doesn't have any difference (the average is the sum divided by a constant number, which doesn't affect the optimization very much).


text:
Couldn't an AI cheat when trying to follow its goal?
Hello, I was reflecting about what implications might building a strong AI have and I came across some ideas which I find disturbing, I'd love to have some external thought on that :
1) If we ever managed to create an AI say nearly as smart as a human, It would probably have been programmed with some concrete goals, as the AIs we are programming right now : Reinforcement learning allow an agent to try and increase a "reward" variable, regression is all about getting closer to a certain goal function etc..
But then a strong AI, which would undoubtedly be able to understand how it is built, just as we understand (partly at least) how our brains work, because it would be as smart as its creators and we don't tend to build machines that are as hard to understand as brains.
Then couldn't such an agent figure out the best way to achieve its goals would actually not be, say, pleasing and protecting the humans like we would've wanted it to do, but to get control of its own program and maximize whatever reward it was set to pursue ? Just as we could decide to branch electrods to our brain if we were able to find out how exactly our brain was built.
I really don't see how this scenario could ever be avoided if we were to build such an AI, apart from finding a perfect security preventing anyone from accessing the code of the said AI (including itself).
2) On the same note, I also wondered, could it try to not only satisfy its goals by "cheating" (updating its reward variables for example) but also to change itself, or commit suicide ? After all, we humans have never been able to figure out what goals we were meant to pursue (by that I mean what reward variable in our brain we are trying to increment), and many philosophers reflecting upon that matter thought about death as an escape from once goals. So my question is : could it try to change its code or kill itself ?
I have other questions and thoughts I would like to discuss, but I think this a good start, to test whether I'm in the right place for this kind of discussion.
Looking forward to your thoughts.

These are two good questions. To address the first, this is a known problem in AI safety and is called "Wireheading". There is currently no known solution other than to somehow prevent an AI agent from being able to alter its internal state in this fashion.
If you are interested in this area in general then I recommend this paper as an entry point into the literature.

If you want to read about all the other various risks around AI then https://www.amazon.co.uk/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111 will keep you up at night.


text:
LSTM text classifier shows unexpected cyclical pattern in loss
I'm training a text classifier in PyTorch and I'm experiencing an unexplainable cyclical pattern in the loss curve. The loss drops drastically at the beginning of each epoch and then starts rising slowly. However, the global convergence pattern seems OK. Here's how it looks:

The model is very basic and I'm using the Adam optimizer with default parameters and a learning rate of 0.001. Batches are of 512 samples. I've checked and tried a lot of stuff, so I'm running out of ideas, but I'm sure I've made a mistake somewhere.
Things I've made sure of:

Data is delivered correctly (VQA v1.0 questions).
DataLoader is shuffling the dataset.
LSTM's memory is being zeroed correctly
Gradient isn't leaking through input tensors.

Things I've already tried:

Lowering the learning rate. Pattern remains, although amplitude is lower.
Training without momentum (plain SGD). Gradient noise masks the pattern a bit, but it's still there.
Using a smaller batch size (gradient noise can grow until it kinda masks the pattern, but that's not like solving it).

The model
class QuestionAnswerer(nn.Module):

    def __init__(self):
        super(QuestionAnswerer, self).__init__()
        self._wemb = nn.Embedding(N_WORDS, HIDDEN_UNITS, padding_idx=NULL_ID)
        self._lstm = nn.LSTM(HIDDEN_UNITS, HIDDEN_UNITS)
        self._final = nn.Linear(HIDDEN_UNITS, N_ANSWERS)

    def forward(self, question, length):
        B = length.size(0)
        embed = self._wemb(question)
        hidden = self._lstm(embed)[0][length-1, torch.arange(B)]
        return self._final(hidden)

This looks an awful lot like you are plotting error at the end of each batch, rather then at the end of each epoch. I think you'll find if you plot just the epoch error, you'll get a nice curve. This is normal behaviour for a neural network if you simply plot each batch error, rather then epoch.

Yes, I do plot sub-epoch wise. My concern is not the niceness of the curve, but what the curve is telling me. And what it's telling me is that I'm missing something, either a cause for this behaviour or that I'm unadvertedly interferring on the training. Such an abbrupt fall of the loss is not common while using momentum. But you gave me an idea: I have a ton of output classes, so maybe this is a sign of the model memorizing samples. I'll test more on monday.

The cause for this behaviour is because you are plotting sub-epoch wise. This is meant to happen, it's normal. Try plotting only the first batch's error, and I'm sure you'll notice a nice curve. Then try only the second, and so on. Then, if you were to plot each of these curves sequentially, the graph in your question is what you would get.

Sorry, but that's not the cause. It's the reason for me being able to see it, and if I'd plot only once per epoch the thing would still be there, only I wouldn't notice it. Plotting sub-epoch wise is a good practice because it helps noticing coding errors, as well as analyzing the convergence and optimization nature of the net. So I guess you don't know why this pattern shows up and that's ok, me neither. I'll test more soon.

It turns out that the zig-zag pattern is an inherent effect of using a word embedding layer. I don't fully understand the phenomenon, but I believe it has a strong correlation with the embeddings acting as a sort of memory slots, which can change relatively quickly, and the LSTM generating a summary of the sequence, so that the model can remember past combinations.
I found this plot of a training loss curve of word2vec and it exhibits the same per-epoch pattern.

Edit
After conducting several experiments, I've isolated the causes. It seems that this is an indirect effect of having a large model capacity. In my case, I had too large word embeddings (size 1024) and too many classes (2002), which also increases model capacity, so the model was doing an almost per-sample learning. Reducing both resulted in a smooth-as-silk learning curve and a better generalisation.

This weird pattern can be caused by a big learning rate. Check this: https://stackoverflow.com/a/49095437/13164928


text:
Create an AI to solve a puzzle (by deduction)
Some puzzle games have a unique solution that can be solved by deduction rather than guesswork (e.g. Slitherlink, Masyu). Using a computer to solve this puzzle it's pretty easy, we can use a backtracking method to find the best solution in second (in general, the puzzle size is not too big).

Is it possible to train a bot to solve this kind of puzzle by deduction?

I think by train it to watch a previous step-by-step solution several times the bot can find some implicit rules/patterns to solve a specific puzzle. Is this possible? are there any references for this method?

Deductive solutions work by constraining the search space and then executing a suitable search algorithm. As such they are typically solved by describing the puzzle in a notation your solver can read.

The alternative is to still describe the problem and use reinforcement learning, which will find good search paths for similar domains.

Hi @PaulBrown could you give a more detailed explanation as an answer maybe?

What you are trying to achieve sounds a lot like inductive logic programming:

Given an encoding of the known background knowledge and a set of examples represented as a logical database of facts, an ILP system will derive a hypothesised logic program which entails all the positive and none of the negative examples.

nice reference! thank you, I'll wait some time before giving you a green tick to look for another answer

You're welcome. I guess statistical methods, deep learning and all that are more en vogue these days, but once in while it certainly can't hurt to take a look at what various people have done in other fields such as ILP. In particular the class of problems that you are interested in (deterministic, fully observable logic puzzles) should lend itself very well to it.


text:
Effect of rescaling of inputs on loss for a simple neural network
I've been trying out a simple neural network on the fashion_mnist dataset using keras. Regarding normalization, I've watched this video explaining why it's necessary to normalize input features, but the explanation covers the case when input features have different scales. The logic is, say there are only two features - then if the range of one of them is much larger than that of the other, the gradient descent steps will stagger along slowly towards the minimum.
Now I'm doing a different course on implementing neural networks and am currently studying the following example - the input features are pixel values ranging from 0 to 255, the total number of features (pixels) is 576 and we're supposed to classify images into one of ten classes. Here's the code:
import tensorflow as tf

(Xtrain, ytrain) ,  (Xtest, ytest) = tf.keras.datasets.fashion_mnist.load_data()

Xtrain_norm = Xtrain.copy()/255.0
Xtest_norm = Xtest.copy()/255.0

model = tf.keras.models.Sequential([tf.keras.layers.Flatten(),
                                    tf.keras.layers.Dense(128, activation="relu"),
                                    tf.keras.layers.Dense(10, activation="softmax")])

model.compile(optimizer = "adam", loss = "sparse_categorical_crossentropy")
model.fit(Xtrain_norm, ytrain, epochs=5)
model.evaluate(Xtest_norm, ytest)
------------------------------------OUTPUT------------------------------------
Epoch 1/5
60000/60000 [==============================] - 9s 145us/sample - loss: 0.5012
Epoch 2/5
60000/60000 [==============================] - 7s 123us/sample - loss: 0.3798
Epoch 3/5
60000/60000 [==============================] - 7s 123us/sample - loss: 0.3412
Epoch 4/5
60000/60000 [==============================] - 7s 123us/sample - loss: 0.3182
Epoch 5/5
60000/60000 [==============================] - 7s 124us/sample - loss: 0.2966
10000/10000 [==============================] - 1s 109us/sample - loss: 0.3385
0.3384787309527397

So far, so good. Note that, as advised in the course, I've rescaled all inputs by dividing by 255. Next, I ran without any rescaling:
import tensorflow as tf

(Xtrain, ytrain) ,  (Xtest, ytest) = tf.keras.datasets.fashion_mnist.load_data()

model2 = tf.keras.models.Sequential([tf.keras.layers.Flatten(),
                                    tf.keras.layers.Dense(128, activation="relu"),
                                    tf.keras.layers.Dense(10, activation="softmax")])

model2.compile(optimizer = "adam", loss = "sparse_categorical_crossentropy")
model2.fit(Xtrain, ytrain, epochs=5)
model2.evaluate(Xtest, ytest)
------------------------------------OUTPUT------------------------------------
Epoch 1/5
60000/60000 [==============================] - 9s 158us/sample - loss: 13.0456
Epoch 2/5
60000/60000 [==============================] - 8s 137us/sample - loss: 13.0127
Epoch 3/5
60000/60000 [==============================] - 8s 140us/sample - loss: 12.9553
Epoch 4/5
60000/60000 [==============================] - 9s 144us/sample - loss: 12.9172
Epoch 5/5
60000/60000 [==============================] - 9s 142us/sample - loss: 12.9154
10000/10000 [==============================] - 1s 121us/sample - loss: 12.9235
12.923488986206054

So somehow rescaling does make a difference? Does that mean if I further reduce the scale, the performance will improve? Worth trying out:
import tensorflow as tf

(Xtrain, ytrain) ,  (Xtest, ytest) = tf.keras.datasets.fashion_mnist.load_data()

Xtrain_norm = Xtrain.copy()/1000.0
Xtest_norm = Xtest.copy()/1000.0

model3 = tf.keras.models.Sequential([tf.keras.layers.Flatten(),
                                    tf.keras.layers.Dense(128, activation="relu"),
                                    tf.keras.layers.Dense(10, activation="softmax")])

model3.compile(optimizer = "adam", loss = "sparse_categorical_crossentropy")
model3.fit(Xtrain_norm, ytrain, epochs=5)
model3.evaluate(Xtest_norm, ytest)
------------------------------------OUTPUT------------------------------------
Epoch 1/5
60000/60000 [==============================] - 9s 158us/sample - loss: 0.5428
Epoch 2/5
60000/60000 [==============================] - 9s 147us/sample - loss: 0.4010
Epoch 3/5
60000/60000 [==============================] - 8s 141us/sample - loss: 0.3587
Epoch 4/5
60000/60000 [==============================] - 9s 144us/sample - loss: 0.3322
Epoch 5/5
60000/60000 [==============================] - 8s 138us/sample - loss: 0.3120
10000/10000 [==============================] - 1s 133us/sample - loss: 0.3718
0.37176641924381254

Nope. I divided by 1000 this time and the performance seems worse than the first model. So I have a few questions:

Why is it necessary to rescale? I understand rescaling when different features are of different scales - that will lead to a skewed surface of the cost function in parameter space. And even then, as I understand from the linked video, the problem has to do with slow learning (convergence) and not high loss/inaccuracy.  In this case, ALL the input features had the same scale. I'd assume the model would automatically adjust the scale of the weights and there would be no adverse effect on the loss. So why is the loss so high for the non-scaled case?
If the answer has anything to do with the magnitude of the inputs, why does further scaling down of the inputs lead to worse performance?

Does any of this have anything to do with the nature of the sparse categorical crossentropy loss, or the ReLU activation function? I'm very confused.

One of the factors that rescaling gives a better result is on your weight initialization of the model. Your experiments run on the same model that using the same weight initialization (tf.keras.dense using Glorot Uniform). If your input is too big or too small the model needs time to adjust the weight.

Why is it necessary to rescale? I understand rescaling when different features are of different scales - that will lead to a skewed surface of the cost function in parameter space. And even then, as I understand from the linked video, the problem has to do with slow learning (convergence) and not high loss/inaccuracy. In this case, ALL the input features had the same scale. I'd assume the model would automatically adjust the scale of the weights and there would be no adverse effect on the loss. So why is the loss so high for the non-scaled case?

Slow learning means you can't compare the loss of two models even at the same epoch. As the model using the same initial value of weights, the second model may need time (epoch) to adjust its weight to get the same result with the first model.

If the answer has anything to do with the magnitude of the inputs, why does further scaling down of the inputs lead to worse performance?

It has the same reason, very small values of the inputs also lead to the difficulty to adjust the weight.

Thanks so much! I need just one clarification. The reason for the slow convergence as given in the video is a skewed loss function surface in the parameter space. But in this case, since all the inputs are of the same scale, wouldn't the loss function surface be more or less symmetric? Why then is the rate of convergence slow in this case?


text:
Relation between size of parameters and complexity of model with overfitting
I'm reading the book Pattern Recognition and Machine Learning by Bishop, specifically the intro where he covers polynomial regression model. In short, let's say we generate $10$ data points using the function $\sin(2\pi x)$ and add some gaussian random noise to each observation. Now we pretend not knowing the generating function and try to fit a polynomial model to these points.
As we increase the degree of the polynomial, it goes from underfitting ($d=1,2$) to overfitting ($d=10$). One thing the author notes is that the higher the degree of the polynomial, the higher the values of the coefficients (parameters). This is my first doubt: why does the size of the coefficients increase with the polynomial degree? And why is the size of the parameters related to overfitting?
Secondly, he states that even for degree $10$, if we get sufficiently many data points (say $100$), then the high degree polynomial will no longer overfit the data and should have comparatively better generalization performance. Second doubt: Why is this so?

Regarding your first question, this is domain/task specific, and not always the case. My guess of why it happens in your situation (You did not specify your domain, so ill assume it's sometimes outside $(-1,1)$) is that higher order polynomials increase much faster than lower order ones so it may fall under the trap where the coefficients have to be large to handle it. (e.g. if $x=2.0$ then $ 2^5*x^2 = x^7 \rightarrow 32*x^2 = x^7$)
Now to answer why more points will cause less overfitting is sheerly because you have more equations. Your loss is generally some form of average distance accross points and predictions. Given $N$ points I can always find an $N-1$ order polynomial that fits it exactly. Instead of a formal proof I will provide a form in which you can easily generate this polynomial from $N$ $(x_i, y_i)$ pairs
$$ f(x) = \sum_{i=1}^{N}y_i\prod_{j=1\atop j \ != i}^N\frac{(x-x_j)}{(x_i-x_j)}$$
As you can see, for each $x_i$ it will zero out all terms in the sum except one where the product become 1 making $f(x_i) = y_i$ by construction. So adding more points will prevent whatever optimization process you are learning to directly solve each point exactly, but instead try to find an algorithm that best generalizes to each of the points.


text:
How are edge features implemented in Geometric Deep Learning?
The work I've seen so far have the nodes containing features. Any resources for how to use a GCN on a graph where the edges are the ones that contain features rather than the nodes?

one practice ive seen used is to define a new graph where each edge in the previous is a node and connection are determined by if there was a vertex between two edges. This is an invertible process

@mshlis Where did you see this? Maybe you could also provide your answer!

@nbro i saw it used on at a project at my work and turns out it only works for DAGs with a couple of additional assumptions about the graph/task. so my comment isnt general enough. i.e. was probably a mishap of me messaging on something I wasn't too familiar with

In the paper Neural Message Passing for Quantum Chemistry (2017), the authors (from Google, Google Brain and Google DeepMind) introduce a framework called message passing neural network (MPNN), which generalizes previously proposed geometric deep learning models. In section 2 of the paper, they describe this MPNN framework and they state that edge features can also be learned, using the MPNN framework, by introducing hidden states for all edges in the graph, $\mathbf{h}_{e_{vw}}^t$, where $t$ is the iteration number and $e_{vw}$ is the edge from node $v$ to node $w$.
See also the paper Machine learning prediction errors better than DFT accuracy, which is cited in the MPNN paper as an example where the authors also learn the edge features. In section 2 of the MPNN paper, they briefly describe this specific instantiation of the MPNN framework, called the Molecular Graph Convolutions.


text:
Should RL rewards diminish over time?
Should a reward be cumulative or diminish over time?
For example, say an agent performed a good action at time $t$ and received a positive reward $R$. If reward is cumulative, $R$ is carried on through for the rest of the episode, and summed to any future rewards. However, if $R$ were to diminish over time (say with some scaling $\frac{R}{\sqrt{t}}$), then wouldn't that encourage the agent to keep taking actions to increasing its reward?
With cumulative rewards, the reward can both increase and decrease depending on the agents actions. But if the agent receives one good reward $R$ and then does nothing for a long time, it still has the original reward it received (encouraging it to do less?). However, if rewards diminish over time, in theory that would encourage the agent to keep taking actions to maximise rewards.
I found that for certain applications and certain hyperparameters, if reward is cumulative, the agent simply takes a good action at the beginning of the episode, and then is happy to do nothing for the rest of the episode (because it still has a reward of $R$).

RL agents - implemented correctly - do not take previous rewards into account when making decisions. For instance value functions only assess potential future reward. The state value or expected return (aka utility) $G$ from a starting state $s$ may be defined like this:
$$v(s) = \mathbb{E}_{\pi}[G_t|S_t=s] = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^kR_{t+k+1}|S_t=s] $$
Where $R_t$ is the reward distribution at time $t$, and $\mathbb{E}_{\pi}$ stands for
expected value given following the policy $\pi$ for action selection.
There are a few variations of this, depending on setting and which value function you are interested in. However, all value functions used in RL look at future sums of reward from the decision point when the action is taken. Past rewards are not taken into account.
An agent may still select to take an early high reward over a longer term reward, if:

The choice between two rewards is exclusive
The return is higher for the early reward. This may depend on the discounting factor, $\gamma$, where low values make the agent prefer more immediate rewards.

If your problem is that an agent selects a low early reward when it could ignore it in favour of something larger later, then you should check the discount factor you are using. If you want a RL agent to take a long term view, then the discount factor needs to be close to $1.0$.
The premise of your question however is that somehow a RL agent would become "lazy" or "complacent" because it already had enough reward. That is not an issue that occurs in RL due to the way that it is formulated. Not only are past rewards not accounted for when calculating return values from states, but there is also no formula in RL for an agent receiving "enough" total reward like a creature satisfying its hunger - the maximisation is applied always in all states.
There is no need to somehow decay past rewards in any memory structure, and in fact no real way to do this, as there is no data structure that accumulates past rewards used by any RL agent. You may still collect this information for displaying results or analysing performance, but the agent doesn't ever use $r_{t}$ to figure out what $a_t$ should be.

I found that for certain applications and certain hyperparameters, if reward is cumulative, the agent simply takes a good action at the beginning of the episode, and then is happy to do nothing for the rest of the episode (because it still has a reward of R

You have probably formulated the reward function incorrectly for your problem in that case. A cumulative reward scheme (where an agent receives reward $a$ at $t=1$ then $a+b$ on $t=2$ then $a+b+c$ on $t=3$ etc) would be quite specialist and you have likely misunderstood how to represent the agent's goals. I suggest ask a separate question about your specific environment and your proposed reward scheme if you cannot resolve this.


text:
What does it mean for AlphaZero's network to be "fully trained"
Reading this blog post about AlphaZero:
https://deepmind.com/blog/article/alphazero-shedding-new-light-grand-games-chess-shogi-and-go
It uses language such as "the amount of training the network needs" and "fully trained" to describe how long they had the machine play against itself before they stopped training. They state training times such as 9 hours, 12 hours, and thirteen days for chess, shogi, and Go respectively. Why is there a point at which the training "completes?" They show plots of AlphaZero's performance on the Y axis (its Elo rating) as a function of the number of training steps. Indeed, the performance seems to level out as the number of training steps increases beyond a certain point. Here's a picture from that site of the chess performance vs training steps:

Notice how sharply the Elo rating levels off as a function of training steps.

First: am I interpreting this correctly? That is, is there an asymptotic limit to improvement on performance as training sessions tend to infinity?
If I am interpreting this correctly, why is there a limit? Wouldn't more training mean better refinement and improvement upon its play? It makes sense to me that the millionth training step may yield less improvement than the very first one, but I wouldn't expect an asymptotic limit. That is, maybe it gets to about 3500 Elo points in the first 200k training steps over the course of the first 10 hours or so of playing ches. If it continued running for the rest of the year, I'd expect it to rise significantly above that. Maybe double its Elo rating? Is that intuition wrong? If so, what are the factors that limit its training progress beyond the first 10 hours of play?

Thanks!

Neural network will eventually reach limit of it's approximation power. You can't expect to learn more and more things infinitely long with the same amount of learnable parameters. Also, if you eventually reach optimal performance, you can't play more optimal than what optimal is (not saying that it reached optimal performance but possibly something close to optimal for it approximation abilities). So probably combination of those two leads for the performance increase to reach it's limit.


text:
How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?
This is related to my earlier question, which I'm trying to break down into parts (this being the first). I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding:
If we have a vocabulary $V$, the naive way to represent words in it would be via one-hot-encoding, or in other words, as basis vectors of $R^{|V|}$ - say $e_1, e_2,\ldots,e_{|V|}$. We want to map these to $\mathbb{R}^n$, via some linear transformation such that the images of similar words (more precisely, the images of basis vectors corresponding to similar words) have higher inner products. Assuming the matrix representation of the linear transformation given the standard basis of $\mathbb{R}^{|V|}$ is denoted by $\mathcal{V}$, then the "embedding" of the $i$-th vocab word (i.e. the image of the corresponding basis vector $e_i$ of $V$) is given by $\mathcal{V}e_i$.
Now suppose we have a context "The cat ____ over a", CBoW seeks to find a word that would fit into this context. Let the words "the", "cat", "over", "a" be denoted (in the space $V$) by $x_{i_1},x_{i_2},x_{i_3},x_{i_4}$ respectively. We take the image of their linear combination (in particular, their average):
$$\hat v=\mathcal{V}\bigg(\frac{x_{i_1}+x_{i_2}+x_{i_3}+x_{i_4}}{4}\bigg)$$
We then map $\hat v$ back from $\mathbb{R}^n$ to $\mathbb{R}^{|V|}$ via another linear mapping whose matrix representation is $\mathcal{U}$: $$z=\mathcal{U}\hat v$$
Then we turn this score vector $z$ into softmax probabilities $\hat y=softmax(z)$ and compare it to the basis vector corresponding to the actual word, say $e_c$. For example, $e_c$ could be the basis vector corresponding to "jumped".
Here's my interpretation of what this procedure is trying to do: given a context, we're trying to learn maps $\mathcal{U}$ and $\mathcal{V}$ such that given a context like "the cat ____ over a", the model should give a high score to words like "jumped" or "leaped", etc. Not just that - but "similar" contexts should also give rise to high scores for "jumped", "leaped", etc. For example, given a context "that dog ____ above this" wherein "that", "dog", "above", "this" are represented by $x_{j_1},x_{j_2},x_{j_3},x_{j_4}$, let the image of their average be
$$\hat w=\mathcal{V}\bigg(\frac{x_{j_1}+x_{j_2}+x_{j_3}+x_{j_4}}{4}\bigg)$$
This gets mapped to a score vector $z'=\mathcal{U}\hat w$. Ideally, both score vectors $z$ and $z'$ should have similarly high magnitudes in their components corresponding to similar words "jumped" and "leaped".
Is my above understanding correct? Consider the following quote from the lectures:

We create two matrices, $\mathcal{V} \in \mathbb{R}^{n\times |V|}$ and $\mathcal{U} \in \mathbb{R}^{|V|\times n}$, where $n$ is an arbitrary size which defines the size of our embedding space. $\mathcal{V}$ is the input word matrix such that the $i$-th column of $\mathcal{V}$ is the $n$-dimensional embedded vector for word $w_i$ when it is an input to this model. We denote this $n\times 1$ vector as $v_i$. Similarly, $\mathcal{U}$ is the output word matrix. The $j$-th row of $\mathcal{U}$ is an $n$-dimensional embedded vector for word $w_j$ when it is an output of the model. We denote this row of $\mathcal{U}$ as $u_j$.

It's not obvious to me why $v_i=\mathcal{V}e_i$ should be the same as or even similar to $u_i$. How does the whole backpropagation procedure above ensure that?
Also, how does the procedure ensure that basis vectors corresponding to similar words $e_i$ and $e_j$ are mapped to vectors in $\mathbb{R}^n$ that have high inner product? (In other words, how is it ensured that if words no. $i_1$ and $i_2$ are similar, then $\langle v_{i_1}, v_{i_2}\rangle$ and $\langle u_{i_1}, u_{i_2}\rangle$ have high values?)

Unlike in skip-gram, the reason similar words have similar embeddings in CBOW is because the words show up in the same contexts of other skipped words.
lets assume two words $e_i$ and $e_j$ pop up in the exact same context of some word $e_k$ with 3 other context words as well. An example would be:

He leaped over the truck
He jumped over the truck

Where the italics represent the words with similar meanings but the bolded words above are the ones being predicted/skipped in CBOW. Lets now show this.
Let the rest of the context be denoted $\{e_r\}_r$ and the skipped word as $e_s$ and so the loss will try to minimize both $-e_s^T log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_i + \sum_re_r)]))$ and $-e_s^T log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_j + \sum_re_r)]))$.
Assuming a long run withlarge enough batch-size (so ignoring catastrophic forgetting) we can essentially say it will be minimizing
$$-e_s^T [log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_i + \sum_re_r)])) +\\ log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_j + \sum_re_r)]))]$$
Now let $\mathcal{U}\mathcal{V}\frac{1}{R+1}\sum_re_r$ be denoted as $c$, $\mathcal{U}\mathcal{V}\frac{1}{R+1}e_i$ as $\hat u_i$, and $\mathcal{U}\mathcal{V}\frac{1}{R+1}e_j$ as $\hat u_j$. So we have to minimize
$$-e_s^T[log(softmax(\hat u_i+c)) + log(softmax(\hat u_j+c))]$$
or equivalently
$$-e_s^T\ log(softmax(\hat u_i+c) * softmax(\hat u_j+c))$$
As we know in crossentropy, we get our critical point when the inside of the log here would equal $e_s$, which is only possible if both $softmax(\hat u_i+c)$ and $softmax(\hat u_j+c)$ were to be equal to $e_s$. softmax is non-invertible and is invariant under addition only, therefore to achieve this we would need $\hat u_i + c$ to equal $\hat u_j + c + K$ where $K$ is just some arbitrary constant vector ($K=[k,k,k,...]$. Subtracting $c$ and multiplying by $R+1$ we make our model want
$$\begin{align}
u_i &= u_j + K \\
\implies K &= \mathcal{U}\mathcal{V}(e_i -e_j) \\
\end{align}$$
This would mean for all common words, we see this activity that $u_{i-j}$ would be a constant, but this is difficult because that means a $\delta$ must exist such that $\mathcal{U}\delta = k*\vec 1$ and that all similar word pairs indexed by $(a,b)$ would have $v_a \approx v_b + const*\delta$ (because $u_a = u_b + K \implies v_b = v_a + \mathcal{U}^{-1}K$ and we denote $\delta = \mathcal{U}^{-1}K$). This constraint is heavy, tough to learn, and would also be difficult to process on multiple word senses that would appear in the vocabulary. This is because if it were to learn this, it enforces constraints on the images of $\mathcal{U}$ and $\mathcal{V}$ along with creating word vectors with highly different magnitudes making the learning process difficult. This would indicate that this constant would be low in practice. Therefore we would get $v_i \approx v_j$.

Thanks for the nice answer! I have a few doubts: there's a couple of assumptions underlying the whole treatment - 1. The central (skipped) word as well as all other words in the context are exactly the same. 2. The definition of "similar" words assumed by your answer is ones that occur in the context of the same word (in this case, "over"). The definition is not: similar words are those that occur in same/similar contexts. Am I right that these are the assumptions? In their current form, aren't they quite restrictive/unlikely?

(cont'd) When you say both $softmax(\hat u_i+c)$ and $softmax(\hat u_i+c)$ should both be equal to $e_s$ - do you mean $e_s^{1/2}$ (where this denotes a vector obtained on taking element-wise square root of $e_s$)? I'm guessing by $softmax(\hat u_i+c)*softmax(\hat u_i+c)$ you mean the vector obtained by element-wise multiplication of $softmax(\hat u_i+c)$ and $softmax(\hat u_i+c)$.

Finally, I'm sorry I didn't quite understand your last paragraph. I guess by $\delta$ you mean $\mathcal{V}(e_i-e_j)$. I didn't understand from "and that all similar word pairs" onwards. Why do you say $v_a \approx v_b+const*\delta$? Is this $\delta$ different from the one in $\mathcal{U}\delta=k*1$? I don't see the reasoning behind the part "This constraint is heavy... ...would be low in practice". Sorry to be a pain but I want to understand all this properly, so I'd be really grateful if you could elaborate. Thanks again for the answer!

@Shirish regarding your first 2 doubts: This is restriction isnt reality, its just an approximation to demonstrate the mathematics. We dont expect this to occur so plaoinly but as an expectation we expect it to be approximate

@Shirish no i did mean $e_s$. Its multiplication, not addition: $e_s \odot e_s = e_s$

@Shirish in essence all imilar words would have to have this connection linearly in the vector space, but because words have multiple senses and multiple meanings this is difficult for the optimization process to learn, if not impossible. Therefore it will probably do best in a scenario where the vectors magnitudes are not so different (i.e. $K=0, \delta=0$)

Oh my bad about the $e_s$ doubt: stupid of me not to see that. Regarding your last comment - why $\delta=0$? Also, is the $\delta$ in $v_a\approx v_b+const*\delta$ the same as the one in $\mathcal{U}\delta=k*1$, or are they different? Did you mean to say that practically $v_a\approx v_b+\epsilon K$? Finally, I'm still not clear why the fact that this constraint is heavy, tough to learn and difficult to process would imply that $\epsilon$ is low. Sorry if I'm being dense.

@Shirish No they are the same comment, one leads to the other and no $delta$ does not equal $\epsilon *K$ because $\delta = \mathcal{U}^{-1}K$, its heavy for various reasons: forces $K$ to be in image of $\mathcal{U}$ and $\delta$ to be in $\mathcal{V}$, along with this structure that would mean that all magnitudes would differ heavily which make the loss landscape difficult to traverse in a single order method like gd

Could you edit the answer to show how one leads to the other (as in how we get $v_a\approx v_b+const*\delta$) and elaborate on the final paragraph in the answer? Just for future reference or if anyone else is to come across the same question. In any case, I've accepted the answer. Appreciate the help!


text:
Dice loss gives binary output whereas binary crossentropy produces probability output map
On recommendation of Kanak on stackoverflow I am posting this question here:
Currently I am experimenting with various loss functions and optimizers for my binary image segmentation problem. The loss functions that I use in my Unet however give different output segmentation maps.
I have a highly imbalanced dataset, thus I am trying dice loss for which the customized function is given below.
    def dice_coef(y_true, y_pred, smooth=1):
        """
        Dice = (2*|X & Y|)/ (|X|+ |Y|)
             =  2*sum(|A*B|)/(sum(A^2)+sum(B^2))
        ref: https://arxiv.org/pdf/1606.04797v1.pdf
        """
        intersection = K.sum(K.abs(y_true * y_pred), axis=-1)
        return (2. * intersection + smooth) / (K.sum(K.square(y_true), -1) + K.sum(K.square(y_pred), -1) + smooth)

    def dice_coef_loss(y_true, y_pred):
        return 1 - dice_coef(y_true, y_pred)

Binary cross entropy results in a probability output map, where each pixel has a color intensity that represents the chance of that pixel being the positive or negative class. However, when I use the dice loss function, the output is not a probability map but the pixels are classed as either 0 or 1.
My questions are:
1.How is it possible that these different loss functions have these vastly different results?

Is there a way to customize the dice loss function so that the output segmentation map is a probability map similar to the one of binary crossentropy loss.

The probability map / output isn't produced by your loss function, but your output layer, which is activated either by softmax or sigmoid.
In other words, your dice loss output is also a probability map. It's simply very confident in itself. If you forget about the problem with potential overfitting for a moment and train your binary crossentropy model longer, the probability values will eventually all converge to the 2 ends (0 and 1).
In my experience, dice loss and IOU tend to converge much faster than binary crossentropy for semantic segmentation, so if you stop the training early on, dice loss will produce a probability map that resembles a binarized output more so than binary crossentropy.


text:
Is "dataset size" and "model size" same thing?
I mean what is determine my model size, connection amount between layers and neurons, or size of my dataset?

Dataset and model refer to different things. Dataset means part of data available for training (training dataset) or validation (validation dataset). Model is the learning process goal, the state of the computer "brain" after it has been fully educated (or made its learning). Model size refers to size of the container which contains the model. In deep learning it can be measured by width and depth of the network used, I also found a site comparing different models by npy file size, that physically contains the generated model as computer code. In that case model contained a more complex structure which was documented and size in bytes was for comparison purposes.
So in short, it is roughly speaking the size of layers and neurons, if I have to take one of your options. Dataset is a different thing.
More precise explanation about what is model and what is dataset:
https://www.quora.com/What-are-different-models-in-machine-learning

Thank you for your answer
I understand that size of dataset doesn't effect the size of model.
For example I have keras sequential model with 3 layers of Dense() with 64 units each. This amount of layers and units detmermine the size of model and this size is not depend on what size of dataset I use (1 GB or 50GB) right?

That is my understanding. The model is more accurate the more you train it, but that does not add the size, at least on Deep Learning where I am familiar with.


text:
TensorFlow estimator DNNClassifier fails to fit simple data
The ready-to-use DNNClassifier in tf.estimator seems not able to fit these data:
X = [[1,2], [1,12], [1,17], [9,33], [48,49], [48,50]]
Y = [ 1,     1,      1,      1,      2,       3     ]

I've tried with 4 layers but it's fitting to 83% (=5/6 sampes) only:
hidden_units = [2000,1000,500,100]
n_classes    = 4

The sample data above are supposed to be separated by 2 lines (right-click image to open in new tab):

It seems stuck be cause of Y=2 and Y=3 are too close. How to change the DNNClassifier to fit to 100%?

Normalise your inputs.
Neural networks work poorly outside of relatively small numerical ranges on input. An ideal range is for each feature to be drawn from $\mathcal{N}(0,1)$ i.e. a Normal distribution with mean $0$ and standard deviation $1$. In your case, divide both parts of $\mathbf{x}$ by $25$ and subtract $1$ would probably suffice.
Your neural network architecture is completely overblown for the problem at hand. That may be because you were trying to force it to fit this data (and failing because of lack of normalisation). Try something more like: hidden_units = [20,10]

my inputs are now divided by 50 to be in range [0,1], but accuracy is still 83% (=5/6)

any other changes that i should apply?

@datdinhquoc: I'll take a deeper look. You probably need to play with other hyperparameters, such as learning rate, in order to get the classifier to split class 2 and 3 given just that one example. It may be an interesting toy example because of that, although typically you would not use a NN on real data with only 6 rows and 4 classes (one of which is not even in the data set)

@datdinhquoc: I can get 100% accuracy (in Keras) reliably using a simple network as I suggested, and around 500 epochs. It did take a few attempts with different hyperparameters. Probably one thing worth knowing is that tanh activations on hidden layers works better than relu for smaller networks. However, relu will work, you just need to make the NN slightly larger.

tks, i'll try switching the code to keras, may i have your 100%ac code? maybe in google colab?

@datdinhquoc OK, here is a gist: https://gist.github.com/neilslater/6053e3729045e575f40f4d884f4d65dc - it is just hacky code. I replicated your data as train, cv and test - it is important to not do that for real data if you want to test whether your NN has generalised correctly. Instead you should split your dataset properly into separate datasets. If you want to know more about that. please research it or ask a new question on the site

tks for the gist


text:
Why tf object detection api needs so few pictures?
I am wondering why tf object detection api needs so few picture samples for training while regular cnns needs many more?
What I read in tutorials is that tf object detection api needs around 100-500 pictures per class for training (is it true?) while regular CNNs need many many more samples, like tens of thousands or more. Why is it so?

I haven't work with the tf object detection api so I might be totally wrong but I guess that they need so little data because their models are already trained on huge datasets, and they are just transferring the learning.

You actually might be right :) haven't thought about it, thanks!

you are right. Actually from scratch OD needs more images than Classifiers because its a more difficult task

@razvanc92 you can put your comment as an answer for this question :)

I guess that they need so little data because their models are already trained on huge datasets, and they are just transferring the learning (using those pre-trained models as starting point).