text:
What is the concept of channels in CNNs?
I am trying to understand what channels mean in convolutional neural networks. When working with grayscale and colored images, I understand that the number of channels is set to 1 and 3 (in the first conv layer), respectively, where 3 corresponds to red, green, and blue.
Say you have a colored image that is $200 \times 200$ pixels. The standard is such that the input matrix is a $200 \times 200$ matrix with 3 channels. The first convolutional layer would have a filter that is size $N \times M \times 3$, where $N,M < 200$ (I think they're usually set to 3 or 5).
Would it be possible to structure the input data differently, such that the number of channels now becomes the width or height of the image? i.e., the number of channels would be 200, the input matrix would then be $200 \times 3$ or $3 \times 200$. What would be the advantage/disadvantage of this formulation versus the standard (# of channels = 3)? Obviously, this would limit your filter's spatial size, but dramatically increase it in the depth direction.
I am really posing this question because I don't quite understand the concept of channels in CNNs.

Say you have a colored image that is 200x200 pixels. The standard is such that the input matrix is a 200x200 matrix with 3 channels. The first convolutional layer would have a filter that is size $N×M×3$, where $N,M<200$ (I think they're usually set to 3 or 5).
Would it be possible to structure the input data differently, such that the number of channels now becomes the width or height of the image? i.e., the number of channels would be 200, the input matrix would then be 200x3 or 3x200. What would be the advantage/disadvantage of this formulation versus the standard (# of channels = 3)? Obviously, this would limit your filter's spatial size, but dramatically increase it in the depth direction.

The different dimensions (width, height, and number of channels) do have different meanings, the intuition behind them is different, and this is important. You could rearrange your data, but if you then plug it into an implementation of CNNs that expects data in the original format, it would likely perform poorly.
The important observation to make is that the intuition behind CNNs is that they encode the "prior" assumption or knowledge, the "heuristic", the "rule of thumb", of location invariance. The intuition is that, when looking at images, we often want our Neural Network to be able to consistently (in the same way) detect features (maybe low-level features such as edges, corners, or maybe high-level features such as complete faces) regardless of where they are. It should not matter whether a face is located in the top-left corner or the bottom-right corner of an image, detecting that it is there should still be performed in the same way (i.e. likely requires exactly the same combination of learned weights in our network). That is what we mean with location invariance.
That intution of location invariance is implemented by using "filters" or "feature detectors" that we "slide" along the entire image. These are the things you mentioned having dimensionality $N \times M \times 3$. The intuition of location invariance is implemented by taking the exact same filter, and re-applying it in different locations of the image.
If you change the order in which you present your data, you will break this property of location invariance. Instead, you will replace it with a rather strange property of... for example, "width-colour" invariance. You might get a filter that can detect the same type of feature regardless its $x$-coordinate in an image, and regarldess of the colour in which it was drawn, but the $y$-coordinate will suddenly become relevant; your filter may be able to detect edges of any colour in the bottom of an image, but fail to recognize the same edges in the top-side of an image. This is not an intuition that I would expect to work successfully in most image recognition tasks.

Note that there may also be advantages in terms of computation time in having the data ordered in a certain way, depending on what calculations you're going to perform using that data afterwards (typically lots of matrix multiplications). It is best to have the data stored in RAM in such a way that the inner-most loops of algorithms using the data (matrix multiplication) access the data sequentially, in the same order that it is stored in. This is the most efficient way in which to access data from RAM, and will result in the fastest computations. You can generally safely expect that implementations in large frameworks like Tensorflow and PyTorch will already require you to supply data in whatever format is the most efficient by default.

Thanks for the detailed answer. To clarify, when I said to "rearrange the data," I mean that the CNN is ALSO trained under the new format. I am not suggesting that we use a CNN trained on the previous format and use it to make a predictions on an image in the new format. Apologies for the miscommunication. As a side note, this question was inspired by my question in https://datascience.stackexchange.com/questions/43291/wrangling-data-for-cnn/43303#43303  where I am trying to use CNNs on non-image data.

@Iamanon Yes that's what I assumed in my answer as well, that you'd do both training and evaluation with the "rearranged" data. I still expect that would generally perform less well. The dimensions that your filters are "sliding" along should by the dimensions where you want the "invariance". That should be in the spatial dimensions (width and height), but NOT in other dimensions (e.g. channels for different colours).

@Iamanon So, intuitively you could say that dimensions for which you expect the invariance property to be beneficial are not "channels", and all other dimensions are "channels". For example, TensorFlow has conv2d and conv3d for inputs with invariance in 2D or 3D spaces (and any other input dimensions are "channels"). It also has convolution (https://www.tensorflow.org/api_docs/python/tf/nn/convolution) with an arbitrary number input_spatial_shape of "spatial" inputs, and an arbitrary number in_channels of remaining input dimensions.

@Iamanon For your physics application... it's difficult for me to judge where you'd want the property of invariance. I could easily see it being a useful property in all of the dimensions. Spatial and temporal invariance both can make sense. You have to ask yourself: whatever kind of high-level features I expect my NN to "learn inside hidden layers", should the manner in which it detects them be invariant to time and/or space? If yes, filters should slide along those dimensions. If no, treat them as "channels".

Thanks again. The idea that the channels are NOT the dimension(s) in which you expect the invariance property makes sense! What happens if there is more than one dimension that fits under this idea of channels? It seems these deep learning libraries only allow for a single input as the "channels." With one dimension, say, of size 5, you would simply set channels = 5. What if you had some data where you had 2 dimensions of size 5 and 10, respectively, in which you are NOT expecting the invariance property, how would you define the number of channels?

Would you map the 2 dimensions to 1-D? So basically channels = 5*10 = 50?

For my physics application, I think both spatial and temporal invariance makes the most sense -- I think this requires a little bit of domain knowledge. So I would have only a single channel.

@Iamanon Ehm I think you'd just want to concatenate them, so 5 + 10 = 15 in your example. How many features would you have in your example per instance/data-point? That's how many channels you'd have. In 2D RGB images, you have three "features" per pixel: a Red feature, a Green Feature, and a Blue feature. So, also three channels. That's how it works basically.

If you have a gray scale image, that means you are getting data from one sensor.
If you have an RGB image, that means you are getting data from three sensors.
If you have a CMYK image, that means you are getting data from four sensors.
So, channels can be considered as same information seen from different perspective. (here color)
If you see how the kernel (for example 5*5*3) moves, it moves only in XY direction and not in the channel direction. So, you are trying to learn features in XY direction from all the channels together.
But, if you exchange the dimensions like you mentioned your XY dimensions become 200*3 or 3*200 and your channels become 200. In this case you are moving kernel not in the actual XY spatial space of image. So, it doesn't make any sense according to me. You are contradicting the basic concept of CNN by doing so.
The concept of CNN itself is that you want to learn features from the spatial domain of the image which is XY dimension. So, you cannot change dimensions like you mentioned.


text:
What is a "logit probability"?
DeepMind's paper "Mastering the game of Go without human knowledge" states in its "Methods" section on its "Neural network architecture" that the output layer of AlphaGo Zero's policy head is "A fully connected linear layer that outputs a vector of size 19^2+1=362, corresponding to the logit probabilities for all intersections and the pass move" (emphasis mine).  I am self-trained regarding neural networks, and I have never heard of a "logit probability" before this paper.  I have not been able by searching and reading to figure out what it means.  In fact, the Wikipedia page on logit seems to make the term a contradiction.  A logit can be converted into a probability using the equation $p=\frac{e^l}{e^l+1}$, and a probability can be converted into a logit using the equation $l=\ln{\frac{p}{1-p}}$, so the two cannot be the same.  The neural network configuration for Leela Zero, which is supposed to have a nearly identical architecture to that described in the paper, seems to indicate that the fully connected layer described in the above quote needs to be followed with a softmax layer to generate probabilities (though I am absolutely new to Caffe and might not be interpreting the definitions of "p_ip1" and "loss_move" correctly).  The AlphaGo Zero cheat sheet, which is otherwise very helpful, simply echoes the phrase "logit probability" as though this is a well-known concept.  I have seen several websites that refer to "logits" on their own (such as this one), but this is not enough to satisfy me that "logit probability" must mean "a probability generated by passing a logit vector through the softmax function".
What is a logit probability?  What sources can I read to help me understand this concept better?

Indeed I haven't seen the term "logit probability" used in many places other than that specific paper. So, I cannot really comment on why they're using that term / where it comes from / if anyone else uses it, but I can confirm that what they mean by "logit probability" is basically the same thing that is more commonly referred to simply as "logits": they are the raw, unbounded scores of which we generally push a vector through a softmax function to generate a discrete probability distribution that nicely adds up to $1$.
This definition fits the one you linked from wikipedia (although that link only covers the binary case, and AlphaGo Zero would have multinomial logits since it has more than two outputs for the policy head).
In the AlphaGo Zero paper, the described architecture has a "linear output layer" (i.e. no activation function for the outputs, or the identity function as activation function for the outputs, or however you like to describe it) for the policy head. This means that these outputs are essentially unbounded, they could be any real number. We know for sure that these outputs cannot directly be interpreted as probabilities, even if this isn't stated quite explicitly in the paper.
By calling them logits (or logit probabilities for reasons unknown to me), they are essentially implying that these outputs will still be post-processed by a softmax to convert them into a vector that can be interpreted as a discrete probability distribution over the actions, even if they do not explicitly describe a softmax layer as being a part of the network.
It is indeed possible that in Leela Zero they decided to make the softmax operation explicitly a part of the Neural Network architecture. Mathematically they end up doing the same thing... the AlphaGo Zero paper implies (by using the word "logit") that a softmax is used as a "post-processing" step, and in Leela Zero they explicitly make it a part of the Neural Network.
Here are a couple more sources for the reasoning that usage of the word "logit" basically implies usage of a softmax, though indeed they do not cover the term "logit probability":

https://stats.stackexchange.com/a/52836/144483
https://stackoverflow.com/a/34243720/6735980

Thanks for your answer.  That Wikipedia link to multinomial logistic regression especially helped since it ultimately and explicitly uses softmax to generate the probability distribution.  I will likely accept your answer, but I will wait to see if anyone else might have more information about the logit probability phrase in particular.

@sadakatsu Of course :) I'd love to see if anyone else knows more about where that phrase comes from myself too, I was also wondering about that when I first tried implementing something similar and was looking into all the tiny details. Note that the original AlphaGo paper does explicitly mention using a softmax, and in AlphaGo Zero they also must be doing something to convert their linear outputs into something that can be interpreted as a probability distribution... so it's a fairly safe bet that they're still doing softmax.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to deal with large (or NaN) neural network's weights?
My weights go from being between 0 and 1 at initialization to exploding into the tens of thousands in the next iteration. In the 3rd iteration, they become so large that only arrays of nan values are displayed.
How can I go about fixing this?
Is it to do with the unstable nature of the sigmoid function, or is one of my equations incorrect during backpropagation which makes my gradients explode?
import numpy as np
from numpy import exp
import matplotlib.pyplot as plt
import h5py

# LOAD DATASET
MNIST_data = h5py.File('data/MNISTdata.hdf5', 'r')
x_train = np.float32(MNIST_data['x_train'][:])
y_train = np.int32(np.array(MNIST_data['y_train'][:,0]))
x_test = np.float32(MNIST_data['x_test'][:])
y_test = np.int32(np.array(MNIST_data['y_test'][:,0]))
MNIST_data.close()

##############################################################################
# PARAMETERS
number_of_digits = 10 # number of outputs
nx = x_test.shape[1] # number of inputs ... 784 --> 28*28
ny = number_of_digits
m_train = x_train.shape[0]
m_test = x_test.shape[0]
Nh = 30 # number of hidden layer nodes
alpha = 0.001
iterations = 3
##############################################################################
# ONE HOT ENCODER - encoding y data into 'one hot encoded'
lr = np.arange(number_of_digits)
y_train_one_hot = np.zeros((m_train, number_of_digits))
y_test_one_hot = np.zeros((m_test, number_of_digits))
for i in range(len(y_train_one_hot)):
  y_train_one_hot[i,:] = (lr==y_train[i].astype(np.int))
for i in range(len(y_test_one_hot)):
  y_test_one_hot[i,:] = (lr==y_test[i].astype(np.int))

# VISUALISE SOME DATA
for i in range(5):
  img = x_train[i].reshape((28,28))
  plt.imshow(img, cmap='Greys')
  plt.show()

y_train = np.array([y_train]).T
y_test = np.array([y_test]).T
##############################################################################
# INITIALISE WEIGHTS & BIASES
params = { "W1": np.random.rand(nx, Nh),
           "b1": np.zeros((1, Nh)),
           "W2": np.random.rand(Nh, ny),
           "b2": np.zeros((1, ny))
          }

# TRAINING
# activation function
def sigmoid(z):
  return 1/(1+exp(-z))

# derivative of activation function
def sigmoid_der(z):
  return z*(1-z)

# softamx function
def softmax(z):
  return 1/sum(exp(z)) * exp(z)

# softmax derivative is alike to sigmoid
def softmax_der(z):
  return sigmoid_der(z)

def cross_entropy_error(v,y):
  return -np.log(v[y])

# forward propagation
def forward_prop(X, y, params):
  outs = {}
  outs['A0'] = X
  outs['Z1'] = np.matmul(outs['A0'], params['W1']) + params['b1']
  outs['A1'] = sigmoid(outs['Z1'])
  outs['Z2'] = np.matmul(outs['A1'], params['W2']) + params['b2']
  outs['A2'] = softmax(outs['Z2'])

  outs['error'] = cross_entropy_error(outs['A2'], y)
  return outs

# back propagation
def back_prop(X, y, params, outs):
  grads = {}
  Eo = (y - outs['A2']) * softmax_der(outs['Z2'])
  Eh = np.matmul(Eo, params['W2'].T) * sigmoid_der(outs['Z1'])
  dW2 = np.matmul(Eo.T, outs['A1']).T
  dW1 = np.matmul(Eh.T, X).T
  db2 = np.sum(Eo,0)
  db1 = np.sum(Eh,0)

  grads['dW2'] = dW2
  grads['dW1'] = dW1
  grads['db2'] = db2
  grads['db1'] = db1
#  print('dW2:',grads['dW2'])
  return grads

# optimise weights and biases
def optimise(X,y,params,grads):
  params['W2'] -= alpha * grads['dW2']
  params['W1'] -= alpha * grads['dW1']
  params['b2'] -= alpha * grads['db2']
  params['b1'] -= alpha * grads['db1']
  return

# main
for epoch in range(iterations):
  print(epoch)
  outs = forward_prop(x_train, y_train, params)
  grads = back_prop(x_train, y_train, params, outs)
  optimise(x_train,y_train,params,grads)
  loss = 1/ny * np.sum(outs['error'])
  print(loss)

```

Apart from the answer given by Aman Shenoy, take a look at this blog post for more details. https://machinelearningmastery.com/exploding-gradients-in-neural-networks/ Cheers!

This problem is called exploding gradients, resulting in an unstable network that at best cannot learn from the training data and at worst results in NaN weight values that can no longer be updated.
One way to assure it is exploding gradients, is if loss is unstable and not improving, or if loss shows NaN value during training.
Apart from the usual gradient clipping and weights regularization that are recommended, I think the problem with your network is the architecture.
30 is an abnormally high number of nodes for 2 layer perceptron model. Try increasing number of layers and reducing nodes per layer. - This is under the assumption that you're experimenting with MLP's, because for the problem above, convolutional neural networks seem like an obvious way to go. If unexplored - definitely check out CNN's  for digit recognition, two layer models will surely work better there.
Hope this helped!

If you were using a non-standard data set, it could for future reference also be useful to look into data preprocessing and normalization to remedy this!

Try using float64 instead of float32; int64 instead of int32; increasing the bits of memory gradually increases the weights that can be stored

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to fill NaNs in Cross-Validation?
I have been searching this but did not find the answer, so sorry if this is a duplicated question.
I was working with cross-validation, where some doubts came to my mind, and I am not sure which is the correct answer.
Let's say I have a mixed dataset, with numerical and categorical features. I want to perform a K-Fold Cross-Validation with it, with a K=10. Some of these numerical features are missing, so I decided that I will replace those NaNs with the average of that feature.
My steps are the following ones:

Read the entire dataset
Perform One Hot Encoding to categorical features.
Divide my data into different folds. Let's say that I will use 90% for training, 10% for validating.
For every different combination of folds, I replace the missing values from the training and validating sets separately. This means, on one hand, I get the average of the missing values of the training part, and on the other hand the average of the missing values of the validating part.
Normalize the data of the training and validating sets between [0, 1] separately, as I did before.
Train the correspondant model.

So let's put a simple example of a dataset of 20 rows with N columns. Once I do steps 1 and 2, in the first iteration I will select the 18 first rows as a training set, and the last two rows as validating set.
I fill the missing values of the 9 first rows with the average of those 18 rows. Then the same for the 2 last rows.
Then, again, normalize in the same way, separately. And do this for every combination of folds.
I am doing it like this, because otherwise, from my understanding, is that you are training your model with biased data. You should not have access to the validation data, thus you should not be able to do the average with those numbers. Hence I am using only the numbers of the training part. If I do the average with the entire dataset, this will make my model overfitting.
I am not so sure about the normalization step, as I do not really think this will have the same impact. But here I do not really know...
Is this approach correct? Or should I do the average and normalization with the entire dataset? Why?

I would do the exact same thing as you are describing! One of the main reasons that you would want to do cross-validation is to prevent that your model is unable to generalize later. Therefore, you take out a random small subset which will be your new small validation set and do all the 'operations' on that which you are also doing on your training set. This way, you can check that these 'operations' are thus also generalizable (if they work for all different cross-validations).


text:
How to define loss function for Discriminator in GANs?
To train the discriminator network in GANs we set the label for the true samples as $1$ and $0$ for fake ones. Then we use binary cross-entropy loss for training.
Since we set the label $1$ for true samples that means $p_{data}(x) = 1$ and now binary cross-entropy loss is:
$$L_1 = \sum_{i=1}^{N} P_{data}(x_i)log(D(x)) + (1-P_{data}(x_i))log(1-D(x))$$
$$L_1 = \sum_{i=1}^{N} P_{data}(x_i)log(D(x))$$
$$L_1 = E_{x \sim P_{data}(x)}[log(D(x))]$$
For the second part, since we set the label $0$ for fake samples that means $p_{z}(z) = 0$ and now binary cross-entropy loss is:
$$L_2 = \sum_{i=1}^{N} P_{z}(z_i)log(D_{G}(z)) + (1-P_{z}(z_i))log(1-D_{G}(z))$$
$$L_2 = \sum_{i=1}^{N} 1-P_{z}(z_i)log(1-D_{G}(z))$$
$$L_2 = E_{z \sim \bar{P_{z}(z)}}[log(1-D_{G}(z))]$$
Now we combine those two losses and get:
$$L_D = E_{x \sim P_{data}(x)}[log(D(x))] + E_{z \sim \bar{P_{z}(z)}}[log(1-D_{G}(z))]$$
When I was reading about GANs I saw that the loss function for discriminator is defined as:
$$L_D = E_{x \sim P_{data}(x)}[log(D(x))] + E_{z \sim P_{z}(z)}[log(1-D_{G}(z))]$$
Should not it be $E_{z \sim \bar{P_{z}(z)}}$ instead of $E_{z \sim P_{z}(z)}$ ?

I doubt whether your derivation is correct. Your are trying to apply binary-cross entropy for data on each label separately, which is not the correct way to do.
The procedure for calculating binary cross entropy is as follows

Pass the input $x$ whose label is $y \in \{0, 1\}$ to your model $M$.
Obtain $\hat{y} \in [0, 1]$ as output of your model $M$ instead of actual label $y$.
Calculate binary cross-entropy loss using the equation

$$L_{CE} = y \log \hat{y} + (1-y) \log (1 - \hat{y})$$
It is true that there are two types of inputs to a discriminator: genuine and fake. Genuine data is labelled by 1 and fake data is labelled by 0. Use the variable $x'$ to represent the input to the discriminator module $D$. If the input $x'$ is genuine then its label is 1 and if your input $x'$ is fake then its label is 0. Note that it is better to avoid the unnecessary details regarding the generator or noise vector while formulating the binary cross-entropy loss of discriminator. Just see discriminator as a module taking two classes of inputs: genuine and fake. Suppose the discriminator outputs $\hat{y} \in [0, 1]$ for the input $x'$ instead of actual label $y \in \{0, 1\}$ then the binary cross-entropy loss is given by
$$L_{CE} = y \log \hat{y} + (1-y) \log(1-\hat{y})$$
$$\implies L_{CE} = y \log D(x') + (1-y) \log(1 - D(x'))$$
Suppose the input $x'$ is a genuine one $x$ then $y = 1$ and
$$\implies L_{CE} = \log D(x)$$
Suppose the input $x'$ is a fake one $G(z)$ then $y = 0$ and
$$\implies L_{CE} = \log (1-D(G(z)))$$
Since the labels are clear from the input of the discriminator $D$, we can write the binary cross-entropy loss for $2m$ samples $\{x_1, x_2, x_3, \cdots, x_m, z_1, z_2, z_3, \cdots, z_m\}$ as
$$\implies L_{CE}^{2m} = \dfrac{1}{2m} \sum\limits_{i = 1}^{m} \log D(x_i) +  \sum\limits_{i = 1}^{m} \log (1-D(G(z_i)))$$
Later, we need to perform some mathematical analysis, which I am not sure about whether it is due to law of large numbers or some other 1, 2 we equate the mean to the actual expectations on probability distribution and hence
$$ L_{CE}^{2m} = \dfrac{1}{2} \sum\limits_{i = 1}^{m} \dfrac{1}{m} \log D(x_i) +  \sum\limits_{i = 1}^{m} \dfrac{1}{m} \log (1-D(G(z_i)))$$
$$ = \dfrac{1}{2} {\LARGE(} \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +  \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))] {\LARGE)}$$
Since removing $\dfrac{1}{2}$ does not matter while optimizing the loss function, the final loss function is given by
$$  L_{D} =  \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +\mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]$$

You are wrong about the binary cross-entropy for GANs. While this is true for the common classification problem, in the GAN we have both real and fake data at the same time. So both labels are correct and equal to 1 (true for real and true for fake). I also wrote it in my answer.

@ArayKarjauv I don't think so. Discriminator has only two labels  1 for genuine $x$ and 0 or fake $G(z)$. If you are sure enough, please point the exact step where the derivation is wrong.

Your conclusion that the loss becomes $L_{CE} = \log D(x)$ when $y = 1$ and $L_{CE} = \log (1-D(G(z)))$ when $y = 0$  is wrong. The intuition is that these 2 terms have different labels. The first one is $y_{real}$, whereas the second is $y_{fake}$, which always equal to 1, hence, the loss is always $\implies L(D, G) =  \log D(x_i) +  \log (1-D(G(z_i)))$.

You can think of it as if the loss were a combination of 2 cross-entropy loss functions: one for real data and one for fake. But since the labels there would be constant, we simplify it.

@ArayKarjauv I can't figure out where can it be wrong. In case of your analysis, it seems to me that there are four labels and in the analysis I did, there are only two labels.


text:
Can any area of math come into play in Machine Learning Research?
As I read online following areas in mathematics come into play in ML research

Linear Algebra
Calculus
Differential Equations
Probability
Statistics
Discrete Mathematics
Optimization
Analytic Geometry
Topology
Numerical and Real Analysis

Can / Are any other areas of math used in ML research? If so what other areas? ex: Number theory

Probably not number theory, except for combinatorics.

Convex geometry, can it be?

Here is a related question.


text:
What is the reason for taking tuples as vectors rather than points?
Across the literature of artificial intelligence, especially machine learning, it is normal to treat the tuples of datasets as vectors.
Although there is a convention to treat them as data points. Treating them as vectors is also considerable.
It is easy to understand the tuples of datasets as points over space. But what is the purpose of treating them as vectors?

They are equivalent. When we consider a particular instance as a vector, we are not literally imagining it as an arrow with it's head at the point coordinates and tail at the origin. It's just when you are working with a tuple of numbers in a mathematical context, it is conventional to call it a vector. This language follows into machine learning which is usually based on associated linear algebra.


text:
What is the purpose of the DAMSM loss for the generators in AttnGAN?
I am confused about the training part in AttnGan.
If you observe page 3. There are two types of losses for generator network: one involving the Deep Attentional
Multimodal Similarity Model (DAMSM) loss $(L_{DAMSM})$ and the others for individual generator $(L_{G_i})$ for $i= 1, 2, 3$.
My doubt is: if each generator has its own loss function that is useful in training, what is the purpose in using $L_G$, i.e., with DAMSM loss function? Is my assumption wrong?


text:
How can a probability density value be used for the likelihood calculation?
Consider our parametric model $p_\theta$ for an underlying probabilistic distribution $p_{data}$.
Now, the likelihood of an observation $x$ is generally defined as $L(\theta|x) = p_{\theta}(x)$.
The purpose of the likelihood is to quantify how good the parameters are. How can the probability density at a given observation, $p_{\theta}(x)$, measure how good the parameters $\theta$ are?
Is there any relation between the goodness of parameters and the probability density value of an observation?

The probability density is used to 'measure how good' the parameters are because it is a natural way of quantifying if these parameters are good for the observed data.
Also, as the notation often causes some confusion, $L(\theta | x)$ denotes the probability of all of your observed data, not just one value. Also the "$|$" may cause confusion as it looks like we are conditioning on $x$ but this is not the case - it may be better practice to use $L(\theta; x)$ which is the notation used when I was learning likelihood. Further, as you have written $L(\theta | x) = p_\theta(x)$ I would like to clarify that if we are being precise in our definitions then this is only correct if you have one observed data point, assuming that you meant $p_\theta(x)$ is the density of $X$.
In my example below I use $p_\mu(x)$ to denote the density of the normal distribution with mean parameter $\mu$, but the density of the likelihood is the product of all the densities (because we assumed iid data). It is crucial that you understand that in general the likelihood is the probability of your observed data and not just the density or the product of densities as you may not always have iid and so it will not always boil down to taking the product of some densities.
The idea of Maximum Likelihood is to maximising the (log-)likelihood for a given set of data. This means that we need to choose a probability distribution that is parameterised by some parameters $\boldsymbol{\theta}$ and then optimise the parameters such that the likelihood is maximum.
Assuming we don't remove any constants then this makes intuitive sense as maximising the likelihood would be maximising the probability that the data came from this distribution -- i.e. the data we observed is most likely to have come from the given distribution with the given parameters. This is important as it means we then have the most likely model of our data which allows us to use this distribution to make inference about our data.
As an example, imagine if I had some iid data $x_1, x_2, ..., x_n \sim \mathcal{N}(0, 1)$. Now I could try to fit a $\mathcal{N}(\mu, 1)$ to this data and optimise for $\mu$. If I chose e.g. $\mu = -1000000$ then the likelihood (again assuming we don't remove any constants) would be $\approx 0$. If I chose a value of $\mu = 0.1$ then the likelihood would be much higher because this parameter is closer to the true parameter value.
To see why it is higher, recall that the likelihood for iid data is given by
$$\prod_{i=1}^n p_\mu(x_i)$$
and if we evaluated our likelihood at $\mu=-10000000$ then you're going to be taking the product of lots of numbers that are $\approx 0$ - if you think about the bell curve shape of a Normal distribution that is centred at $-10000000$ with variance 1 then the density at the true $x_i$ values (recalling they are simulated from a unit Normal) would be approx $0$ - whereas if we evaluated the $x_i$ at the density of a Normal distribution centred at $0.1$ then the density will be non-zero and so your likelihood will have a higher value.
To summarise, the density value can be used to measure how good parameters are for a set of data as maximising wrt the parameters is analogous to maximising the probability that your data arose from said distribution.
As an aside, note that the definition of likelihood is the probability of observing your data under some assumed distribution. For discrete random variables this is fine, but for continuous distributions we have to be a more subtle. For any continuous random variable $X$ we have $\mathbb{P}(X=x) = 0$. However, for a very small $\delta$ we can say that
$$\mathbb{P}(x - \frac{\delta}{2} < X \leq x + \frac{\delta}{2}) = \int_{x - \frac{\delta}{2}}^{x + \frac{\delta}{2}} f_X(x) dx \approx \delta p_X(x) \; ;$$
you can think of this approximation by visualising an integral and recalling that an integral represents the area under the curve, and so for small $\delta$ this integral can be approximated by taking the area of a rectangle which is width $\times$ height, where the width is $\delta$ and the height is $f_X(x)$. This justifies our use of the density function for continuous random variables. Note that typically in maximum likelihood we omit any multiplicative constants as they don't depend on the parameters which is what happens with the $\delta$ from this justification.

Let us continue this discussion in chat.


text:
What is the fundamental difference between an ML model and a function?
A model can be roughly defined as any design that is able to solve an ML task. Examples of models are the neural network, decision tree, Markov network, etc.
A function can be defined as a set of ordered pairs with one-to-many mapping from a domain to co-domain/range.
What is the fundamental difference between them in formal terms?

What is the difference between a multiplication operator and a function?

A model as a set of functions
In some cases in machine learning, a model can be thought of as a set of functions, so here's the first difference.
For example, a neural network with an arbitrary vector of parameters $\theta \in \mathbb{R}^m$ is often denoted as a model, then a specific combination of these parameters represents a specific function. More specifically, suppose that we have a neural network with 2 inputs, 1 hidden neuron (with a ReLU activation function, denoted as $\phi$, that follows a linear combination of the inputs), and 1 output neuron (with a sigmoid activation function, $\sigma$). The inputs are connected to the only hidden unit and these connections have a real-valued weight. If we ignore biases, then there are 3 parameters, which can be grouped in the parameter vector $\theta = [\theta_1, \theta_2, \theta_3] \in \mathbb{R}^3 $. The set of functions that this neural network represents is defined as follows
$$
f(x_1, x_2) = \sigma (\theta_3 \phi(x_1 \theta_1 + x_2 \theta_2)) \tag{1}\label{1},
$$
In this case, the equation \ref{1} represents the model, given the parameter space $\Theta = \mathbb{R}^3$. For any specific values that $\theta_1, \theta_2,$ and $\theta_3$ can take, we have a specific (deterministic) function $f: \mathbb{R} \rightarrow [0, 1]$.
For instance, $\theta = [0.2, 10, 0.4]$ represents some specific function, namely
$$
f(x_1, x_2) = \sigma (0.4 \phi(x_1 0.2 + x_2 10.0)) \tag{2}\label{2}
$$
You can plot this function (with Matplotlib) for some values of the inputs to see how it looks. Note that $x_1$ and $x_2$ can be arbitrary (because those are just the inputs, which I assumed to be real numbers).
This interpretation of a model is roughly equivalent to the definition of a hypothesis class (or space) in computational learning theory, which is essentially a set of functions. So, this definition of a model is useful to understand the universal approximation theorems for neural networks, which state that you can find a specific set of parameters such that you can approximately compute some given function arbitrarily well, given that some conditions are met.
This interpretation can also be applied to decision trees, HMM, RNNs, and all these ML models.
A model in reinforcement learning
The term model is also sometimes used to refer to a probability distribution, for example, in the context of reinforcement learning, where $p(s', r \mid s, a)$ is a probability distribution over the next state $s'$ and reward $r$ given the current state $s$ and action $a$ taken in that state $s$. Check this question for more details. A probability distribution could also be thought of as a (possibly infinitely large) set of functions, but it is not just a set of functions, because you can also sample from a probability distribution (i.e. there's some stochasticity associated with a probability distribution). So, a probability distribution can be considered a statistical model or can be used to represent it. Check this answer.
A function as a model
A specific function (e.g. the function in \ref{2}) can also be a model, in the sense that it models (or approximates) another function. In other words, a person may use the term model to refer to a function that attempts to approximate another function that you want to model/approximate/compute.

Any model can be considered to be a function. The term "model" simply denotes a function being used in a particular way, namely to approximate some other function of interest.

Every model is a function. Not every function is a model.
A function uniquely maps elements of some set to elements of another set, possibly the same set.
Every AI model is a function because they are implemented as computer programs and every computer program is a function uniquely mapping the combination of the sequence of bits in memory and storage at program start up, plus inputs, to the sequence of bits in memory and storage, plus output, at program termination.
However, a 'model' is very specifically a representation of something. Take the logistic curve:
$$
f(x) = \frac{L}{1 + e^{k(x-x_{0})} }
$$
Given arbitrary real values for $L$, $k$, and $x_{0}$, that's a function. However, given much more specific values learned from data, it can be a model of population growth.
Similarly, a neural network with weights initialized to all zeros is a function, but a very uninteresting function with the rather limited codomain $\{0\}$. However, if you then train the network by feeding it a bunch of data until the weights converge to give predictions or actions roughly corresponding to some real world generating process, now you have a model of that generating process.

This is an interesting perspective of what a model and functions are, which is slightly different than what my answer says. Essentially, you're saying that a model is a function that describes (or indeed "models") some physical or not phenomenon and that every function that doesn't currently describe anything or is apparently useless is not a "model". In other words, according to you, functions that are not "interesting" (i.e. they don't explain anything that we can relate to in the real world or maybe in a virtual world, such as a game) are not models.

That's a bit an anthropomorphic definition of a model, i.e. it is relative to the human. I don't know if that's a good idea, but I understand your points. In any case, you should note that, if you had a neural network initialized to all zeros, it would probably not be able to learn anything. Read this.

In simple terms, a neural network model is a function approximator which tries to fit the curve of the hypothesis function. A function itself has an equation which will generate a fixed curve:

If we have the equation (i.e., the function), we do not need neural network for its input data. However, when we only have some notion of its curve (or the input and output data) we seek a function approximator, so that for new, unseen input data, we can generate the output.
Training this neural network is all about getting as close an approximation to the original (unknown function) as possible.

"A function itself has an equation which will generate a fixed curve in the dimensions of its input space." I don't see any interpretation of that statement that is correct. A function can be considered to define a subset of the Cartesian product of its input and output space, but not of just its input space.


text:
What's the difference between architectures and backbones?
In the paper "ForestNet: Classifying Drivers of Deforestation in Indonesia using Deep Learning on Satellite Imagery", the authors talk about using:

Feature Pyramid Networks (as the architecture)
EfficientNet-B2 (as the backbone)

Performance Measures on the Validation Set. The RF model that only inputs data from the visible Landsat 8 bands achieved the lowest
performance on the validation set, but the incorporation of auxiliary
predictors substantially improved its performance. All of the CNN
models outperformed the RF models. The best performing model, which we
call ForestNet, used an FPN architecture with an EfficientNet-B2
backbone. The use of SDA provided large performance gains on the
validation set, and land cover pre-training and incorporating
auxiliary predictors each led to additional performance improvements.

What's the difference between architectures and backbones? I can't find much online. Specifically, what are their respective purposes? From a high-level perspective, what would integrating the two look like?

That paper gives the email of authors. Why don't you ask them by email?

Architecture is the entire model.
Backbone is the first few hidden layers. This is containing low-level features which generalize the knowledge. Practically, it's commonly used in pretrain where this backbone weights are frozen (non-updatable). This is make sense since it's trained something general.
Another terms are Neck and Head.
Neck is the intermediary layers between Head and Backbone. It's commonly used in transition for multi-task learning.
While the Head is simply the output layer for certain task.

The vocabulary is definitely non-standard and a bit confusing, but Feature Pyramid Networks is used as a feature extractor, and its output is then fed into EfficientNet-B2 to be used to classify the image. One neural network model is concatenated at the end of the other.
So it seems like "architecture" is the front half of the neural network model which takes as input the satellite image and extracts image features, and then is directly connected to the back half of the model (hence "backbone"), which takes the features extracted from the "architecture" and makes a classification.
This terminology is definitely non-standard here, at least in the AI community, and if you ask anyone here I think it will be uncommon for them to naturally think about the words "architecture" vs "backbone" in this way unless they specialize in a similar field to the authors.

I don't know if it's standard or not, but I've seen the term "backbone" being used in several CV papers. For example, take a look at this famous paper (famous siamese networks) or this other one (mask R-CNN). I am pretty sure there are other examples, so I think that this terminology is not that uncommon, at least, in papers that introduce DL-based CV techniques.

There's also this SO question or this Stats question. In any case, I don't know if this terminology is being used consistently or not across all these papers. I am not familiar with the paper mentioned in this post (and I don't really have time to read it now), but I think that your interpretation of the terms is different than the usage of these terms in other contexts (such as mask R-CNN, although it's been a long time since I skimmed through that mask R-CNN papers).

So, if you have some time, I suggest that you go through these papers (and questions/posts) more carefully and try to understand what they really mean by that and if they are being used consistently across these papers.

Thanks! I'll give it a read when I get the chance

The backbone term may be borrowed from other fields of software development. I work in web development and would understand them differently. The weird thing for me here is that "architecture" seems clearly used in the ai/NN sense, whilst "backbone" seems to be used in a more general software Dev sense.

I've taken an NVIDIA course on the portal and it said that ResNet, VGG, GoogleNet were model architectures , and that DetectNet_V2,FasterRCNN,SSD, UNET were model backbones, so I think it's a common terminology

It's actually the opposite for me. ResNet, VGG and GoogleNet are convolutional backbones, while FasteRCNN, SSD and DetectNet are typical Object Detection architectures, and UNET is a FC architecture for image-to-image tasks (e.g object segmentation).


text:
Why is my GAN more unstable with bigger networks?
I am working with generative adversarial networks (GANs) and one of my aims at the moment is to reproduce samples in two dimensions that are distributed according to a circle (see animation). When using a GAN with small networks (3 layers with 50 neurons each), the results are more stable than with bigger layers (3 layers with 500 neurons each). All other hyperparameters are the same (see details of my implementation below).
I am wondering if anyone has an explanation for why this is the case. I could obviously try to tune the other hyperparameters to get good performance but would be interested in knowing if someone has heuristics about what is needed to change whenever I change the size of the networks.

Network/Training parameters
I use PyTorch with the following settings for the GAN:
Networks:

Generator/Discriminator Architecture (all dense layers): 100-50-50-50-2 (small); 100-500-500-500-2 (big)
Dropout: p=0.4 for generator (except last layer), p=0 for discriminator
Activation functions: LeakyReLU (slope 0.1)

Training:

Optimizer: Adam
Learning Rate: 1e-5 (for both networks)
Beta1, Beta2: 0.9, 0.999
Batch size: 50

How many data points did you used? Maybe increasing dataset n times reduces this effect.

I use 100k data points in total for training. With a batch size of 50, that's 2000 updates for each epoch, and I have 600 epochs. I think 100k is big enough so that increasing it shouldn't change too much.

Maybe the bigger network is perceiving more random noise and replicating it better. Just to be sure, I'd take it easy with the dropout and/or increase the batch size.

Generally I think I agree with Andre above. Out of curiosity, have you tried increasing the number of layers instead of increasing the number of neurons in each layer?

Variance
What it seems to be taking place here is a form of overfitting. Specifically when using larger layers, the model may become too complex and start to fit the noise in the training data, rather than the underlying patterns. This can lead to poor generalization performance and, as seen here unstable behavior.
Other
Other complexities that may arise when using larger layers is diversity issues. Specifically, more complex GAN models may produce less diverse output from the generator, as the network is prone to get stuck in a particular mode of the data distribution and fail to explore other areas.
To address this issue, regularisation techniques such as dropout or noise introduction can be deployed to encourage the generator to produce more diverse output. Additionally modifying the loss function can encourage the generator to explore different modes of the data distribution.


text:
What are the purposes of pooling in CNNs?
There are at least three questions on this site related to this

What is the effect of using pooling layers in CNNs?
Is pooling a kind of dropout?
What are the benefits of using max-pooling in convolutional neural networks?

I got the following useful information regarding the purpose of pooling. As per my understanding, the purposes of pooling, based on priority, in general, are as follows:

To decrease the size of the feature maps
To make the model stronger in feature extraction

Are there any other purposes of pooling in CNN other than them?

Without pooling, how do you recommend making the image smaller?

@user253751 It can be one of the purpose.


text:
Why does this paper say that the Nash-equilibrium of GAN is given by a discriminator which is 0 everywhere on the data distribution?
I am facing difficulty in understanding the bolded portion of the following statement from this paper

GANs are defined by a min-max two-player game between a discriminative network $D_\Psi(x)$ and generative network $G_\theta(z)$. While the discriminator tries to distinguish between real data point and data points produced by the generator, the generator tries to fool the discriminator. It can be shown that if both the generator and discriminator are powerful enough to approximate any real-valued function, the unique Nash-equilibrium of this two-player game is given by a generator that produces the true data distribution and a discriminator which is 0 everywhere on the data distribution.

My understanding is that discriminator gives $\dfrac{1}{2}$ for any further inputs after training. But, what is the $0$ mentioned?

What this paper is not saying is that the discriminator, $D_{\phi}$, always returns a scalar value of zero. What they are saying is that the generator, $G_{\theta}(z)$, has accurately learned the distribution of the input data, and that the discriminator produces the correct answer for each input from the generator. It's a mathematical description of the global optimum for a GAN. This Medium article by Jonathan Hui talks more about Nash equilibria, the Kullback-Leibler, and more.

Still cannot understand. So it means the generator produces images from real distribution and the discriminator label anything from the generator (also from the true images) as fake?


text:
What model to train to restore MNIST test dataset
I came across this problem, and am not sure where to start. What model would work best for this problem and why?
Imagine the digits in the test set of the MNIST dataset (http://yann.lecun.com/exdb/mnist/) got cut in half vertically and shuffled around. Implement a way to restore the original test set from the two halves, whilst maximizing the overall matching accuracy.

What have you tried so far to solve this problem? Have you found some solution online? Share with us your findings so far.

So far i have found nothing. I tried to implement resnet38 to classify each of the halves as a number, but then im not sure how to attach two 1's or two 2's.
Also resent38 gives bad predictions


text:
What is meant by "two action selections" in SARSA?
I have some difficulties understanding the difference between Q-learning and SARSA. Here (What are the differences between SARSA and Q-learning?) the following updating formulas are given:
Q-Learning
$$Q(s,a) = Q(s,a) + \alpha (R_{t+1} + \gamma \max_aQ(s',a) - Q(s,a))$$
SARSA
$$Q(s,a) = Q(s,a) + \alpha (R_{t+1} + \gamma Q(s',a') - Q(s,a))$$
I know that SARSA is for on-policy learning while Q-learning is off-policy learning. So, in Q-learning, the epsilon-greedy policy (or epsilon-soft or softmax policy) is chosen for selecting the actions and the greedy policy is chosen for updating the Q-values. In SARSA the epsilon-greedy policy (or epsilon-soft or softmax policy) is chosen for selecting the actions and for updating the Q function.
So, actually, I have a question on that:
On this website (https://www.cse.unsw.edu.au/~cs9417ml/RL1/algorithms.html) there is written for SARSA

As you can see, there are two action selection steps needed, for determining the next state-action pair along with the first.

What is meant by two action selections? Normally you can only select one action per iteration. The other "selection" should be for the update.

In my view, the best way to understand these algorithms is to read the pseudocode (multiple times, if necessary!).
Here's the pseudocode of Q-learning.

Here's the pseudocode of SARSA.

So, as you can see, in SARSA, we choose one action before the episode starts, and, during the episode, we choose (and take) again more actions. In both cases, we choose these actions with the same policy (e.g. $\epsilon$-greedy), which is derived from $Q$. In Q-learning, we do not choose an action before the episode starts. We only choose and take an action at each step of the episode (like in SARSA). Hence, in SARSA, we choose actions in two places (but only take an action at each step of the episode). Note the difference between choosing/selecting an action and taking an action in the environment (you may just choose an action to update the Q-function, i.e. without taking it into the environment!).

Thanks for your answer nbro. Unfortunately I have to admit that I still have problems understanding it and I have several follow-up questions. 1) Why are we choosing more than 1 action in SARSA? One for going into the next state and the other one for updating the Q function? 2) In SARSA do we only select one action at the very beginning and then we always choose the same action for each step? Does it really make sense to choose the same initally choosen action a regardless of the sate s

Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm.  4) How can we calculate the Q(s', a') in both SARSA and Q-learning for updating the Q-function. After having taken an action a at state s we get the reward r which we can observe. But we cannot observe Q(s',a') from the environment as far as I see it.

@PeterBe Create one post for each of these specific questions and I or someone else will answer them. Please, really only one question per post and provide the context for each question in their separate post.

Thanks for your answer and effort. I understand that generally each question should be asked in a separate post. However, at least the first 2 questions of mine are direct follow-up questions to your answer. It does not make sense to ask question 1) and 2) in a separate post, because I would exactly ask the same think as in this post (question 1) or react directly to something you mentioned in your answer (question 2).

@PeterBe In this answer, I am answering the question "What is meant by two action selections?" in the context of SARSA. This is actually different than explaining why we have 2 action selections in SARSA. If it's a follow-up question or not, it doesn't matter. Your questions could be useful for future readers, that's why I am asking you to ask each of your specific questions in their separate post. More people might have those same questions (but not exactly the same combination of questions: that's why it's important to ask each specific question in a separate post).

Thanks for your comment. Actually my first 2 questions were also targeting at "what" are the 2 actions. I still don't understand what they are. As asked in my question 1) is one action for going into the next state and the other one for updating the Q function? and as asked in question 2) what is the first action really about? Is it one action that will not change througout all episodes? What is the first action doing and what is the second action doing? I have problems understanding this from your answer. This is why I think that at least those 2 questions are directly related

@PeterBe I will answer all your questions, if you just ask them in a separate post.


text:
Are there any recommendations on initialising a single parameter in deep learning?
I want to initialize a parameter, which is a single real number in my model. If you want the role of the parameter in the model, you can assume it as the parameter to multiply with the output of the neural network and the resultant product will be the final output.
Does initialization value matter? If yes, are there any guidelines on initializing the single parameter?

Initialization always matters. There's countless papers on parameter initialization, however a common method (last I checked) is to use Kaiming initialization. In this case since it's only one parameter it should be fairly simple to initialize using this method

In this case, the most suitable initialization may also depend on the specific role of this parameter. You say you multiply it with the output of the model, but why are you doing this? Maybe you can provide more details about that.

Ha @nbro I will try to do.


text:
What are the types of inputs used for RNN in literature given sentences?
Suppose there are $m$ sentences in a text file and the number of distinct words is equal to $n$. The goal is to get word embeddings using RNN.
We know that it is impossible to pass any word, which is in text format, as an input to RNN. We need to convert each word into some number and then pass it to the RNN to get word embeddings.
I know only the following method, if correct:

Assign an index to each word. So, the index ranges from $0$ to $n-1$.
Use the indices as input to RNN.

Is it the only technique used in the literature? If not, what are the names of other techniques that are used in the context of RNN encoders?

I don't understand why you're asking this question. What's the problem with having a one-to-one mapping between integers and words in the vocabulary in order to retrieve the embeddings?

@nbro I don't know whether it is the only method or it is one of the method of passing text input to rnn.

@nbro I want to know if there exist other methods.


text:
Does the Frechet Inception Distance (FID) consider color?
I was wondering if the Frechet inception distance
for two colored datasets would be the same than the FID calculated for the same datasets converted to grayscale.
I know that it depends on the feature extraction, which is the Inception network. And that is the question, I don't fully understand the role of color in CNNs. I thought that all color information is lost in the feature extraction, and in this case I grayscale and colored datasets would produce the same FID. But I am not sure about that.

Can you please provide us with a link to an article that talks about this "Frechet Inception Distance". I know it exists, but not all people may know what exactly you're referring to. Moreover, it may be a good idea to tell us what you know about the FID. What is it usually used for? What's the equation to compute it? Generally, it's a good idea to do a little bit of research before asking a question.

Hi, I added more info


text:
Why batch normalization before upsampling is giving worse results?
I am training a model to generate images.
The model contains 5+5 layers:
Conv2D -> Upsample -> Conv2D -> Upsample -> Conv2D -> Upsample -> Conv2D -> Upsample -> Conv2D -> Upsample

I am modifying it as
Conv2D -> BatchNorm -> Upsample -> Conv2D -> BatchNorm -> Upsample -> Conv2D -> BatchNorm -> Upsample -> Conv2D -> BatchNorm -> Upsample -> Conv2D -> BatchNorm -> Upsample

I am applying the batch normalization layers just before upsampling as shown above and hence I am not getting the results that are at least comparable to the results by the model without any batch normalization layer.
Is my placement of the batch normalization layer wrong? If yes, then why?

I don't know if it makes any difference, but it may be a good idea to tell us which specific model you're using.

@nbro Okay, I will update today.


text:
Textbook for CNN-LSTM networks of predictions of numerical data
I am learning NN algorithms because I'd like to create my own project.
What I found on the internet, is that for my type of project which I have in mind CNN-LSTM neural network would be ideal.
But now I have a question - I don't know if it's against the rules of this forum or not. So pardon me if I violated them.
So, now I am learning NN algorithms from a couple of books that "classify" them like: Classification NN, LSTM, Convolutional - each neural network is a separate topic in each book.
But I am looking for a book that teaches the reader about Convolutional Long-Short Term Memory Neural Network. Does someone know such a book where such hybrid NN is the main topic?

Hello, welcome to AI stack exchange. Asking for textbooks related to AI is on-topic here. So, no need to worry about this question. Read here for more details.

I edited the tile of your question, please re-edit if you are not comfortable with it.

It's difficult to find a textbook that contains all hybrid neural networks. Since the field is changing so frequently, even if someone writes a book about it, it will likely become obsolete in a few years.
There is actually a nice paper (with lots of citation) that developed the network you mentioned:
Convolutional LSTM Network: A Machine Learning
Approach for Precipitation Nowcasting
You are more likely to find different kinds of neural network architectures in journal or conference papers. Hope this helps!


text:
Can I apply reparametrization trick on "any" deep neural network?
I came across the  "reparametrization trick" for the first time in the following paragraph from the chapter named Vector Calculus from the test book titled Mathematics for Machine Learning by Marc Peter Deisenroth et al.

The Jacobian determinant and variable transformations will become
relevant ... when we transform random variables and probability
distributions. These transformations are extremely relevant in machine
learning in the context of training deep neural networks using the
reparametrization trick, also called infinite perturbation analysis.

The trick has been used in the context of neural networks training in the quoted paragraph. But when I search about the reparametrization trick, I found it only or widely in training autoencoders.
In the context of training a traditional deep neural network, is the trick useful?

The reparameterization trick (also known as the pathwise derivative or infinitesimal perturbation analysis) is a method for calculating the gradient of a function of a random variable. It is used, for example, in variational autoencoders or deterministic policy gradient algorithms.
If you plan on working with models that involve random variables, you definitely need to understand what the reparameterization trick is.
You will also need to understand the other method to calculate gradients for functions of random variables, which is known as the likelihood ratio (also known as the score function or  the REINFORCE gradient).
If your definition of a "traditional" neural network does not involve random variables, then such a method is irrelevant.

Note that the re-parametrization trick is used to reduce the variance (i.e. it's a variance reduction technique). The authors of the VAE state this in the VAE paper (if I remember correctly). So, is the re-parametrization really needed or only needed to reduce the variance? Moreover, are you sure that "REINFORCE gradient" is generally a synonym for "likelihood ratio"? Maybe in the context of reinforcement learning and the REINFORCE algorithm, but isn't the likelihood ratio something more general?

Yes, REINFORCE gradient definitely means the same thing as score function or likelihood ratio (at least when the latter two are talking about gradient estimation).

The reparameterization trick has lower variance than likelihood ratios. I guess you could call it a variance reduction technique in this sense (assuming that either method can be applied), but typically variance reduction is referring to the use of some sort of control variate.

I agree with @nbro, REINFORCE gradient is not synonymous with the likelihood ratio. The likelihood ratio is something extremely general from statistics that has been around much longer than REINFORCE.

https://arxiv.org/abs/1506.05254 page 3, nomenclature, "What we call the score function
estimator (via [3]) is alternatively called the likelihood ratio estimator [5] and REINFORCE [26]."

@Taw The point is: REINFORCE is a reinforcement learning algorithm, while the term "score function" is used more generally. Do you understand what we want to say? The score function appears e.g. in variational inference too.  Saying that REINFORCE is the score function is similar to saying that gradient descent is the chain rule. No. Gradient descent is an algorithm that uses a specific version of the chain rule applied to neural nets. So, REINFORCE is an algorithm, it's not a score function. It uses a score function.

See my answer here, for instance, and the linked paper about Monte Carlo gradient estimation.

Yes, the reparametrization trick can be useful in the context of variational Bayesian neural networks, although other more effective variance reduction techniques are more commonly used (in particular, the flipout estimator). See this implementation of BNNs that uses Flipout, but TensorFlow Probability, the library used to implement that example, also provides layers that implement the reparametrization trick.
Note that the reparametrization trick is used in the context of variational auto-encoders (VAEs) (so not in the context of deterministic auto-encoders). VAEs and BNNs have a lot in common: both are based on stochastic variational inference (i.e. variational inference combined with stochastic gradient descent). So, whenever you have some sampling or some stochastic operation, the reparametrization trick could turn out to be useful. However, right now, I am only familiar with these two types of models that use it.


text:
Using GAN's to generate dataset for CNN training
I'm doing bachaleor thesis on traffic sign detection using single shot detector called YOLO. These single shot detectors can perform detection of objects in image and so they have specific way of training, ie. training on full images. Thats quite problem for me, because the biggest real dataset with full traffic sign images is Belgian one with 9000 images in 210 classes, which is unfortunately not enough to train good detector.
To overcome this problem, I've created DatasetGenerator, which does quite good job in generating synthetic datasets, you can see in the results directory.
Recently I came across GAN's which can (besides others) generate or extend existing dataset and I would like to use these networks to compare with my dataset generator. I've tried this introduction to GANs succesfully.
The problem is it's unsupervised learning and so there are no annotations. It means it's able to extend my dataset of traffic signs, but the generated dataset won't be annotated at all, which is problem.
So my question is: Is there any way how to use GAN's to extend my dataset of full traffic sign images with annotations of traffic sign class and position? Actually the class is not important, because I can do it separately for each class, but what matters is the position of traffic sign in generated image.

I think you'll enjoy this work from Apple on improving the realism of synthetic images.  Essentially what you need to do is generate a synthetic image then have your GAN modify the synthetic image so that a 1) a discriminator thinks it is real while also 2) not changing the gross structure of the image very much (so the traffic sign doesn't move) - yes, this loss function is going to take a little work!
Making synthetic data realistic enough to allow models to generalize successfully in the real world is a very active and exciting area of research, not least with respect to robotics, and so the work you are doing now should make you very attractive indeed to the right employer.

That's amazing!

Glad you liked it!  I would be most interested in reading your thesis when you are finished - I think there is a good chance that lots of other folk will want to read it too.

Indeed, I think these dataset generative tools/papers would help a lot of guys at least training some nets. Unfortunately after reading work from Apple Iam not sure I can do such complex system in just few months but I'll do my best and let you know :-)

A Floridian MSc student by the name of Connor Shorten - active on Twitter, Medium, etc. - has very similar interests.  Probably you two should have a Skype call.  I think you under-estimate yourself (and also, since the Apple work, there'll be many others on the same road).

You could add the desired traffic sign location to the latent vector and then arrange that the generator incurs loss if the traffic sign is not at the right place in the generated image.

If I knew position of traffic sign in the generated image, I could have just use that position as annotation of image. But since it's generated image based on let's say 1000 real images, I down know position of traffic sign in the generated image.

The sign location could be random and the generator's task would be to generate the sign at that particular location. This would be similar to a conditional GAN: instead of the desired class label (CGAN), here you would  supply the desired object location.

Thats nice idea, but still the YOLO detection network learns (besides others) postion of the object in the full image. That means if all the images in the training dataset would have traffic sign on the same place, YOLO would learn that feature and then, most-likely, would detect traffic signs in that desired location only (but I am not entirely sure with that tho). I will give it a try and we shall see :)


text:
Traffic signs dataset
I'm looking for annotated dataset of traffic signs. I was able to find Belgium, German and many more traffic signs datasets. The only problem is these datasets contain only cropped images, like this:

While i need (for YOLO -- You Only Look Once network architecture) not-cropped images.

I've been looking for hours but didn't find dataset like this. Does anybody know about this kind of annotated dataset ?
EDIT:
I prefer European datasets.

Direct Answer
The Belgium TS Dataset may be helpful, as well as The German Traffic Sign Detection Benchmark.
Additional Notes Based on Question Author's Idea
The idea in the question author's addendum of placing signs onto street sides and corners is a good one, but to do it repeatably and in a way that doesn't bias the training is its own research project.  However, it is a good research direction.  What would be of benefit to AV researchers worldwide is a multi-network topology and equilibrium strategy with the objective to create the following data generation features.

Street sign symbol inputs in image form, with or without cropping, as movie frame sequences or single still shots, or from SVG files.
Annotation generation using partially human-labelled data.
3D analysis of sign angle and perspective setting so that the images appear exactly as they would from a vehicle's imaging system.
Matching of lighting between the superimposed sign and the background scene.
Automatic blue-screening for the sign image.

This is obviously not a basic data hygiene problem. It is its own AI project, but the return on this research project in terms of furthering the AV technology is immense and may have significant data set statistical advantages over collecting data from the vendors that supply images to Google maps and other Big Data aggregators.

I thought the German/Belgium dataset contains only cropped traffic signs, but recently i found out there are also datasets for detection purposes. Thank you for your answer. I think these datasets are going to do the work.

Really appreciate additional notes @Douglas Daseeco. Definitely you have got the point. The 3D analysis and matching of lighting came also to my mind. As you said, it's best for later down the road. Now I will try to train models on datasets mentioned in your answer and hope the results will be good. Then i would like to work on this artifical dataset generator and compare both models -- one trained on real dataset, another one on artifical.

Look at Google's Open Image Dataset @ https://storage.googleapis.com/openimages/web/index.html
They provide image-level labels, object bounding boxes, object segmentation masks, and visual relationships.
Here is the link for the traffic signs dataset.

Check this one by UCSD.
It contains both video as well as images related to traffic signs. The annotations are present in csv

It seems like great annotated dataset. Actually i forgot to mention i prefer european traffic signs. Still its better than nothing, thank you.

My bad, I assumed you were looking for the US signs. Let me check if I can find European ones.

Actually I've made simple "dataset generator", which places cropped signs into images of road, on the positions where signs usually are placed. It also create annotation file at the same time. You may check it here: https://github.com/kocica/training-dataset-generator, in the results folder. The results of neural network learned on this pseudo-dataset are surprisingly pretty well.

I searched the web but there are no such dataset published but Check this out


text:
Getting better results in improving the configuration
Currently, I found the right recipe for a time series regression problem to finally get acceptable to good results.
Here is the config file
{
    "data": {
        "sequence_length":45,
        "train_test_split": 0.85,
        "normalise": false,
        "num_steps": 10
    },
    "training": {
        "epochs":30,
        "batch_size": 32
    },
    "model": {
        "loss": "mse",
        "optimizer": "adam",
        "layers": [
            {
                "type": "lstm",
                "neurons": 161,
                "input_timesteps": 45,
                "input_dim": 161,
                "return_seq": true,
                "activation": "relu"
            },
            {
                "type": "dropout",
                "rate": 0.1
            },
            {
                "type": "lstm",
                "neurons": 161,
                "activation": "relu",
                "return_seq": false
            },
            {
                "type": "dense",
                "neurons": 128,
                "activation": "relu"
            },
            {
                "type": "dense",
                "neurons": 1,
                "activation": "linear"
            }
        ]
    }
}

Here is the results I got

What can be good improvements I can bring to my model so that I can get better results? There are a lot of small spikes and other places where the curve is rather constant that I should get a rise or fall of the curve.


text:
Price difference predictions curve almost vanished
With a team, we are studying how it is possible to predict the price movement with high-frequency. Instead of predicting the price directly, we have decided to try predicting price difference as well as the features. In other words, at time t+1, we predict the price difference and the features for time t+2. We use the predicted features from time t+1 to predict the price at time t+2.
We got very excited, because we thought getting good results with the following graph

We got problems in production and we wasn't known the problem till we plot the price difference.

Here is the content of the config file
{
    "data": {
        "sequence_length":30,
        "train_test_split": 0.85,
        "normalise": false,
        "num_steps": 5
    },
    "training": {
        "epochs":200,
        "batch_size": 64
    },
    "model": {
        "loss": "mse",
        "optimizer": "adam",
        "layers": [
            {
                "type": "lstm",
                "neurons": 51,
                "input_timesteps": 30,
                "input_dim": 101,
                "return_seq": true,
                "activation": "relu"
            },
            {
                "type": "dropout",
                "rate": 0.1
            },
            {
                "type": "lstm",
                "neurons": 51,
                "activation": "relu",
                "return_seq": false
            },
            {
                "type": "dropout",
                "rate": 0.1
            },
            {
                "type": "dense",
                "neurons": 101,
                "activation": "relu"
            },
            {
                "type": "dense",
                "neurons": 101,
                "activation": "linear"
            }
        ]
    }
}

Prices don't change very fast. Therefore, the next price is almost always very close to the last price. In other words, P_{t+1} - P_{t} is very often close to zero or zero directly. If there is too many zeros then the network will only recognize the zeros. The model has picked up on that.
I guess the model learned almost nothing except the very simple relationship that the next price is close to the last price. There is not necessarily anything wrong with the model. Predicting stock prices should be a very hard problem.
So a straightforward improvement should be of taking the features as a whole instead of their difference.
I want to keep working with price difference instead of the price in itself because we are making the series potential more stationary.
What might be a good solution to deal with the repetitive zeros related to our "price difference" problem? Does applying the log-return is a better idea than applying price differences?
Does a zero inflated estimators is a good idea? First predict whether it's gonna be a zero. If not predict the value. https://gist.github.com/fonnesbeck/874808 ?

I have encountered similar problem while trying to predict forex prices. I understand it this way:

The data based on which you try to model price differences are so poor that the lowest error is achieved by zeroing the predicted values

In other words, due to poor "correlation" in data, zero values are the best solution.
My advice would be to look for more correlated data to be used.


text:
IQN bellman target: using Z vs using Q
IQN paper (https://arxiv.org/abs/1806.06923) uses distributional bellman target:
$$ \delta^{\tau,\tau'}_t = r_t + \gamma Z_{\tau'}(x_{t+1}, \pi_{\beta}(x_{t+1})) - Z_{\tau}(x_t, a_t) $$
And optimizes:
$$ L = \frac{1}{N'} \sum^{N}_i \sum^{N'}_j \rho^\kappa_{\tau_i} \delta^{\tau_i,\tau_j}_t $$
But similar quantiles can be got just from Q values, when doing so:
$$ \delta^\tau_t = r_t + \gamma \frac{1}{N'} \sum_{j}^{N'} Z_{\tau_j}(x_{t+1}, \pi_{\beta}(x_{t+1})) - Z_\tau(x_t, a_t) \\ = r_t + \gamma Q (x_{t+1}, \pi_\beta(x_{t+1})) - Z_\tau(x_t, a_t) $$
optimizing:
$$ L = \sum^N_i \rho^{\kappa}_{\tau_i} \delta^{\tau_i}_t $$
Both lead to similar performance on CartPole env. The loss function of the 2nd one is more simpler and intuitive (atleast to me). So i was thinking if there are any obvious reason why authors didin't use it?

Replacement you suggest is replacement of random variable by its expectation in forward part of TD. It would make IQN into modification of C51 with randomly sampled function approximator instead of discrete distribution. Both distribution produced and especially exploration behavior with your replacement would be very different. The authors of paper explicitly said that "more randomness" in their opinion benefit training, so reducing randomness would go aginst spirit of the paiper.  That they produce similar results on single toy test mean very little. IQN could be better then C51 or it could be worse then C51 but single toy example is not enough to say they are close. Nevertheless I agree that IQN looks overly complex and may require more training time, C51 approach could be more practical.


text:
Keywords to describe people counting from a camera?
The subject matter is to count the number of people in a large room, wherein a camera is placed in a very high ceiling: an example would be Grand Central Station.   Faces are not visible: the scalp (top of the head) is visible to the camera as shown in the link's video.
The goal: I would like to perform a Google literature search to assess the work that has been performed on overhead head recognition,  however, I am not sure what the best  keyword pairs: (scalp? head? people?) to describe the object that is to be recognized from a camera positioned in the ceiling (overhead? bird's eye? satellite?).  I'd like the search to return leading-edge (AI) techniques that benchmark results

Welcome to SE:AI!  At present, this question seems to be about keywords used in search.  Can you elaborate on how this relates to an AI algorithm?

Good question: image recognition (counting heads from a photo) is a class of problems for AI.  Hopefully my question is appropriate for this forum.

I usually start with some papers and look at the references:

Counting people using video cameras
Sheng-Fuu Lin, Jaw-Yeh Chen, Hung-Xin Chao, Estimation of Number of People in Crowded Scenes Using Perspective Transformation,  IEEE Transactions on Systems, Man and Cybernetics, November 2001, Part A, Vol. 31, Issue 6, pp. 645-654.
A.  C.  Davies,  J.  H.  Yin,  and S.  A.  Velastin,  Crowd monitoring using image processing,  Electronics  & Communication Engineering Journal, February  1995, Vol. 7, Issue 1, pp. 37-47.

By this technique you only find older work.
To find more recent articles, use Google scholar and find who cites the articles you investigated.
Always write them down (e.g. in your bibtex file) to keep track of what you looked at already.
Good luck!


text:
Cropping image using ML?
Is there a way to train AI to find aspecific line or symbol in a image and crop it?
OpenCV scripts finds a face and crops it: how can I add my annotations?
Lest say I have a image like this:

 +------+
 |  *  *|
 |      |
 |  *  *|
 |      |
 +------+

I want it to find the * and crop it.

I very much like simple problems like these, and have no doubt ML could be effectively applied to this.  Which it wouldn't be the most efficient solution to the stated problem, I could see how it would be very useful when extended to similar, non-trivial problems. Welcome to AI!

Yes, this is possible.
There is actually a pretty easy way that doesn't even require machine learning and can be implemented with a small amount of code. You just use a framework for image processing (e.g. PIL for Python), find the marks by going over your image with an appropriate filter and use the implemented crop function, that the framework hopefully provides (or write i yourself with 2 nested for loops copying the area). Your filter will react strongest in the areas of your image that look like the mark you defined. This could actually be implemented like a single layer conv+RELU layer in a CNN (see this introduction for details about conv and RELU layers).
Because you are asking explicitly about machine learning, I will also give you the hard version to solve this problem using a full CNN to identify the bounding box. Once you have found the bounding box, the actual cropping can be done like in the example above and is not part of the CNN. You could call it post processing if you like.
The technique you are looking for is called Object Localization and Detection. The linked article should tell you all the details you need. You basically build and train your CNN to localize and identify interesting objects in your image. The easy way is to train your CNN to identify the 4 marks individually and return the coordinates of the 4 areas with the highest likelihood to contain your marks. You calculate the center of those 4 individual points and feed it to your cropping function. The hard way is to find the smallest bounding box containing all 4 marks. This is a much harder problem, because the neural network needs to learn that it should ignore everything except those 4 marks during detection, although they will be a small portion of the bounding box. It can be done, but looking for the 4 marks individually would be much easier.
Which approach is best for you? Depends on your goal. If you need a robust and efficient algorithm for this problem, forget about ML and implement it with the straight forward approach I described first. No need for fancy learning CNNs. You can implement this with a few dozen lines of code.
If this is a research or training project to study ML, take my second approach. The easy and the hard way are both fine, the hard one obviously more challenging for your CNN. If you are new to CNN, I recommend following this Stanford Course. It teaches you all you need for your project in 25 to 30 hours (excluding homework).

Edit: Concerning your question, how do you add your own marks. I would simply do it using with the same framework you use for cropping. Your program needs some kind of image manipulation capabilities and this will allow pixel manipulation as well (at least the framework PIL which I suggested has this capability). Just draw your mark using this method. You can also copy a sample mark with a transparent background using a tool like Gimp manually, if you prefer that.

If you are so sure that ML is the way to go ahead, then try following.

Tensorflow code for object detection
YOLO
This is especially for real time object detection.

You can train these model for your object in question & see the results.There are some other localization models that you can try.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
How to improve testing accuracy when training accuracy is high?
Following-up my question about my over-fitting network
My deep neural network is over-fitting :

I have tried several things :

Simplify the architecture
Apply more (and more !) Dropout
Data augmentation

But I always reach similar results : training accuracy is eventually going up, while validation accuracy never exceed ~70%.
I think I simplified enough the architecture / applied enough dropout, because my network is even too dumb to learn anything and return random results (3-classes classifier => 33% is random accuracy), even on training dataset :

My question is : This accuracy of 70% is the best my model can reach ?
If yes :

Why the training accuracy reach such high scores, and why so fast, knowing this architecture seems to be not compatible ?
My only option to improve the accuracy is then to change my model, right ?

If no :

What are my options to improve this accuracy ?

I'v tried a bunch of hyperparameters, and a lot of time, depending of these parameters, the accuracy does not change a lot, always reaching ~70%. However I can't exceed this limit, even though it seems easy to my network to reach it (short convergence time)
Edit
Here is the Confusion matrix :

I don't think the data or the balance of the class is the problem here, because I used a well-known / explored dataset : SNLI Dataset
And here is the learning curve :

Note : I used accuracy instead of error rate as pointed by the resource of Martin Thoma
It's really ugly one. I guess there is some problem here.
Maybe the problem is that I used the result after 25 epoch for every values. So with little data, training accuracy don't really have time to converge to 100% accuracy. And for bigger training data, as pointed in earlier graphs, the model overfit so the accuracy is not the best one.

Did you have a look at chapter 2.5 of https://arxiv.org/abs/1707.09725 ? Please post the results of those analysis techniques, especially the learning curve and the confusion matrix.

I did read it, but I didn't apply it since I didn't understand all. I will try to apply it and come back here with the results.

If something is unclear, let me know.

I identified the origin of this overfitting..
Origins
I tried a lot of models, putting more and more dropout, simplifying as much as I could.
No matter what I did, after a few epoch of good learning, invariably my loss function was going up. I tried simpler and simpler models, always the same overfitting behavior.
What bugged me at that moment is that no matter what kind of model I used, how deep or how complex, always the accuracy was fine, stabilized at some nice level.
So I tried the simplest model I could imagine : Input => Dense with 3 hidden units =>  Output. Finally I got random results, with a 33% accuracy !
From here, I guilt again my network, layer by layer, to see which one was causing the overfitting.
And it was the Embedding layer.
Even with a simple network like Input => Embeddings => Dense with 3 hidden units =>  Output, the model was overfitting.
How to solve it
In Keras, simply instantiate the Embeddings layer with trainable=False. After doing this, no more overfit.
In my opinion, this is quite counter-intuitive : I want my embeddings to evolve with the data I show to the network. But look like I can't...

I think sometimes it can also help to examine your test and training sets.
Fundamentally, your data was produced by an underlying process/system that has certain properties. The system can have many "states" and all the possible states form the state space. If you have really tried things like dropout and regularization, my guess would be that the test set is somehow different from your train set. It is possible that your training set only takes samples from one part of the state space (AKA, your samples might all be similar in the training set and the test set has different samples - imagine you are classifying humans and all of your training samples have a class label of 1 meaning all the training samples have humans in them -> and all your test samples have no humans in them Good luck with that!).  Some questions to ask:

Are you combining datasets from different sources? If so:
   If you have "n" sources of data, you need to make sure that your training set has many samples from each of the "n" sources of data and your test set has samples from each of the "n" sources.
Are you shuffling your data enough and randomly putting samples in both the training and test sets? This relates to the human example I gave, make sure your training set has a little bit of everything (different combinations of inputs and/or outputs) and your testing set has a little bit of everything (different combinations of inputs and/or outputs).

Thanks for your answer. However I don't think the problem is from the data : I am using the SNLI Dataset, and only this one. This is a good and well balanced dataset.
The test and validation set are already defined by SNLI, and I always shuffle my data.


text:
Will LMS always be convex function? If yes, then why do we change it for neural networks?
In LMS(least mean square) since, we use a quadratic error function, and quadratic functions are generally parabola in (some convex like shape). I wonder whether that is the reason why we use least square error metric? If that is not the case(its not ALWAYS convex or reason WHY we use LMS), what is the reason then? why this metric changes for deep learning/neural networks but works for regression problems?
[EDIT]: Will this always be a convex function or is there any possibility that it will not be convex?

Square loss is fine for regression, since minimizing it is the same as maximizing the likelihood of the model parameters (under assumption that the error is Gaussian). However, if the model directly produces probabilities, then it is natural to use these probabilities directly within the loss. Hence, in all classification models we prefer to minimize negative log-likelihood of the correct class.
Note that choosing a natural loss leads to several practical advantages. In particular, applying a quadratic loss after a sigmoid activation leads to very poor gradients when the sigmoid is saturated in the wrong direction. The negative log-likelihood loss has no such problems.
This issue is not specific for neural networks. Logistic regression has used the negative log likelihood loss ever since 1958.

Minimize negative log-likelihood, or maximize log-likelihood, right? :D


text:
Can AI 'fix' heavily compessed videos/photos?
So let's say you had a really nice day in a flight simulator and you are getting videos of this type of quality:

This is Full HD (1080p), but heavily compressed. You can literally see the pixels. Now I tried to use something like RAISR, and this python implementation, but it only scales the image up and does not 'fix the thicc pixels'.
So is there a type of AI that does fix this kind of video/photo into a reasonable quality video? I just want to get rid of those pixels and image artefacts that was generated during the compression.

Although there are AI techniques that can do this sort of thing, this Stack Exchange is not for questions looking for implementations. It is for questions about the nuts and bolts or how they work, or even the philosophy of whether or not we could or should build "intelligent" machines. If you are willing to learn enough about neural networks and invest a few weeks or months into researching and writing an algorithm to de-pixelate images, then you are in the right place. If you just want to fix your image, then sorry no - I believe there are projects out there that might help though.

@NeilSlater I am always open to design new algorithms. I just wondered if there was already such algorithm out there.

Yes the algorithm is out there, it is not much different to super-resolution, and implementations are available if you search. At one point I am pretty sure I have used a NN trained to de-block an image on my own graphics. If you are interested in studying the algorithm, could you please make that clearer in the question, explaining what you know already?

Here's one example of an algorithm built to do what you want: https://arxiv.org/pdf/1709.06229.pdf

@NeilSlater Very interesting! Never heard of it lol.

Check out source code to DeepImagePrior it does a remarkable job guessing what's missing to repair images with a variety of damage.

I've answered a part of the question below, i.e. can the photo/video be decompressed in theory?
If the RGB values are continuous (well behaved function), interpolation should generate the pixels in between them in a predictable pattern like say in old pictures of models where skin color shades and data is available in plenty for models today and they resemble each other whereas in outlier cases like this picture where the designs of cockpit circular meter maybe different for each plane or maybe too much data is lost say of a distant object, no neural network should be able to fix it because the possibilities of an orange blur can be an orange, a sun or many other things.
One of the solutions to such things can be recognizing context but finding sufficiently dense pixels so exact should be difficult (I think impossible) because sun will be very bright but orange might have some spots.
In short, average things can be done, not outliers.


text:
What are the ways to calculate the error rate of a deep Convolutional Neural Network, when the network produces different results using the same data?
I am new to the object recognition community. Here I am asking about the broadly accepted ways to calculate the error rate of a deep CNN when the network produces different results using the same data.
1. Problem introduction
Recently I was trying to replicate some classic deep CNNs for the object recognition tasks. Inputs are some 2D image data including objects and the output are the identification/classification results of the object. The implementation involves the use of Python and Keras.
The problem I was facing is that, I may get different validation results among multiple runs of the training even using the same training/validation data sets. To me, that made it hard to report the error rate of the model since every time the validation result may be different.
I think this difference is because of the randomness involved in different aspects of deep CNN, such as random initialization, the random ‘dropout’ used in the regulation, the ‘shuffle’ process used in the choosing of epochs, etc. But I do not know yet the “right” ways to deal with this difference when I want to calculate the error rate in object recognition field.
2. My exploration – online search
I have found some answers online here. The author proposed two ways, and he/she recommended the first one shown below:

The traditional and practical way to address this problem is to run your network many times (30+) and use statistics to summarize the performance of your model, and compare your model to other models.

The second way he/she introduced is to go to every relevant aspect of the deep CNN, to "freeze" their randomness nature on purpose. This kind of approach has also been introduced from Keras Q&A here. They call this issue the “making reproductive results”.
3. My exploration – in academia community (no result yet, need your help!)
Since I was not sure whether the two ways mentioned above are the “right” ones broadly accepted, I was going further exploring in the object recognition academia community.
Now I just begin to read from imageNet website. But I have not found the answer yet. Maybe you could help me knowing the answer easier. Thanks!
Daqi


text:
Is it possible to spot photoshoped or edited photos using AI?
I have this question in my head: does the current level of AI development allow us to spot faked or photoshoped images? (i.e forged ID card or personal documents).
If it is possible, what is such a process to follow in order to build an AI that achieves this task?

It is possible! Here is an article by Adobe where they explain how they do it: https://theblog.adobe.com/spotting-image-manipulation-ai/
The algorithm for this would almost certainly be a Convolutional Neural Net trained on a dataset of real and manipulated images (labeled as such).

Thank you @axon, I'll check it and let you know if I'll reach any achievements.