text:
How to design fitness function for multiple objectives?
I am currently building a neural network with genetic algorithms that learns to fly a 2D drone to a target. My goal is that it achieves all tasks as fast as possible, but I want the drone to also fly stable and upright. The way I tried to calculate the fitness was to create a function that has the greatest value when the drone does everything I want right.
fitness += 1/distToTarget + cos(drone_angle)

My current inputs are:
difference_target_X
difference_target_Y
velocity_X
velocity_Y
angular_velocity (degree per second)
drone_angle     | = 0;     |_ = 90     _| = -90

The output (I don't think it is important but)
left_thruster_angle
left_thruster_boost
right_thruster_angle
right_thruster_boost

The NN is programmed in unity and the drone uses a 2D rigid body and the NN adds a force to the thruster at the right angle.
How do I get the drone to set the best weights to fulfill all tasks: fly stable, fly fast, fly to the target?

Your fitness function has two objectives that are added together, but they are not necessarily on the same scale. The component cos(drone_angle) must have a value from 0..1. The component 1/distToTarget will have a range that depends on how you measure distToTarget; e.g. if distToTarget has a range 0..1000, then this part of the fitness function will always be small far from the target (e.g. distance of 500) and massive when it gets very close (e.g 0.1 distance from the target). So the contribution of both components may not always be equal. Another potential complication is that the cosine function is nonlinear and make a very rapid transition between 0 and 1, as opposed to a smooth transition.
I recommend reworking the fitness function to make the two components more equal, e.g. something like
fitness(angle, distance)=
    w1 * |desired_angle - angle| / maximum_angle_error
  + w2 * distance / maximum_distance

In this function that should be minimised to zero, both components contribute linearly to the final fitness, both components have a range of 0..1, and you can tune w1 and w2 to give different importance to different components depending on your preference (e.g. you may prefer w1>w2 if staying upright is the most important).

Thanks a lot, u helped quite well. But I not sure what to do about the max distance because if the target is just far away how am I supposed to calculate that that or how to set a max distance?  any idea?

set it to the starting distance? would that be possible?


text:
Are there architectures to generate pictures from four labels? (VAEs, GANs)
I want to try something with image creation via NNs. I have come across Variational Autoencoders and Generative Adversarial Networks as possible solutions but have only found image creatinon with CIFAR-100 and GANs (two class-labels, non continuous)
Im looking for an idea for generating a picture with four continuous labels (Age and Rotation around X-, Y-, Z-axis). Is there anything usable for this?
Im especially looking for an VAE-Model on this task.

This is quite a difficult task and is still an open area of research. The idea behind the GANs is to map latent features (e.g. rotation, age) to an output image, but the problem is that the source data comes from an unknown probability distribution. GANs aim to approximate it by sampling this latent feature vector from a simple known distribution (e.g. normal).
However, such a distribution usually won't fit the target distribution and hence the latent features become entangled as they will reproduce statistics of the real data (e.g. gender and beard)
There are several attempts to solve this problem. One of them is Conditional GAN. For example, with a conditional GAN it can be possible to translate a segmentation mask into an image (Pix2Pix). Another approach has been proposed in StyleGAN. The authors introduce an intermediate feature vector that the model learns to disentangle in an unsupervised manner. Each value in this intermediate represents a meaningful attribute of the resulting image, but it is still not possible to control the latent features in a supervised manner. That is, after training you should manually test each of them to identify its relations.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to train my model using transfer learning on inception_v3 pre-trained model?
I am trying to train my model to classify 10 classes of hand gestures but I don't get why am I getting validation accuracy approx. double than training accuracy.
My dataset is from kaggle:
https://www.kaggle.com/gti-upm/leapgestrecog/version/1
My code for training model:
print(x.shape, y.shape)
# ((10000, 240, 320), (10000,))

# preprocessing
x_data = x/255
le = LabelEncoder()
y_data = le.fit_transform(y)
x_data = x_data.reshape(-1,240,320,1)
x_train,x_test,y_train,y_test = train_test_split(x_data,y_data,test_size=0.25,shuffle=True)
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)

# Training

base_model = keras.applications.InceptionV3(input_tensor=Input(shape=(240,320,3)),
                                           include_top=False,
                                           weights='imagenet')
base_model.trainable = False

CLASSES = 10
input_tensor = Input(shape=(240,320,1) )
model = Sequential()
model.add(input_tensor)
model.add(Conv2D(3,(3,3),padding='same'))
model.add(base_model)
model.add(GlobalAveragePooling2D())
model.add(Dropout(0.4))
model.add(Dense(CLASSES, activation='softmax'))
model.compile(loss='categorical_crossentropy',
              optimizer=optimizers.Adam(lr=1e-5), metrics=['accuracy'])

history = model.fit(
x_train,
y_train,
batch_size=64,
epochs=20,
validation_data=(x_test, y_test)
)

I am getting accuracy like:
Epoch 1/20
118/118 [==============================] - 117s 620ms/step - loss: 2.4571 - accuracy: 0.1020 - val_loss: 2.2566 - val_accuracy: 0.1640
Epoch 2/20
118/118 [==============================] - 70s 589ms/step - loss: 2.3253 - accuracy: 0.1324 - val_loss: 2.1569 - val_accuracy: 0.2512

I have tried removing the Dropout layer, changing train_test_split, but nothing works.
EDIT:
On changing the dataset to color images from https://www.kaggle.com/vbookshelf/v2-plant-seedlings-dataset
, I am still getting higher validation accuracy in initial epochs, is it acceptable or I am doing something wrong?

If you set the validation set size to 50% instead of 25% do you get the same behaviour? Could be that dropout is turned off for validation, so validation loss will be slightly higher, perhaps for the first few epochs anyway

@MikeNZ Yes I get same behavior on setting validation size to 50%

The problem is that you're not creating a model with InceptionV3 as the backbone. What you want to do is this:
input_tensor = Input(shape=(240,320,1) )

base_model = tf.keras.applications.InceptionV3(input_tensor=Input(shape=(240,320,3)),
include_top=False, weights='imagenet')

base_model.trainable = False

x = base(input_tensor)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dense(1024, activation='relu')(x)
x = tf.keras.layers.Dense(N_CLASSES, activation='softmax')(x)

...

model = tf.keras.Model(inputs = input_tensor, outputs = x)

model.compile(...)

The inputs and outputs parameters are of importance here.

The  problem is I am using grayscale dataset but inception_v3 accepts rgb images, that's why I converted grayscale to 3 channel images by convolution operation model.add(Conv2D(3,(3,3),padding='same')) before inserting inception model layers in my model.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Unable to 'learn' a rotational angle by parametrising the angle as a neural network layer
I'm trying to implement a neural network that can capture the drift in a measured angle as a way of dynamic calibration. i.e, I have a reference system that may change throughout the course of the data gathering and would like to train a network layer which actually converts the drifting reference to the desired reference by updating the angle parameter.
For example: Consider the 2d case. We would have a set of 2d points $X\in \mathbb{R}^2$ and a trainable parameter called $\theta$ in the layer. The output of the layer would then be:
$$X_o = XR$$ where
$$R = \begin{bmatrix}
\cos(\theta) & -\sin(\theta) \\
\sin(\theta) & \cos(\theta)
\end{bmatrix}$$
Using Adam optimizer I then try to find the $\theta$ which transforms a given angle to the desired reference.
However, the $\theta$ value seems to fluctuate around the initial value probably because of a diverging gradient(?). How can I overcome this issue?
The code is below.
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

class Rotation2D(tf.keras.layers.Layer):
  def __init__(self):
    super(Rotation2D, self).__init__()

  def build(self, input_shape):
    self.kernel = self.add_weight("kernel", initializer=tf.keras.initializers.Constant(90),
                                  shape=[1, 1])

  def call(self, input):
    matrix = ([[tf.cos(self.kernel[0, 0]), -tf.sin(self.kernel[0, 0])],
              [tf.sin(self.kernel[0, 0]), tf.cos(self.kernel[0, 0])]])
    return tf.matmul(input, tf.transpose(matrix))

layer = Rotation2D()

t = np.arange(0, 1000)/200.

y_in = np.array([np.sin(t), np.cos(t)]).T
y_ta = np.array([np.cos(t), np.sin(t)]).T

model = tf.keras.Sequential()
model.add(layer)

model.compile(tf.keras.optimizers.SGD(lr=1.), loss='MSE')
model.fit(y_in, y_ta, epochs=1)
for i in range(100):
  print(layer.get_weights())
  model.fit(y_in, y_ta,verbose=0, batch_size=5)
y_out = (model.predict(y_in))

fig, axes = plt.subplots(2, 1)

for i in range(2):
  ax = axes[i]

  ax.plot(y_in.T[i], label = 'input')
  ax.plot(y_ta.T[i], label = 'target')
  ax.plot(y_out.T[i], label = 'prediction')

plt.legend()

plt.show()```

There are two basic problems with your code:

The functions sin(t) and cos(t) (both in numpy and tensorflow) take radians as inputs. Seeing Constant(90) in your code, and the learning rate of 1. I'm guessing that you assume that it is in degrees - that's incorrect.

In your training data y_ta is not a rotation of y_in:
y_in = np.array([np.sin(t), np.cos(t)]).T
y_ta = np.array([np.cos(t), np.sin(t)]).T

It is a reflection about y=x diagonal. No wonder it fails to find an appropriate rotation.
I just had to change y_ta to:
  y_ta = np.array([-np.cos(t), np.sin(t)]).T

And train with more sensible learning rate:
  model.compile(tf.keras.optimizers.SGD(lr=0.1), loss='MSE')
  model.fit(y_in, y_ta, epochs=10)

To get the angle (which I then convert to degrees):
  (180 * model.layers[0].kernel[0,0].numpy() / np.pi) % 360
  > 90.00187079311581

Thanks and yes, I believe I messed up with the target value which was not actually a rotation.


text:
What is remembering in Hopfield network?
Hopfield is a simple and traditional network. We feed into the network some patterns (Learning/Training). There is no training in Hopfield as the weight calculation adds up all the strength between neurons. The network goes into remembering mode by feeding a new unseen pattern (partially corrupted), and then the input is deactivated. The network iterates until it reaches a global or local minimum.
My question is that it finally remembers anything. It means that it remembers one of those patterns (combinations).
For example, we have five neurons. It remembers one of those $2^5=32$ patterns. So, one can say OK, this is what I am looking for, but it is not. What mechanism is available in the Hopfield network to check whether the found pattern is identical or similar to the input pattern?

The Hopfield net has no mechanism to check that the pattern that it generates in response to a stimulus pattern is the same or similar to any of the patterns in its "memory".
The "memorized" patterns are only attractors in the dynamics of the Hopfield net as it updates itself, in which the neurons influence each other. There is no guarantee that a Hopfield net will generate one of the original input patterns in response to any given stimulus pattern. In fact, Hopfield nets (the original kind from 1982) usually have attractors at many more patterns than the original input patterns: there's an attractor at the reverse of every original pattern (i.e. in which black and white pixels are swapped), and usually attractors at many patterns that include a mix of many of the original input patterns.


text:
Is there any specific SW framework, libraries or algorithms designed for implementing a practical artificial general intelligence (AGI) system?
Any (AGI)-KERAS like libraries useful to develop a real Artificial General Intelligence (AGI) system? Any deep-learning framework to develop AGI applications? any real Transformer-based (like Chat-GPT) libraries that can be used to achieve a minimum level of Artificial General Intelligence?
Existing frameworks/algorithms used in NN, NLP, ML, etc are not enough in my opinion. In my opinion any framework has to be based on building blocks from: Cognitive Science, Neuroscience, Mathematics, Artificial Intelligence, Computer Science, psycology, sociology, etc.

Short answer: no.

To me, it's not fully clear what you're asking here. What's the purpose of this question? Moreover, what does it mean "a theory designed for implementing a practical AGI system"?. Please, edit your post to clarify the rationale behind this post/question. Are you just curious? Have you done some research before asking the question? Please, take a look at this page too: https://ai.stackexchange.com/help/how-to-ask.

I think so . I have rephrased it asking  for info on tools,libraries and algorithms really focusing on AGI and not on weak AI.

@RaulAlvarez Ok, then, please, revise also your title, because in the title you're asking for "theories", which are different from a software library, which is different from an "algorithm". You should also clarify whether you're looking for a "theoretical framework" or if by "framework" you mean a "software library". In any case, note that asking for tools here is generally off-topic, although this question will be kept, as this is about AGI (which is central to this website). Moreover, to improve this question, you should explain why you're interested in this (as I said in the last comment).

In any case, the closest thing to what you're asking really seems to be something like OpenCog, as suggested by one person below.

Well, nor really. I have reviewwed OpenCog websites and aproach and it only seems to me to be a "single" example.... the comunity should be able to find out other solutions (whatever they are) and compare them. For the time being (imho)  I don´t think this question is similar to any other on Stackexchange nor has been really anwered.... so I have rephrased it again to be in line with your suggestions...

Not to my knowledge. The problem is that this is such an enormous task, it cannot really be tackled at once. So the obvious solution is to reduce the scope. In early AI people were using toy domains, whereas nowadays AI systems work more generally (but still perform better if the domain is restricted).
So while (slow) progress is being made putting the individual building blocks together, we're still a long way off building an overarching general system from them.

Yep, any solution has to deal with some kind of reduction in the "mind complexity" to make this problem manageable. Thanks!

Would OpenCog fit the bill? I have had tremendous amounts of trouble building up the demos, which include some non-AGI stuff, but if I’ve read the manual correctly, I think there’s something here — https://opencog.org/

Thanks! It may be a good start where some interesting issues are mentioned:    "One of the critical issues confronted in doing logical inference on large spatiotemporal knowledge bases is the fact that most real-world logical relationships are contextual in nature. Contextuality must be explicitly taken into account to make real-world inference tractable."  https://wiki.opencog.org/w/Real_World_Reasoning


text:
Are the Word2Vec encoded embeddings available online?
I am trying to do an NLP project and was wondering if there is anywhere online where the Word2Vec embeddings are stored (the actual n-dimmensional vectors).
I want to search up a word and see what its encoding is. I have tried looking but couldn't find anything.
Thank you

Hello. Are you looking for a tool? If yes, this question is off-topic here. You may want to ask your question here: https://softwarerecs.stackexchange.com/.

Hi, not necessarily software, I would just imagine that maybe some websites allows you to search the embedding of a given word

The gensim library for Python provides both a convenient way of manipulating Word2Vec embeddings, but also downloadable data sets and pretrained models. That last part is what you're looking for. Wikipedia2Vec also has multilingual embeddings. Then this guy (whom I've never heard of) has an assortement of languages I haven't seen other places. There's another 44 models hosted by the European NLPL. You can also search Kaggle, but a lot of this looks like training data, models I've already linked, or just cruft.


text:
What are the typical sizes of practical/commercial artificial neural networks?
I'm interested in artificial neural networks (ANN) and I wonder how big ANNs in practical use are, for example, Tesla Autopilot, Google Translate, and others.
The only thing I found about Tesla is this one:

"A full build of Autopilot neural networks involves 48 networks that
take 70,000 GPU hours to train. Together, they output 1,000 distinct
tensors (predictions) at each timestep."

It seems like most companies don't publish clear information about their ANN sizes. I really can't find anything detailed on this subject.
Is there any information about the size of big practical/commercial ANNs that include something like the amount of neurons/connections/layers etc.?
I'm looking for a few examples in this scale with more precise information on the size of the neural networks.

NLP Domain
You can easily find such open-source neural networks in NLP applications that have been published by Companies like Google. For example, in BERT models, you can see the BERT-Base has the following specifications:

BERT-Base, Multilingual Cased: 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters

You can find more data about other versions of BERT in the same link.
Another example is GPT models, like GPT-3:

All GPT-3 models use the same attention-based architecture as their GPT-2 predecessor. The smallest GPT-3 model (125M) has 12 attention layers, each with 12x 64-dimension heads. The largest GPT-3 model (175B) uses 96 attention layers, each with 96x 128-dimension heads.

Image Procssing Domain
Another useful domain for your expectation is image processing tasks such as image classification. Pretrained models such as VGG, ResNet, and Inception. These are mostly used for image classification tasks in different companies, and you can find their specification many where. For example for VGG-16, we can see the followings:

Speech Processing
Another practical domain is Auto-Speech Recognition or ASR in short. One of the renowned models in this context is DeepSpeech(2) by Baido Research center. For example, you can find some info like the number of its parameters and its structure in this github link.
Summing up
Note that one regular metric to measure the size of neural networks is "the number of parameters" of the network that is required to be learned in the training phase. Hence, you can compare the size of models even between cross domains by knowing their number of parameters (instead of going to more details about the number of hidden layers and their types). Although, sometimes the length (number of layers) and height (number of neurons in each layer) of the network are very important in the matter of performance and capability of the network.

I hope this helps. Disclaimer: the info is extracted from Computer Vision at Tesla, though aditional references may be needed....

Tesla is using 8 cameras per vehicle fused with radars.
To detect road lanes, vehicles, pedestrians etc., Tesla have to run
at least 50 neural networks simultaneously.
The neural networks are trained using PyTorch. Each image (for each
camera I guess) is of dimension (1280, 960, 3). RGB image it seems.
Regarding the neural network architecture, "the backbone is a modified ResNet 50", that is, a convolutional network.
The network is 50 layers deep. The network depth is defined as the largest number of sequential convolutional or fully connected layers on a path from the input layer to the output layer. In total, ResNet-50 has 177 layers. The ResNet-50 has over 23 million trainable parameters
Every camera is processed through a single neural network. Then everything is combined into a middle neural network.

As interesting as this are the hyperparameters and data used to train these networks. Any info on this?

Use the benchmarked algorithms or research papers will be a good start. Addition to that use the open sourced Bert GPT 2 like architectures is a good start.

The size of the model depends on the domain. I am currently working with a model that is used for real time inference on an embedded device. Speed of computation is critical.
The model size is a 5 layer CNN, about 700k parameters and it's about 12MB in size on disk.


text:
When exactly am I overfitting -- contradicting metrics
I am training an object detection machine learning pipeline. Among the many metrics provided out of the box by tensorflow object detection API, I look at total_loss and DetectionBoxes_Precision/mAP@.75IOU:

Here the x-axis is the number of steps i.e. model experience. The orange line is for the training data and the blue line is for the validation data.
From the loss graph I would conclude, that at approx 2k steps overfitting starts, so using the model at approx 2k steps would be the best choice. But looking at the precision graph, training e.g. until 24k steps would be a much better model.
Which one is the best model?
Here, loss and the precision metric where picked just for illustrating the dilemma, there are many more metrics available, leading to multiple conclusions about when overfitting actually starts.

From the loss graph I would conclude, that at approx 2k steps overfitting starts, so using the model at approx 2k steps would be the best choice. But looking at the precision graph, training e.g. until 24k steps would be a much better model. Which one is the best model?

Different metrics will lead you to different conclusions. To detect overfitting under any metric you do need to plot training and validation/dev of the same metric, so it is harder to spot on your second graph.
The reason why loss/cost functions and other metrics can disagree on what the best model is, is due to sensitivity to different types of error. Often loss functions will be sensitive to outliers - large differences in predictions versus ground truth in the training or validation data. So it is common to see loss diverging between validation and train first - probably due to a very few bad predictions caused by difficult edge cases that the model is not coping well with - whilst metrics such as accuracy may continue to improve on validation and test sets.
The way to resolve this is to pick the metric that is important to you for business reasons (ideally ahead of the training, so you are not biased in your own decision about what to do), and use that to make judgement calls. This is easier to rely on the larger and cleaner your dataset is.
One other thing you can do is to some error analysis - find the worst cases for loss function, and check that they are labelled correctly.
If things still look inconclusive to you, and dataset size is small enough that results are noisy, then consider using k-fold cross validation to reduce effects of sample bias. This takes more time, but may improve your confidence in your model selection process.

Generally, a model is considered to be overfitted when there's a huge gap between training and test/validation performances.
So during the training, you monitor the loss on validation data, and training data, and stop training if validation loss stagnates/increases given the training loss keeps decreasing.
In your scenario, I'm not sure what the total loss metric corresponds to. As I said, you've to measure loss on a held-out data other than training data to detect, and prevent overfitting.

Thank you @SpiderRico. According to your statement, I would stop training at around 2k steps. My question is: why should I stop training if validation based (held-out data) mean average precision (mAP) further increases?


text:
PPO in continuous control not working
I have PPO agent for discrete action space for LunarLander-v2 env in gym and it works well. However, when i am trying to solve continuous version of the same env - LunarLanderContinuous-v2 it is totally failing. I guess i made some mistakes in converting algorithm to continuous version. So, my steps of changing at algorithm are:

Change network: return mu and  var of shape n_actions. I have 2 different last layers for that, for mu i return Tanh of logits and for var i return Softplus of logits.
Choosing action: sampling action from normal distribution with expectation mu and variance var - torch.distributions.multivariate_normal.MultivariateNormal(torch.squeeze(mu), torch.torch.diag_embed(var))
For log of action probability i am using dist.log_prob(actions)

With this small changes my algorithm totally doesn't work. Is it right steps to convert algorithm with discrete action space to algorithm with continuous action space? I really confused, because my PPO for discrete action space work very well and with only this changes it is failing. Could you please suggest what i am doing wrong here?

It sounds like you did the right changes though I will look more closely at changes I did with the same environment and algorithm. One thing that might be happening is that learning with continuous actions is just harder and you could have the wrong parameters. I think you should try changing the learning rate and other hyper parameters, increase the learning time, and maybe keeping the variance constant to make sure it’s not a problem with your code.

@S2673 Thank you for response! I found a solution but unfortunately i haven't found reason yet. So, if i have 2 distinct neural networks  for 2 actions it works really fine. But, if i have at least 1 common layer(for example first layer is the common one, and 2 next layers are different for 2 action, so i have like 5 layers in my network and 2 tails, sorry if i confused:)) all just stop working at all! It is really curios for me why it happens and may be there are some papers that checking for this problem? About problem - continuous lunar lander.

I don’t think I understand exactly, but it sounds like the problem is with the code (especially in the neural network) instead of the parameters. It still could be possible that the parameters work only for simpler networks with one action output. Try changing the learning rate and size and amount of hidden layers and make sure the code is correct. I think I have PPO code somewhere that works with LunarLander and I could share it with you if you want.

@S2673 Yes, please can you share your network, thanks!

I put the code in here. It is written in Python with PyTorch and there are no explaining comments. You probably want to focus on the hyperparameters, and the Agent class, especially the initialization, storage, and updating. Run exactly as it is, it did pretty well on continuous LunarLander after 50 training loops. Holding space at any time will stop the training. agent.run(1) will animate the environment and show you how the agent is doing and agent.train(50) will train for 50 training loops.


text:
Exploration for softmax should be binary or continuous softmax?
Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like (0,0,1,0,0,0) or continuous softmax like (0.1, 0.15, 0.45, 0.25, 0,5, 0.1)??
if the answer is continuous, what algorithm do you suggest? like generating random numbers between 0 and 1 and then using softmax? (this algorithm mostly provides close numbers and I think it's not the correct way)

This post is unclear to me. What do you mean by "what random samples should we provide?" Provide to what? Moreover, what "random samples" are you talking about? Samples of what?

Random samples of possible actions for exploration. In the exploitation, the neural network will provide the action for us but in the exploration, we have to generate these samples of action ourselves and feed them to the algorithm. The actual question is for a neural network with softmax in output, these random actions should be binary or continuous.

If by samples you mean "actions" (which is what you just said), then I don't understand the question "actions should be continuous or binary".

Firstly, I would suggest you do not use softmax for exploration, because it does not imply the model's uncertainty. Training with softmax and cross-entropy, your model may be very confident, but wrongly, because of overfitting.
Another reason why you should not use softmax to measure uncertainty is that your estimate of the variance (optimal estimate of the variance is the average error) may be quite low, explaining the low entropy of your model. However, there maybe exist regions (holes), where your model is highly uncertain (large error).
Thus, you should try other options for measuring uncertainty.

One way to measure model (or epistemic) uncertainty is to use a Bayesian Network estimating the whole distribution $p(\theta | D)$, where \theta are your parameters and D is the collected dataset of experience. The entropy of $p(\theta | D)$ tells us the model uncertainty.
Why Bayesian Network? Intuitively, if our estimator says all θs are equally likely given your data, then it means that you have no idea of what the model really is. On the other hand, if it says there is one only θ, that can possibly explain D, then you have high confidence in the model.

Another way to measure epistemic uncertainty is to use bootstrap ensembling. Instead of estimating the uncertainty of every single parameter in the Bayesian Net, learn N different networks! Use bootstrapping in order to generate N datasets (of same size as D) by resampling with replacement. Then train each network on a corresponding generated dataset. Then in testing, you can either sample uniformly a network ($p(\theta | D) = \frac{1}{N}\sum_i^N{\delta({\theta}_i)}$)  or just average all networks predictions.

For exploration (in Deep RL), there are a lot of different options. I recommend you check information gain, Thompson sampling and pseudo-counts. However, if you were using the standard deep RL agents, like DQN etc, you could begin with the standard ε-greedy behavior policy, ignoring that this is not an optimal exploration policy for the most cases.


text:
Is intersection of labels acceptable in computer vision?
I have a dataset, where objects are very close to each other. So, the question is: what is the best approach to label them?
There are two possible options:

mark objects so that they will not intersect (it is difficult, surroundings are not included in the label area)
mark a larger area of objects, but labels will intersect

What is more practical?

I've always seen it not intersecting in examples of CV datasets. it's probably better if you can avoid intersection between labels. Yet I thinks the most important part when labelling is that it does not take you days and weeks to label your data, so you shoudn't worry if sometimes it intersects because you're drawing swiftly around your objects. Having an example of image in your post may help understand the situation and give a more precise answer than this one tho.

Hello. Welcome to AI SE. Maybe you should clarify what kind of task will your dataset be used for? Object detection?

my dataset is proprietary, I give an idea of the problem, pictures were uploaded

In my opinion, the second option will be more general. You can refer to some famous datasets for object detection task such as COCO or Pascal VOC, they usually accept the intersect annotations. As the image below, image from this link where they process the annotation of COCO dataset.
I think the reason is that the model will be easier to separate the intersect patterns in the bounding box than interpolate the missing patterns of the object to understand it


text:
Which approach best suits vector encodings?
I want to build a model that when given two vectors, outputs the probability of one vector being the encoded form of the other. I have 2 strategies for this: (Dataset available)

I can directly feed them in concatenation to a neural network and take the output as the probability.

I can train a conditional GAN with the conditional vector being the encoded vector and using the original vector as the generated one. In this case, the discriminator works as the network that I train in the first approach.

Which approach is better? Am I thinking in the right direction?

I haven't dealt with vector encodings, but I would suggest changing the title of your post to be more relevant. Maybe something like "What model best suits vector encoding classification"

Okayy! I'll do that


text:
In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?
In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted for all the possible $N$ actions.
Would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?
With cheaper I mean cheaper in terms of the time complexity of a single training step of the network.

Hello. I tried to clarify your question. Make sure that I didn't change the meaning of the post. Please, could you also clarify what you mean by "cheaper"? Cheaper in terms of what?

Hi, thank you. The meaning is the same. With "cheaper" I mean cheaper in terms of operations required.

Please, edit your post to clarify what you mean by "cheap". In terms of operations, do you mean the time complexity of one iteration of training DQN as described in the original paper?

Yes, I mean that. I have edited.

But, if that's what you mean, why did you accept the answer below? It doesn't really answer your question (which was not specific enough indeed). You should discuss your edited version with the author of that answer.

@nbro: Rating learning on time complexity alone is not useful. You can easily trade accuracy or efficiency of learning for time complexity, up to the extremes of not learning anything very fast per iteration. So there needs to be some further constraints. However, it is very hard to phrase such trade-offs in a question like this. I think my answer (or some variant of it) is indeed what the OP is looking for - validation that the idea has some merit, with caveats on which scenarios it could be useful in.

The only improvements I can see is that both question and answer could be extended to cover more technical details on the applied scenarios and degree to which the idea has merit. However, that is hard to phrase correctly. Definitely time complexity per step alone, without specifying the variables of interest, is not a useful metric.

@NeilSlater You're right that the time complexity of these stochastic algorithms may not be the first metric that one may be interested in, but I don't think it's completely useless (in particular, in the cases the proposed algorithms converge to the same solutions, which doesn't seem will be the case here). Anyway, my point was that the question wasn't clear/specific enough, so I think using the word "cheap" here is very vague. Maybe the OP can confirm that he was actually not interested in the time complexity. In that case, you may want to suggest another word, or rephrase the question.

Would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?

I think the "No Free Lunch" theorem applies here, or something like it.
Your proposed architecture would be an unusual choice in many cases, but might be more efficient in others. For instance, it could be more efficient in the following scenario:
The long term value is highly dependent on the immediate action choice, and in a way that relies on state variables differently, depending on the specific action. That means it would be difficult for a single NN to create shared features in its layers, and you could save processing by treating each action as a different prediction problem.
This is only an educated guess.
As usual, the only way to find out for sure is to try different approaches and compare them. I don't think there is anything other than experience and a little intuition to guide you.


text:
How to compute circle loss?
I have read the paper about circle loss for neural networks. I may have missed something, but I didn't find the way to compute the positive similarity and the negative similarity in the case of circle loss.
The paper gives the equations for AM-SoftMax and Triplet loss, that can be computed from a unified loss function:

For example, in the case of AM-SoftMax, the positive and negative similarities are computed by the following way, w being the weights of the last fully connected layer, and x being the inputs of this last layer:

So my questions are:

Should I compute the similarities for circle loss by the same way the paper compute them for AM-SoftMax ?

Should I compute the forward pass of the last fully connected layer with a standard product: output[j] += weight[i][j] * input[i], followed by a standard SoftMax function ? Or by another way ?

Hello. Welcome to AISE. In general, each post should contain exactly one question. As a rule of thumb, if you can't put your specific question in the title (because maybe you have more than 1), then your post might be too broad. Moreover, note that programming questions are off-topic. However, if you're only trying to understand how the theory is connected to the code, then your questions are fine for our site.

I have reformulated my question, focalizing on the two problems I have with circle loss computation.


text:
Normalization of possibly not fully representative data
I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:

outliers, but I could fight them or use RobustScaler instead.
I am not sure that some features in my dataset include all possible ranges. Like I have max(feature_A) == 10, but with the data update, it could become 20. And if I'll preprocess data the same way I will get bad prediction results.
Some features do not have a limit at all and will only increase with time, like how many years patients were treated, for example. I could suppose that this value is !>100years, for example, but if my mean value is 10 years, it will squeeze feature values a lot.

My dataset is pretty large, like millions of observations, thus there is a pretty good chance that it is representative, though. But I am concerned with the small-time range, like all those observations are for the 2 years only, thus, some feature values (like how many years patients were treated) could still grow their bounds.
How should I handle this?
My concerns example:
import pandas as pd
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

#### like, initial state
df1 = pd.DataFrame({'A': [1, 2, 3, 4, 5], 'B': [10, 40, 60, 80, 100]})
""" output:
   A    B
0  1   10
1  2   40
2  3   60
3  4   80
4  5  100
"""

scaler.fit_transform(df1)
""" output:
array([[0.        , 0.        ],
       [0.25      , 0.33333333],
       [0.5       , 0.55555556],
       [0.75      , 0.77777778],
       [1.        , 1.        ]])
"""

#### new data arrived, while preprocessing is the same
df2 = pd.DataFrame({'A': [1, 2, 3, 4, 5, 10, 10], 'B': [10, 40, 60, 80, 100, 120, 140]})
""" output:
    A    B
0   1   10
1   2   40
2   3   60
3   4   80
4   5  100
5  10  120
6  10  140
"""

# now 5 in "A" scaled to 0.4 instead of 1, same in "B"
scaler.fit_transform(df2)
""" output:
array([[0.        , 0.        ],
       [0.11111111, 0.23076923],
       [0.22222222, 0.38461538],
       [0.33333333, 0.53846154],
       [0.44444444, 0.69230769],
       [1.        , 0.84615385],
       [1.        , 1.        ]])
"""

PS: I've duplicated this question in different communities (question in ai got most of views):

https://stats.stackexchange.com/questions/523097/normalization-of-possibly-not-fully-representative-data
https://datascience.stackexchange.com/questions/94095/normalization-of-possibly-not-fully-representative-data

A friend of mine answered on this question in different social media on different language, I'll post his answer here:

1. scaler should be saved in this case.
You do fit_transform in the example the second time you run it, but you should just transform. Scaler should be "fit" once on data train and not change it afterwards. Then you will get 5 in 1 in both cases. And when 10 appears, scaler will convert it to 2.
2. It's not a fact that this will break the model.
I think everything is clear here, but just in case:
Let us consider the simplest case. Suppose there is a linear dependence y = 3 * x, we don't know about it, but there is a dataset:
X Y
1 3
2 6
3 9

If we assume that the model has the form y = a * x, in the process of learning it turns out that
a = 3 gives the best score and when suddenly x = 100 appears, it has no problem mapping to 300, even though 100 is strongly out of dataset (1,2,3).
3. You can check if this breaks the model now.
Suppose there is a chip that takes values 0, 1, 2, 3, 4, 5, 6, 7, 8 ...
You are afraid that after some time samples with values greater than the current maximum (9, 10, 11, ...) will appear and the model will behave strangely on them. You can try to "simulate" this situation.
Split dataset into train and test so samples with values from 0 to 6 will go into train, and samples with values from 7 to 8 will go into test. If accuracy doesn't fail at the test stage, then we can assume that when all the 0-8 ranks get into train, then the appearance of 9-11 won't break anything.

Also, there is answer for the second question: https://stackoverflow.com/questions/46744076/machine-learning-normalizing-features-with-no-theoretical-maximum-value


text:
Why don't integrated gradients explain samples correctly?
I have a linear tabular dataset made of floats. The dataset follows a simple rule like:
if features A and B are in a certain range then target class is 1, otherwise target class is 0.

Since I want to get some interpretability from my ANN model, I opted for using the integrated gradients method implemented by alibi.
Unfortunately, most of individual samples don't show A and B as the leading features as expected. Even more weird is the fact that, when I average the attributions of all the individual samples, A and B get the highest score. In other words, local explanations fail but, on average, the global explanation is correct.
Can anyone help me out to understand why this happens? Isn't integrated gradients method suitable for tabular datasets?
By the way, my baseline is based on a uniform distribution of random floats ranging from 0 to the maximum of each column.

After several tests, I've opted for the following configuration:
Approximation mode: riemann trapezoidal
Steps: 1000
Baseline: average mean of each column (feature)
Although the global score remains good, there're important variations in the number of correctly explained local samples. For example, an execution returns 200 out of 390 samples explained as expected, but the following execution only 17.
Is there a reason for that? And, is there a way to make the algorithm predictable?


text:
Update Rule with Deep Q-Learning (DQN) for 2-player games
I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great results for Connect 4. The agent is learning to win quickly, if it gets the chance, but it only plays in the centre. It is unable to detect threats in the first and last columns. I am using DDQN with memory replay. Also teacher and student refer to two agents at different strengths, while the teacher is frequently replaced by a new student. My algorithm looks simplified as follows:
for i in range(episodes):
    observation = env.reset()
    done = False
    while not done:
        if env.turn == 1:
            action = student.choose_action(observation)
            observation_, reward, done, info = env.step(action)
            loss = student.learn(observation, action, reward, observation_, done))
            observation = observation_
        else:
            action = teacher.choose_action(-observation)
            observation_, reward, done, info = env.step(action)
            observation = observation_

The observation is -1 for player "o", 1 for player "x" and 0 for empty. The agent learns to play as player "x" and through action = teacher.choose_action(-observation) it should find the best move for player "o". I hope that is clear.
The update rule looks as follows:
# Get predicted q values for the actions that were taken
q_pred = Q_eval.forward(state, action)
# Get Q value for opponent's next move
state_ *= -1.
q_next = Q_target.forward(state_, max_action)
# Update rule
q_target = reward_batch - gamma * q_next * terminal
loss = Q_eval.loss(q_pred, q_target)

I am using -gamma * q_next * terminal, because the reward is negative, if the opponent wins in the next move. Am I missing anything important or is it just a question of hyperparameter tuning?

What does terminal represent, and what values can it take?

it is 0 if the next state is terminal (done == True) and 1 otherwise. This is, because all terminal states have Q-Value of zero in Q-Learning.

You also asked this question here.

yeah I did, and?


text:
What are the roles of the prior $\mathrm{p}(\mathbf{z})$ in a VAE?
I know the encoder is variational posterior $q_{\phi}(\mathbf{z} \mid \mathbf{x})$.
I also know that the decoder represents the likelihood:
$p_{\theta}(\mathbf{x} \mid \mathbf{z})$.
My question is about the prior $\mathrm{p}(\mathbf{z})$.
I know ELBO can be written as:
$$E_{q_{\phi}(\mathbf{z} \mid \mathbf{x})}[\log (p_{\theta}(\mathbf{x} \mid \mathbf{z}))]-\mathrm{D}_{\mathrm{KL}}( q_{\phi}(\mathbf{z} \mid \mathbf{x}) \| \mathrm{p}(\mathbf{z})) \leq \log (p_{\theta}( \mathbf{x}))$$
And for the VAE, the variational posterior is
$$ q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x}^{(i)})= \mathcal{N}( \boldsymbol{\mu}^{(i)}, \boldsymbol{\sigma}^{2(i)} \mathbf{I}),$$
and prior is
$$ \mathrm{p}(\mathbf{z})=\mathcal{N}( \boldsymbol{0}, \mathbf{I}).$$
So
$$\mathrm{D}_{\mathrm{KL}}\left(\mathrm{q}_{\Phi}(\mathbf{z} \mid \mathbf{x}) \| p_{z}(\mathbf{z})\right)=\frac{1}{2} \sum_{j=1}^{J}\left(1+\log \left(\sigma_{j}^{2}\right)-\sigma_{j}^{2}-\mu_{j}^{2}\right)$$
That's one way I know the prior plays a role, in helping determine part of the loss function.
Is there any other role that the prior plays for the VAE?

The prior $p(z)$ is assumed as part of the problem formulation. A typical case is where $z$ is a vector of iid normal random variables. The ELBO involves a regularization term which encourages $q(z \, | \, x)$ to have a similar distribution to $p(z)$ (the way you've written it, that's the KL term). Thus $q(z \, | \, x)$ will end up having a similar shape to $p(z)$. For example, again assuming $z$ is a vector of iid normals, if you plot samples of $z$ drawn from $q(z \, | \, x)$ you will find it has a roughly spherical shape. If you scroll down to the [16] code block and look at the figure you'll see what I mean. The figure is plotting samples of $z$, colored according to what $x$ is (MNIST example). This is just some random figure I found, and I don't endorse this code, but the image is what you'd expect to see.
The way we end up with a distribution $p(x, z)$ is by using the prior. We sample $z$ according to $p(z)$; we've trained the decoder $p(x \, | \, z)$, and by definition $p(x, z) = p(x \, | \, z) p(z)$.

Thanks, I will think about what you said..

Also, I moved my 2nd question to a separate post here https://ai.stackexchange.com/questions/32161/how-does-the-vae-learn-px-z


text:
Are the authors of the VAE paper writing the PDFs as a function of the random variables?
Usually, I see the conventions:

discrete random variable is denoted as $X$,
the pmf is written as $P(X=x)$ or $p(X=x)$ or $p_{X}(x)$ or $p(x)$, where $x$ is an instance of $X$
a continuous random variable is denoted as $X$,
the pdf is denoted as $f_{X}(x)$ or $f(x)$, where $x$ is an instance of $X$; sometimes $p$ is used here too instead of $f$.

However, the VAE paper uses slightly different notation that I'm trying to understand

Let us consider some dataset $\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N}$ consisting of $N$ i.i.d. samples of some continuous or discrete variable $\mathrm{x}$. We assume that the data are generated by some random process, involving an unobserved continuous random variable $\mathbf{z}$. The process consists of two steps: (1) a value $\mathbf{z}^{(i)}$ is generated from some prior distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{z}) ;(2)$ a value $\mathbf{x}^{(i)}$ is generated from some conditional distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$. We assume that the prior $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ and likelihood $p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$ come from parametric families of distributions $p_{\boldsymbol{\theta}}(\mathbf{z})$ and $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$, and that their PDFs are differentiable almost everywhere w.r.t. both $\boldsymbol{\theta}$ and $\mathbf{z}$. Unfortunately, a lot of this process is hidden from our view: the true parameters $\theta^{*}$ as well as the values of the latent variables $\mathrm{z}^{(i)}$ are unknown to us.

So I am looking at these:

$p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$
$p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$
dataset $\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N}$

So I know the subscript for $\theta$ denotes those are the parameters for the pdf. It says "discrete variable $\mathrm{x}$", "unobserved continuous random variable $\mathbf{z}$", and  "latent variables $\mathrm{z}^{(i)}$".  In the top, where I wrote " discrete random variable $X$", seems like that's the equivalent of "discrete variable $\mathrm{x}$" in this paper.
So, it looks like they're writing the PDFs as a function of the random variables. Is my assumption correct? Because it is different than the typical conventions I see.
edit: looks like his other paper has a notation guide, in the appendix, though it seems like he's conflating both random vector and instances of vector in the notation?
https://arxiv.org/pdf/1906.02691.pdf

no they’re not writing the pdf/pmf as a function of a random variable. The capital X here is just denoting your dataset of realisations of random variables

Sorry I rewrote my comment, that particular part about p(X) might have made things more confusing.

I would say that here they’re saying ‘discrete variable’ to denote a realisation of the random variable, so $p(x)$ in this setting is still in line with standard notation. $p(\cdot)$ is a function of $x$ (discrete in this case), it’s just that $x$ are some random realisations of a random variable $X$.

One other possibility I'm considering is that they could just be using it for both the instance and random variable?

When it comes to notation/terminology, often, people in machine learning are (a bit?) sloppy, which causes a lot of confusion, especially for newcomers to the field or people not very math-savvy. I was also confused about this notation at some point (see my last questions here, which are all about this confusing topic). See also this answer.
In the VAE paper, $\mathbf{X}$ is a dataset, as the authors write.
Your confusion also arises because the authors vaguely use the term "probability distribution", rather than pdf or pmf, to refer, for example, to $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$, which thus does not refer to a pdf or pmf. In fact, the authors also write

their PDFs are differentiable almost everywhere w.r.t. both $\boldsymbol{\theta}$ and $\mathbf{z}$

The $\mathbf{z}$ can refer to

a random variable, or
an input to the function $p_{\boldsymbol{\theta}^{*}}$

If it's the first case, then $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ is the composition of 2 functions (because a rv is also a function).
If it's the second case, then $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ is the evaluation of $p_{\boldsymbol{\theta}^{*}}$ at $\mathbf{z}$.
I think the 2nd case is the most likely. In addition, people are being sloppy here and use the notation $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ (rather than just $p_{\boldsymbol{\theta}^{*}}$) to emphasize $p_{\boldsymbol{\theta}^{*}}$ is a function of some input variable (not random variable!), which we denote with the letter $\mathbf{z}$ to remind ourselves that $\mathbf{z}$ is associated with a random variable denoted with the same letter (and maybe also in bold and lowercase).
So, in this case, let's say we denote the random variable associated with $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ with $\mathbf{z}$, then we could refer to this associated prior more explicitly as follows $p_\mathbf{z}(\mathbf{z})$ (but that would even be more confusing). It would have been a better idea to use $\mathbf{Z}$, but then we may use the upper case letters to denote matrices or sets (like the VAE paper), so we end up with this mess (which is one of the 2 mythical difficult problems well-known in Computer Science, i.e. naming things), which we need to learn to deal with or just ignore.
Conclusion: when I look at $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$, which has been referred to as a probability distribution, I think there's also some associated random variable, which people, in that same context, will probably denote as $\mathbf{z}$ or $\mathbf{Z}$. There may also be some input variable (not a random variable), which we denote by $\mathbf{z}$ or $z$. If they are not mentioned, then I just ignore that. I never think that $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ is the composition of 2 functions (even if that's the case), because that case was never useful in my readings.

I just noticed in Appendix A of his other paper on VAEs, he goes over the notation.  Seems like he's using the same notation for random vectors and instances of random vectors. https://arxiv.org/pdf/1906.02691.pdf

and yeah, figuring out which notation to use is a mess. I was thinking italics, using mathcal in latex , and something like this:https://tex.stackexchange.com/questions/520412/is-there-a-latex-font-correspond-to-the-symbol-an-x-with-bars-at-the-top-and-bo  gives more variety there.

Machine learning papers are often somewhat confused about the distinction between a distribution and its probability density. I would rewrite this

The process consists of two steps: (1) a value $\mathbf{z}^{(i)}$ is generated from some prior distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$; (2) a value $\mathbf{x}^{(i)}$ is generated from some conditional distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$.

as follows

The process consists of two steps: (1) a value $\mathbf{z}^{(i)}$ is generated from some prior distribution. The probability density of selecting $\mathbf{z}^{(i)}$ is known and denoted as $p_{\boldsymbol{\theta}^{*}}(\mathbf{z}^{(i)})$. (2) a value $\mathbf{x}^{(i)}$ is generated from some conditional distribution. The probability density of selecting $\mathbf{x}^{(i)}$ given $\mathbf{z}^{(i)}$ is known and denoted as $p_{\boldsymbol{\theta}^{*}}(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i)})$.

As for the uppercase/lower case notation, this notation is not used in machine learning. $z, \, x$ are both random variables. In this paper, the authors use $z^{(i)}$, $x^{(i)}$ to indicate specific realizations of the random variables $z, \, x$. Probably a notation like $x_i, z_i$ is more common in general.
The notation/explanation is quite bad, because they should not say $p_{\theta}(z)$ is the distribution. $p_{\theta}(z)$, or if you wanted to be more precise, $p_{\theta}(z^{(i)})$, refers to the probability density.

I noticed Kingma added notation guide in his other paper on VAEs, in appendix A.
https://arxiv.org/pdf/1906.02691.pdf  Seems like he uses the same notation for random vectors and their instances here too?

You can read $X=\{x^{(i)}\}_{i=1}^N$ as $X$ represents the sequence of all values of $x$ from $x_i$ to $x_N$ where $i$ is all values from 1 to $N$.
To me, the notation is confusing since my experience tells me that curly braces are used for sets, but this seems to be the best interpretation.

I guess its more like a sequence than a set?

Ah yes, that would be a good interpretation.  Let me update the answer.


text:
Do we use two distinct layers to compute the mean and variance of a Gaussian encoder/decoder in the VAE?
I am looking at appendix C of the VAE paper:
It says:

C.1 Bernoulli MLP as decoder
In this case let $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Bernoulli whose probabilities are computed from $\mathrm{z}$ with a fully-connected neural network with a single hidden layer:
$$
\begin{aligned}
\log p(\mathbf{x} \mid \mathbf{z}) &=\sum_{i=1}^{D} x_{i} \log y_{i}+\left(1-x_{i}\right) \cdot \log \left(1-y_{i}\right) \\
\text { where } \mathbf{y} &=f_{\sigma}\left(\mathbf{W}_{2} \tanh \left(\mathbf{W}_{1} \mathbf{z}+\mathbf{b}_{1}\right)+\mathbf{b}_{2}\right)
\end{aligned}
$$
where $f_{\sigma}(.)$ is the elementwise sigmoid activation function, and where $\theta=\left\{\mathbf{W}_{1}, \mathbf{W}_{2}, \mathbf{b}_{1}, \mathbf{b}_{2}\right\}$ are the weights and biases of the MLP.
C.2 Gaussian MLP as encoder or decoder
In this case let encoder or decoder be a multivariate Gaussian with a diagonal covariance structure:
$$
\begin{aligned}
\log p(\mathbf{x} \mid \mathbf{z}) &=\log \mathcal{N}\left(\mathbf{x} ; \boldsymbol{\mu}, \boldsymbol{\sigma}^{2} \mathbf{I}\right) \\
\text { where } \boldsymbol{\mu} &=\mathbf{W}_{4} \mathbf{h}+\mathbf{b}_{4} \\
\log \sigma^{2} &=\mathbf{W}_{5} \mathbf{h}+\mathbf{b}_{5} \\
\mathbf{h} &=\tanh \left(\mathbf{W}_{3} \mathbf{z}+\mathbf{b}_{3}\right)
\end{aligned}
$$
where $\left\{\mathbf{W}_{3}, \mathbf{W}_{4}, \mathbf{W}_{5}, \mathbf{b}_{3}, \mathbf{b}_{4}, \mathbf{b}_{5}\right\}$ are the weights and biases of the MLP and part of $\boldsymbol{\theta}$ when used as decoder. Note that when this network is used as an encoder $q_{\phi}(\mathbf{z} \mid \mathbf{x})$, then $\mathrm{z}$ and $\mathrm{x}$ are swapped, and the weights and biases are variational parameters $\phi$.

So, it seems like, for a Bernoulli decoder, it only outputs a vector $\mathbf{y}$, which gets plugged into the log-likelihood formula.
But then, for the Gaussian decoder, it outputs both $\boldsymbol{\sigma}$ and $\mu$. So, is it like 2 parallel layers, one calculating  $\boldsymbol{\sigma}$  one calculating $\mu$?
Similar to how we get the $\mu$ and $\sigma$ of the encoder (which I am assuming the encoder ones are different from the decoder ones)?
And we plug it into the formula I derived in this link
here,  the log-likelihood to get the reconstruction loss?
This is the intuition I am getting, but I haven't seen it explicitly all in one place.

You should accept my answer if answers your question. If it doesn't answer your question, can you explain why? Note also that, ideally, each post should contain only one question, so, please, next time, ask one question per post. If you have multiple questions, ask each of them in a separate post (but make sure to provide the necessary context to understand each question in their separate post).

In particular, your last question is quite distinct from the other ones, so, please, ask it in a separate post. It seems that that last question is: "What the loss should be when the decoder is also a Gaussian rather than a Bernoulli?" This question may pop multiple times, so I think it deserves its own post with a good answer.

Yes, in the case of the Gaussian, you have two distinct layers (so weights and biases), one for the mean and the other for the variance, as the equations are telling us.
The mean is calculated with the weights $\mathbf{W}_{4}$ and bias $\mathbf{b}_{4}$ from $\mathbf{h}$ as follows
$$\boldsymbol{\mu} =\mathbf{W}_{4} \mathbf{h}+\mathbf{b}_{4},$$
while the variance (actually, equivalently, the log of the standard deviation) is calculated from $\mathbf{W}_{5}$ and $\mathbf{b}_{5}$ from $\mathbf{h}$ as follows
$$\log \sigma^{2} =\mathbf{W}_{5} \mathbf{h}+\mathbf{b}_{5}$$
Here you have a PyTorch implementation that uses 2 distinct linear/dense layers for doing this, but note that it is doing this only for the encoder to produce the latent vector $\mathbf{z}$.
Yes, generally, these layers (so the mean and variance) are not the same for the encoder and decoder. However, it would not be surprising to me if someone already tried to share some layers between the encoder and decoder for some specific task.


text:
Is a genetic algorithm efficient for a snake game?
I am working on a DIY project in which I want to be able to train a neural network to play Snake.

Is a genetic algorithm an efficient way of training a network for this application?

For a GA, what should the inputs of the network be? (distance to walls and fruit or the squares in the proximity of the snake head as a vector)

What would the difference in efficiency be depending on the algorithm and what limitation does each one have? Are there any other alternatives I should consider?

Q-Learning and Genetic algorithm are both good algorithms to create an IA that plays Snake.
The one you use depends mostly on how you understand and model your IA environment.

Q-Learning algorithm is an algorithm that needs a State (give by
the environment), Actions it can take, and Rewards to give
him according to how it performs.
Genetic algorithm needs to have intrinsic parameters (could be caracteristics of the network taking the decisions, or more simple things like leg size / muscle strenght if you want to make an IA that runs), how to merge 2 parents to create children, and a Metric to evaluate how the network performed.

I assume your snake is make of a grid (like old snake where you only have 4 directions, you can't go in diagonal directions).
In snake example, here is how I would define it in both case (careful, this is how I would model the problem, there might be more efficient models) :
Q-Learning

State : what is in each block of the grid (snake tail, snake head, food)
Actions : Up, Down, Right, Left
Rewards : Not dying : +0.01; dying : -10; eating piece of food: +1.

Genetic Algorithm

Intrinsic Parameters : Parameters of the network that takes decisions (inputs of the network could be whole state of the environment as described in Q-Table).
How to merge : ?
Metric : Lenght of the snake or time lived.

I can't find exactly how to make a genetic algorithm, but it should be possible.
Hope it helps. Q-Learning is usually easier so this is what I would use, but feel free to use whatever you want.


text:
What is a model lineage?
We know about the lineage of datasets. Is there anything called "(ML) model lineage". What are all the works that had been remarkable regarding "model lineage"?
There are few links available on internet which talk about model lineage. According to one of the articles[1], the definition of model lineage is as follows:

Model Lineage keeps the history of a model: when it was trained, using which data, algorithms, and parameters. This should be automatically generated each time a new version of a model is generated.

The next reference that I could find on the internet is the session by data bricks[2].
Apart from these two links, I could not find many resources or standards regarding model lineage. It will be helpful if anyone could provide more resources or pointers towards this topic.

https://blog.tail.digital/en/we-need-to-talk-about-model-lineage/#:~:text=Model%20Lineage%20keeps%20the%20history,of%20a%20model%20is%20generated

https://databricks.com/session_na20/machine-learning-data-lineage-with-mlflow-and-delta-lake

Can you please give some additional context? Perhaps some references where the phrase is used?

Hi Sathya. As pointed out by the pointer in the comment above, you write "We know about the lineage of datasets", but I also don't know what you mean by "lineage of datasets", so please edit your post to include more context (which may also include links and references to other resources where you read this term and that describes it, in addition to what you should do, i.e. describe what "lineage of models" are and why you're interested in "model lineage", whatever that means).

Hi, I have added some details to the question. I hope the additions I made will be useful.


text:
Neural Network for Picking Parameters of a Nonlinear Function to Data Points
I'm trying to make a neural network in pytorch that picks the parameters of a nonlinear function, the radius and (x,y) center of a circle in the example below, based on a sample of values from the nonlinear function.
More concretely, the neural network trained in the code below takes as input 100 (x,y) points on a circle and outputs radius, x_center, y_center of the circle.
I don't consider this a very difficult problem, but the trained neural network doesn't work very well, as you can see from two example plots after the code. How can the code be improved?
And in case this informs your recommendation, the goal is not to fit circles, which no one needs a neural network to do. I'm trying to use a neural network to calculate 9 parameters in a nonlinear function taking a single real valued input and outputting a complex number f(t) -> a + b*sqrt(-1). The input into the neural network is 54 complex values, and the output is 9 parameter values. I am guaranteed that the 54 complex input values can always be well approximated by f(t) with an appropriately picked 9 parameters. The parameters can easily be guessed by a human because different parameters intuitively change the shape of the complex curve, but I've been unable to use a minimization math algorithm for curve fitting. The problem is there are a lot of local minima the minimization algorithms can encounter before reaching the global minimum. The goal of the neural network is to get a good guess of the 9 parameters at the global minimum for a minimization math algorithm to be close to the global minimum initially, and thus converge to the global minimum rather than get stuck at a local minima.
You probably guessed that I know a bit of math, but I don't know much about machine learning. I was able to pick it up pretty quickly because of my math background, but I am severely lacking in practical experience. I don't know what to do at this point other than randomly changing the number of samples on a circle, number of examples circles, adding more layers to the neural network, adding different types of layers to the neural network, changing the loss function, changing the learning rate, changing the optimizer, changing the loss function, et cetera, but I have no method to my madness.
Post Script
I've found someone who did almost what I need. This paper paired with this github repo used 1,000 samples in a set of 100,000 with 1% failure rate, so there's hope. I have to dig deeper for the innards of their neural network training.
import torch
import numpy as np
import math
import matplotlib.pyplot as plt

#circle parameterized by t, < x(t) , y(t) >
t_parameter = np.linspace(-math.pi, math.pi, 100)

#create random radius,(x,y) center or circle paired with points on circle evaluated at all t in t_parameter
examples = 1000
max_radius = 4
random_rxy = np.random.rand(examples,3)
input_list = []
for i in range(examples):
  r_rand = random_rxy.item(i,0) * max_radius
  x_rand = random_rxy.item(i,1) * 7 - 2 #-2 < x_rand < 5
  y_rand = random_rxy.item(i,2) - 2 #-2 < y_rand < -1
  x_coordinates = [r_rand*math.cos(t) + x_rand for t in t_parameter]
  y_coordinates = [r_rand*math.sin(t) + y_rand for t in t_parameter]
  input_list.append(x_coordinates + y_coordinates)
input_tensor = torch.Tensor(input_list)
output_tensor = torch.Tensor(random_rxy)

print(input_tensor)
'''
tensor([[ x_0_0,   x_0_1,   ..., x_0_99,   y_0_0,   y_0_1,   ..., y_0_99   ],
        [ x_1_0,   x_1_1,   ..., x_1_99,   y_1_0,   y_1_1,   ..., y_1_99   ],
        [ x_2_0,   x_2_1,   ..., x_2_99,   y_2_0,   y_2_1,   ..., y_2_99   ],
        ...,
        [ x_997_0, x_997_1, ..., x_997_99, y_997_0, y_997_1, ..., y_997_99 ],
        [ x_998_0, x_998_1, ..., x_998_99, y_998_0, y_998_1, ..., y_998_99 ],
        [ x_999_0, x_999_1, ..., x_999_99, y_999_0, y_999_1, ..., y_999_99 ]])
'''
print(output_tensor) #radious, x circle center, y circle center
'''
tensor([[r_0,   x_0,   y_0  ],
        [r_1,   x_1,   y_1  ],
        [r_2,   x_2,   y_2  ],
        ...,
        [r_997, x_997, y_997],
        [r_998, x_998, y_998],
        [r_999, x_999, y_999]])
'''

#define model and loss function.
model = torch.nn.Sequential(
  torch.nn.Linear(200, 200),
  torch.nn.Tanh(),
  torch.nn.Tanh(),
  torch.nn.Linear(200, 3)
)
loss_fn = torch.nn.MSELoss(reduction='mean')

#train model
learning_rate = 1e-4
optimizer = torch.optim.Adagrad(model.parameters(), lr=learning_rate)
for t in range(10000):
  # Forward pass: compute predicted y by passing x to the model.
  output_pred = model(input_tensor)

  # Compute and print loss.
  loss = loss_fn(output_pred, output_tensor)
  if t % 100 == 99:
    print(t, loss.item())
    '''
    99   0.0337635762989521
    199  0.0285916980355978
    299  0.025961756706237793
    399  0.024196302518248558
    499  0.022839149460196495
    ...
    9799 0.004136151168495417
    9899 0.0040830159559845924
    9999 0.004030808340758085
    '''

  #typical procedure
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

print(output_tensor[0].tolist())
print(output_pred[0].tolist())
#[0.7722834348678589, 0.46600303053855896, 0.5080233812332153 ]
#[0.7921068072319031, 0.46946045756340027, 0.49222415685653687]

plt.xlabel('x')
plt.ylabel('y')
r_rand, x_rand, y_rand = output_tensor[0].tolist()
plt.scatter([r_rand*math.cos(t) + x_rand for t in t_parameter],[r_rand*math.sin(t) + y_rand for t in t_parameter],label="Measured Data")
r_rand, x_rand, y_rand = output_pred[0].tolist()
plt.scatter([r_rand*math.cos(t) + x_rand for t in t_parameter],[r_rand*math.sin(t) + y_rand for t in t_parameter],label="Fit Data")
plt.legend(loc='upper right')
plt.tight_layout()
plt.show()

You would probably have better luck using an RNN, and that way you can feed an arbitrary number of inputs in and the network will slowly refine it's guesses of the parameters you want. You probably don't even need to use a complicated structure, as there aren't really any long term relationships in this problem.

@Recessive I read the full PhD thesis of the author and he used a CNN. His rationale was a sufficient number of consecutive points effectively determines the entire curve, and then lots of sets of consecutive points to train on. For my toy circle problem, three consecutive points with exact coordinates can be used to solve for the radius and (x,y) center of the circle.
I want to emphasize that I'm not saying a RNN would or would not perform better, as I have no idea, but I'll be trying out the CNN first since the paper author at least got that working.

I gave up. I was able to get better results, but at least one of the radius, x center, y center values were off by at least 10% too often to be useful and some of the predicted values were way off. I doubt my circle problem is an unsolved problem, but I couldn't find relevant search results, and then most neural network results, and libraries, are geared toward image, audio, and text processing, which makes sense since that's where the money is. If anyone does find a solution, please let me know. That would help me, and others interested in numerical applications, know where to look.


text:
Unet Overfitting for binary segmentation of fake images
I am working on a project where I am trying to detect and localize forgeries in images. I am using the CASIA v2 dataset and using Unet model for the task. I have the binary masks of all the images in the CASIA v2 dataset. The metric I am using for the model are F1 score.
The issue with the model is that it is highly overfitting, the validation loss plateaus up.
Batch size is 128 and Learning rate is 0.000001. Image size is 128 x 128.
Updated graph for batch size 16 with the changes mentioned by @spb is as follows:

I have also tried using Learning rate scheduler to decrease the learning rate(starting with high learning rate) on plateaus but that didn't help much.
I am also using the package Albumentations for data augmentation of both the images and its masks. I load the images and the masks and then apply the augmentations and save the augmented images and masks in a separate arrays and finally extend the original images and masks with the augmented images and masks. So technically I have original plus the augmented images and masks that I use for training the model. The augmentations I am using are:
Augment = A.Compose([
A.VerticalFlip(p=0.5),
A.RandomRotate90(p=0.5),
A.HorizontalFlip(p = 0.5)
])

I have split the dataset into 70% Training, 20% Validation and 10% for testing.
Here is a snippet of my model. Updated Code below
def conv2d_block(input_tensor, n_filters, kernel_size = 3, batchnorm = True):
"""Function to add 2 convolutional layers with the parameters passed to it"""
# first layer
x = Conv2D(filters = n_filters, kernel_size = (kernel_size, kernel_size),\
          kernel_initializer = 'he_normal', padding = 'same')(input_tensor)
if batchnorm:
    x = BatchNormalization()(x)
x = Activation('relu')(x)

# second layer
x = Conv2D(filters = n_filters, kernel_size = (kernel_size, kernel_size),\
          kernel_initializer = 'he_normal', padding = 'same')(input_tensor)
if batchnorm:
    x = BatchNormalization()(x)
x = Activation('relu')(x)

return x

def get_unet(input_img, n_filters = 16, dropout = 0.1, batchnorm = True):
"""Function to define the UNET Model"""
# Contracting Path
c1 = conv2d_block(input_img, n_filters * 1, kernel_size = 3, batchnorm = batchnorm)
p1 = MaxPooling2D((2, 2))(c1)
#p1 = Dropout(dropout)(p1)

c2 = conv2d_block(p1, n_filters * 2, kernel_size = 3, batchnorm = batchnorm)
p2 = MaxPooling2D((2, 2))(c2)
#p2 = Dropout(dropout)(p2)

c3 = conv2d_block(p2, n_filters * 4, kernel_size = 3, batchnorm = batchnorm)
p3 = MaxPooling2D((2, 2))(c3)
#p3 = Dropout(dropout)(p3)

c4 = conv2d_block(p3, n_filters * 8, kernel_size = 3, batchnorm = batchnorm)
p4 = MaxPooling2D((2, 2))(c4)
#p4 = Dropout(dropout)(p4)

c5 = conv2d_block(p4, n_filters * 16, kernel_size = 3, batchnorm = batchnorm)
p5 = MaxPooling2D((2, 2))(c5)
#p5 = Dropout(dropout)(p5)

c6 = conv2d_block(p5, n_filters = n_filters * 32, kernel_size = 3, batchnorm = batchnorm)

# Expansive Path
u7 = Conv2DTranspose(n_filters * 16, (3, 3), strides = (2, 2), padding = 'same')(c6)
u7 = concatenate([u7, c5])
u7 = Dropout(dropout)(u7)
c7 = conv2d_block(u7, n_filters * 16, kernel_size = 3, batchnorm = batchnorm)

u8 = Conv2DTranspose(n_filters * 8, (3, 3), strides = (2, 2), padding = 'same')(c7)
u8 = concatenate([u8, c4])
u8 = Dropout(dropout)(u8)
c8 = conv2d_block(u8, n_filters * 8, kernel_size = 3, batchnorm = batchnorm)

u9 = Conv2DTranspose(n_filters * 4, (3, 3), strides = (2, 2), padding = 'same')(c8)
u9 = concatenate([u9, c3])
u9 = Dropout(dropout)(u9)
c9 = conv2d_block(u9, n_filters * 4, kernel_size = 3, batchnorm = batchnorm)

u10 = Conv2DTranspose(n_filters * 2, (3, 3), strides = (2, 2), padding = 'same')(c9)
u10 = concatenate([u10, c2])
u10 = Dropout(dropout)(u10)
c10 = conv2d_block(u10, n_filters * 2, kernel_size = 3, batchnorm = batchnorm)

u11 = Conv2DTranspose(n_filters * 1, (3, 3), strides = (2, 2), padding = 'same')(c10)
u11 = concatenate([u11, c1])
u11 = Dropout(dropout)(u11)
c11 = conv2d_block(u11, n_filters * 1, kernel_size = 3, batchnorm = batchnorm)

outputs = Conv2D(1, (1, 1), activation='sigmoid')(c11)
model = Model(inputs=[input_img], outputs=[outputs])
return model

Currently I am not using the dropout as it leads to higher validation loss plateaus in my case.
The F1 score and  F1 loss I am calculating are as follows
def f1(y_true, y_pred):

y_pred = K.round(y_pred)
tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)
tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)
fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)
fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)

p = tp / (tp + fp + K.epsilon())
r = tp / (tp + fn + K.epsilon())

f1 = 2*p*r / (p+r+K.epsilon())
f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)
return K.mean(f1)

def f1_loss(y_true, y_pred):

tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)
tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)
fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)
fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)

p = tp / (tp + fp + K.epsilon())
r = tp / (tp + fn + K.epsilon())

f1 = 2*p*r / (p+r+K.epsilon())
f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)
return 1 - K.mean(f1)

I have also tried using other losses like focal_tversky but have a similar result.
What can be the issue  and how can I solve it?
Is it

Issue with my data like presence of outliers
Model related issue
Batch size and Learning rate related issue
Or anything else?

Please your help in this regard is really appreciated as I really need to solve it soon.

Data augmentations is usually done on the fly during training, meaning before each you apply the random augmentation for the entire dataset, because of the randomness there will be different transformation of the same image in each epoch.
Shuffle the dataset before batching in each epoch, so that each epoch will not have minibatch of same images, which will reduce overfitting. Learning rate usually 1e-4 works fine for me.
Your UNet is not wide enough, why are you using only 16 filters in first conv block, original UNet paper had 64 filters in first conv block. Also you have only one convolution block in each layer, why? original unet has 2 conv blocks in each layer. I suggest you to try with unet given in here
https://github.com/zhixuhao/unet/blob/master/model.py
Dice loss is usually prefered for segmentation, check code here
from keras import backend as K

def dice_score(y_true, y_pred, smooth=1e-7):

    intersection = K.sum(K.abs(y_true * y_pred), axis=-1)
    return (2. * intersection + smooth) / (K.sum(K.square(y_true),-1) + K.sum(K.square(y_pred),-1) + smooth)

def dice_loss(y_true, y_pred):
    return 1-dice_score(y_true, y_pred)

Thanks for your in detail answer. Just to clear some things, so random augmentations should be applied before each epoch on the entire dataset, right?
Also I updated the code in the post above, I am using two convolution layers, conv2d_block is actually a function i made for each layer, i have put that code above as well for your reference.

Yes, apply random augmentations before each epoch on the entire dataset.

What about number of filters in the first layer, did you try changing to 64? Also try removing dropouts in contracting path, it worked well for me.

Yes, I am not using dropouts they are commented, so you used dropouts in the expanding path?
I am going to try changing filters to 64 now and get back to you.

Yes, dropout(0.5) in expanding path reduced overfitting

I do not understand why you say that your model is overfitting. An overfit occurs when the validation loss start increasing after diminishing. Here it seems that your model has reaches its potential and cannot improve anymore. What I would recommend here is to make your model bigger: add filters, increase the depth. Also consider trying transfer learning; it is a common base to all tasks.

Ok I will try working with a bigger model. Thanks for your inputs. I also updated my recent graph in the question above considering the inputs given by @spb and got these results. Do you think its overfitting or the model is basically reaching its potential?

Still, I don't see the shadow of an overfitting. However, I checked the dataset that you use, and it says to contain 4795 images. I think that training a model from scratch on a small dataset like this will not give you incredible results. Transfer learning can help a lot with that, combined with data augmentation.

Any recommendations which weights I shall use for transfer learning?

The U-Net architecture is basically an encoder and a decoder with skip connections between them. you can use any pre-trained deep ConvNet for the encoder part and then build your decoder accordingly with transposed convolutions; and don't forget to add skip connections - very important.
Here you have the pre-trained models available in tensorflow that you can use for the encoder.


text:
variational autoencoder - decoder output for images
Following the standard setup/notation for a VAE, let $z$ denote the latent variables, $q$ as the encoder, $p$ as the decoder, and $x$ as the label. Let the objective be to maximize the ELBO, where a single sample monte carlo estimate of the ELBO is given by
\begin{align*}
\log p(x \, | \, z) + \log p(z) - \log q(z \, | \, x)
\end{align*}
Now I want to focus only on the $\log p(x \, | \, z)$ term, where $x$ is an image, and the decoder $p$ outputs the mean/variance of a normal distribution for each pixel.
My understanding is that pixel values should be integer 0-255. Now consider a single pixel: suppose that the ground truth for that pixel is the value 10, and the encoder predicts the mean, variance $\mu, \sigma^2$ respectively. Now when computing the ELBO, we have this term
\begin{align*}
\log p(x \, | \, z) = \log f(10, \mu, \sigma^2) \tag{1}
\end{align*}
where $f$ is the probability density of the normal distribution. My question is why it is justified to compute $\log p(x \, | \, z)$ using the density considering that image data should be discrete valued. It seems to me that any sampled output between 9.5-10.5 would all get mapped to the correct value of 10. Then it seems that you should take the term in the ELBO as
\begin{align*}
\log p(x \, | \, z) = \log \big(F(10.5, \mu, \sigma^2) - F(9.5, \mu, \sigma^2)\big) \tag{2}
\end{align*}
where $F$ is the CDF of the normal distribution.
It seems that all references calculate the ELBO as (1) and none as (2). Why is this justified?

So you mean you want to use a (discrete) gaussian distribution to model a quantity that has finite support (i.e. {0, 1, .., 255}) ? This doesn't sound like a good design choice. Is there any reference you can provide, or is it your own design ?

I'm not sure you understand the question. I am describing the typical setup e.g. https://proceedings.neurips.cc/paper/2016/file/eb86d510361fc23b59f18c1bc9802cc6-Paper.pdf or the original paper https://arxiv.org/abs/1312.6114. the decoder uses a gaussian distribution. The question is asking why it is justified to use the probability density of the normal when considering that the output is an image, which as far as I understand is not technically continuous.

I also don't understand this question, but maybe because it's been a while I had to do with VAEs. What do you mean by "compute p(x | z) using the probability density corresponding to the value of 10".

@nbro I have edited the question to be more precise now.

@Taw Thanks. Now, it's clearer. I've quickly checked the VAE paper and they say that they use a Gaussian and constraint the output to [0, 1] for images. Note that pixels can also been seen as having values in the range [0, 1]. Maybe this partially answers your question. Anyway, I would also advise you to put your specific question in the title now (because right now it's not a question and not specific)


text:
In TD(0) with linear function approximation, why is the gradient of $\hat v(S^{\prime}, \mathbf w)$ wrt parameters $\mathbf w$ not considered?
I am reading these slides. On page 38, the update for the parameters for the linear function approximation of TD(0) is given. I have a doubt regarding this.
The cost function (RMSE) is given on page 37.
My doubt is: why is the gradient of $\hat v(S^{\prime}, \mathbf w)$ with respect to parameters $w$ not considered?
I think the parameter update should be:
$$\mathbf w \leftarrow \mathbf w +\alpha [R + \gamma \hat v(S', \mathbf w) - \hat v(S, \mathbf w)] (\nabla \hat v(S, \mathbf w)- \gamma \nabla \hat v(S', \mathbf w))$$
Instead in the material it is given as:-
$$\mathbf w \leftarrow \mathbf w +\alpha [R + \gamma \hat v(S', \mathbf w) - \hat v(S, \mathbf w)] \nabla \hat v(S, \mathbf w)$$
Could someone please explain?

This is formally known as a semi-gradient method.
What we would like to do is to minimize $\big(v(S) - \hat v(S, w)\big)^2$, where   $v(S)$ is the true value function. This would give the gradient descent update
\begin{align*}
w \leftarrow w + \alpha[v(S) - \hat v(S, w)]\nabla \hat v(S, w) . \tag{1}
\end{align*}
Of course we don't have access to $v(S)$. So instead, we could use the monte carlo return (the observed, discounted, episodic return). The other option is to use a bootstraped estimate of $v(S)$, e.g.  the estimate $r + \gamma \hat v(S', w)$, which would give the update
\begin{align*}
w \leftarrow w + \alpha[r + \gamma \hat v(S', w) - \hat v(S, w)]\nabla \hat v(S, w) \tag{2}
\end{align*}
As you correctly point out, Eq. 2 is no longer a true gradient descent method. To directly quote Sutton and Barto, section 9.3 page 165,

This step [Eq. 1] would not be valid if a bootstrapping estimate were
used in place of v(S). Bootstrapping methods are not in fact instances
of true gradient descent (Barnard, 1993). They take into account the
effect of changing the weight vector w on the estimate, but ignore its
effect on the target. They include only a part of the gradient and,
accordingly, we call them semi-gradient methods.

The 'estimate' here is refering to $\hat v(S, w)$, whereas the 'target' is $v(S)$ (or its approximation). Indeed the parameter update you give would be the true gradient descent update for minimizing $\big(r + \gamma \hat v(S', w) - \hat v(S, w)\big)^2$


text:
Policy Gradient ( Advantage actor-critic) for multiple simultaneous continuous actions
i'm trying to solve a problem in which i need to carry out reinforcement learning with multiple simultaneous actions in continuous action space . i checked the multiagent structure; however, im trying to solve a problem in which there are difficulties to set up connection between the agents. for instance, they should take actions simultaneously so there is no way they can be aware of each other's actions. so i decided to go with the multivariate normal solution. has anybody tried that out ever?
first of all i have have difficulties finding the covariance matrix. since it has to be PSD so i decided to assume covariance is zero. something like:
covariance matrix = [[variance1  0][0  variance2]]
but its not everything. the agent doesn't seem to be learning. the problem to be solved by the agent is about resource allocation so the "mean" can not be negative then i decided to go with the "RELU" activation function for the neural network. surprisingly, mean is usually zero so as you can guess its updating the weights in a way to do nothing (negative mean). on the other hand, the variances are on the rise. Though i have checked it a million times there might be a flaw on the code of the environment there is no doubt. i just wanted to to make sure if its mathematically ok to go in this way ? i checked for papers and i found bunch of them but they don't seem to be enough. i would appreciate any guidance.

Sounds like you have several problems with the way your policy is parametrized.
You don't have to use the multivariate normal distribution. It can work, and probably others have done it already (if not with AAC, surely with DDPG, as it'll be easier to derive the policy gradient there). I won't explain how to use the multivariate normal with either case as there is a much simpler solution for the AAC.
Just have a regular normal distribution over each action. Then, you estimate the mean and variance separately for each action. In other words, it's exactly the same as if you had a single action, but instead of outputting a single mean/variance, you output a vector of mean/variance. Note that this is the same as a multivariate normal distribution, with a diagonal covariance matrix.

thanks for the response. i implemented the way you depicted and it worked so i appreciate your guidance. just to be sure please correct me if i'm wrong. right now i have two policy networks each estimating mean and variance of an action separately.  however, i want the actions to be taken, taking into account the fact that their sum should not pass a limit. clearly, i can not handle this limitation through objective function of the policy networks so i'm penalizing the actions with such characteristics through reward. moreover, i make connection between the two policy networks through state.

You probably want to have a single network for the actions. So the output layer would be dimension 4, for the mean/variance for both actions. As for your additional constraint on the sum of actions, this is a bit more tricky. Using the penalty seems reasonable. You could also try manually clipping the actions if they are over the sum.

so would it be fine to modify the objective function of the policy network in this way?         "[log(pi(a1))+log(pi(a2))]*A(s,a)"  since we are assuming the actions are not correlated  we can write it in this way " log(pi(a1,a2))*A". then, because a1 and a2 are not correlated, joint probability of "pi(a1,a2)" can be written as " pi(a1).pi(a2)". then, we can expand the "log(pi(a1).pi(a2))" as "log(pi(a1)) + log(pi(a2))". taking into account that pi(a1) will be calculated with mean1 and varince1 and pi(a2) will be calculated with its related mean and variance.


text:
How do I prepare this 3D data for NN?
How do I prepare the info of 3D models to use with NN? For example, I have thousands of models with boxes similar to the ones in the image below. I can extract the vertices and their normals that make up the faces of these boxes. Similarly, I would like to prepare the info of the red-shaded surfaces, again I have their vertices and their normals. For future studies, I will have more complex shapes such as cylinders, pyramids,...etc. What would be the best way to represent these complex shapes for NN?
Update: These boxes don't stay in the same position, see the second image I added. I will have different geometric models and different red-shaded areas on the surfaces of these objects. The NN output would be a number for each surface of these boxes/objects. The number represents the surface temperature. The input would be the following:
1- Some climate information such as (air temperature, humidity, ...etc.)
3- The location and size of the buildings that are represented in boxes/or maybe other shapes.
4- The size and the location of the red-shaded areas (red-shaded areas represent the shadow cast by buildings.
5- Material of each surface (concrete, brick,...etc).

Why not as a tensor ?

What do you want the NN to look at? A picture is usually treated as pixels; correspondingly a 3D picture could be treated as voxels.

@hanugm what data goes in the tensor?

Intensity values in 3D simulation as you told. @user253751

the advantage of voxels is that they are equivalent to pixels, which we know work well. The disadvantage is that they're a lot of data! It would be convenient if the vertex coordinates etc could somehow be passed to a NN but I can't imagine an NN learning to work with that input.

I just updated my post to include more info

I don't think NNs are suitable to handle this kind of data. As @spiridon_the_sun_rotator mention mesh CNNs, GNNs, pointnet are more appropriate for this task. As you mention, the methodology is not suitable for your task, I don't understand why? You can use the aforementioned feature vector in mesh CNN.

I think, that the answer depends on the application, but a possible choice would be store it as a mesh - a list of vertices $V$ and edges $E$. Instead of edges, one can work with polygons, and define connectivity $F$ - for triplets of vertices $(v_i, v_j, v_k)$.
There is a nice paper on Mesh CNN that can handle various geometric object.
For the special case of boxes maybe there is a more educated approach, but since you would like to handle later more complex shapes, I would suggest to work with this architecture from the start.

Very nice paper but the methodology isn't suitable for my workflow. I just updated my post to include more info on what I want the NN to look for.


text:
How to classify when the label is seldom known
When I think about classification I think of the cancer/not cancer example. You have a bunch of attributes and you know whether the person had cancer during the relevant time period and you determine which attributes predict that result.
I work in a highly-regulated industry that serves the public. There are certain people we are not allowed to do business with, let's say because they will use our service for illegal purposes. Sometimes we tell the (potential) customer "yes" and sometimes "no".
When we say "yes" and the potential customer intends to use our service for illegal activity, they certainly won't inform us of the mistake.
Likewise, when we say "no" the potential customer will sometimes go away, will sometimes complain, but  that potential customer will not self-identify and say "yes, you are correct, I intended to use your service for illegal activity".
Occasionally we will receive a report from a 3rd-party that will label a customer, but these reports are  a tiny fraction of the number of customers. Unlike the cancer classification we almost always don't know the actual label, we only know what we guessed.
What techniques should we consider to measure our accuracy?

Domain knowledge and cluster interpretation
When you have no (or very limited) gold standard labels for your dataset, traditional performance metrics that require knowing how often you're correct (like accuracy, sensitivity, or specificity) simply won't work. If this is the case, you'll need to examine your outliers to see if they make sense in the context of your particular field. You should look at what features are responsible for a sample being classified as an anomaly, for example, you might see that agents classified as spam send a huge number of emails per hour, or that a suspected money laundering bank account interacts frequently with many offshore accounts. In most cases, you will never know with absolute certainty that your classification is correct, but if you can justify the classification in terms of what is known about your particular domain, it can lend credence to a model.
Another approach is to perform unsupervised clustering on your dataset. Hopefully, you should see that your method identifies anomalies which are somehow different from other datapoints, appearing as outliers or a distinct group of samples unlike the others. You will need to do more investigation to be confident that the clusters represent normal people and fraudsters, but clustering can help you identify that there are indeed some samples that appear markedly different from the others. If you can show that certain samples show similar hallmarks of illegal activity, it also helps to gain confidence in what your model is doing.


text:
Musical notes interpretation
Musical notes
Musical notes videos
Piano
Can AI, Machine learning, Data science, Computer vision, image processing technologies assist in interpreting musical notes ?
Input dataset : Musical notes
Output : Sound played.
A python program which can assist in interpreting or converting text to speech ?.
The text inputted is Musical notes captured as a image by camera as input device.

What are your findings so far? Are you back to our site to pollute it with poor questions? Please, don't do that. For the 1 millionth time, please, read https://ai.stackexchange.com/help/how-to-ask and https://ai.stackexchange.com/help/on-topic. To ask a good question, you need to do a little bit of research before asking it and share with us your findings. You can easily find answers to this question on the web. There's no excuse. If you could find our site to ask this question, you could also have find the answer on the web.

Also, why are you linking us to images of notes? Do you think that people here don't know what a musical note is? It makes no sense to me.

Similar if not duplicate question https://ai.stackexchange.com/q/11431/2444.

I am closing this question for being too broad, because it is basically asking about an entire pipeline of a rather big task (image inputs --> sound outputs based on image contents) whether "it is possible using AI/ML". This invites either just "yes"/"no" answers, or 50-page reports on how to handle the entire pipeline. We require more specific and precise questions about well-defined, specific problems.

AI/ML can solve the task described, a solution is as below:

Regular image processing algorithm (pixel row with min black pixels, adjacent rows are considered as 1) to split the sheet music (as image) into rows (red lines)

Regular image processing algorithm (pixel column with min black pixels, adjacent columns are considered as 1) to split a row into notes based on scanning vertically (blue lines)

Do image classification on every cell created by red lines and blue lines to get notes, and play them
Real-time image classification may be slow and not possible to play note by note directly, the programme should process the whole sheet music first.

LMGTFY ;)
The problem is called Optical Music Recognition. Here you can find tutorial that desribes OMR with deep learning and here you have scientific paper. I think it is very good start for your investigations.