text:
Deep Learning with Best-so-far instead of Where-you-are
It is my understanding that when training a Deep NN in Tensorflow/PyTorch/... we only keep the current state of the network in memory, except perhaps when we manually decide to save the current weights to the HDD/SSD.
Now, naively speaking it may seem reasonable to not only remember the current state (i.e. the current values of the trained weights), i.e. "where we are", but also the "best" weights "so far" by some metric, such as the validation error. Immediately, this approach doubles our memory requirements, especially if we want to keep everything in the GPU memory.
Is this done in practice? If not what are arguments against it?

Now there are certainly details that make this question more tricky. If you saved only each epoch it's probably fine to just save your weights to the HDD without significant loss in speed. If you remember the best weights for each step of SGD than you would have to compute the validation error over and over, which is costly. Or you use perhaps an estimate of the training error based on its gradient, which however could be leading to overfitting if you are not careful. You could also mix and match, or you could compute the validation error only on a subset of the data, or only remember the best every $k$ steps, etc. etc. You might also want to go back and restart from a "better" set of weights in the hopes of getting a better trajectory.
I also don't know whether in practice at the of training people typically, deliberately, go back to a previous "checkpoint" they might have saved somewhere.

Is this done in practice?

Yes, this is done normally when using (lack of) improvements to validation metrics as a stop criterion, and many libraries support it as standard. Depending on the library, you may find you need to add a little code to keep a copy of the best-so-far weights, but some will do it automatically by default, or based on setting params on the train or fit function.
For example, Keras' EarlyStopping class has a restore_best_weights parameter. Using this class in your main fit function, and setting the param to true will do what you want automatically with no other code required.

If not what are arguments against it?

Over-fitting to the validation set is a possible concern, as running the validation checks 100s of times to decide the "best" model may lead to some maximisation bias, and make decisions between other hyperparameters than the number of epochs less reliable.

Interesting. Still kinda makes me wonder whether doing it more often than epoch-wise can be beneficial or is rather detrimental. But I suppose probably no one has checked this or maybe the possible benefit is too marginal to matter - I don't know.

You would have to try it and see. I think the concerns about checking too often would apply more heavily (as well as cost of evaluating) if the validation score was tested every mini-batch (or after some number of mini-batches). And yes, maybe not worth it for expected marginal gains where changes to architecture or regularisation can usually make larger differences.

Common practice
Model checkpoints are often saved to the HDD to keep the GPU memory free.
At every epoch, a selection metric (e.g. validation loss) is evaluated after the training stage completes, and we save the:

Top-k best models (by model selection metric)
Latest model at end of epoch

Example training loop:
best_val_loss = np.inf

for epoch in range(num_epochs):
    out_train = train_epoch(model)
    out_val = validate_epoch(model)

    save(model, "last.ckpt")

    if out_val["loss"] < best_val_loss:
        save(model, "best.ckpt")
        best_val_loss = out_val["loss"]

It is possible that there exist models that best minimize the selection metric during an epoch. However, since the gain is likely quite minimal, it is much more practical to choose the strategy mentioned above.

Is there something better?
You propose some interesting strategies. It is possible that there might be a better process than the typical "train/validation/test over shuffled epoch with batches", but it needs to be shown that such a process leads to noticeable improvements. The convenience and comprehensibility of using a common approach should not be underestimated.

You might also want to go back and restart from a "better" set of weights in the hopes of getting a better trajectory.

Restarting from a better model is almost like training using the validation set, though admittedly not as severe as using validation gradients to optimize the model directly. Typical problems to consider when using such an approach are overfitting and difficulty in escaping local minima.


text:
Emergent behavior in AI models that looks similar to natural neural systems
"ImageNet Classification with Deep Convolutional Neural Networks" by Krizhevsky & Sutskever & Hinton describes very interesting emergent behavior of the AlexNet.
It was trained on 2 GPU's:

specialization exhibited by the two GPUs ... The kernels on GPU 1 are largely color-agnostic, while the kernels on on GPU 2 are largely color-specific. This kind of specialization occurs during every run and is independent of any particular random weight initialization

Likewise our brain mostly processes color with left side of the brain.
Are there other examples of emergent behavior in AI models that looks similar to natural neural systems? Any kind from coordination of several neurons to high-level function, usefull or detrimental, like optical illusions?
So far I found only some articles with optical illusion examples.

Yes, there is a whole field where people compare the emergent representations of biological and artificial systems. It is very often found that the role of individual neurons is similar. For example, in the early layers of the visual pathway (retina, LGN, V1), we see receptive fields that are sensitive to local contrast, edges, center-surround and similar simple patterns. The same holds for early layers of CNNs. Intermediate layers both of the visual cortex and of CNNs are sensitive to increasingly complex stimuli such as grids, circles, complex patterns, etc., up to features like eyes, wheels, dog ears. At the last layer, CNN representations are semantic, i.e. coincide with labels (by construction). Equivalently, you find areas of the brain where there are semantic representations, i.e. cells that are sensitive to the concept of (famously) Jennifer Aniston, independently of how we see/hear/read of her.
Just a couple of examples of literature I'm familiar with: Banino et al. found grid cells, similar to those for which the Mosers won the Nobel prize for physiology, in a neural network trained for a spatial navigation task. People like James DiCarlo spent their careers on this kind of thing applied to the visual cortex, a summary can be found in this paper, where they propose a challenge for finding the computer vision model that is most similar to the brain; but all of his work is relevant. In fact, an amazing fact is that CNNs trained to do the same task as an animal (vision), can linearly predict the activity of neurons in the animal's visual cortex better than if the network was directly trained to predict those neurons' activities. People also did similar things for language models and for audio.
These are all works that compare single-neuron activations (receptive fields, preferred stimuli) between biological systems and artificial models. Of course other methods of comparison could be devised but it sounds like this is what you were interested in.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to deal with small reward values
In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale up the rewards, say multiple by 100?

The numbers that a value-based neural network will predict are usually based on expected returns (sum of rewards by end of an episode, or a discounted infinite sum), although in some cases they might be based on average reward. You will generally know which is in use if you are building the environment. For instance, if you typically run episodes that are 1000 time steps long, then rewards in $[-0.01, 0.01]$ on each time step might be fine.
Neural networks are less sensitive to scale of outputs for regression problems than they are to scale of inputs. However it does affect things such as the scale of gradients for loss functions.
You could adjust for small expected returns by increasing the learning rate to compensate. But it should also be OK to re-scale the rewards so that the min and max returns are somewhere in e.g. $[-10, 10]$. The exact range is not a big deal, so you could pick a range where you are comfortable interpreting the values that you see quickly - multiplying by some factor of 10 makes it easier to map back to the original (and presumably meaningful) values in your head.
In the original Atari Game DQN paper, the authors normalised game scores for each of the games. This was mainly so that the same hyperparameters for the neural network worked in all the games (the agent was not tuned specially for each game). However, it may have also had some small benefit of keeping gradients within bounds for more efficient learning.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
'Advancing' basic models
Good morning.
I am a student running a project using medical data, predicting if the patient will or won't get a disease. The data has about 50k cases and 70 features.
I proposed to train 5 models- SVM, KNN, LR, RF and a neural network on this data, using cross-validation to optimize the hyperparameters and then report the best AUROC.
However I've been told by my professor that using these models isn't 'advanced' enough and I need to do something at a higher level.
First- I must admit I am not an expert at ML- i have run similar projects before successfully but nothing more advanced. I have also read a lot of literature on disease prediction using tabulate data and I am struggling to find options that are more 'advanced' and yet still feasible for myself- I have only been learning computer science for 9 months so my abilities are limited. However, I do have about 5 weeks to develop these models so I know I can learn lots.
Please can anybody suggest where I can start? I am really worried about failing the project based on the reaction of my professor to my proposal.
Thanks in advance

what have you found so far that was not advanced enough or advanced enough but not feasible tacking into account your skills? especially the latter is quite difficult to figure out by your question as it is.

@EdoardoGuerriero thanks for your reply. My suggestions were SVM, KNN, Logistic Regression and Random Forest using Sklearn, however she told me 'Anybody can use Sklearn' and it wasn't appropriate. But all i've ever done is use Sklearn, so anything more challenging than that is above my skill level!

If some of your input features are categorical, you could consider a TabTransformer model (see paper and blog post).
There is an official Keras implementation and tutorial here and a third-party Pytorch version on Github. It may count as advanced enough because it uses Transformers, while still having a relatively simple neural network architecture.
There are other examples of structured data classification on the Tensorflow/Keras websites that you can use for inspiration or comparison/baselines, e.g. GBDTs. Another option is to search past Kaggle competitions that are similar to your project, see what kind of advanced models scored highly and learn from their code.
Personally I think anybody who can use Sklearn can use Keras, and I don't have a computer science background. Just look for a beginner-friendly tutorial that suits your needs, ideally one for the latest versions of TF/Keras from the official websites (they've changed quite a bit over the last few years).


text:
Rationalle behind SE3 Transformer?
I have just finished reading the SE3 transformer paper (SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks) by Fuchs et-al and although I'm sure I understand less than 50% of the math involved, I have a lingering question:
Is there a reason one can't use distance matrix representation, e.g.:
$$
 M[i,j]=d(i,j),  \forall i > j
$$
as a valid rotation translation equivarient representation for point cloud data? Why is there a need for a neural network to encode such representation?
(of course, it is not a one - to - one representation, inverted/flipped point clouds can still map to the same distance matrix, but in most practical applications this distinction carry little significance)

After thinking about it for a while, I noticed that distance matrices are rotation and translation invariant rather than equivariant.
For equivariant representation, you'd want that a rotation will produce a rotated output.


text:
what is `Normalize` for in PyTorch transfer learning tutorial?
in this pytorch tutorial, there is transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), what is the purpose of this?
(i removed it and the code still works)

Those are mean and standard deviation used to standardize each channel of the images from IMAGENET used to train the torchvision pretrained models.
Since the models were trained using this preprocessing step, it is useful to apply it also when using those models for transfer learning on new data.


text:
Is batch size of 1 a valid choice for a very deep neural network with high memory requirement?
I am training a very deep neural network (Panoptic-DeepLab) with a ResNet34 backbone on Google Colab on CityScapes dataset for Panoptic Segmentation, and noticed that, with a big crop size, the batch size has to be decreased to 1 image per batch, otherwise CUDA out of memory issues start to occur. While I know that this can create skewness in the training and it will likely be very hard to attain good convergence, can I ask this question in general to the experts: how valid is a batch size of 1 generally considered in image-based processing? The images in consideration can be considered large (high resolution). The optimizer used is Adam alongwith a warm up polynomial learning rate (with base around 0.00005), and 90k iterations.
(I understand that it would possibly be a good idea to try out a smaller crop size and bigger batch size, but would like to know the feedback from the community anyway)

You can do gradient accumulation to artificially use a higher batch size even if only one image can fit in the memory.

After more research, I found that a batch size of 1 is quite common in deep-learning image processing use-cases where there are high memory/GPU requirements for model training. In fact, it gives better results at times.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
Has there been a study done in tuning hyper-parameters for off-policy reinforcement learning?
I am interested in learning about hyper-parameter tuning for off-policy reinforcement learning (specifically DQN). Could someone point me to papers published or empirical observations in this area?

Yes and we can do it in a similar way we normally perform hyper-param optimization. See this paper from Google
In this framework we can also attempt to jerry-rig some of our classic algorithmic tuning techniques with better or worse success depending on the context.


text:
Can I use discrete data in the same model as continuous data?
In my dataset, I have some data that is continuous - eg. Age and BMI. I also have some data that is discrete- for example, occupation is labelled as 1 ="Homemaker"
2="Working"
3="Unemployed"
4="Retired"
5="Extended Sick Leave"
6="Disabled"
7="Other".
My question is- is it correct to pre-process this data the same (0-1 scale, fill in missing values with mean, SMOTE etc) to then put into my model (SVM, KNN, LR, RF). Or do I need to perform different preprocessing with the categorical data? I have looked into One Hot Encoding however concluded it is not appropriate since the data is not string like, but I am unsure.

It's definitely wrong to threat categorical data as continuous. It is also wrong to encode the variable occupation with increasing numerical values before feed them to the models you listed.
Reason being that those numbers have no meaning in relation to those categorical values, for example 3 is bigger than 2 but it has no sense at all to say that the class unemployed is bigger than the class working.
One hot encoding is definitely the way to go to encode categorical variables like occupation.
Filling in missing values with mean of course will not work cause you don't have a mean, and usually that's a very bad way to fill in missing values, even for continuous variables. If you have lot of instances with missing values that you can't discard you can add a new value called "Unknown" and fill in the missing data with that class. Conceptually is the most reasonable thing to do since you have no information.
SMOTE is a whole different story, not related to continuous or categorical variables, and you should consider it only if you have classes (i.e. variables values) heavily underrepresented. It is though a risky move that might lead to over fitting and misinterpretation or your models performances, so if you don't understand it conceptually I would skip it.


text:
How are Neural Networks protected from false training data?
Suppose the training data there exist an element of some data being misleading and some being right, how could the Neural network be trained so that it could filter the right data from the wrong one?
Excuse me if this question is written as a bit non technically my knowledge is not that high of this topic.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

For all intents and purposes, the neural networks are not protected against bias dataset unless the dataset is explicitly curated wrt to an arbitrary bias.
This is very much an open problem and a busy research area with many researchers attempting to address this problem in academic and production environments.
Some of the work on solving this is specific circumstances involves weighting certain data differently and utilizing statistical techniques to otherwise counteract the biases.


text:
Question regarding matlab computer vision application and color recongnition

I am thinking of choosing a computer vision project for my school project(detect crack on surface) and the duration I have is roughly 4 months. With no prior knowledge in neural network, is matlab computer vision application consider to be "user friendly" to beginner?

When detecting a defect by comparing standard image to product image that work by checking the RGB contents of pixels to pixels. Does such method still involve the use of neural network(ie. the RGB information of a pixel can be set not to exceed certain value, otherwise it will mark that pixel as defect)?

2 Yes, cracks and in general defects detection is a task performed with neural networks and deep learning, but depending on your specific use case it might be an overkill. You talk about pixel to pixel comparison, if your data are images taken from a production line with standardize equipment that capture images of the same piece always with same lighting, angle, etc then you'll probably want to investigate analytic methods for more explainable and efficient results. Neural nets become almost compulsory when the images to compare differs a lot between each other, making hard for an expert to came up with analytic rules that are always applicable.
1 I never played a lot with matlab and its deep learning libraries, but I know that at the very least it allows to use models trained with python libraries like tensorflow, this might be an advantage since at least you know you can search models and repositories not exclusively written in matlab.


text:
Why does sklearn perceptron converge for linearly inseparable data points?
I learned that the perceptron algorithm only converges if the dataset is linearly separable. I am implementing this algorithm using scikit learn.
The blue and orange points are from the training set, while red and green are from the test set. Clearly the blobs are inseparable, but why does my perceptron algorithm converge and give me a decision boundary?

Line 30 of your jupyter notebook:
Perceptron(max_iter=40, random_state=0)
The perceptron does not converge, it simply stops after going through the data 40 times. Always read carefully the code and especially the documentation.

Wow, that instantly made sense. Thank you so much! I actually thought the convergence was just naturally achieved within the max_iter.


text:
Which paper describes the effect of learning_starts in Reinforcement Learning?
I have seen many popular RL libraries have a learning_start parameter. This allows the agent to collect enough experiences before training on the replay_buffer. However, I am unable to find the paper that introduced this parameter. This post describes the parameter quite well.
This paper does go a little into this parameter but it seems to be in the context of deep learning and not reinforcement learning.

The replay buffer allows breaking the temporal dependence of the data and thus makes them more i.i.d. (which is what we want). The replay buffer needs to be at least filled with enough experience to compute a single batch, but can also be filled completely before the actual training begins (which avoids 'oversampling' the early experiences). Those implementation details were probably left out in most methods sections.
This question here provides further details on the replay buffer in the context of DQNs.

Thank you for the answer. "(which avoids 'oversampling' the early experiences", do you have any source for that?

I think this follows from the following: Let's say our experiences are in a bag of size N, and currently, we have k (k << N) items in that bag. Each time we sample (e.g. 1 element as batch size) from the bag we also add one new element to the bag (Sampling with Replacement + adding items). After e.g. 1000 iterations we stop. The experiences we added last are the ones we will have sampled the least, the ones, originally in the bag, the most. How much we oversample depends on how different k and N are. Does that make sense?


text:
fondamental question about regularization techniques to solve overfitting problem in neural networks
I have a text classification neural network based on BERT that overfits. The accuracy on the training dataset is 95%, whereas it is 68% on the validation dataset.
Using some classical regularization techniques (dropout=0.5) and weight_decay=0.7, I have an accuracy of 84% on the training dataset and 70% on the validation dataset.
My question is: Do regularization techniques usually improve the validation accuracy? Or they will only reduce the training accuracy to a closer level to the validation accuracy?
As my objective is improve the validation accuracy to 90%, and I am wondering if there is any hope that solving the overfitting problem would increase validation accuracy, so should I continue investigating on the regularization techniques or thinking about changing the whole model.

Regularization techniques reduce overfitting. This is why they tend to reduce training accuracy when applied to a model: they prevent the model to learn noise from the training data. For the same reason the validation accuracy increases: learning less noise means better generalization on unseen data. These two aspects are sides of the same coin.
There would be no point in developing a technique that only reduce training accuracy to match for aesthetic reasons the validation one.
So yes, you can keep investigating regularization techniques and you should always use them. But be aware that since you have a strict goal you want to achieve you might also want to try other architectures or expand your dataset. Regularization will mostly help you to get a better grasp on the best scores you can achieve (if you get 84% accuracy on training, I argue you'll probably never get to 90% validation, unless the model was far from converging) and they boost a bit good results, but they will not help you making a huge jump up.


text:
How to build the actor policy of Soft-Actor-Critic after sampling from a Multivariate normal distribution?
I'm trying to solve LunarLanderContinuous-v2 (https://www.gymlibrary.ml/environments/box2d/lunar_lander/) using Soft Actor-Critic algorithm (following the pseudocode above)

To update the actor policy (step 14) I understood that I need to:

sample an action from a normal distribution using mean and variance that are the output of the policy network
squash the action with tanh to have action bounds [-1, 1]
compute the log_prob, the log probability density of the sample
do the following operation in tensorflow

# STEP 14
bounded_actions_sample, predicted_next_actions_sample = policy_network.predict_actions(batch_observation)
pdf = get_pdf(predicted_next_actions_sample, batch_observation)
q_min = tf.math.minimum(q_network_1(batch_observation, bounded_actions_sample), q_network_2(batch_observation, bounded_actions_sample))

loss_policy_network = tf.reduce_mean(alfa * tf.math.log(pdf) - q_min))

where "pdf" is the value of the probability density function of the normal distribution computed from predicted action (not clipped), mean and variance.
Since the action space of LunarLanderContinuous-v2 is continuous and the possible actions are 2, my idea is to predict means μ, variances σ and a correlation ρ parameter in order to build a Multivariate normal distribution using tfp.distributions.MultivariateNormalTriL and sample actions from it.
My problem is that, according to the Appendix C of the paper https://arxiv.org/pdf/1801.01290.pdf, I have to compute the likelihoods of the bounded actions using

but I don't know how to do that because the first term is a single number due to it come from a Multivariate normal distribution, instead the second term contains 2 elements, one for each action.
Has anyone any idea to code it?

I don't really understand the confusion, as both terms are scalars (where $u_i$ are the components of $u$).
So if $u\sim \mathcal{N}(\mu, \Sigma)$, then the first term would be
$$\log \mu(u\mid s)=-\frac{D}{2}\log(2\pi)-\frac{1}{2}\log\vert \Sigma\vert - \frac{1}{2}(u-\mu)^T\Sigma^{-1}(u-\mu) $$
However, SAC assumes that $\Sigma$ is diagonal, which makes the expression easier, that is
$$\log \pi(a\mid s)=-\frac{D}{2}\log(2\pi)-\frac{1}{2}\sum_{i=1}^D\log\sigma_i - \frac{1}{2}\sum_{i=1}^D\frac{(u_i-\mu_i)^2}{\sigma_i}-\sum_{i=1}^D\log(1-\tanh ^2u_i) $$
If there is a correlation between the individual actions, you would have to implement the first expression.
An explicit implementation I have come across in Pytorch is the following: (where $\sqrt {2\pi} \approx2.5066$)
sampled_u = self.sample_z(mua_tensor.data, siga_tensor.data).data
sampled_a = torch.tanh(sampled_u)

log_pi_exp = torch.sum(- (mua_tensor.data - sampled_u.data).pow(2)
                       / (siga_tensor.data.pow(2)) / 2
                       - torch.log(siga_tensor.data * torch.tensor(2.5066)),
                       dim=-1, keepdim=True)
log_pi_exp -= torch.sum(torch.log(1.0 - sampled_a.pow(2) + EPS), dim=-1, keepdim=True)

However, this is not efficient and there are undoubtedly specific functions in Tensorflow that make this implicitly, but I hope this provides more intuition.

You are right, I got confused with the second term of the equation. Thanks for your explanation!


text:
Do I have enough images? Fine-tuning pre-trained models for bvinary image classification
I am developing a binary image classifier, and my dataset size is of 90 images. From a theoretical point of view, are they enough for fine-tuning a pre-traiend classifier? I plan to test the models listed here.

As a rule of thumb, a dataset in computer vision is considered small when it contains less than 1000 images per class. This holds also for finetuning pretrained models.
This doesn't mean you won't get better results with less than that, but in your case 90 is truly a small amount that will almost certainly lead to overfit those images without any good  generalization.
If the classes you want to classify are not part of any of the main "big" datasets like ImageNet, then unfortunately you'll have to sit down, prepare some coffee and start scraping and annotating images as much as you can.
If instead the classes you want to classify are a subset of some already existing dataset, then your problem becomes more of a transfer learning problem, in which case it will be more useful for you to start investigating preprocessing step that will match your images with those of ImageNet (or whatever dataset was used to train the pretrained weights you're planning to use. Some annotation work will still be beneficial though, since 90 images are very few even just for evaluation, i.e. you'll might find nice tricks like normalization and histogram matching that boost the performance of a pretrained model on your 90 images just to see those step brutally fail in a real production environment.

Thanks for your reply! Do you have any reference for the 1000 images per class rule-of-thumb?


text:
Markov's Decision Process - calculate value in each iteration
I have the following decision tree:

I calculated the value of the plan using the following paramenters (given):
{0 → 1 , 1 → 3 , 2 → 4 },
Discount factor ()= 0.2
I used this formula to calculate the linear equations to find the value of the plan:
 =  +  ∑ ( ,  ,  )
Linear equations:
V0 = 1 + 0.2 (0.5 V0 + 0.5 V2 )
V1 = -1 + 0.2 (0.7 V0 + 0.3 V1)
V2 = -2 + 0.2 (1 V1 )
Now I need to calculate the first three iterations of the value-iteration algorithm, if a discount factor of 0.2 is used and starting initially (iteration 0) with state values all equal to 0. And write it in below format:
S0 = {Value at iteration1, value at iteration2, value at iteration3}
S1 = {Value at iteration1, value at iteration2, value at iteration3}
S2 = {Value at iteration1, value at iteration2, value at iteration3}
How can I find the value at different iterations?
I know that we can use the Bellman Equation:

But how do I use this equation?
Thanks!

The value iteration algorithm defines the following update rule (reference is slide 11 in this MIT course):
$$V_{i+1}(s) = \max_{a}\{R(s,a) + \gamma E_{s'\sim T(.|s,a)} V_i(s')\}$$
for all states $s$, with $E_{s'\sim T(.|s,a)} V_i(s') = \sum_{s'} T(s'|s,a)V_i(s')$. I adapted the transition probability name from $P$ to $T$ to conform with your notation.
Note the index $i$ for the value function on the right hand side of the equations. That's what enables you to do the computations: the value of the state at iteration $i+1$ is calculated recursively from the values at iteration $i$.
We also have $\max_a R(a,s) =$ reward for reaching the state, given that the reward assigned to each state doesn't depend on the action which was taken in your example. Therefore the value iteration update rule simplifies to:
$$V_{i+1}(s) = R(s) + \gamma \max_a \{\sum_{s'} T(s'|s,a)V_i(s')\}$$
If you initialize $V_0(s)$ to the value of the reward function for that state, we have: $V_0(S_0) = 1$, $V_0(S_1) = -1$, $V_0(S_2) = -2$.
Then for $S_0$:
$$\begin{align} V_1(S_0) = R(S_0)  + \gamma \max \{ & T(S_0| a_1, S_0) V_0(S_0) + T(S_2 | a_1, S_0)V_0(S_2); \\ & T(S_0 | a_2, S_0) V_0(S_0) + T(S_1| a_2, S_0) V_0(S_1)\}\end{align}$$
where the first term in the $\max$ is the expected value if you take action $a_1$ from state $S_0$ and the second term is the expected value if you take action $a_2$ from state $S_0$.
This gives, with $\gamma = 0.2$
$$\begin{align}V_1(S_0) & = 1 + 0.2 * \max \{0.5*1 + 0.5 *-2; 0.2*1 + 0.8*-1\} \\ & = 1 + 0.2 * \max \{-0.5; -0.6\} \\ & = 1 - 0.2*0.5 = 0.9\end{align}$$.
Similarly for $S_1$:
$$\begin{align} V_1(S_1) = R(S_1)  + \gamma \max \{ & T(S_1| a_3, S_1) V_0(S_1) + T(S_0 | a_3, S_1)V_0(S_0); \\ & T(S_1 | a_5, S_1) V_0(S_1)\}\end{align}$$
which gives:
$$\begin{align}V_1(S_1) & = -1 + 0.2 * \max \{0.3*-1 + 0.7*1; 1*-1\} \\ & = -1 + 0.2 * \max \{0.4; -1\} \\ & = -1 + 0.2*0.4 = -0.92\end{align}$$
And finally for $S_2$:
$$\begin{align} V_1(S_2) = R(S_2)  + \gamma \max \{ & T(S_2| a_3, S_2) V_0(S_2) + T(S_0 | a_3, S_2)V_0(S_0); \\ & T(S_1 | a_4, S_2) V_0(S_1)\}\end{align}$$
which gives:
$$\begin{align}V_1(S_2) & = -2 + 0.2 * \max \{0.2*-2 + 0.8*1; 1*-1\} \\ & = -2 + 0.2 * \max \{0.4; -1\} \\ & = -2 + 0.2*0.4 = -1.92\end{align}$$
To get the values at the next iteration, you can reuse the equations above incrementing the indices by one.


text:
In a neural network's neuron that has no activation function, to calculate the delta for the neuron during back propagation do you use a derivative?
I have a neural network that is composed of an input layer, two hidden layers and an output layer. The topology is [151, 200, 100, 1] I am using ReLU activation function on the neurons that are in the hidden layers and no activation function on the neuron that is in the output layer. I am wondering if when calculating the delta value of the neuron in the output layer, I should be using a derivative or if I should just subtract the expected output? Here I will put my line of code that this question concerns:
this.deltas[this.NN.length-2][0] = this.NN[this.NN.length-1][0] - expected;
//In forward propagation this neuron has no activation function.

Having no activation function means your activation function is the identity, namely $g(z)=z$.
Therefore, any derivative of $g(z)$ wrt to a parameter is simply the corresponding derivative of $z$, with no extra factor.
You would get the same result if you include the derivative, as it is multiplying by $1$.


text:
"Tweaking" the cost function to penalize rarer cases more severely
I have a very unbalanced data set that I am running a CNN on for regression.  Most of the values are 0, while it is possible for the values to range from 0 to 32.
Is it possible to "tweak" the cost function to penalize rarer cases more severely?
I'm using torch.nn.MSELoss(reduction='none') as my loss function.

Yes, you can use weighted MSE by applying different coefficients to each data point.
Here is an implementation given by Francisco Massa at https://discuss.pytorch.org/t/how-to-implement-weighted-mean-square-error/2547.
def mse_loss(input, target):
    return torch.sum((input - target) ** 2)

def weighted_mse_loss(input, target, weight):
    return torch.sum(weight * (input - target) ** 2)


text:
Is the discriminator of a GAN network embedded in VAE?
From what I understand, a Generative Adversarial Network (GAN) is composed of an encoder (generator), some synthetic data (fake data) and a discriminator that will penalize any distinguishable real data from the fake ones. This will result in 2 training phases where one tries to beat the other one and vice versa.
This force the generator to mimic a distribution given by the synthetic data, thus creating a latent space similar the fake data.
However, in Variational Autoencoders (VAE), the usual loss function also includes a penalty in the latent space when the data do not follow a normal distribution. From what I've learned, this also strongly encourages the latent space to follow a normal distribution.
My question is how does the penalty in the VAE is different from training a whole new network, the discriminator, that in the end penalize the generator for not following the synthetic data distribution?
More generally, is it similar to penalize a wrong latent space distribution directly in the model's loss function than it is with a discriminator network?
Maybe it's easier with GAN to make new distributions, at the cost of training another network, than it is to model an adapted loss function. (?)

To train a GAN you need only 2 networks, the generator and discriminator, none of which require to be an encoder. You also don't require synthetic data, those are generated by the generator network, you need instead real data.
"This force the generator to mimic a distribution given by the synthetic data, thus creating a latent space similar the fake data"
We're not interested in mimic synthetic data, what a GAN is suppose to learn is to generate data that follow a similar distribution of some real data. So initially the generator will create only noise, if the training is successful then the data generated by the generator should become closer and closer, eventually indistinguishable from the real data. Also, there's no requirements regarding to the latent space, even though it's true that GANs can be trained to learn latent spaces with specific property to control the features of the generated images in a similar fashion to VAE.
The power of GANs when compared to VAEs is that on the paper they can map any kind of distribution to any other kind of distribution. This is possible cause rather than using a single loss that compare a network output to a ground truth label, GANs use two losses combined that mimic a distance between distributions. The combined loss nowhere include a direct comparison of the generated images with a target. Rather it just leverage the discriminator to compute how far (in terms of probability distributions) the generated and real images are.


text:
What's the relationship between Bayesian RL and POMDPs?
Bayesian RL seems concerned with having uncertainty over the transition function of the environment, while POMDPs try to capture uncertainty over the state one is currently in. However, both end up reducing to something like a belief MDP: an MDP in which everything is known, but the states are points in belief space.
Could anyone explain what the differences between these are in more detail?

After looking more into this, I think I have a better understanding.
In Bayesian RL, one has uncertainty over the transition function of the environment. For example, we might know that our robot can slip when transitioning state, but we don't know what the probability of slip $\theta$ is (although we have some prior over it).
In POMDPs, one has uncertainty over the current state. For example, we might not know exactly where the robot is located, and we try to understand that from observations (which are a function of the state, e.g. sensor readings).
In both POMDPs and Bayesian RL, a similar trick applies: one can "tuck one's uncertainty into the state", reducing both problems to MDPs. This is nice because we already have a bunch of methods developed to solve MDPs, so we don't have to reinvent the wheel. However, these are a particular type of MDP: in which each state corresponds to a specific "belief-state".

In POMDPs, each belief-state will only contain a belief over which state one is (e.g. a distribution over x-y locations).
In Bayesian RL, each belief-state will contain both a state and a belief over the unknown parameters of the transition dynamics (e.g. a Beta distribution over $\theta$).

How the transition dynamics are computed in POMDPs and Bayesian RL are obtained is also subtly different, but the idea is similar: one simply incorporates the latest information one gains through observations, e.g. "did the robot slip this last timestep?", or "what were my last sensor readings this last timestep?".
For more information on POMDPs, see "Planning and acting in partially observable stochastic domains" by Kaelbling. For more info on Bayesian RL, this lecture seems like a good starting point.


text:
Storing training dataset in a platform like mlflow
I am pretty new to Machine learning and would like to know whether there are experiment management platforms that also allow storing and managing training datasets (images, in my case). I am familiar with the ML-Flow, but IMHO it doesn't support such an option, am I right? Anyways if there are any, please tell me which. If there are no platforms like this, how would you suggest managing training datasets in combination with existing platforms? I appreciate any help you can provide.

What you are looking for is a DVC (Data version control) tool. One such tool is available at dvc.org

You can go through their documentation to get started.
https://dvc.org/doc/start


text:
Binary Classification Model Comparison - Interpretation of Training, Test and Validation Set Performance
I am looking for some advice regarding the best choice of binary classification model based on training, validation and test set results. Model 1 (results in 1st image) shows better test set results than Model 2, but Model 2 (results in 2nd image) shows results that seem more intuitive to me with better training set performance than its test set performance. I feel as if the Model 1 test set results might have been a bit of a fluke, whereas Model 2 appears more like a well trained model with more long-term reliability.
Any advice on this is much appreciated.

Generally if a model A performs worse than model B on "Test set" while performing better on "Train/Val set" then model A has suffered from overfitting.

So, in your case Model 1 should give you long term reliability against unseen data.

Thank you very much for your answer Marib.

If you are satisfied with the answer, can you mark it as accepted


text:
What does it mean for an episode to start in a state-action pair?
In Sutton and Barto on chapter 5 (p.96), they talk about estimating state-action values with Monte Carlo:

For policy evaluation to work for action
values, we must assure continual exploration. One way to do this is by specifying that
the episodes start in a state–action pair, and that every pair has a nonzero probability of
being selected as the start. This guarantees that all state–action pairs will be visited an
infinite number of times in the limit of an infinite number of episodes. We call this the
assumption of exploring starts.

What does it mean for an episode to start in a state-action pair? Examples are welcome

It simply means that you start an episode in any possible state-action combination. Consider a gridworld type environment: an example of starting in a state-action pair would be to start in the top left most square with the action selected as being 'down'. If you do this random initialisation for each episode then in the limit of infinite episodes you will see each possible state-action pair infinitely often, ensuring sufficient exploration.


text:
Are there chatbots which use AI but not machine learning?
I want to see the differences between two chatbots where the first one is based on AI without ML and the second one which is based on ML.

The very first chatbot called ELIZA was developed in 1966, so way before machine learning. It was based on pattern matching and substitution, so very limited range of answers, but still a chatbot.
If you're confident with python you could try this implementation


text:
Why does a PPO agent perform only the action that costs the least?
I am trying to implement an intelligent agent that can perform penetration testing within the nasim (link) environment, a network simulator.
I would like to try to use parametric mode for actions, and therefore I need an agent that can work with OpenAI gym's MultiDiscrete actions since nasim implements OpenAI gym. I have therefore chosen to use PPO's implementation of Stable Baselines3 (link).
Everything seems to work when I run the agent on a small scenario (tiny scenario). Still, when I try to run it on a somewhat larger environment (tiny-small scenario) the agent after a few hundred thousand timesteps decides to run only the action that costs the least. Here is the code of the agent:
import gym
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
import nasim

if __name__ == "__main__":

    # Build the environment
    env = gym.make("nasim:TinySmall-v2")

    # Train the model
    model = PPO("MlpPolicy", env, verbose=1)
    model.learn(total_timesteps=500000)

    # Init the environment
    obs = env.reset()

    # Init variables
    total_reward = 0
    number_of_steps = 0
    done = False

    # Do the test
    while done != True:
        action, _states = model.predict(obs)
        number_of_steps = number_of_steps + 1
        obs, rewards, done, info = env.step(action)
        print("Action: " + str(action) + ", reward: " + str(rewards))
        total_reward += rewards
        print("Total reward: " + str(total_reward))
        print(done)
        print("Steps: " + str(number_of_steps))
        env.render(mode='readable')

I tried looking for the cause of the problem in the algorithm, including watching this explanation video (link)., but I could not identify the cause of the problem.
Step to reproduce the problem:

Install stable-baselines3: pip install stable-baselines3
Install version 0.9.1 of nasim (version 0.10 is too recent to be compatible with the PPO implementation of stable-baselines 3): pip install -Iv nasim==0.9.1
Install the last required component: sudo apt install python3-tk
Save the agent code written above in a .py file and run it.

The agent has probably learned that it's not worthwhile to explore further, and greedily minimizes cost by taking the least expensive action. I haven't looked at the environment, but one simple test could be to increase the entropy bonus of PPO, and see if the policy explores more (you can keep track of the policy's entropy to see if this is indeed the issue).

The agent has probably learned that it's not worthwhile to explore further, and greedily minimizes cost by taking the least expensive action. I haven't looked at the environment, but one simple test could be to increase the entropy bonus of PPO, and see if the policy explores more (you can keep track of the policy's entropy to see if this is indeed the issue).

You were right. I just modified the line in which I invoke the PPO in this way: model = PPO("MlpPolicy", env, verbose=1, ent_coef=0.05). Now everything works fine


text:
Reward design or Inverse reinforcement learning?
I'm working on a reinforcement learning project where I only have demonstrations (i.e. set of states and actions). During my research on how handle the reward signal, I noticed that research papers often design their reward functions, based on heuristics and human/expert knowledge. Meanwhile, when I read papers on Inverse RL, they claim that such approach can be used in applications where designing the reward function is not trivial, although this approach is not common. I'm not experienced in the field, but I'm wondering how one can decide whether to go for designing the reward function or using IRL, instead?

It depends on the domain you are in.
Inverse RL (IRL) would be most advantageous in domains in which:

It's hard to specify the reward by hand: for example, it would be hard to hand-specify a reward for "safe driving", and
Demonstrations from experts are available: one could find "safe drivers" and collect some of their data, and then use IRL to obtain a more nuanced reward than what we could hand-specify.

That being said, IRL can't provably recover the true reward function (many reward functions will always be consistent with the observed data), and it tends to be very computationally intensive. This is why it's not used very often in practice.
If it's easy to specify the reward (e.g. "win at chess"), hand-specifying it is the simplest way to go about it.
See this related question about ways to hand-design reward functions.


text:
How do I interpret this loss function?
In this AI note from https://deeplearning.ai, the loss function below is used for a regression problem. However, I don't know how to interpret this loss function.

First, does the author take the square of the difference between y-hat (prediction) and y (ground truth), so that positive and negative numbers don't cancel each other out? If so, why do we take the norm or distance, as well? Isn't the norm positive anyway? Or does he take the square so that it's more convenient to calculate the derivative?
Second, what does the other 2 in subscript mean? It's not explained in the note and I was also not able to derive it from the context. All I know is that y ∈ R.
It seems like I find it difficult to read mathematical notations. If you know a resource that explains these notations, please let me know.

This is the sum of squared residuals, and it uses notation from the mathematical subfield of linear algebra (arguably functional analysis).
The double vertical bars indicate that we use the mathematical operation of a norm, which is a special kind of measure of distance that holds familiar properties.
The subscript $2$ means that it is the $L_2$ norm. Until you know this notation, there’s no way to know that, but it’s so common that it’s worth remembering what an $L_p$ norm means. Let $x=(x_1,\dots,x_n)\in\mathbb R^n$ be a vector of real numbers.
$$
\vert \vert
x
\vert\vert_p
=
\left(
\sum_{i=1}^n\vert x_i\vert^p
\right)^{1/p}
$$
For the $2$-norm in your equation, this is just the usual Euclidean distance.
The $2$ superscript has its usual meaning, that you square the $L_2$ norm once you calculate it.
Finally, $y-\hat y$ means that you take the vector of true $y$-values and subtract the vector of predicted $y$-values, denoted by $\hat y$.
In total:

Take the difference between the true and predicted values.

Square each component of that difference.

Add up all of those squared differences.

Take the square root of that sum.

Square that square root.

As the last step undoes the fourth step to get you back to the third step, you can stop at step $3$.
EDIT
It is common to refer to the residuals as errors. From a statistical standpoint, this is not quite correct, though it seems to be a reasonable slang that does not cause confusion among experienced practitioners.


text:
Does the Weights of Discriminator get updated when traning Generators in GANs?
When we train the GAN we usually train the discriminator first then the generator, first we stop the generator from updating its weight by removing it from the computation graph, using fake_image.detach()
noise=get_noise(num_images,z_dim,device=device)
fake_images=gen(noise)

disc_fake_pred = disc(fake_images.detach())
disc_fake_loss=criterion(disc_fake_pred,torch.zeros_like(disc_fake_pred))
disc_real_pred = disc(real)
disc_real_loss=criterion(disc_real_pred,torch.ones_like(disc_real_pred))

disc_loss = (disc_real_loss + disc_fake_loss)/2
disc_loss.backward()
disc_opt.step()

But when we want to train the discriminator we don't stop discriminator weights all we do is just leave the generator weights in the computational graph, this means discriminator and generator weights will be updated since we didn't stop it
gen_opt.zero_grad()
z_noise = get_noise(num_images, z_dim, device)
fake_images = gen(z_noise)
disc_fake_pred = disc(fake_images)
gen_loss = criterion(disc_fake_pred, torch.ones_like(disc_fake_pred))
gen_loss.backward()
gen_opt.step()

But in

Since you are using Pytorch, an optimizer can only update parameters wrapped inside it since initializing.
Therefore, even if the gradient of discriminator parameters exists while updating generator, discriminator parameters are still stable if you don't call disc_opt.step().

you are correct


text:
Can YOLOv3 architecture be clearly separated into feature extractor and classifier parts?
I am new to machine learning and am confused about whether its architecture has clearly defined boundaries which demarcate the feature extraction and classification part. Or is it that it classifies the features extracted at different scales throughtout the network ?

The feature extraction part is described in section 2.4 of the original paper. The proposed darknet-53 is an improvement over the previous darknet-19 which didn't include residual layers.

After the darknet backbone, three detection blocks are responsible for predicting bounding boxes and probability scores for each object class. notice that YOLO is an end-to-end architecture, so bounding boxes and objects are predicted at the same time.

Thank you for your reply. It has cleared all my doubts about this topic.


text:
How to detect that the fitness landscape of a genetic algorithm is changing over time?
I understand that in each generation of a genetic algorithm, that generation must re-prove it's fitness (and then the fittest of that population is taken for the next population).
In this case, I guess it's a presumption that if you take the fittest of each generation, and use them to form the basis of the next generation, that your population as a whole is getting fitter with time.
But algorithmically, how can I detect this? If there's no end goal known, then I can't measure the error/distance from goal? So how can you tell how much each generation is becoming fitter by?

I'm not sure I fully understand your concerns, but one natural way of measuring the fitness of the population, as a whole, is to calculate the average fitness (or something similar) and plot that as a function of time (number of generations). Is this what you're asking? I don't actually know if this is the best way of doing it, but this is what comes to my mind right now. The other thing I don't understand is why you're confused about how to calculate the fitness. The definition of the fitness really depends on the problem. If you need to define how to calculate it for your problem.

There is no exact way to assess that a genetic algorithm has located a global optima. Indeed there may be multiple global optima. You must fall back to heuristic methods. The fitness of a population is the maximum fitness of any individual. Unless specific measures are taken to maintain diversity the population will converge to an optima, local or global. At that point all individuals will, except for mutation, be identical. You could take the fittest individual of such a population as your solution, but you will not know if the solution is a global or local optima.
Two reasonable heuristics are these. First, run the algorithm till it converges and maintains its fitness for a number of further generations. Or second run the algorithm multiple times, and take the fittest of all the located solutions. Neither is exact.

Interesting thanks. I'm thinking of genetic algorithms that are used in adversarial problems, so something that was once fit....could now not be fit, and vice versa, and how that could be detected with an algorithm. So I appreciate what you're saying.

There is no general method to detect a change in the fitness landscape, since changes can be very local and can occur in just a small area of the fitness landscape. For this reason nature inspired optimization algorithms usually maintain a diversified population to cope with environmental changes. a common mechanism is using several sub-populations and ensuring that these sub-populations do not overlap. Also, there are some heuristics proposed that can help the algorithms in detecting changes. for instance if you are using multiple sub-populations, you can double test one element of each sub-population at some generation to find out if an environmental change has occurred or not. albeit, there are some comprehensive heuristics have been proposed for change detection, like the one proposed in [R. Mukherjee, S. Debchoudhury, S. Das, Modified differential evolution with locality induced genetic operators for dynamic optimization]. In my opinion, Use simple methods. for instance, at the keep a small by diverse set of population members at some k generations, and reevaluate them after k generations to detect change.

This answer can be improved by providing links to references that support your claims.


text:
Should we use a pre-trained model or a blank model for custom entity training of NER in spacy?
Further to my last question, I am training a custom entity of FOODITEM to be recognized by Spacy's  Name Entity Recognition engine. I am following tutorials online, following is the advise given in most of the tutorials;

Load the model or create an empty model

We can create an empty model and train it with our annotated dataset or we can use the existing spacy model and re-train with our annotated data.

But none of the tutorials tell how/why to choose between the two options. Also, I don't understand how will the choice affect my final output or the training of the model.
How do I make the choice between a pre-trained model or a blank model? What are the factors to consider?

Would you add reference or weblink for retraining using spaCy tool?

The reason you would load a pre-existing model is that it offers something of value to your task (e.g. named entity recognition for food) and the cost of training it from scratch is not worth it.  For example, to train GPT-3 from scratch would cost several million dollars.  Typically someone will use a model like BERT and fine tune it.  This is called transfer learning.  With spaCy you will typically use en_core_web_sm which was trained on the OntoNotes corpus and includes named entities.  Making a custom food NER using en_core_web_sm should be more accurate than making one from scratch. You should be able to build a good model with and without transfer learning fairly quickly if you have a GPU.

Would you add reference or web link of transfer learning? For the spaCy tool, I found only https://spacy.io/usage/embeddings-transformers.


text:
How does DDPG algorithm know about my action mapping function?
I am using DDPG to solve a RL problem. The action space is given by the Cartesian product $[0,20]^4\times[0,6]^4$. The actor is implemented as a deep neural network with an output dimension equals to $8$ with tanh activation.
So, given a state s, an action is given by a = actor(s) where a contains real numbers in [-1,1]. Next, I map this action a into a valid action valid_a that belongs to the action space $[0,20]^4\times[0,6]^4$. Than, I use valid_a to calculate the reward.
My question is: how does the DDPG algorithm know about this mapping that I am doing? In what part of the DDPG algorithm should I specify this mapping? Should I provide a bijective mapping to guarantee that the DDPG algorithm learns bad from good actions?

I would recommend doing is allowing your network to output any real number and then clipping the output. For instance, I was working with an agent that had to learn an angle between $[0, 2\pi]$ and $[0, 1]$. If the network outputted e.g. 10 in the first dimension then this would just be clipped to $2\pi$.
This way the agent only learns about actions within the action space and the weights of the network would eventually be adjusted to only output actions within this action space, provided that the boundaries aren't the optimal actions.


text:
Can a convolutional network predict states for a RL Agent
During the course of training a DQN agent, all visited states are stored in a replay buffer. Therefore would it be practically possible for a CNN, given a reasonable amount of data, to predict the next RL state (in the form of an image) for a given action?
For example, take the game of Atari as shown below -

Here the agent can take 2 major actions - go left and go right. Would a CNN be generate the image of the bar going right/left for the respective actions?
My practical knowledge of CNNs is quite limited and therefore I'm trying to gauge the abilities of CNNs before I take up a project.

It sounds like what you're suggesting is similar to what is done in methods that use a planner. These methods looks to learn the dynamics of the MDP to use to plan during training; that is they want to be able to learn the transition probabilities $p(s'| s, a)$.
In this paper that I read recently they note that learning to predict environment dynamics when the state/action space is high dimensional, as is the case with images, is difficult; so whilst it may be possible in theory it would be difficult to do and if you were predicting many steps into the future then the error would compound.
A way around this, as is done in the referenced paper, is to use predict environment dynamics in a latent space. This means that they use a latent variable to predict the next state using e.g. Variational Autoencoders.


text:
Is it possible to modify or replace the basic network of YOLO?
I have an idea to adapt YOLO algorithm to my application, the original YOLO algorithm is for image classifications, which have 24 convolutional layers with output class of 1000, is it possible to replace the basic network of YOLO with Alexnet or Resnet or the custom network structure designed by myself? Noted that my application have input shape of 500 * 10000 * 1 and only 4 classes for classification.

No, the original (or any) YOLO is for object detection. You can easily replace the feature extractor (DarkNet53, if I'm not mistaken) with any other, as long as you maintain the correct number of weights in the detection layer.

The basic network behind YOLO is CNN, why it's cannot be replaced?

Pls read my answer again


text:
Factors that causing totally different outcomes from an exactly same model and datasets
Here is a model that trains time series data in (batch, step, features) way.
I have kept the random state for train test split function the same. Every parameter below the same, running the model training yields different outcomes every time and the outcomes are drastically different.
What may be the factors that led to this? Regularization?

X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=666)
def attention_model(X_train, y_train, X_test, y_test,num_classes,dropout=0.2, batch_size=68, learning_rate=0.0001,epochs=20,optimizer='Adam'):

    Dense_unit = 12
    LSTM_unit = 12

    attention_param = LSTM_unit*2
    attention_init_value = 1.0/attention_param

    u_train = np.full((X_train.shape[0], attention_param),
                      attention_init_value, dtype=np.float32)
    u_test = np.full((X_test.shape[0],attention_param),
                     attention_init_value, dtype=np.float32)

    with keras.backend.name_scope('BLSTMLayer'):
        # Bi-directional Long Short-Term Memory for learning the temporal aggregation
        input_feature = Input(shape=(X_train.shape[1],X_train.shape[2]))
        x = Masking(mask_value=0)(input_feature)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005), activation='relu')(x)
        x = Dropout(dropout)(x)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005),activation='relu')(x)
        x = Dropout(dropout)(x)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005),activation='relu')(x)
        x = Dropout(dropout)(x)
        x = Dense(Dense_unit,kernel_regularizer=l2(0.005), activation='relu')(x)
        x = Dropout(dropout)(x)

        y = Bidirectional(LSTM(LSTM_unit,activity_regularizer=l2(0.000029),kernel_regularizer=l2(0.027),recurrent_regularizer=l2(0.025),return_sequences=True, dropout=dropout))(x)
#         y = Bidirectional(LSTM(LSTM_unit, kernel_regularizer=l2(0.01),recurrent_regularizer=l2(0.01), return_sequences=True, dropout=dropout))(y)

    with keras.backend.name_scope('AttentionLayer'):
        # Logistic regression for learning the attention parameters with a standalone feature as input
        input_attention = Input(shape=(LSTM_unit * 2,))
        u = Dense(LSTM_unit * 2, activation='softmax')(input_attention)

        # To compute the final weights for the frames which sum to unity
        alpha = dot([u, y], axes=-1)  # inner prod.
        alpha = Activation('softmax')(alpha)

    with keras.backend.name_scope('WeightedPooling'):
        # Weighted pooling to get the utterance-level representation
        z = dot([alpha, y], axes=1)

    # Get posterior probability for each emotional class
    output = Dense(num_classes, activation='softmax')(z)

    model = Model(inputs=[input_attention, input_feature], outputs=output)

    optimizer = opt_select(optimizer,learning_rate)
    model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer=optimizer)

    hist = model.fit([u_train, X_train],
                     y_train,
                     batch_size=batch_size,
                     epochs=epochs,
                     verbose=2,
                     validation_data=([u_test, X_test], y_test))

#kernel_regularizer=l2(0.002),recurrent_regularizer=l2(0.002),
    return hist

batch_size= 150
#217
epochs = 1000
learning_rate = 0.00081
optimizer = 'RMS'
num_classes = y_train.shape[1]
dropout=0.22

tf.keras.backend.clear_session()

history = attention_model(X_train, y_train, X_test, y_test, num_classes,dropout = dropout,batch_size=batch_size, learning_rate=learning_rate,epochs=epochs,optimizer=optimizer
)

Hello. Welcome to AI SE. I don't know if this is "just" a programming issue or not, but I would like you to note that programming questions (like "Why am I getting this programming error/bug?") are generally off-topic here. These questions are better suited for Stack Overflow. I don't know if your question is a pure programming issue (because I didn't read it, but it contains code...), but just keep this in mind. You should read https://ai.stackexchange.com/help/on-topic for more info.

@nbro Hi nbro, I was not sure what was causing the problem. My initial thought was that the problem was caused by some randomness of models not because of codes so I post it here.

By default, Keras sets shuffle argument True, so you should set the numpy seed before importing Keras.
CPU
from numpy.random import seed
seed(25)
from keras.models import Sequential

GPU
tf.random.set_seed(seed)

Hi yahkyo, I set the random seed to a fixed value and rerun the whole model but the outcomes still changed every training. So I then set the shuffle = False, the outcomes were still not exactly the same but this time the trends were generally the same.

Hi Leo, I think random shuffle in your code is on GPU. However numpy works on CPU. Have you tried  tf.random.set_seed(seed) ?

You are right, I was running GPU. tf.random.set_seed(seed) this code doesn't give the 100% same outcomes, but I think  95% similarity is good for parameter adjustment. Thanks.


text:
What should the initial UCT value be with MCTS, when leaf's simulation count is zero? Infinity?
I am implenting a Monte Carlo Tree Search algorithm, where the selection process is done through Upper Confidence Bound formula:
def uct(state):
        log_n = math.log(state.parent.sim_count)
        explore_term = self.exploration_weight * math.sqrt(log_n / state.sim_count)
        exploit_term = (state.win_count / state.sim_count)

        return exploit_term + explore_term

I have trouble however choosing the initial value for UCT, when the sim_count of the node is 0. I tried with +inf (which would be appropriate as approaching lim -> 0 from the positive side would give infinity), but that just means the algorithm will be always choosing an unexplored child.
What would you suggest as a initial value for the uct?
Thank you in advance!

Assigning a value of $\infty$ to unvisited nodes is indeed the "default" or most basic choice, and it indeed ensures that the search never visits a node for a second time if it also still has siblings that have not had any visits. But many other kinds of values have been tried in the literature too.
Gelly and Wang, in "Exploration exploitation in Go: UCT for Monte-Carlo Go" referred to the parameter as "First-play Urgency" (FPU), and indeed really treated it as a hyperparameter; they simply tried various different values and some were found to work better than others.
Other more specific values that you may want to consider (but in most cases we can't really say much about which one will be better than which other ones, without empirical evaluations) are:

The value of a loss; treating nodes that have not had any visits as losing nodes is probably the most "pessimistic" initialisation you could pick. It's not very clearly described in the papers, but there is some evidence out there that this was used by AlphaGo Zero and AlphaZero. Those are not simply vanilla UCTs though, those programs had very strong trained Deep Neural Networks to make additional recommendations for actions. If you don't have such strong neural networks to introduce additional biases in your selection, I would not recommend treating unvisited nodes as losing nodes.
The value of a draw; treating unvisited nodes as draws means that you may prioritise re-visiting nodes that look like winning nodes, but you won't prioritise re-visiting nodes that look like losing nodes.
The value of a win; this probably produces similar behaviour in practice as a value of $\infty$
The average value backpropagated into the parent (correcting for differences in player-colour-to-move); you may prioritise re-visiting nodes that perform better than average in this subtree, but won't prioritise re-visiting nodes that perform worse than average in this subtree

Thank you Dennis for your thorough answer!
Similarly to your third suggestion, I have experimented with setting each node's sim_count = 2 and win_count = 2, as the evaluation is comparative anyway. For my implementation this also means the easiest change.


text:
How do you calculate KL divergence on a three-dimensional space for a Variational Autoencoder?
I'm trying to implement a variational auto-encoder (as seen in Section 3.1 here: https://arxiv.org/pdf/2004.06271.pdf).
It differs from a traditional VAE because it encodes its input images to three-dimensional latent feature maps. In other words, the latent feature maps have a width, height and channel dimension rather than just a channel dimension like a traditional VAE.
When calculating the Kullback-Liebler divergence as part of the loss function, I need the mean and covariance that is the output of the encoder. However, if the latent feature maps are three-dimensional, this means that the output of the encoder is three-dimensional, and therefore each latent feature is a 2D matrix.
How can I derive a mean and covariance from a 2D matrix to calculate the KL divergence?

Your three dimensional latent representation consists of two images of mean pixels and covariance pixels as shown in Fig. 3. Which represents a Gaussian distribution with the mean and covariance for each pixel in the latent representation. Each pixel value is a random variable.
Now, have a close look at KL-loss Eq. 3 and it's corresponding description in the paper:
$$\mathcal{L}_{KL} = \frac{1}{2 \times (\frac{W}{16} \times \frac{H}{16}) } \sum^M_{m = 1}[\mu^2_m + \sigma^2_m - \log(\sigma^2_m) - 1]$$

Finally, $M$ is  the  dimensionality  of  the  latent  features $\theta \in \mathbb{R}^M$ with  mean $\mu  = [\mu_1,...,\mu_M]$ and covariance matrix $\Sigma = \text{diag}(\sigma_1^2,...,\sigma_M^2)$, [...].

The covariance matrix is diagonal, thus all pixel values are independent of each other. That is the reason why we have this nice analytical form for the KL-divergence given by Eq. 3. Therefore you can treat your 2D random matrix simply as a random vector of size $M = \frac{W}{16} \times \frac{H}{16}$ ($\times 3$ if you like to include color dimension). The third dimension (RGB channel) can be considered independent as well, therefore it can be also flattened to a vector and appended. Indeed this is what is done in the paper indicated by the second half of the sentence from above:

that are reparameterized by via sampling from a standard multivariate Gaussian $\epsilon \sim \mathcal{N}(0,I_M)$, i.e. $\theta = \mu + \Sigma^{\frac{1}{2}}\epsilon$.

To clarify, will flattening the W/16 x H/16 matrix into a vector of size M maintain the spatial correspondence because the pixel values are independent of each other? When I say spatial correspondence, I refer to this part of the paper: "we use three-dimensional latent feature maps, i.e., channel, height and width dimensions, rather than one-dimensional latent vectors with only channel dimension, for improving the reconstruction quality and preserve more spatial information."

As an additional point of clarification, does this mean that each mean pixel will get its own covariance? Since the covariance is diagonal, would this mean that the covariance matrix is (H/16 x W/16) x (H/16 x W/16) in size?


text:
What kind of word embedding is used in the original transformer?
I am currently trying to understand transformers.
To start, I read Attention Is All You Need and also this tutorial.
What makes me wonder is the word embedding used in the model. Is word2vec or GloVe being used? Are the word embeddings trained from scratch?
In the tutorial linked above, the transformer is implemented from scratch and nn.Embedding from pytorch is used for the embeddings. I looked up this function and didn't understand it well, but I tend to think that the embeddings are trained from scratch, right?

If you were to implement a Transformer or Word2Vec yourself with PyTorch, in both cases you will probably be using nn.Embedding. Agreed, the original paper is not clear on this. (The other parts of your question have already been answered well by you and in comments.)

I have found a good answer in this blog post The Transformer: Attention Is All You Need:

we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some information about the word. We can do this using nn.Embedding in Pytorch, or, more generally speaking, by multiplying our one-hot vector with a learned weight matrix W.

There are two options for dealing with the Pytorch nn.Embedding weight matrix. One option is to initialize it with pre-trained embeddings and keep it fixed, in which case it’s really just a lookup table. Another option is to initialize it randomly, or with pre-trained embeddings, but keep it trainable. In that case the word representations will get refined and modified throughout training because the weight matrix will get refined and modified throughout training.
The Transformer uses a random initialization of the weight matrix and refines these weights during training – i.e. it learns its own word embeddings.

This response is confusing because it basically says it could be either trained or not trained. But I assume there is a single overall approach taken by the initial transformers paper, gpt-*, BERT, BLOOM, etc. The original Vaswani paper and GPT papers don't mention anything about an initial word embedding, nor do they mention anything about a trainable embedding matrix -- only trainable projections. So which one is it in those specific cases? Is the initial embedding trained or not?

When you set up a transformer you can choose if you want to train from scratch or with pre-trained embeddings, and for the latter, whether it should stay fixed or continue to be trained, just like the answer says. Embeddings and projections are the same concept, the first applied to tokens (non-numbers), the second to numbers, but sometimes you see the terms used synonymously.

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms.  Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings.
The embeddings are trained from scratch.

Maybe you should provide more details about the embeddings used by the transformers (apart from saying that they are learned from scratch).

Embeddings in Word2Vec are also trained from scratch

And how are they trained from scratch then? In the tutorials I have read yet that part is never mentioned. Every time just the attention part is focussed and the embedding part is skipped.

This answer is deceptive. After an input sequence has been tokenized, the transformer's first step will be to use a STATIC matrix to find that token's embedding. Bert Gayus' answer gives some insight into how that static matrix is trained.

Re. static (Word2Vec) and dynamic/contextual (Transformer) embeddings, for the Transformer you would still use PyTorch nn.Embedding ("static") but the attention mechanism disambiguates homonyms (words with different meaning but spelled the same) using the context of the  words.

@BertGayus yeah, so true. seem like they treat it as highest secret.

As "initial"  word embeddings (those without any positional or context information for each word or sub word)  are used from the very beginning  It seems to me that someone has to provide a trained embedding for each word at the very beginning.

They can and will be trained from random starting values.


text:
Number Series Continuation?
I am new to AI.
I have a series of numbers ranging from x to y and I have a lot of data to train with
What I am trying to do is, let's say from 0 to 1, I train it with data calculated over time and predict what may happen next, training it with my data and then feeding it the last few days and continue the pattern.
I have been thinking about using char-rnn, but from what i understand the data exported is arbitrary and not a continuation of a series. I oftentimes see videos on youtube "AI continues this song" so I'm wondering which I can use and where I can get started to do this myself.
Thank you and have a nice day ☺

I don't actually understand your question, but if your data is completely arbitrary, there is nothing to predict, it have no patterns to recognize or something like that.
But if you say that you are working with time-series data and it have some patterns, then you could start by trying to implement just the forward propagation of a simple RNN. I am really new as well at AI and the first RNN I code was a Elman RNN, and they were very simple equations to implement. I recommend you trying that, and then implement the backpropagation to that RNN.
Something that really helped me starting was searching simple github scrips for RNNs (around 100 lines of code) so that you can see their architecture.
About the example you give, the one about songs, they are actually really predictable, because you have the rime, rithm and tempo, and also they use to repeat the same secuence of notes every stanza.
Hope it helps :)


text:
How to deal with evolutionary/genetic fitness function that can have both negative and positive values?
I am optimising function that can have both positive and negative values in pretty much unknown ranges, might be -100, 30, 0.001, or 4000, or -0.4 and I wonder how I can transform these results so I can use it as a fitness function in evolutionary algorithms and it can be optimised in a way that, for example, it can go from negative to positive along the optimisation process (first generation best chromosome can have -4.3 and best at 1000 generation would have 5.9). Although the main goal would always be to maximise the function.
Adding a constant value like 100 and then treating it simply as positive is not possible because like I said, the function might optimise different ranges of results in different runs for example (-10000 to +400 and in another run from -0.002 to -0.5).
Is there a way to solve this?

What make you think that you can't use negative number in your fitness function ?
The objective of the fitness function is to tell how much your solutions are close to the optimum solution so that you can rank them. From what you are describing it look like your function is doing exactly this or am I missing something ?

@kirua in evolutionary algorithms, choice of chromosomes to the new population is based on their scores from the fitness function, to achieve we take fitness function score of each chromosome and divide it by sum of all fitness function results and if we add negative values it will simply not make sense, because sum will be lower than it really should be.

You can always try and see if taking the logarithm of the objective function improves your situation. This makes them positive and their distribution uniform (if it was exponential before). Might make sense if you have such a huge range of values. Also maybe you can somehow normalize (e.g. divide by the range between highest and lowest value) if the range changes that much from iteration to iteration. After all, you only need to compare the members of each generation to each other.

If I understand correctly your problem, you always want to maximize some function $h(x)$, which is defined as follows $h: \Gamma \rightarrow \mathbb{R}$, where $\Gamma$ is the space of genotypes. However, at every generation $g$, you don't know exactly $h(i)$ of each individual $i$, i.e., maybe in one generation $g$, all $h(i)$, for $i=1, \dots, N$ (where $N$ is the size of the population), are negative, but in the next generation $g+1$, due to the mutations and crossovers, that may not be the case anymore, so you don't know how to shift $h(i), \forall i$, so that they are all positive and you can compute the probability of being selected, assuming you are using the fitness proportionate selection.
If my intepretation is correct, then, at every generation $g$, you just need to find $w = \operatorname{min}_i h(i)$, then shift all $h(i)$ by adding $|w| + \epsilon$ (where $\epsilon$ can be zero or a very small number) to all $h(i)$, so you would compute the fitness as follows $f(i) = h(i) + |w| + \epsilon$, for all $i$. Here are all the steps to compute the probability of individual $i$ being selected $p(i)$

$w = \operatorname{min}_i h(i)$
$f(i) \leftarrow h(i) + |w| + \epsilon, \forall i$
$p(i) \leftarrow \frac{f(i)}{\sum_{j=1}^{N} f(j)}, \forall i$

This technique is known as windowing, which I have used in the past to solve the exact same problem that you seem to be trying to solve (check it here).

In one generation there can be chromosomes with both positive and negative values, not as you assumed all must have one sign inside one generation. But it looks like the solution of adding absolute value of the generation's minimum would actually work.

@Makintosz I assumed that there can be both positive and negative values of $h(i)$ in one generation. If you shift by the absolute value of the smallest $h(i)$, you make all positives, and that allows you to convert the fitness into a probability of being selected. That's the idea.

ahhh ok, I was misslead because you wrote: "ALL h(i), for i=1,…,N (where N is the size of the population), are negative" and "so that they are ALL positive and you can compute the probability". But it doesnt matter, I will try the solution later.


text:
Sentiment analysis does not handle neturals
I'm writing some financial tools,  I've found highly performant models for question and answering but when it comes to sentiment analysis I haven't found anything that good.  I'm trying to use huggingface:
from transformers import pipeline
classifier = pipeline('sentiment-analysis')
print(classifier("i'm good"))
print(classifier("i'm bad"))
print(classifier("i'm neutral"))
print(classifier("i'm okay"))
print(classifier("i'm indifferent"))

Which returns results

[{'label': 'POSITIVE', 'score': 0.999841034412384}]

[{'label': 'NEGATIVE', 'score': 0.9997877478599548}]

[{'label': 'NEGATIVE', 'score': 0.999396026134491}]

[{'label': 'POSITIVE', 'score': 0.9998164772987366}]

[{'label': 'NEGATIVE', 'score': 0.9997762441635132}]

The scores for all of the neutral words come up very high in a positive or negative direction,  I would of figured the model would put the score lower.
I've looked at some of the more fine-tuned models yet they seem to perform the same.
I would assume there would be some pretrained models which could handle these use cases.  If not, How can I find neutral sentiments?

@nbro, I was fine with conceptually or programmatically, but you can leave this closed since I already found an answer which met my needs.

Yes, there is. You can try Spacy. Here you go.
import spacy
from spacytextblob.spacytextblob import SpacyTextBlob

nlp = spacy.load('en_core_web_sm')
spacy_text_blob = SpacyTextBlob()
nlp.add_pipe(spacy_text_blob)

text = "i'm good"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.7

text = "i'm bad"
doc = nlp(text)
print(doc._.sentiment.polarity) # -0.6999999999999998

text = "i'm neutral"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.0

text = "i'm okay"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.5

text = "i'm indifferent"
doc = nlp(text)
print(doc._.sentiment.polarity) # 0.0

Thanks, this looks promising, I just need a few hours to check it out when im off work and then ill accept the answer!


text:
Why is tf.abs non-differentiable in Tensorflow?
I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is not applied to tf.abs? Whenever I tried to use it in my custom loss implementation TF was throwing errors about missing gradients.

By convention, the $\mathrm{ReLU}$ activation is treated as if it is differentiable at zero (e.g. in [1]). Therefore it makes sense for TensorFlow to adopt this convention for tf.nn.relu. As you've found, of course, it's not true in general that we treat the gradient of the absolute value function as zero in the same situation; it makes sense for it to be an explicit choice to use this trick, because it might not be what the code author intends in general.
In a way this is compatible with the Python philosophy that explicit is better than implicit. If you mean to use $\mathrm{ReLU}$, it's probably best to use tf.nn.relu if it is suitable for your use case.
[1] Vinod Nair and Geoffrey Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. ICML'10 (2010). URL.

Thanks a lot. That makes sense. Do you know then what is the easiest way to convert tf.abs into differentiable operation with a gradient at 0 defined as 0?

@zedsdead Could you just substitute in tf.nn.relu in the place you're using tf.abs?

How about relu(x) + relu(-x)? :)

@htl Well, they are not the same thing but NikoNyrh’s answer below is probably the easiest solution.

Creating custom gradient for tf.abs may solve the problem:
@tf.custom_gradient
def abs_with_grad(x):
  y = tf.abs(x);

  def grad(div): # Derivation intermediate value
    g = 1; # Use 1 to make the chain rule just skip abs
    return div*g;

  return y,grad;

Use 1 as above to skip thru' abs or, use the actual abs grad (Samual K):
g = tf.where(x<0, -1, 1) #now the gradient at 0 would be one. This way u dont have dead  weights.
# With/without:
g = tf.where(x==0, 0, g) #if you realy want the gradient 0 at 0 add  this.


text:
Is there a methodology for splitting up annotated orthophotos into smaller photos that retain the original bounding boxes?
I'm trying to train an object detection algorithm (i.e. YOLOv4 Scaled, Faster R-CNN) on data taken from large orthophotos. Let's say I have one class, and I label the entire orthophoto with bounding boxes. After labeling, is there a way to slice up the entire image into individual photos of specified pixel sizes (i.e. 416x416 pixels) while keeping the bounding boxes? I can easily slice the photo into the specified dimensions, but the problem I am having is keeping the bounding boxes in these new images.
That way, I would not be exhausting my GPU's memory requirements.

You can reduce your photo size and scale the corresponding boxes to the new dimensions (416x416).
Or if you want to go with your technique, you can slice the image and then, check if the bounding box lies in the slice, then, reorient it according to the slice you took.
Take a look at albumentations library for this.


text:
setting up last layer in tensoflow for class type of label
I am creating a NN in tensorflow keras. the inputs are all float and the output is a class.
The output currently encoded as a float, but only has 4 values (0,1,2,3).
My model is similar to this:
model = tf.keras.Sequential([
    normalize,
    layers.Dense(128, activation='relu'),
    layers.Dense(254, activation='relu'),
    layers.Dense(512, activation='relu'),
    layers.Dense(512, activation='relu'),
    layers.Dense(512, activation='relu'),
    layers.Dense(512, activation='relu'),
    layers.Dense(512, activation='relu'),
    layers.Dense(512, activation='relu'),
    layers.Dense(254, activation='relu'),
    layers.Dense(128, activation='relu'),
    layers.Dense(4)
])

model.compile(loss = tf.keras.losses.SparseCategoricalCrossentropy(), metrics=['accuracy'],
                           optimizer = tf.optimizers.Adam())

history=model.fit(data_set_features, data_set_labels,validation_split=0.33, epochs=100)

is the model last layer correct?
what type of activation should I use and loss function?

Programming questions are off-topic here. Please, edit your post to clarify how this is not a programming question.

Since you're using categorical cross-entropy loss, the last layer (output layer) should come with softmax activation instead of identity (as being blank in layers.Dense(4)).
model = tf.keras.Sequential([
    normalize,
    ...,
    layers.Dense(4, activation=tf.keras.activations.softmax)
])

And SparseCategoricalCrossentropy is different from CategoricalCrossentropy, for categorical cross-entropy, the shape of output and label should be the same, for example:
When using categorical cross-entropy (example labels in one-hot):
Output: [[.1,.0,.9,.0],...] <-- Prediction of 90% for the 3rd class, sum=100%
Label:  [[ 0, 0, 1, 0],...] <-- Third class holds 100% classification possibility

When using sparse categorical cross-entry (labels are class indices):
Output: [[.1,.0,.9,.0],...] <-- Prediction of 90% for the 3rd class, sum=100%
Label:  [[2]          ,...] <-- Third class index is 2


text:
Does feature scaling have any benefits if all features are on the same scale?
By scaling features, we can prevent one feature from dominating the decisions of a model. For example, say heights (cm), and age (years) are two features in my data. Since range of heights is larger than of years, a trained model could weight importance of heights much more than years. This could result in a poor model in return.
However, say that all of my features are binary, they take a value of either 0 or 1. In such a case, does feature scaling still have any benefits?

If all you features are binary, then, you don't need to apply normalization on them. Since their values are on the same scale already.


text:
How to use mixed data for image segmentation?
I have a task for which I have to do image segmentation (cancer detection on MRIs). If possible, I would also like to include clinical data (i.e. numeric/categorical data which comes in the form of a table with features such as age, gender, ...).
I know that for classification purposes, it's possible to create a model that uses both numeric data as well as image data (as mentioned in the paper by Huang et al. : "Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case‑study in pulmonary embolism detection"
The problem I have is that, for image segmentation tasks, it doesn't really make sense to me as to how to use both types of data.
In the above-mentioned paper, they create one model with only the image data and another with only the numeric data, and then they fuse them (there are multiple strategies for fusing them together). For classification tasks, it makes sense. However, for my task, it does not make sense to have a model which only uses the clinical data for image segmentation and that's where I get confused.
How do you think I should proceed with my task? Is it even possible to mix both types of data for image segmentation tasks?

You can try doing image segmentation the traditional way, just using the image data. If you want to use the non-image data, then, you can introduce classification as another task for your network. It will provide some regularization to your model. But, this is one way you can still use non-image data whilst still working with image outputs.


text:
What is a good convergence criterion for Q-learning in a stochastic environment?
I have a stochastic environment and I'm implementing a Q-table for the learning that happens on the environment. The code is shown below. In short, there are ten states (0, 1, 2,...,9), and three actions: 0, 1, and 2. The action 0 does nothing, action 1 subtracts 1 with a probability of 0.7, and action 2 adds 1 with a probability of 0.7. We get a reward of 1 when we are in state 5, and 0 otherwise.
import numpy as np
import matplotlib.pyplot as plt

def reward(s_dash):
    if s_dash == 5:
        return 1
    else:
        return 0
states = range(10)
Q = np.zeros((len(states),3))
Q_previous = np.zeros((len(states),3))
episodes = 2000
trials = 100
alpha = 0.1
decay = 0.995
gamma = 0.9
ls_av = []
ls = []
for episode in range(episodes):
    print(episode)
    s = np.random.choice(states)
    eps = 1
    for i in range(trials):
        eps *= decay
        p = np.random.random()
        if p < eps:
            a = np.random.randint(0,3)
        else:
            a = np.argmax(Q[s, :])

        if a == 0:
            s_dash = s
        elif a == 1:
            if p >= 0.7:
                s_dash = max(s-1, 0)
            else:
                s_dash = s
        else:
            if p >= 0.7:
                s_dash = min(s+1, 9)
            else:
                s_dash = s
        r = reward(s_dash)
        Q[s][a] = (1-alpha)*Q[s][a] + alpha*(r + gamma*np.max(Q[s_dash]))
        s = s_dash
    ls.append(np.max(abs(Q - Q_previous)))
    Q_previous = np.copy(Q)
print(Q)
for i in range(10):
    print(i, np.argmax(Q[i, :]))
plt.plot(ls)
plt.show()

When I plot the absolute value of the maximum change in the Q-table at the end of each episode, I get the following, which indicates that the Q-table is constantly being updated.

However, I see that when I print out the action with the max Q-value for each state, it shows what I expect to be the optimal policy. For each state, the best action is given as shown below:
(0, 2)
(1, 2)
(2, 2)
(3, 2)
(4, 2)
(5, 0)
(6, 1)
(7, 1)
(8, 1)
(9, 1)

My question is: why do I not have convergence in the Q-table? If I had a stochastic environment for which I didn't know before-hand what the optimal policy is, how will I be able to judge if I need to stop training when the Q-table isn't converging?

To obtain guarantees of convergence for Q table values, you need to decay the learning rate, $\alpha$, at a suitable rate. Too fast and convergence will be to inaccurate values. Too slow and convergence never happens.
For sticking with theoretical guarantees, the learning rate decay process should generally follow the rule that $\sum_t \alpha_t = \infty$ but $\sum_t \alpha_t^2 \ne \infty$ - an example of a learning rate schedule that does this is $\alpha_t = \frac{1}{t}$, although in practice that specific choice could lead to very slow convergence.
Choosing a good starting $\alpha$ and a good decay schedule will depend on the problem, and you may want to base it on experience with similar problems. However, it is not that common to need to gurantee convergence of action values in value-based reinforcement learning. In control problems you often care more about have finding an optimal policy than about perfectly accurate action values. Further to that, many interesting control problems are too complex to solve perfectly in tabular form, so you expect some approximation. It seems relatively common just to pick a learning rate for the problem and stick with it.
If you make your learning rate lower, the Q table will converge to more stable Q values, but possibly at the expense of taking longer to converge on the optimal policy.

Thanks Neil for you answer. It's helpful. So now I think a constant learning rate is only suitable for the non-stationary case?

@O'Jhene A constant learning rate is often fine. If you want really good approximations to Q values, it may just need to be lower. But scaling it down slowly is also good.