text:
Reinforcement learning - calculating policy gradient using cross entropy loss
I am writing a program that uses reinforcement learning and the policy gradient method to play Pong. It basically extends Andrej Karpathy's version (https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5) to 3 possible actions (up/down/stay instead of up/down).
I changed the policy_forward() so that it ends with a softmax and not sigmoid since there are 3 probs for the 3 actions now. However, I have some difficulties calculating the gradient. I don't know whether it should be y-aprob (used in Andrej's version with 2 actions where one neuron is sufficient - high activation=UP, low activation=DOWN) or aprob-y (or maybe even something else). y is the one hot encoded vector of the taken action (e.g. [0, 0, 1]) and aprob are the post-softmax probabilities of the actions (e.g. [0.1, 0.25, 0.65]).
After doing some research, it seems to me that aprob-y is the correct derivative of the cross entropy loss. The problem is, the network doesn't learn at all - actually I think it starts to perform worse and worse (just like if it was trying to minimize the reward?). At the beginning, the agent sometimes manages to score a point, but after some learning, the agent just loses every single time. The network with y-aprob seems to work though and I have no idea why, because the formula is different than what I found on the internet (actually I found one piece of code on github that also uses y-aprob: https://github.com/berkott/SpaceInvadersAI/blob/master/Pong/pong_simple.py ---> "gradients = stacked_actions - stacked_predictions").
Does anyone have an idea of what the gradient should be and why? I would especially love to hear the "why" part because for now I am using the y-aprob gradient but have completely no intuition on why it's more or less working.
Forward pass:
def policy_forward(x):
  # shapes -> numpy matmul docs
  h = x @ model['W1'] # (6400,) @ (6400, 300) ---> (1, 6400) @ (6400, 300) ---> (1, 300) ---> (300,)
  h[h<0] = 0 # ReLU nonlinearity
  logits = h @ model['W2'] # (300,) @ (300, 3) ---> (1, 300) @ (300, 3) ---> (1, 3) ---> (3,)
  p = softmax(logits) # instead of the sigmoid
  return p, h # return probability of taking each of the 3 actions and the hidden states

Confusing backprop part:
RIGHT, LEFT = 2, 3

# forward the policy network and sample an action from the returned probability
aprob, h = policy_forward(x)
action = random.choices(range(len(aprob)), weights=aprob, k=1)

# record various intermediates (needed later for backprop)
xs.append(x) # observation
hs.append(h) # hidden state

# cross entropy loss derivative
y = np.zeros_like(aprob)
y[action] = 1
dlogps.append(y - aprob)

if action[0] == 0: a = RIGHT
elif action[0] == 1: a = LEFT

in policy gradient you need to maximise the log-likelihood of your chosen action, weighted by the returns/advantage. minimising the cross entropy is equivalent to maximising the log-likelihood, so calculate your cross entropy, weight by the returns/advantage, and then minimise.

Oh, so the difference in the sign might be because my parameter update looks like that: model[k] += learning_rate * g / (np.sqrt(rmsprop_cache[k]) + 1e-5)? I'm moving in the positive gradient direction (because of the +=), so the loss, in order for it to be minimized, needs to have a negative sign? And that's how it becomes y-aprob instead of aprob-y, which is the actual cross entropy loss derivative? Btw. I do multiply the gradient with the advantage, just forgot to add the code snippet here

honestly, I did not read your code, I was more telling you what your code should do. looking at your code, I can't see where you are taking the log of aprob? looks like you take the softmax and never log it.

Ohh, yeah, for some reason I thought it wouldn't be necessary. But now that I'm reading about it, I should probably take the log of aprob. Where should I do it though? Doing dlogps.append(np.log(aprob) - y) causes the model not to learn anything and causes an error after some episodes: action = random.choices(range(len(aprob)), weights=aprob, k=1)   File "/usr/lib/python3.10/random.py", line 537, in choices     raise ValueError('Total of weights must be finite') ValueError: Total of weights must be finite. Maybe some probability was 0 and the log jumped to infinity

Or actually, do I need the log? aprob - y is already the derivative of the cross entropy loss (or at least I think so). I only calculate the softmax, because I need softmax probabilities to choose an action. I could later calculate the actual loss (I would have to take the negative log of the softmax probs, right? and later maybe the mean value?) but I don't think need it. I only need its derivative for the backprop

if you are calculating the derivative 'by hand' then you may not need to compute the log. though, if you are compute the derivative by hand, why not just differentiate log of the policy and then perform a gradient ascent step using this gradient (scaled by the advantage)?

I'm not sure if I know what you mean by that. aprob - y is the derivative of the cross entropy loss which already contains the log of the softmax output. I'm just not calculating the loss itself because I don't need it, the derivative is enough for the backprop. I think that I'm using the same formula as in this lecture: https://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec21.pdf (slide 13, last line of "To apply REINFORCE..."). I might be wrong though

cross entropy loss is defined as $-\log p_\theta$, where $p_\theta$ is the predicted probability of the true class distribution. differentiating this wrt $\theta$ gives $-\frac{\nabla_\theta p_\theta}{p_\theta}$. that is your gradient of the cross entropy loss. how does this relate to aprob? is aprob the derivative of your weights? to me it looks like it is just the probability value of the selected action. can you clarify?


text:
"The single scalar stored by an LSTM or GRU memory cell" - Deep learning book
I am reading Deep Learning by Goodfellow, Bengio, and Courville, and on page 413, they discuss how to store information using a framework such as a neural Turing machine. Quote:

Neural networks excel at storing implicit knowledge, but they struggle to memorize facts. Stochastic gradient descent requires many presentations of the same input before it can be stored in neural network parameters, and even then, that input will not be stored especially precisely. ...

To resolve this diﬃculty, Weston et al. (2014) introduced memory networks that include a set of memory cells that can be accessed via an addressing mechanism. Memory networks originally required a supervision signal instructing them how to use their memory cells. Graves et al. (2014) introduced the neural Turing machine, which is able to learn to read from and write arbitrary content to memory cells without explicit supervision about which actions to undertake, and allowed end-to-end training without this supervision signal, via the use of a content-based soft attention mechanism ...

These memory cells are typically augmented to contain a vector, rather than the single scalar stored by an LSTM or GRU memory cell.

(emphasis by me)
Why do they claim that LSTM or GRU only stores a scalar? I thought they could also contain a vector. Am I misunderstanding what they are talking about?

"LSTM" can refer to a layer or a unit. In one case, you will have vectors, in the other, you will have scalars. I'd need to double check the specifics, though, because it's been a while since I dealt with this. I'll let someone else provide a more complete answer.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
When to stop a DQN agent train?
Hello AI Stack people,
I'm in doubt to when i should stop my DQN agent train.
I'm traning a DQN agent and i will use a hyperparameter optimization method (probabily random search). So, i need a function that verify if the train iteration has ended, but as i'm trying to find the optimal solution, i want to get the local optimum performance of that train iteration, as that, i think the best approach would be to only end the train when the loss comes to zero, a minimum objective performance is reached and the episode reward has stabilized, all these conditions, for at least some X episodes.
What do you think about my approach ?

To compare hyperparameters, IMO go ahead with random search. But stop each run after a fixed amount of compute time. (Or a fixed number of steps or episodes, as a proxy for compute time. But keep in mind that some hyperparams may affect the compute cost of a step or episode.)
With a clever rule to detect when a run has stopped to make progress, you will always wonder whether there would have been a sudden break-through later on. Then you have to justify or debug your stopping criteria, instead of just comparing the runs.
If you care more about finding a good solution (instead of finding the optimal hyperparameters), then IMO you should use a stopping criterion that compares between different in-progress runs. Common choices for this are ASHA and PBT (population-based training). Reinforcement learning is often very unstable, and stopping bad runs has the additional benefit of filtering out the lucky initializations, which may be more important than finding the exact optimium of the hyperparameters.
This has the same drawback as above: you can no longer draw any simple conclusion about the hyperparameters. You will never know if one of the "bad" configurations was just a slow starter.

I  got a bit confused with your anwser. My goal is to find the best solution by random search a "search space of hyperparameters", hence i think that is good idea to stop the run when the agent probably found the local minimum to that solution, so i include the zero loss and the stabilized reward on the stopping criteria.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to modify the bellman operator for in-place iterative policy evaluation?
The iterative update rule for policy evaluation that is, approximating the value function for a given policy is:
$$v^{k+1} = r_{\pi} + \gamma P_{\pi}v^{k}$$
This is the simultaneous update rule where the new values of the value function vector are calculated only using the old estimates.
An in-place (Gauss-Seidel type) update would use the new values of the value vector as they are calculated. I know how to do this using for loops but how can we modify the matrix equation above to make this update in a vectorized way in code? That is, how should the vector $r_{\pi}$ and the matrix $P_{\pi}$ be changed to produce the required outcomes?


text:
Trained model on cifar10 performs poorly on real images
So I'm trying to train a model using the CIFAR10 dataset.
The problem is that while the performance of the model on validation and test sets are good (about 95-96%), the model fails to predict images downloaded from the internet (preprocessed like with the inputs). I know that there are a lot of similar questions already, as a matter of fact, I’ve already tried implementing those suggestions but none works for me so far. I don’t know what I did wrong.
Here's my approach:

Data: test – 80%, validation – 10% (input of val_dataset in model.fit), test – 10% (input of model.evaluate)
Model (transfer learning): I use ResNet50 as the base model, use “imagenet” weights and freeze them.
UpSampling2D((7,7)) to resize images from 32x32x3 to 224x224x3 --> ResNet50 --> Flatten --> Dense (relu) --> Dropout --> Dense(10, softmax).

Is it because the resolutions of images from this dataset is too low, while the resolutions of images downloaded from the internet are much higher? I don’t know how to describe this, but even after resizing the downloaded images to 32x32x3 to feed in model.predict, they still look kinda different from the images in the dataset?
An image from the dataset:

Downloaded image (original):

Downloaded image (resized to 32x32x3):

Is that the issue? If not, can you tell me what I did wrong, or what I could do so that my model can perform well on real life images?
Many thanks.


text:
Validating bundles of XMLs with inconsistent structure
The problem:
We have a large number of XML bundles, and each bundle needs to meet certain criteria to be considered valid; specifically, certain values (or types of values) should belong to certain tags in the XMLs within a bundle. One option is, of course, to simply parse the files and substring/regex-search for the data we're interested in. I'm hesitant to proceed this way for a couple of reasons:

A hard-coded set of acceptable values/regular expressions might be comically large, and still incomplete.
Some data have multiple representations in the bundle (i.e. different tags in different files that, for our business needs, have logically equivalent values), and again, not all representations are necessarily known.

The ideal solution (?):
We'd be able to construct a neural network that receives the XML strings, one input neuron for the concatenated bundle or one for each file within, and bada bing bada boom, a probability is output for each of our validation criterion. If all the criteria are above a certain threshold, the bundle is considered valid. I believe binary cross-entropy and sigmoid would be the appropriate loss and output activation functions, respectively, for such a perfect-world network, but I'd happily take correction there too.
The subsequent problem:
As I understand it, NN's aren't exactly literate beings, at least in the conventional sense. One approach I can (kind of) understand to make text digestible to them is the bag-of-words model. It seems to me, though, that this approach is better suited for "regular" text than something like XML. This led to me deliriously contemplating an adapted "bag-of-tags" model, in which the "vocabulary" consists of all the available XML tags, and the text nodes are embedded (somehow) in matrix/tensor positions corresponding to their parent tags, but I haven't started experimenting with that yet, and I'm not sure it even makes any sense.
Finally, my question(s):
Is this impossible? Should I go back to school for several years to solve the problem at hand? Or are there recommended ways of preprocessing XML inputs to validate them with neural networks?


text:
What are these special "AI chips" actually used for?
To make an AI / LLM, like ChatGPT, you need two things:

To create the LLM. This includes training it, etc. Very expensive from computation perspective.
Run the LLM to answer user queries. For instance, I go to Chatgpt.com and ask "what is a dog", the LLM created in step 1 processes that question and spits out an answer.

Apparently, Nvidia (and other such companies) make special "AI" chips that are "good at AI". What does this really mean? In practice (i.e., in the real world), are Nvidia "AI chips" solving step 1, step 2, or both?
Can step 2 not be done with regular cpu chips, once the step 1 is completed? Or do we want special AI chips for both because the performance is so much better?

Special AI chips historically and presently are to address step 2 for one or both of two reasons:

The host CPU is not powerful enough for timely inference.  This is a typical concern in very small devices such as mobile and battery powered.
To lower power consumption.  In the case of always on AI in battery powered devices, keeping the main CPU running can mean poor battery life.

I have designed systems 5-6 years ago when running YOLO on an embedded CPU was 2 FPS.  We plugged in the first gen Intel Neural Compute stick and it jumped to 15-20FPS.  CPUs may have caught up to YOLOv3 but there will always be bigger models.


text:
How to interpret this training
I'm still learning ml/ai and I'm running a training where the curves look like this.
I was told that this looks good by some and that it doesn't look good by others... But none told me exactly why, I was told that I should look for both lines to indicate a reduction in loss, but does it look good or not?
thanks!

this is with smoothing set to 0.6:

Loss curves of generative adversarial networks (GANs) are very difficult to interpret. It depends on the type of GAN used, the type of data used, the size of the networks, the size of the dataset, etc. etc..
I have trained a ton of GANs and have seen all types of loss curves work. I know this recommendation is not going to be a fun one but the best way to gauge whether your GAN is converging is just to look at the output it generates. If that is improving, then it likely is going well.
There are metrics for assessing GAN quality which are more interpretable. If you are working with images, there are quite good metrics that depend on pretrained image classifiers (such as FID).

thank you! when I check the output from epoch 2000 and compare it with the output of epoch 3000 I can't see any difference... maybe these metrics don't matter much then. Thanks!


text:
Correctly applying softmax in self attention layer
I'm trying to understand how to apply softmax in self attention layer.
Let's say we have Query and Key matrix where the last row is for Paddings

In this case Z = Q*K_t would be something like this:

When we add the Mask we have:

Now the problem. When I apply the softmax function to each row, I get that the last one is not going to be populated with 0s. By defining S = softmax(Z) what we get is:

To get the output Y of the self attention layer, we have to multiply S by V (Values matrix):

And as a result, we have that y41 and y42 are both functions of Padding values.
How to get rid of this problem?


text:
What is the input size and sequence length for lstm when the input is audio data where each audio file is sampled at the rate of 3000hz?
I have music data which contains raw vocals of some indian classical and devotional songs and I have segmented each file into 20 seconds files. I have used librosa.load function with a sample rate of 3000 to load the audio file. The output returned by the function is of type ndarrray and I have converted it to pytorch tensor of type float and reshaped it to(1,60001). Each audio file is labeled. I want to do the classification task using lstm. If I consider batch_size as 32 then input will be of shape(32,1,60001). I want to know what is the input size and sequence length in this case. Considering audio as a single channel input will the input size be 1 and sequence length be 60001.The model ran faster each epoch when the input size was 60001.Model ran very very slow when the input size was 1.Which is the correct way of considering the input size and sequence length of the lstm model. Can lstm be trained using raw audio files for classification task??

"Input size and sequence length for lstm" is not a question. Can you please put your specific question in the title?


text:
Computing variance of gradient of infinite horizon MDP
This comes from CS285 2023Fall homework2.
I'm self-learning RL, but analysis question seems a little hard for me.

In question one, we could get
$\mathbb{E}_{\tau \sim \pi_{\theta}}R(\tau)=\theta(1-\theta)+2\theta^2(1-\theta)+3\theta^3(1-\theta)+...=\theta+\theta^2+\theta^3+...=\theta/(1-\theta)$, where $|\theta|<1$. Then take its gradient w.r.t $\theta$ we simply arrive at $1/(1-\theta)^2$.
\
But Question 2 is still confusing me.
The variance I compute as fllowed:
$$Variance = \sqrt{\sum_{t=1}^{\infty } ((t-1)\theta^{t-1}(1-\theta)\nabla_{\theta }\ln \theta^{t-1}(1-\theta ) - \frac{1}{(1-\theta)^2})^2}$$
But how to simplify it to eyeball the min/max?
Q3:I compute policy gradient use reward-to-go as followed, but I don't know why this is unbiased to $1/(1-\theta)^2$. And How to compute the variance...
$$
\nabla_{\theta}J_{\theta}=\sum_{T=1}^{\infty}\theta^{T-1}(1-\theta)\sum_{t=1}^{T-1}\nabla_{\theta}\ln\theta(T-t)
$$
$$
= \sum_{T=1}^{\infty}\theta^{T-1}(1-\theta)\frac{T^2 - T}{2\theta}
$$

Does this answer your question? Policiy Gradient Infinite-Horizon Analysis

The question was also queried by me. That question below only focus on the Q1 part and this question focus on Q2 and Q2.


text:
Bias of multi-step Q Learning on/off policy
This is comes from cs2852023Fall, hw3.
I'm learning RL by myself and I cann't find answers related to this question.
Althrough it's from a homework, I believe it would be beneficial to solve the question and get a deeper understanding for q-learning.
The resource could be found in
https://rail.eecs.berkeley.edu/deeprlcourse/.
Backgorund
Consider the N-step variant of Q-learning described.
\begin{align}
y_{i,j} \gets (\sum_{t'=t}^{t+N-1 }\gamma^{t'-t}r_{j,t'} + \gamma^N \max_{a_{j, t+N}}Q_{\phi_{k}}(s_{j,t+N}, a_{j,t+N}) (1)\\
\phi_{k+1} \gets \arg\min_{\phi\in\Phi}\sum_{j,t}(y_{j,t} - Q_{\phi}(s_{j,t}, a_{j,t}))^2 (2)
\end{align}
In these equations, $j$ indicates an index in the replay buffer of trajectories $_{k}$.  We first roll out a batch of B
trajectories to update  $_{k}$ and compute the target values in (1). We then fit $Q_{\phi_{k+1}}$ to these target values with (2). After estimating $Q_{\phi_{k+1}}$, we can then update the policy through an argmax:
$$
\pi_{k+1}(a_t | s_t) \gets
\begin{cases}
  & 1\text{ if } a_{t} = \arg\max_{a_{t}}Q_{\phi_{k+1}}(s_{t}, a_{t})\\
  & 0\text { otherwise }
\end{cases}(3)
$$
We repeat the steps in eqs.(1) to (3) K times to improve the policy. In this question, you will analyze some properties of the algoriyhm, which is summarized in Algorithm 1.
Question
At each iteraction of the algorithm above after update from eq.(2), $Q_{\phi_{k}}$ can be viewed as an estimate of the true optimal $Q^\star$. Consider the following statements:
I. $Q_{\phi_{k+1}}$ is an unbiased estimate of the $Q$ function of the last policy, $Q^{\pi_{k}}.$
II. As $k\to\infty$ for some fixed $B$ (replay_buffer), $Q^{\pi_{k}}$ is an unbiased estimate of $Q^{\star}$, i.e., $\lim_{k\to\infty}\mathbb{E}[Q_{\phi_{k}}(s,q) - Q^{\star}(s,a)] = 0$
III. In the limit of infinite iterations and data we recover the optmimal $Q^{\star}$, i.e.,  $\lim_{k,B\to\infty}\mathbb{E}[||Q_{\phi_{k}} - Q^{\star}||] = 0$
We make the additional assumptions:

The state and action spaces are finite.
Every batch contains at least one experience for each action taken in each state.
In the tabular setting, $Q_{\phi_{k}}$ can express any function, i.e., ${Q_{\phi_{k}}:\phi\in\Phi}=\mathbb{R}^{S \times A}$

When updating the buffer $_{k}$ with $B$, we say:

When learning on-policy, $_{k}$ is set to contain only the set of $B$ new rollouts of $\pi$ (so $|_{k} = B|$). Thus,
we only train on rollouts from the current policy
When learning off-policy, we use a fixed dataset $_{k} = $ of $B$ trajectories from another policy $\pi'$.

Indicate which of the statements I-III always hold in the following cases.
\begin{align}
1. &\text {N = 1 and...  }&I\qquad &II\qquad &III\\
&a. on-policy in tablular setting. &\Box\qquad &\Box\qquad &\Box \\
&b. off-policy in tabular setting.&\Box\qquad &\Box\qquad &\Box \\
2. &\text { N > 1 and...}\\
&a. on-policy in tablular setting.&\Box\qquad &\Box\qquad &\Box \\
&b. off-policy in tabular setting.&\Box\qquad &\Box\qquad &\Box \\
3. &\text {In the limite as $N\to\infty$ and...}\\
&a. on-policy in tablular setting.&\Box\qquad &\Box\qquad &\Box \\
&b. off-policy in tabular setting.&\Box\qquad &\Box\qquad &\Box \\
\end{align}
My Understanding

When N = 1

I: $
Q_{\phi_{k+1}}(s, a)=r(s,a) + \gamma \max_{a}Q_{\phi_{k}}(s_{t+1}, a_{t+1})
$, which is the Q function of the last policy, so this is the unbiased estimate of last policy?
II: Because of the contraction of Bellman operator, and the expression of Q is suffcient for Q value function, this is also right?
III: Statement is also right because of the contraction of Bellman Operator.

When you say "Don't know how to analysis", what exactly is unclear? Please, ask one question per post, even if they are related to the same homework. Also, don't provide screenshots but quote the relevant unclear part of the text

Sorrry for not clarification. I've edited my question. Would your mind helping me?@nbro

I’m voting to close this question because its a homework


text:
Policiy Gradient Infinite-Horizon Analysis
Source: CS285 2023Fall homework2 analysis.

In the part B, the expectation $\mathbb{E}[R(\tau)] = \theta + \theta^2 ... = \theta/(1-\theta)$.
Derivate this with respect to $\theta$ is $1/(1-\theta)^2$.
But how to compute the gradient using policy gradients?(Part A)

I'm really confused for computing the expectation of the gradient of log of $\pi_\theta$(Trajectory).

Hi, welcome to the site! What's the source of your first picture? And the source of the second one?

Both pictures come from CS285 2023Fall homework2 analysis. I added this into question. : )

For part B, from this simple state diagram you're right we can directly compute the expected return without discounting as $\mathbb{E}_{\tau \sim \pi_{\theta}}R(\tau)=\theta(1-\theta)+2\theta^2(1-\theta)+3\theta^3(1-\theta)+...=\theta+\theta^2+\theta^3+...=\theta/(1-\theta)$, where $|\theta|<1$. Then take its gradient w.r.t $\theta$ we simply arrive at $1/(1-\theta)^2$.
For part A it seems supposed to use policy gradient theorem for a generic MDP to calculate the same, which is usually expressed as $\nabla_{\theta}J(\theta)=\mathbb{E}_{\tau \sim \pi_{\theta}}[\nabla_{\theta}\log\pi_{\theta}(\tau)R(\tau)]$ following your reference's notation with a random trajectory $\tau$. Similar to part B above to calculate the expected return, we can simplify the policy gradient as $$\nabla_{\theta}J(\theta)=\theta(1-\theta)·\nabla_{\theta}\log\theta(1-\theta)·1+\theta^2(1-\theta)·\nabla_{\theta}\log\theta^2(1-\theta)·2+...=\nabla_{\theta}\theta(1-\theta)·1+\nabla_{\theta}\theta^2(1-\theta)·2+...=\nabla_{\theta}(\theta(1-\theta)·1+\theta^2(1-\theta)·2+...)=\nabla_{\theta}(\theta/(1-\theta))$$, which obviously matches the result of the above Part B. Note here my answer didn't use or need the hint of your reference since only elementary convergent geometric series sum formula and basic calculus formula involving logarithm and division are actually involved.

Thanks for the answer which really helps a lot!

By MDP, do you mean Markov Decision Process? And, what does the letter J stand for?

@evaristegd J os the same as in your question, the loss

@Alberto , thanks. Do you know where did OP got those two pictures from? Is it a book?

@evaristegd section 8 https://rail.eecs.berkeley.edu/deeprlcourse/deeprlcourse/static/homeworks/hw2.pdf

@Alberto , thank you. How is it that you were able to find the source just by looking at the picture?

@evaristegd exact match on google, just search "and verify that this matches the policy gradient" on google with quotes and you will find it

May I ask, how did you come up to the computation the expected return without discounting? Why did $\theta$ get higher exponents?

@Dave see the above question stated explicitly that Do not use discounting in bold font. Higher exponents for $\theta$ is just following multiplication rule of probability for the random trajectory $\tau$. Hope this clarifies and can get your upvote.

@cinch do you mean the joint probability formula?


text:
What are the obstacles for compositional images generation from text (text-to-image)?
I've tried to find the answer and could not find it. Why still no model (AFAIK) is capable of correct compositional images generation from text? I.e. "simple" (but uncommon) compositions of objects connected with prepositions under/in/on/...
P.S. I suspect some might argue that this question is not answerable so not allowed here. But papers discussing difficulties of compositional images generation might had been published even as failures are usually not published in the current scientific community. E.g. AI alignment is not solved, but there is a lot of material on that topic.


text:
Trying to understand some derivation in the paper: Deep Unsupervised Learning using Nonequilibrium Thermodynamics
I have recently been learning about diffusion models and trying to derive all the results in the paper by Sohl-Dickstein, et. al, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" (2015): https://arxiv.org/pdf/1503.03585.pdf
In the Appendix B of this paper, they define the log-likelihood lower bound term as, $$K = \int{dx^{(0...T)}} q(x^{(0...T)}) \text{log} \Bigg[p(x^{(T)}) \prod_{t=1}^{T} \frac{p(x^{(t-1)}|x^{(t)})}{q(x^{(t)}|x^{(t-1)})} \Bigg]$$.
They further try to isolate the $p(x^{(T)})$ in the square bracket. The next step of the calculation should be,
$$K = \int{dx^{(0...T)}} q(x^{(0...T)}) \text{log} \sum_{t=1}^{T}\Bigg[ \frac{p(x^{(t-1)}|x^{(t)})}{q(x^{(t)}|x^{(t-1)})} \Bigg] + \int{dx^{(0...T)}} q(x^{(0...T)}) \text{ log } p(x^{(T)})$$
The first integral above looks the same as the first term in the paper, I am trying to make the second integral above go to $$\int dx^{(T)} q(x^{(T)}) \text{ log } p(x^{(T)})$$
For this, I am using the fact that $q(x^{(0...T)})$ can be written as,
$$q(x^{(0...T)}) = q(x^{(0)}) \prod_{t=1}^{T} q(x^{(t)}|x^{(t-1)})$$
Then I am writing the second integral above as,
$$\int{dx^{(0...T)}} q(x^{(0...T)}) \text{ log } p(x^{(T)}) \\= \int dx^{(0)} q(x^{(0)}) \int dx^{(1)}q(x^{(1)}|x^{(0)}) \ldots \int dx^{(T-1)}q(x^{(T-1)}|x^{(T-2)}) \int dx^{(T)}q(x^{(T)}|x^{(T-1)}) \text{ log } p(x^{(T)}) \\ = \int dx^{(0)} q(x^{(0)}) \Bigg(\prod_{t=1}^{T-1} \int dx^{(t)}q(x^{(t)}|x^{(t-1)}) \Bigg) \int dx^{(T)}q(x^{(T)}|x^{(T-1)}) \text{ log } p(x^{(T)})$$
Question 1: Is the above expression correct?
Moreover, I think we can also express any $q(x^{(t)}|x^{(t-1)}) = \frac{q(x^{(t)}, x^{(t-1)})}{q(x^{(t-1)})}$ (using conditional probability rule). Does that mean that each of the integrals within the parenthesis in the previous step can be written as,
$$ \int dx^{(t)}q(x^{(t)}|x^{(t-1)}) = \int dx^{(t)} \frac{q(x^{(t)}, x^{(t-1)})}{q(x^{(t-1)})} = \frac{1}{q(x^{(t-1)})} \int dx^{(t)} q(x^{(t)}, x^{(t-1)}) = \frac{1}{q(x^{(t-1)})} q(x^{(t-1)}) = 1$$
Also, we have $\int{dx^{(0)}} q(x^{(0)}) = 1$. Following the above line of thought, the second integral becomes,
$$\int dx^{(T)}q(x^{(T)}|x^{(T-1)}) \text{ log } p(x^{(T)})$$
I am not able to proceed further than this step.
Question 2: Can someone point out if there is a mistake in one of my previous steps and how to proceed with getting the required expression for the second integral term of $K$?

Question 1 is indeed correct. From there, the integral for $\int dx^{(t)} q(x^{(t)}|x^{(t-1)})$ is also correct. The problem is that this expression does not appear in your intended integral.
In
$$\int dx^{(0)} q(x^{(0)}) \Bigg(\prod_{t=1}^{T-1} \int dx^{(t)}q(x^{(t)}|x^{(t-1)}) \Bigg) \int dx^{(T)}q(x^{(T)}|x^{(T-1)}) \text{ log } p(x^{(T)})$$
if you want to integrate out the $x^{(t)}$ variable you actually need to simplify:
$$ \int dx^{(t)} q(x^{(t)}|x^{(t-1)}) q(x^{(t+1)}|x^{(t)}) $$
for $t \notin {0, T}$. For a Markov chain this is doable, using the Chapman–Kolmogorov equation, which implies:
$$ \int dx^{(t)} q(x^{(t)}|x^{(t-1)}) q(x^{(t+1)}|x^{(t)})  = q(x^{(t+1)}|x^{(t-1)}) $$
Doing the above for every single variable except $x^{(T)}$ and $x^{(0)}$ leads to:
$$\int dx^{(T)} \int dx^{(0)} q(x^{(0)}) q(x^{(T)}|x^{(0)}) \log p(x^{(T)}) = \int dx^{(T)} q(x^{(T)}) \log p(x^{(T)})$$
where in the end I integrated on $x^{(0)}$ and obtained the term you wanted.

Thank you, ThunderSmotch. This was helpful! :)


text:
input_tensor is not connected to anything in onnx but can still do inference. How?
I converted an effdet model trained on custom dataset from tensorflow to onnx and tried to further convert it to tensorrt engine file. However, while doing so, I got uint8 issue (tensorrt does not support it). So, I thought of checking out the onnx file using netron.app. I see that the input_tensor data type is uint8 that explains this error but I also see that it is not connected to anything and I can still do inference using this onnx file. As per my understanding, the input_tensor should be connected to any other node to perform further calculations. Please correct me if I am wrong.
I can share the onnx file if required.
Any idea what is happening here? Below is a screenshot from the netron app.

Thanks in advance!

Are you sure there isn't a non compliant implementation that assumes the unconnected input tensor to be Connected anyway?

@CoffeDeveloper No idea about that honestly. Any pointers on how to find that out?
I used this command to convert the model from tf to onnx
python3 -m tf2onnx.convert --saved-model /path/to/saved_model --output /path/to/save/onnx --opset 12


text:
Why do we add noise when training VAEs, instead of just insisting each training batch is Normally distributed at hidden layer?
I had come to think of VAEs as having 2 loss terms - A reconstruction loss that  tries to ensure that the input matches the output (just as a regular Auto Encoder), and a KL-Divergence Loss term that tries to ensure each training batch is mapped to a smaller dimensional space where the data being mapped is normally distributed at the latent encoding layer with a std normal distribution of $\mu$=0 and $\sigma^2$=1. The output of the encoder is a $\mu$ vector and a $\sigma^2$ vector.
But, I can't avoid noticing that we add $\mathcal{N}(0,1)$ noise to each sample at the latent encoding layer. So, if our data has an $\mathcal{N}(0,1)$ distribution and we then add exactly this type of noise to each sample in the representation layer, why do we expect successful reconstruction? It seems like we've added a lot of noise, given the region over which our data is distributed.
Is it just because on 'average' no noise has been added?
Why not train by just letting the kl-loss term insist that each batch have $\mu$=0 and $\sigma^2$=1 and only take a $\mathcal{N}(0,1)$ sample from the hidden layer when we actually do want a new hidden sample?

You claimed "I can't avoid noticing that we add $\mathcal{N}(0,1)$ noise to each sample at the hidden layer", in what reference you noticed this unit Gaussian noise is added at the hidden layer? Encoder or decoder's hidden layer?

You can just insist each training batch is normally distributed—this is the main idea behind normalizing flows, where you try to maximize the likelihood of sampling the encoding assuming it follows a normal distribution.
People mostly use VAEs because

VAEs became popular first, which is because
Normalizing flows require more math/numerical methods, and
Take more compute to train.

Unfortunately, (1) has gotten to the point where everyone is using VAEs, simply because everyone else is! Research and compute is slowly making normalizing flows more viable. For example, you can pair them with neural ODEs for smoother, dynamic flows, but (as of March 2024) you'd likely have to write your own GPU kernels to get an efficient ODE solver.
A recent success with normalizing flows is voice cloning: you can flow from an arbitrary voice to the international phonetic alphabet, which removes timbre. Flowing backward lets you clone someone else's voice.
Now, back to VAEs. You're absolutely correct that you should not expect perfect reconstruction after adding noise. The issue with traditional encoders is they may get perfect reconstruction, but only for your sample of images. For example, perhaps the encoder only needs to signal "this is a horse", and because there is only one horse in your dataset the decoder can perfectly reconstruct it. The point of the noise term in VAEs is to splat out your distribution so the representation has to actually be about the image. See this previous StackExchange answer.


text:
Predict outputs based on a variable subset of inputs
To simplify this: I have 5 columns in my dataset -> A, B, C, D and E. I want the neural network to predict the rest of the outputs based on a subset of inputs.
For example:
Case #1
Inputs -> (A) | Outputs -> (B, C, D, E)
Case #2
Inputs -> (E) | Outputs -> A, B, C, D
Case #2
Inputs -> (A, C) | Outputs -> (B, D, E)
Case #3
Inputs -> (B, D, E) | Outputs -> (A, C)
Ensemble models do not scale well as I would have to train a model with all the permutations and combinations (in this case 5! = 120 models).
What I have tried:
MICE Imputation: To fill in the missing data for the features based on the given inputs. Not sure this is a good approach.
Autoencoder: To mask and encode the features but with the variable nature of inputs, I would require to eventually resort to permutations and combinations as I actually have ~200 features / variables in my dataset.
Multiple models: With ~200 features / variables (not sure about whether to call them variables or features) in my dataset, having to create all the possible permutations and combinations is not possible. Ideally, I would like a single model to have the shape of the input number of features and the shape of the output to be the rest of the variables.
Thank you for the help!


text:
Proof of bellman optimality equations
I am studying RL and have a hard time proving the existence of an optimal policy. I found some resources online, and I am trying to prove the following theorem:
If there exists a policy $\pi$, state $s' \in \mathcal{S}$ and action $a' \in \mathcal{A}(s')$ such that $q_\pi(s', a') \geq v_\pi(s')$ then the policy $\pi'(a|s) = \pi(a|s)(1- \delta_{s, s'}) + \delta_{a, a'}\delta_{s, s'}$ satisfy
$$
v_{\pi'}(s) \geq v_\pi(s) \; \forall s \in \mathcal{S}
$$
Now there is a proof in this article on medium(theorem 1), but I dont fully understand it and would like to write it out more explicitly, using the definition of expectaton with respect to the different policies and so on, but I always end up being really confused and struggling to see where the inequalities in line 4 and 5 comes from in the article.
I would really appreciate if someone could take a look at it and give me a more explicit derivation of this fact.
Thanks:)

This theorem is basically the policy improvement theorem in S&L's RL book page 78, though expressed differently. The reasons for your confused line 4 & 5 are essentially same as your reference's proof comment stated:

and then repeatedly used the fact that the value of the pair of state and action $(s^*, a^*)$ is greater than or equal to the value of state $s^*$.

Since this theorem's condition is that there exists $s^*,a^*$ such that $q_\pi(s^*,a^*) \geq v_\pi(s^*)$, following your reference's notation and line 1 which you already understood, line 4 can be formally derived as:
$$\mathbb{E}_{\pi'}(R_{t+1}+\gamma v_\pi(S_{t+1})|S_t=s) = \mathbb{E}_{\pi'}(R_{t+1}+\gamma \mathbb{E}_{\pi}(q_\pi(S_{t+1},A_{t+1}))|S_t=s)$$, where $S_{t+1}$ is the (random) entering state after $s$ determined solely by the MDP environment.
Then follow the same proof step for line 2, the above $$\text{RHS} \leq \mathbb{E}_{\pi'}(R_{t+1}+\gamma \mathbb{E}_{\pi'}(q_\pi(S_{t+1},A_{t+1}))|S_t=s) = \mathbb{E}_{\pi'}(R_{t+1}+\gamma q_\pi(S_{t+1},A_{t+1})|S_t=s) = \mathbb{E}_{\pi'}(R_{t+1}+\gamma (R_{t+2}+\gamma v_\pi(S_{t+2}))|S_t=s) = \mathbb{E}_{\pi'}(R_{t+1}+\gamma R_{t+2}+\gamma^2 v_\pi(S_{t+2})|S_t=s)$$
Finally line 5 is similar to above derivation of line 4.

Thanks for the reply! I just have some minor questions. When I try to obtain $\mathbb{E}{\pi'}(R{t+1}+\gamma v_\pi(S_{t+1})|S_t=s)$ I arrive at $\mathbb{E}{\pi'}(R{t+1}+\gamma\mathbb{E}\pi( v\pi(S_{t+1}))|S_t=s)$. Is it true that $\mathbb{E}\pi( v\pi(S_{t+1})) = v_\pi(S_{t+1})$? It probably answers my second question as well; why can you remove the expectation around $q_\pi$ when you go from $\mathbb{E}{\pi'}(R{t+1}+\gamma \mathbb{E}{\pi'}(q\pi(S_{t+1},A_{t+1}))|S_t=s) $ to $ \mathbb{E}{\pi'}(R{t+1}+\gamma q_\pi(S_{t+1},A_{t+1})|S_t=s)$?

Yes since by definition both state and action value functions of a specific state $s$ are expectation statistics conditioned on $S=s$ following your notation, and note here all expectations are conditioned on $S_t=s$. So to transform from action value to state value there's no need for extra expectation operator.


text:
Proof of existence of optimal policy
I have been trying to prove existence of an optimal policy for RL. I have proved that the Bellman optimality operator, $B: \mathbb{R}^{|\mathcal{S}|} \to \mathbb{R}^{|\mathcal{S}|}$ given by
$$B(v_\pi)(\; \cdot\; ) = \max_{a \in \mathcal{A}(\; \cdot \;)} \sum_{r \in \mathcal{R}} \sum_{s' \in \mathcal{S}} p(s', r|\;\cdot\; , a) \big[ r + \gamma v_\pi(s') \big]$$
Is a contraction under the infinity norm. As per usual $\mathcal{S}$ and $\mathcal{A}$ denotes the state and action spaces respectively.
Here is where I am a bit confused. We know that $B$ must have a unique fixed point, by Banach fixed point theorem. We also know that if an optimal policy exist its state value, $v^*$, must be the unique fixed point, since $B(v^*) = v^*$. Is this sufficient to conclude that an optimal policy must existence?
Edit:
Would the following reasoning be correct?
By the Banach fixed-point theorem, $v^*$ is unique and the sequence $\{v_{\pi_n}\}_{n=0}^\infty$ defined by $v_{\pi_{n+1}} = B(v_{\pi_{n}})$ converges to $v^*$ for any initial state-value function $v_0 \in \mathbb{R}^{|\mathcal{S}|}$. In particular $\pi_n \overset{n \to \infty}{\longrightarrow}\pi^*$.

There's still some missing link since from what you described above we merely know that $v^*$ is the unique solution to the BOE with Bellman optimality operator. Whether it is the optimal state value of some existent policy $\pi^*$ is still unclear, thus we need to show $v^∗$ is the optimal state value of a constructed policy $\pi^*$ which is then an optimal policy and may not be unique at all.

I understand.. Can I construct policy $\pi^$ by $$\pi^(s) = \text{argmax}a \sum{r \in \mathcal{R}} \sum_{s' \in \mathcal{S}} p(s', r|;\cdot; , a) \big[ r + \gamma v^*(s') \big]$$

@mNugget sounds good to me since the optimal policy here must attain probability 1 for a best action in DP.


text:
Drum sound classification using RNN issues - help needed
I am new to the field of machine learning, even tho I have solid background in semi-related fields (am control system engineer by trade) and as a hobby project I wanted to work a bit with sound classification, where eventually I would like to have a network that transcribes music - in case of drum it would have to detect when a kick/snare/hi-hat drum has been hit, and for other individual instruments it would need to be able to transcribe them in notes.
At the moment I am trying to solve only the drum classification problem and as far as I understand convolutional networks are the way to go, however one could also solve the problem (probably suboptimally) using RNNs since sound is after all a time sequence and for the purpose of understanding RNNs I would like to follow the approach described in the following article:
https://medium.com/towards-data-science/practical-introduction-to-automation-music-transcription-3ad8ad40eab6
You can see my implementation on the following Link.
I have tried to do the similar thing, albeit on only one of the training sets (hence much less data), however the solution doesnt seem to converge not even close, and I am wondering if it is only about the versatility of the data or am I doing something wrong.
This is not the first time I try to solve a problem using RNN without much success so I am inclined to believe problem may be in my formulation of problem/data.
Same as the author, I have tried to convert each drum sound into a short-term Fourier transform and then generate an array where STFT spectrograms for each time window are vertically stacked together. (Each audio file is converted into a sample_length x 1025 freq_bin array and these are stacked vertically)
Target value is derived from the folder name meaning ['kick', 'snare', 'tom', 'overhead' ...] and then later turned to one-hot representation.
I did not break data down in batches, but as far as I understand that should only affect the speed of fitting the RNN, not the performance itself too much.
Data is hence of (BatchSize = 1, seq_length ~ 4000+ samples, 1025 frequency bins).
Snapshop of the entire batch consisting of one time sequence looks something like what is seen on an image below (I have also inserted periods of silence, labeled as None):

Somehow the problem doesnt seem to converge into a right solution and typically will always give the same output sequence with minor variations in the softmax output, eventho the spectrum of kick drum is so clearly different from the overhead drum, and early stopping typically occurs:
Epoch 26/200
1/1 [==============================] - 7s 7s/step - loss: 4.9557 -    accuracy: 0.2335 - val_loss: 7.1182 - val_accuracy: 0.1133

I have tried playing with hyper-parameters but not with much luck.
Data used for the training can be found here:
https://drive.google.com/drive/folders/1l6lDbER8O_SFRaVGe58781YRK6VprtXm?usp=sharing


text:
Calculating TD($\lambda$) returns, reinforcement learning
I am having some trouble with answering the following question:
A rat is involved in an experiment. It experiences one episode. At the first step it hears a bell. At the second step it sees a light. At the third step it both hears a bell and sees a light. It then receives some food, worth +1 reward, and the episode terminates on the fourth step. All other rewards were zero. The experiment is undiscounted ($\gamma = 1$).
Approximate the state-value function by a linear combination of these features with two parameters: $b · bell(s) + l · light(s)$. If $b = 2$ and $l = −2$, write down the sequence of $\lambda$-returns $v_t^\lambda$ corresponding to this episode, for $\lambda = 0.5$.
The provided answer is this:
$v^\lambda_1 = 0.5 (−2 + 0.5×0 + 0.5×1) = −3/4$
$v^\lambda_2 = 0.5 (0 + 1×1) = 1/2$
$v^\lambda_3 = 0.5 (2×1) = 1$
However, these are my calculations:
$v^\lambda_1 = 0.5 (.5^0×−2 + .5^1×0 + .5^2×1) = 0.875$
$v^\lambda_2 = 0.5 (.5^0×0 + .5^1×1) = 1/4$
$v^\lambda_3 = 0.5 (.5^0×1) = 1/2$
Any help is appreciated!
Edit:
These are the functions used:
$v^{(n)}_t = r_{t+1} +\gamma r_{t+2}+\dots+\gamma^{n−1} r_{t+n} + \gamma^n V_{(s_{t+n})} $
$v^\lambda _t = (1−\lambda)\sum_{n=1}^\infty \lambda^{n−1} v^{(n)}_t$
bell state = $[1 \ \ 0]^T$
light state = $[0 \  \ 1]^T$
bell and light state = $[1 \ \ 1]^T$

Instead of writing "TD ... question", can you please just ask your specific question in the title? Thanks

Could you clarify what formula you are applying in your calculations please, by adding its algebraic form? Most likely you have the wrong formula for the TD plus linear state value approximation, but you could also be making incorrect substitutions.

@NeilSlater I've added the formulas

Thanks. I don't see the linear approximator in your formulae? Typically the trace works directly on the approximator's state description vector.

That's not what I meant. The trace doesn't operate on $v$ directly, it operates on $x$, the state description vector - sometimes noted as $\Phi(x)$. So you will end up with an eligibility trace vector that does not represent a specific state, that you will multiply by $[2, -2]$ to getthe current value approximation. You also have to decide between replacing traces and accumulating traces.


text:
In reinforcement learning policy evaluation, which reward is received?
Assume the agent is in a state $s$ and takes action $a$, intending to move to $s_1$, however the dynamics may drop the agent in $s_1$ or $s_2$: may move to $s_1$ and receive reward $r_1$ with a probability $p_1$, or move to state $s_2$ and receive reward $r_2$ with probability $p_2$.
When calculating $v_{k+1}$ (Sutton eq. 4.4), is it:
(1)  $\pi(a|s) * [p_1 * (r_1 * \gamma * s_1)+ p_2*(r_2 * \gamma * s_2)]$
(2)  $\pi(a|s) * [p_1 * (r_1 * \gamma * s_1)+ p_2*(r_1 * \gamma * s_2)]$
Ie is the reward based on the intention of the agent or the state the agent ends up in?

The reward likely should be based on the dynamics of the environment not the intention of the agent.
Imagine if $r_2$ was a large negative reward and $p_2 > p_1$.  This should be captured in the value of the state.  This way as the values back up the agent can learn a path around a risky set of states.
The most prominent example of this I can think of is epsilon greedy on the cliff grid word.  The agent under high settings of epsilon learns a further path around the cliff to avoid epsilon attempting to explore off of it.  It may have the best intention of walk close to the cliff but under the same epsilon, the agent which is aware of the risk and takes the safer path has a higher average reward.

Depends on the setting, many times you don't know the underlying transition matrix, and thus we resort to a monte carlo estimate of the
$$
\mathbb{E}[\sum \gamma^i r_i]
$$
By running multiple simulations


text:
Why same learning rate for slope and intercept not working in Linear regression?
I'm a new student in AI, currently learning linear regression. I used the california housing dataset for doing my experiments. My goal is to predict the 'population' column based on the 'total_rooms' column. I used the following formula and code to compute the slope 'm' and intercept 'b'.
Formula: $$ m = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2 }$$
$$ c = \bar{y} - m \bar{x} $$
The code is as follows and it works perfectly:
# Linear regression using the above formula
x_vals = np.array(train['total_rooms'])
y_vals = np.array(train['population'])
xm = np.mean(x_vals)
ym = np.mean(y_vals)
def compute_m(x_vals, y_vals):
    n = len(x_vals)
    sum_xy, sum_xx = 0, 0
    for i in range(n):
        sum_xy += (x_vals[i]-xm) * (y_vals[i]-ym)
        sum_xx += (x_vals[i]-xm)**2
    return sum_xy/sum_xx

m = compute_m(x_vals, y_vals)
c = ym - m*xm

xl = np.array([np.min(x_vals), np.max(x_vals)])
yl = m*xl + c
plt.scatter(x_vals, y_vals)
plt.plot(xl,yl, 'r')
plt.show()
print('m, c:', m,c)

To verify that my code is working. I also checked it with built-in linear regression in scikit-learn and it returns the exact same answer:
# Now use scikit learn library
from sklearn.linear_model import LinearRegression
model = LinearRegression()
ans = model.fit(x_vals.reshape(-1,1),y_vals.reshape(-1,1))
score, intercept, coef = ans.score(x_vals.reshape(-1,1),y_vals.reshape(-1,1)), ans.intercept_, ans.coef_

print('results from scikit-learn:', score, intercept, coef)

But the problem arises when I try to use gradient descent for learning the slope m and intercept c. My code is as follows:
# Gradient descent
def loss_func(y, ypred):
    mse = (y - ypred)**2
    return mse

def gradient_loss(y, x, mc, bc):
    n = len(y)
    print(n)
    m_loss, b_loss = 0, 0
    for i in range(n):
        ml = (-2/n) * x[i] * (y[i] - mc*x[i] - bc)
        m_loss += ml
        bl = (-2/n) * (y[i] - mc*x[i] - bc)
        b_loss += bl
    return m_loss, b_loss

ep = 100000
alpha = 0.000000001 # learning rate

m, b = 0, 0
for e in range(ep):
    y_pred = m*x_vals + b
    m_loss, b_loss = gradient_loss(y_vals, x_vals, m, b)
    print("m, b, m_loss, b_loss:", m, b, m_loss, b_loss)
    m = m - m_loss*alpha
    b = b - b_loss*0.001

I took the derivative of the squared loss, and using learning rate $\alpha$ for computing $m$ and intercept $b$ (sorry it is the same as $c$ in the previous block) for 100000 episodes. But note that I have to use different learning rate for m and b. No matter what learning rate I use, it does not work for both m and b. If I write in the last line 'b = b - b_loss*alpha', it never converges for any value of alpha
I've never seen in any article or book to use different learning rate for slope m and intercept b. Can anybody please explain what's happening here? And why I had to use different learning rate? :(

It's called ill-conditioning in optimization, which means that you are a lot sensitive to changes with respect to some parameters than others.

As you can see, the curvature is not the same in the X direction and in the Y direction (in your case, it would be w and m) and it's the reason why adaptive optimizers and momentum exists
For linear regression, since the parameter space usually is small, you can also use the Limited-memory BFGS optimizer, or even the Newton Method, thus using 2-order informations to condition your step

But isn't it obvious that the curvature will be different in x and y direction (m and b)? I never FOUND any real data where the curvature will be same in both direction! But in EVERY BOOK and LINKs I found they are using exact same learning rate for both. But why? For example, please see link1 and link2. Shouldn't it be more general to use learning parameters $\alpha_1$ and $\alpha_2$ instead of just $\alpha$?

@JahidChowdhuryChoton what if you have 10 covariates? then you’ll have to manually tune 10 hyperparametrs… instead what is usually done is to set a stepsize that is fine for every parameter (which indeed is what optimization literature tells you to do with the Lipschitz constant)… in your case, it will (slowly but surely) converge to the minima if you use 0.000000001 for both of them

Thank you for your valuable answer. I cannot upvote because my reputation is low. But I'll wait for a few more answers. You're right it will converge (but VERY SLOWLY) if I use 0.00000001 for both of them. But isn't it impractical since it is very slow? If I have 10 co-variates, it may become more and more impractical to use a single step size (which would be much more slower). Manually tuning can drastically reduce number of episodes. Can you suggest any reading material on optimization? Is it related to stability in systems?

@JahidChowdhuryChoton it’s also impractical to tune manually each single one… instead we usually set an initial guess, and use information about the optimization history to change it (check SGD with momentum , Nesterov Momentum, and AdaGrad)


text:
How are conditional models different from supervised models?
I'm wondering what the difference between conditional learning and supervised learning is - especially in diffusion models? Am I correct to assume that diffusion models are supervised because in training, each noisy image is "labeled" with the original one but conditional learning would mean that in training, further labels (such as text data) would be included? And what do "self-supervised" and "semi-supervised" refer to in contrast?
And is it possible to do inference on a conditionally trained model with only lets say an image and no labels?

well one learns $p(y|x)$ and the other one learns $p(x|y)$

and you can always swap them with the Bayes rule, though it might be intractable

Yes that is pretty much correct. Semi-supervised is an instance where you may have labels on for some of the data points (e.g. in node classification, you may only have labels for a subset of the nodes). Self-supervised is where you don't have any labels for your data. I'd say that self-supervised you use your data to learn useful representations, whereas unsupervised learning (where you also don't have labels) is more about looking for patterns and making predictions based on these patterns, such as clustering.


text:
Synthesising Images from Features
I'm currently trying to understand image generation a bit better. I'm working on a DDPM to generate samples from the MNIST set. My question doesn't really have anything to do with that, it's more just out of interest.
Now I was wondering if it is possible to have a divergence between the number of input and output channels. So for instance, if I have 6 channels, RGB and 3 other values, how would I generate an output that is only the RGB image, so 3 channels? I know that you can use labels to do supervised learning but what if these other 3 channels are floats such as recording time.
My initial thought was to use a table as the input with (R, G, B, x, y, z) being the columns and the rows being each pixel. But that wouldn't make a lot of sense with x, y and z being the same for each pixel in an image.
And now what if I train on these 6 channels but I only want to generate one? So for instance only an R band.
And how would the kernel for such an operation look like?
I would apprechiate any thoughts on this!


text:
Realtime cuboid vs cylinder classification of a 2D mask / object from a 3D scene?
Most realtime SOTA segmentation/detection model can reliably segment an object from a 2D input, and I can get the contours/polylines describing its edges in realtime. By realtime, I'll consider anything in the ~10ms range on a NVIDIA A100.
However, the contours of a cube projected on a 2D input (any IRL camera with perspective projection) are not a square. I know that there are some polyline matching algorithms but I don't have enough contours from all possible perspectives to train a classifier (or possibly fine-tune an existing CV model that has spatial awareness). I could in theory use 3d rendering software to generate a ton of instances in the worst case. I'm considering both supervised and non supervised approaches. For example, there's approxPolyDP in OpenCV, but it wouldn't really work.
Since the objects in questions are household items, 3D reconstruction models like https://one-2-3-45.github.io/ do work in practice (and then I can do trivial operations to classify) but they are quite slow - i.e. >20 seconds.

Are there any datasets, pretrained models or unsupervised linear algebraic approaches that would map a 2D object mask and/or RGB object (viewed from an arbitrary perspective) to a real number [0, 1] representing how close it is to a cylinder vs a cuboid?

Is querying OpenAI CLIP with the image and some 'round' / 'cuboid' word embedding and doing cosine similarity a realistic solution?

I'm aware that this is an ambiguous problem in some cases (ex. a cylinder is a rectangle in some orientations). How much should I focus on the RGB / lighting of the object, assuming I care a lot about the precision?


text:
in the "reward to go" trick in policy gradient methods, I have a question about the proof?
I am specifically talking about this proof Why does the "reward to go" trick in policy gradient methods work?
where Dennis says 'on ith iteration the outer sum of random variable and gradients are independent which means we are allowed to split them', my question is why can't we can do the same with final result of the proof where the only difference is we are summing rewards from t' = t onwards? of course we cannot do this as this would result in gradient J_theta to be zero, I just wanted to understand what am I missing here.
Thanks


text:
How to measure accuracy of learned value function of a fixed policy?
Let's say we've a given policy whose value function is to be evaluated. One way to get the value function can be using expected SARSA, as in this stack exchange answer. However, my MDP's state space is massive, so I am using a modified version of DQN that I call deep expected SARSA. The only change from DQN is that the target policy is changed from 'greedy wrt. value network' to 'the given policy' whose value is to be evaluated.
Now on training a value function using deep expected SARSA, the loss curve that I see don't show a decreasing trend. I've also read online that DQN loss curves needn't show decreasing trend and can be increasing and it's okay. In this case, if loss curve isn't necessarily going to show decreasing trend, how do I measure the accuracy of my learned value function? Only idea I have is to compare output of learned value function at (s,a) with expected return estimated from averaging returns from many rollouts starting from (s,a) and following given policy.
I've two questions at this point

Is there a better way to learn the value function than deep expected SARSA? Couldn't find anything in literature that did this.
Is there a better to way to measure accuracy of learned value function?
Thank you very much for your time!


text:
Different Definitions of Momentum -- which one should I work with?
I'm seeing different manners to define momentum, I'm not sure if there is significant difference or not.
From my thinking, they seem to do a similar thing mathematically and in practice but I'm curious to know if there's a significant difference that i'm not seeing.
1st Definition
Momentum:
$v\theta_{t} = \beta * v\theta_{t-1} + (1 - \beta) * ∂\theta_t$
Update: $\theta = \theta - \alpha * v\theta$
2nd Definition
Momentum: $v\theta_t = \beta * v\theta_{t-1} - \alpha *∂\theta_t$
Update: $\theta = \theta + v\theta_t$
3rd Definition
Momentum: $v\theta_t = \beta * v\theta_{t-1} + ∂\theta_t$
Update: $\theta = \theta - \alpha * v\theta_t$
From my understanding, the first definition explicitly uses the exponentially weighted average with the $\beta$ hyperparameter, which then can increase the stability of the model when using momentum and makes $\beta$ more intuitive to tune.
I'm not sure why I'd want to use the 2nd or 3rd definitions over the 1st, nor if they have any significant difference.

Since they're various kinds of the same idea of leveraging exponentially weighted average of the gradients, they may not have significant difference albeit each is most suitable for different cases.
Nowadays the usually standard definition of momentum is either your 1st or 3rd definition as referenced, and thus they're most similar, though obviously the primary difference is the scaling factor $(1−)$ in the 1st definition which helps in further reducing the contribution of the current gradient versus the past accumulated gradients for the momentum. This makes the 1st definition a bit more stable and less sensitive to the choice of $β$ which in practice is set as $0.9$ in most libraries and it's used in the popular and more adaptive Adam algo considering the propagation of root mean squared gradients as referenced above.
Your 2nd definition as referenced is the original momentum formulation of Polyak's 1964 article on solving functional equations and can be easily converted to a form similar to your 3rd definition, ie, $\theta=\theta - \alpha v_{\theta_t}$, where $v_{\theta_t} = -\frac{\beta}{\alpha} v_{\theta_{t-1}} + ∂\theta_t$. Therefore once you define a new $\beta$ to replace $-\frac{\beta}{\alpha}$, it's equivalent to your 3rd definition.


text:
How to do object detection for 1 object in the image, with 3 possible classes (in a custom dataset)
I am new to deep learning, I hope you will lead me because I have been stuck for a week. I am trying to build a model for identifying a single object in the image.
So, I made my custom dataset, which looks like this:

I have 3 objects e.g dog, cat, cow. (So only 3 pictures which are being pasted)
Object is randomly placed around some (on top of it) background picture.
My dataset has three folders -> Train (has images (jpgs) and labels (txt)). Test with similar and Val with similar.
In my labels, I save the class (0,1,2) and where the object is pasted, so it's coordinates.

Now, I want to do my CNN architecture.
I want the model to have 2 outputs:

one for class probabilities (same size as the number of objects chosen, i.e. 3)
another for bounding box regression (of size 4, due to the box coordinates x,y,w,h)

How do I go on in creating the model? What if I want regularization. Is there an online repo about this? or an article. I can't find something which does training for one object.


text:
How to get Complexity per Layer, Sequential Operations and Maximum Path Length in CNN architecture?

In the paper Attention is all you need, here is Table 1, can someone explain what architecture is referred to in the "Convolution" row and hence describe the other 3 columns in it? The other ones are pretty clear, for example Recurrent, takes $O(d^2)$ operations in one time-step to multiply the hidden state with the weight matrix, and there are $n$ such time-steps, which need to be done sequentially ($O(n)$), also making the information from the first and last tokens in the sentence to travel $O(n)$ steps. For Self-attention, every word in the sentence, attends to every other, so $O(n^2)$ pairs, dot product attention taking $O(d)$ operations. Clearly, the path length is $O(1)$ and there are no sequential operation, all these $O(n^2)$ pairs can be computed independently. The "Convolution" row is not clear to me!

Can you please put your specific question in the title? "CNN's for NLP, Attention is all you need" is not a question and it's also not your question

They don't seem to be sharing any supplementary material with details on this, however as they state, convolution being independent from the input size can be applied to varying size input (see WaveNet for example)
The sequential  part is trivial, as it has no sequential notion the convolution (apart from animations). For the last column, refer to the paper, they explain  at the end of Section 4. Regarding the first one, it's the usual complexity of a convolution


text:
How does this distribution change in "Understanding Diffusion Models: A Unified Perspective"?

In the paper Understanding Diffusion Models: A Unified Perspective, how did we go from equation $(44)$ to $(45)$? I couldn't find the details in the paper. How does the distribtuion for, the expectation change as marked? $q(x_t | x_{t - 1})$ and $p(x_{t - 1} | x_t)$ are Markov processes.


text:
Is token mask masked in attention of encoders of bert?
I have recently researched on Bert structure. And the paper says we will mask some token at the input in 80%, 10% input be changed and 10% left remained. But I wonder if the mask token in the input be masked in attention of encoders layer? In that case, the input token of padding token and mask token is the same (equal 0). is it fine?


text:
why explicit reference of batchsize is required in attention
When I read the code for multi-head attention, I noticed all people consider the batch_size in the forward()
For example, the code below was provided by GPT:
class MultiHeadAttention(nn.Module):
    def __init__(self, embed_size, heads):
        super(MultiHeadAttention, self).__init__()
        self.embed_size = embed_size
        self.heads = heads
        self.head_dim = embed_size // heads

        assert self.head_dim * heads == embed_size, "Embedding size needs to be divisible by heads"

        self.values = nn.Linear(self.head_dim, self.head_dim, bias=False)
        self.keys = nn.Linear(self.head_dim, self.head_dim, bias=False)
        self.queries = nn.Linear(self.head_dim, self.head_dim, bias=False)
        self.fc_out = nn.Linear(heads * self.head_dim, embed_size)

    def forward(self, values, keys, query, mask):
        N = query.shape[0]
        value_len, key_len, query_len = values.shape[1], keys.shape[1], query.shape[1]

        # Split the embedding into self.heads different pieces
        values = values.reshape(N, value_len, self.heads, self.head_dim)
        keys = keys.reshape(N, key_len, self.heads, self.head_dim)
        queries = query.reshape(N, query_len, self.heads, self.head_dim)

        values = self.values(values)
        keys = self.keys(keys)
        queries = self.queries(queries)

        # Dot product attention
        energy = torch.einsum("nqhd,nkhd->nhqk", [queries, keys])
        if mask is not None:
            energy = energy.masked_fill(mask == 0, float("-1e20"))

        attention = torch.softmax(energy / (self.embed_size ** (1 / 2)), dim=3)
        out = torch.einsum("nhql,nlhd->nqhd", [attention, values]).reshape(N, query_len, self.heads * self.head_dim)

        # Apply the final linear layer and return
        out = self.fc_out(out)
        return out

I don't understand why we need N=query.shape[0]. I know the N here represents the batchsize, but I don't know why it is necessary considering pytorch would automatically deal with the batchsize as it did in CNN. Instead, I think values=values.reshape(value_len,self.heads, self.head_dim) is better.
And I asked GPT, he told me the batchsize is important when split the queries into multiple-head. But I think $d_{model}=n_{head}\cdot d_k$, therefore it is nothing to do with the batchsize.


text:
Visual representation(s) for variable length m-tuples
I'm working with a dataset of sequences, each 200 characters long. Within these sequences are embedded 10-character patterns (P1 to P5). The co-occurrence of these patterns varies across sequences - some contain just one pattern (e.g. P1), while others have multiple patterns (e.g. P1, P2, P3).
The current methods like Venn diagrams and basic graph models primarily capture co-occurance frequency of patterns (P1 and P2 vs P1,P2,P3 etc.) and pairwise relationships between patterns but cannot adequately represent complex co-occurrences involving more than two patterns (m-tuples where m>2) for a visualization. While visualizations like upset plots come closer, they still don't provide an insightful visualization of the compositional relationships between P1 to P5.
I need visualization or network analysis techniques that can effectively capture and visualize the frequency and complexity of m-tuple (m>2) pattern co-occurrences within the sequences. I'm particularly interested in methods going beyond pairwise interactions to include higher-order relationships. The goal is to better understand and visualize how these patterns co-exist, potentially revealing intricate relationships or frequent combinations critical for further analysis.
What is a good computational context to look for insight here? Methods focusing on social network analysis? Graph theory? Could this be better suited in the context of natural language processing where P1 to P5 could be posed as words, and sequences as sentences?


text:
NN with user as a teacher
I have similar question to the following one.
Could an AI be built to learn based of interaction with a human?
What category of NN is the situation when the network learns from users feedback? The NN digests data and pass through. Than gives answer and a teacher review it by giving a score: 0.0 - failed, 1.0 - valid (or similar).
I cannot imagine, how the user's answer should be entered to the network to train it.

you are referring to RLHF


text:
How to evaluate the KL divergence between two distributions that may require sampling?
The KL divergence between two distributions is:
\begin{equation}
   \int \mathbf{p}(x;\theta_{1}) \; log \frac{\mathbf{p}(x;\theta_{1})}{\mathbf{p}(x;\theta_{2})} \nu(dx) \\
\end{equation}
If the expression $\mathbf{p}(x;\theta)$ is specified by its canonical parameters, we can simply calculate the KL divergence using the expressions of $\mathbf{p}$. However in the case of (conditional: deleted) distributions (where random vector $\mathbf{x}$ appears: added) $\mathbf{p}(y|x;\theta_{1})$, how would one find the KL divergence? Would it require sampling many $x$'s to estimate this value?
As a concrete example, in the case of Logistic regression we have for some random initialized $\theta_0$:
\begin{equation}
    \mathbf{p}(y = 1|x;\theta_0) = \frac{1}{1+e^{-\theta_0^{\intercal}x}} \quad\quad
    \mathbf{p}(y = 0|x;\theta_0) = \frac{e^{-\theta_0^{\intercal}x}}{1+e^{-\theta_0^{\intercal}x}}.
\end{equation}
and another one with $\theta_1$:
\begin{equation}
    \mathbf{p}(y = 1|x;\theta_1) = \frac{1}{1+e^{-\theta_1^{\intercal}x}} \quad\quad
    \mathbf{p}(y = 0|x;\theta_1) = \frac{e^{-\theta_1^{\intercal}x}}{1+e^{-\theta_1^{\intercal}x}}.
\end{equation}
Now we want to compute the improvement of model $\theta_1$ over $\theta_0$, using KL divergence:
\begin{align*}
    D(P(y|x;\theta_0)|P(y|x;\theta_1)) &= \frac{1}{1+exp(-\theta_0^{\intercal}x)} \cdot ln \biggl(
    \frac{1+exp(-\theta_{1}^{\intercal}x)}{1+exp(-\theta_0^{\intercal}x)} \biggr) \\
    &+\;
    \frac{exp(-\theta_0^{\intercal}x)}{1+exp(-\theta_0^{\intercal}x)} \cdot ln \biggl(
    \frac{1+exp(-\theta_0^{\intercal}x)}{exp(-\theta_0^{\intercal}x)} \cdot
    \frac{exp(-\theta_{1}^{\intercal}x)}{1+exp(-\theta_{1}^{\intercal}x)} \biggr).\\
\end{align*}
How would one evaluate $exp(-\theta_0^{\intercal} x)$? Would you sample the values of x if possible? Now suppose you have many examples of $x$ available, ie in a training set, would it be "fair" to sample from the training set to evaluate $D$?
edit
Removed the "conditional distribution", and replaced it with expressions where random vector $\mathbf{x}$ makes an appearance in the definition. There is a comparable question here https://math.stackexchange.com/questions/3016421/confused-by-kullback-leibler-on-conditional-probability-distributions. But in that case the accepted answer compute the expected value of the KL divergence wrt the distribution $P(x)$. But in my case I do not have this distribution $P(x)$, only a dataset $\{(x_1,y_1),..,(x_n,y_n)\}$. I can evaluate the quantitity $D(P(y|x;\theta_0)|P(y|x;\theta_1))$ for every $(x_i,y_i)$ in the dataset, but how would you compute the expecatation:
\begin{equation}
E_{x \sim P(x)}[ D(P(y|x;\theta_0)|P(y|x;\theta_1)) ]
\end{equation}
without this $P(x)$?. The naive method is to assume $P(x)$ is a uniform distribution, at which point the expectation is just the average... but it seems like a better solutions is available somewhere. The closest I can find is this: https://www.tsc.uc3m.es/~fernando/bare_conf3.pdf, which builds a CMF from the data itself.

The distribution being conditional or not does not change the notion of KL divergence.
Indeed, given $p(x) \sim N(\mu_1, \sigma_1^2)$ and $q(x) \sim N(\mu_2, \sigma_2^2)$, the KL can be estimated in closed form.
However, the KL between $p(y|x) \sim N(\mu_1, \sigma_1^2)$ and $q(y|x) \sim N(\mu_2, \sigma_2^2)$ shares the same closed form with the previous one
The only thing you have to know is what family of distribution does the conditional probability falls in
And in your example, $x$ is the conditioning, thus you are saying that "you know a point estimate $\theta$ and a point estimate $x$", which means that that term can be estimated in closed form as it's a product of two point estimates, aka numbers/vectors

the solution of you (last) formulation is straightforward, as $P(y|x,\theta) ~ Bernoulli$, and the kl of discrete distribution is trivial
The only thing you have to do is to compute the KL sample wise, and then average the KL over all the samples
What you will get is a unbiased monte carlo estimate of your overall KL

Ah yeah that make sense. the question is a bit confused. So really I should ask given the distribution cannot be written in closed form, or if the random vector $\mathbf{x}$ makes an appearance in the distribution like above, how do you estimate the KL divergence? Would you sample out of some $P(\mathbf{x})$ to estimate it? It seems like this is the way: https://arxiv.org/pdf/0809.0853 , https://stats.stackexchange.com/questions/280885/estimate-the-kullback-leibler-kl-divergence-with-monte-carlo.  Does it make sense to say "the KL divergence evaluated at some point $x$"?

@xiaolingxiao conditioning on X means that you already know X, so you would just proceed to use it as if it's a constant, and not a random variable... on the who would you estimate a KL of two non parametric distributions, you need to resort to approximations, and since KL is just an integral, if it's low dimensions you can use quadrature procedures, but if it's somewhat high dimension, something like monte carlo estimate (there are more advanced procedures, however at that point it highly depends on your specific problem)

if i have many X's, ie data = ${(x_1, y_1),..,(x_n, y_n)}$, then the KL divergence would be evaluated at every point $x_i$? But what exactly is this quantity $D(\theta_0||\theta_1) | x_i$, is it the KL divergence between two distributions at the point $x_i$?. And if you computed the set of values $ D(\theta_0||\theta_1) | x_1, ..,  D(\theta_0||\theta_1) | x_n$, then what do you do with the set? Do you just average them? what would be the principle behind the average? Moreover, how do you account for the fact the sample may be biased?

@xiaolingxiao well that's your problem definition, i can't tell you what that means... also, you are increasing the non-clear part, now you start by definition two distribution $y|x,\theta$ and then comparing instead the distribution of the $\theta$ where earlier they were just conditional informations, so the initial distributions seems useless now

@xiaolingxiao I would suggest you to open a new question where you stead the REAL problem, and not a paraphrasis of your, because for the moment, seems like you have a dataset XY, some parameters $\theta$, and you are trying to estimate the distance between two parameters distributions (which makes no sense for the current formalization)

I meant there's 2 distributions with 2 different $\theta$: $\theta_0$ - a random initialization, & $\theta_1$ - a better estimate. The point is to compute the "improvement"  in quality of $\theta_1$ over $\theta_0$. The idea is to evaluate this improvement on the dataset, so the expected KL divergence is given here: https://math.stackexchange.com/questions/3016421/confused-by-kullback-leibler-on-conditional-probability-distributions. But that def assumes there is $P(x)$, which I do not have. Averaging it would be assuming uniform distribution.

I edited the question so it's easier to read.

@xiaolingxiao it's still a bit unclear, because of the "some training" is not well defined. Initially $\theta_0$ is a point estimate (a specific vector say), and after training, it's also still a specific vector, just a different one. At no point in this whole procedure, $\theta$ represents a distribution, thus estimating the KL divergence, it's impossible (it's infinite for two point estimate/lambda distributions)

i changed the notation so it's more clear. $\theta$ is the parameter to a distribution, I'm trying to compute the divergence between two different distributions.

@xiaolingxiao sorry but I have to say it... do you realize how your change in notation completely changes the problem?

@xiaolingxiao answered on the answer, but please consider that wants to be the last answer, as it's probably the 3rd completely different formulation you are posting, and it's 100% against the community rules

why is this true: "What you will get is a unbiased monte carlo estimate of your overall KL"? As in why is it that the sample KL divergence is the unbiased estimate? It's only true in so far as the sample it self is unbiased right?

@xiaolingxiao it's an unbiased estimate for the distribution you have in your hand, that's it. you have a coin that gives you 80% times head and is bised? the MC estimate will tell you that the coin gives head 80% of the times (aka an unbiased estimate)


text:
What's the overall algorithm for population evolution in the NEAT algorithm?
I am implementing NEAT from scratch using Ruby, and I'm having a hard time understanding the necessary steps and overall algorithm of what happens between generations.
I have the Genome (made of Nodes and Connections objects) and the Population (made of Species and Genomes objects) classes.
I am able to speciate the genome into species, calculate fitness for each genome/species, average and adjusted fitness, all of that.
But I'm getting stuck in what happens between generations once I have the genome sorted into species.

Should I keep species around between generations? Should I keep their genomes too?
When generating new offspring from species, should I automatically put them in their parent species? If so, how new species will be found? Won't all genome be part of the same first species?
If I use some elitism logic and discard 80% of the lower fitness genomes from each species, how do I fill in the blank? Should I use only crossover from species or should I introduce new "blank" genomes?

I feel like calculating and generating a completely new population between generations is much easier than keeping things between generations and deciding what stays vs what is replaced.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.


text:
Improve Performance of License Plate Recognition using EasyOCR
I wish to extract the license-plate text from images using EasyOCR.
I am obtaining license-plate images from a 2MP camera's stream placed atop traffic light poles. The final license-plate image's dimensions lie in the range of 70-100px by 25-60px.
The issue is that even in images where text is clearly visible, the OCR is not returning accurate results. How can I improve the OCR's results?
Results I am getting

My pipeline
I am performing the following preprocessing steps, which in order are:

Grayscale conversion
min-max normalization to increase contrast
Detecting straight lines of license-plate's edges using Hough Transform and using them to rotate the image such that the text is straight (most of the time).

Using Binary Thresholding has given me worse outcomes so far. I have also tried using other varieties of thresholding.
Gaussian/Median Blurring is making the problem harder since images are so tiny.
I am calling the EasyOCR's readtext() function and preprocessing like so:
from PIL import Image
import numpy as np
import cv2

image_pil = Image.open('image.jpg')
image = np.array(image_pil)

image_for_ocr = np.dot(image[...,:3],[0.2989,0.5870,0.1140]) ##-- Convert to grayscale

min_val = np.min(image_for_ocr)
max_val = np.max(image_for_ocr)
image_for_ocr = ((image_for_ocr-min_val)/(max_val-min_val))*255  ##-- Increase Contrast

image_for_ocr = np.clip(image_for_ocr, 0, 255).astype(np.uint8)

distance,angle = longest_hough_line(image_for_ocr) ##-- Return the distance from origin, and angle in radian of longest detected line in the image
image_for_ocr  = rotate_image(image_for_ocr, angle) ##-- Performs rotation of image about the centre, using the angle of the longest line

reader = easyocr.Reader(['en'], gpu=False)

detections = reader.readtext(license_plate_crop, decoder='greedy', detail=1, blocklist="~`/?\\|=+-_)(*&^%$#@!).-]{[}'" , contrast_ths = 0.5)
lp_text = ""
for detection in detections:
    print(detection)
    bbox, text, score = detection
    text = text.upper().replace(' ', '')
    lp_text += text

print(lp_text)

I am fixing common errors in detections using dictionaries, because license-plates are of defined format:
dict_char_to_int = {'O': '0',
                    'I': '1',
                    'J': '3',
                    'A': '4',
                    'G': '6',
                    'S': '5',
                    'Z': '7',
                    'B': '8',
                    'Q': '0',
                    'P': '3',
                    'H': '4'}

dict_int_to_char = {'0': 'O',
                    '1': 'I',
                    '2': 'Z',
                    '3': 'J',
                    '4': 'A',
                    '6': 'G',
                    '5': 'S',
                    '7': 'Z',
                    '8': 'B'}

How can I increase the accuracy of the detections ?
Some sample images are as follows:


text:
Which is the probability of JohnCalls given Burglary? Why?
Questions:

P (JohnCalls|Burglary) ?
Why?

Source of the image: Artificial Intelligence: A Modern Approach - Third Edition, by Stuart Russell and Peter Norvig.
What do you know about Bayesian networks?
I know they exist, but don't understand enough of how it works (i.e., I understand almost nothing). This example would help me gain comprehension about the inter-relation of the variables.
I know that P (JohnCalls|Burglary) would result in the probability of JohnCalls being divided by the probability of Burglary if these were the only two variables and there was a contingency table for just these two. But there is the Alarm in the middle impacting JohnCalls and I don't know what to do with Alarm.
My attempt at solving: using Bayes' Theorem
P(JohnCalls|Burglary) = P(J|B) = ( P(B|J) * P(J) / P(B) )

P(B): 0.001
P(J): it depends on the Alarm ringing or not ringing, so I cannot be sure about which probability to take in consideration.
P(B|J): ??????