text:
Can TensorFlow, PyTorch, and other mainstream ML frameworks be used for research-grade work in AI?
Many authors of research papers in AI (e.g. arXiv) write their neural networks from the ground-up, using low-level languages like C++ to implement their theories. Can existing open source frameworks also be used for this purpose, or are their implementations too limited?
Can, for example, TensorFlow be used to craft an original network architecture that shows improvements on existing benchmarks? Can original mathematical work be coded into a high-level framework like TensorFlow such that original research on network architectures/approaches be demonstrated in a paper?
A quick search reveals many papers using C++ in their implementation:

https://arxiv.org/pdf/1904.04174.pdf
https://arxiv.org/pdf/1701.03980.pdf
https://arxiv.org/pdf/2005.07039.pdf
https://arxiv.org/pdf/2007.14236.pdf
https://arxiv.org/pdf/1908.05858.pdf

Hello. Welcome to AI SE! When you say "Many authors of research papers in AI (e.g. arXiv) write their neural networks from the ground-up, using low-level languages like C++ to implement their theories", can you please provide 2-3 examples (links to papers or official implementations) where that has been the case? It seems to me that most AI researchers use frameworks (such as PyTorch and TensorFlow) to implement their models, so I don't think that sentence is true, that's why I'm asking for examples.

Literally nobody does this, the one exception maybe is YOLO where the guy wrote a C based library.

@FourierFlux Literally nobody does this? A 3 second Google search reveals 5 papers off the top. I didn’t say most authors, I said many, which is true.

Did you read the papers you referenced?  Except for the spiking neuron one all of the papers were not all focused on developing AI algorithms, but the implementation on specific hardware. Basically nobody who is interested in developing AI algorithms spends their time developing toolkits.

Look up the papers from top ML/AI conferences such as ICML, ICLR, AAAI, NeurIPS etc. All papers with an experimental content I've read used either Tensorflow, or PyTorch to do their experiments.

Your statement that researchers build their network from the ground-up using C++ or some other low level library couldn't be further from the truth.
You could take a look at this analysis showing the popularity of these two frameworks in the top ML conferences. The following Figure is taken from there.

In CVPR-2020, for example, TensorFlow and pytorch combined for over 500 papers! Furthermore, because the two most active research entities (Google and Facebook) are backing these two frameworks, they are used in some of the most impactful research studies.

I want to give some reasons that support the popularity of these frameworks, but first I'm going to rephrase your question a bit:

Why use TensorFlow/Pytorch in python rather than build your model on your own using C++?

Note: The reason I rephrased the question is because TensorFlow and PyTorch both have a C++ APIs.
Why are these frameworks so popular in contrast to lower-level programming languages?
Some reasons are the following

Rapid prototyping. Languages link C++, have bloated syntaxes, require low-level operations (e.g. memory management) and cannot be run interactively. This means it takes someone much less time to create and test a model in python than it does in C++.

No need to re-invent the wheel. Some operations are common in most networks (e.g. backpropagation), why re-implement them? Other functionalities are hard to implement on your own (e.g. parallel processing, GPU computation). Do data scientists need to have such a strong technical background to research neural networks?

Open-source. They benefit from being opensource and can offer a great deal of tools at your disposal for building neural networks. You want to add batchnorm to your network? No worries, just import it and add it in a single line! Also, they offer the perfect opportunity for sharing pretrained models.

They are optimized. These frameworks are optimized to run as fast as possible on GPUs (if available) or CPUs. It would be virtually impossible for someone to write code that runs as fast on his own.

Your comment “couldn’t be further from the truth” makes no sense. I said many, not most. This is confirmed from a casual Google search. Don’t exaggerate for effect. Read the question.


text:
In the DeepView paper, do they use the same FCN for all depth slices AND all views?
I'm trying to replicate a paper from Google on view synthesis/lightfields from 2019: DeepView: View Synthesis with Learned Gradient Descent and this is the PDF.
Basically the input to the neural network comes from a set of cameras which number is variable, and the output is a stack of images which number is also variable. For that they use both a Fully Convolutional Network and Learned Gradient Descent.
I don't know if I am understanding this correctly: (in each LGD iteration) They use the same network for all depth slices AND all views. Is this correct?
This is the LGD network, not much important to the question but it helps you understand the setup. You can see at least 3 LGD iterations. Part b) is just the calculation they do in the "green gradient boxes" on part a).

This is the inside of the CNNs. On each LGD iteration they use basically the same architecture, but the weights are different per iteration.

For me the confusing part is that they represent each view as a different network, but they don't represent each depth slice as a different network. As you can see in the next image they do say that they use the same parameters for all depth slices, and that the order of the views doesn't matter so it must be that they're also reusing the parameters for all views, right? So if I understand correctly, this is a matter of reusing the same model for all depths and all views. BTW note that the maxpool kind of operation is over each view.
Also I have a question on the practicalities of the implementation. I'll be implementing this with normal 2D convolution layers, so if I want them to run independent of the views and depth slices, I guess I could concatenate views and depth slices in the "batch" dimension? I mean, before the maximum k operation, and then reuse the output.
This is what they say:

Thanks

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
T5 or BERT for sentence correction/generation task?
I have sentences with some grammatical errors , with no punctuations and digits written in words... something like below:

As you can observe, a proper noun , winston isnt highlighted with capital in Sample column. 'People' is spelled wrong and there are no punctuations in Sample column. The date in the first row isnt in right format. I have millions of rows like this and want to train a model to learn punctuations and corrections. Can a single BERT or T5 handle this task? or only option is to try 1 model for each task?
Thanks in advance

BERT might not be best option for this. I have tried both and T5 seems works better than BERT.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What does the outputlayer of BERT for masked language modelling look like?
In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows:

In technical terms, the prediction of the output words requires:

Adding a classification layer on top of the encoder output.

2.Multiplying the output vectors by the embedding matrix, transforming them into the vocabulary dimension.
3.Calculating the probability of each word in the vocabulary with softmax.

In the Figure below this process is visualized and also from the tutorial.
I am confused about what exactly is done. Does it mean that each output vector O is fed into a fully connected layer with embedding_size neurons and then multiplied by the embedding matrix from the input layer?
Update:
In the tutorial The Illustrated GPT-2 (Visualizing Transformer Language Models) I found an explanation for GPT-2 which seems to be similar to my question.
In the tutorial is said that each output vector is multiplied by the input embedding matrix to get the final output.
Does the same mechanic apply to BERT?


text:
Which is the best algorithm to predict the trajectory of a vehicle using lat/lon data?
I'm using Kalman Filter approaches and I've just implemented the extended Kalman filter (EKF) with my object 2D trajectory. However, I have a mess of alternative approaches that may fit better like Unscented Kalman Filter (UFK), particle filters, adaptive filtering, etc.
How can I choose the most suitable algorithm for my case? In addition, are there algorithms that can predict more than one step ahead?

Hello. Welcome to AI SE! I've added a few links to the corresponding Wikipedia pages for more context about these concepts. Please, make sure that I've not changed anything. Feel free to add also a link to some reference that describes "Unscented Kalman Filter (UFK)", which I've not added. Moreover, maybe you should describe more in detail your problem. "2d trajectory prediction" is not very descriptive. Maybe if you provide more details about your specific problem the answer will be more specific.

Thanks for the links. I'll change the title. My question is more advice about which approach to go with.

All predictors can predict more than one step using recursion.

As UKF it's recommended over EKF for efficiency. Which is the best implementation (python vs acc/cpu efficiency?

Somehow related:  https://stats.stackexchange.com/questions/391979/how-to-predict-routes-using-clustering-data,   https://stats.stackexchange.com/questions/125318/machine-learning-algorithm-to-predict-next-users-destination/125331#125331


text:
Reinforcement Learning with constant reward in constant episodes time length
I have a situation where I'm trying to maximize the number of steps in a fixed training time frame. It's possible that specific steps will lead to a delay until the agent can act again, and thus less overall steps by the end of the fixed length episode. I am using a basic DQN.
I want to try to explain to a layman how having a constant reward for every step, if given a fixed time frame during training, can allow the agent to try to maximize the number of steps within the episode.  Conceptually it makes sense to me, but I'm at a bit of loss on how to explain it.
I seem to be missing the right keywords to be searching for examples of this situation online.  Any help with keywords and/or explanation would be helpful.


text:
How to handle invalid actions for next state in Q-learning loss
I am implementing an RL application in an environment with illegal moves. For handling the illegal moves, I am currently just picking an action as the maximum Q-value from the set of legal Q-values.
So, it is clear that when deciding on actions we only pick from a subset of valid Q-values, but, when using the Q-learning algorithm, do we also want to consider the subset of invalid actions for the $\max\limits_{a}Q(s_{t+1},a)$?
My gut tells me that we consider all actions for the max function, purely based on the lack of documentation on the subject, but only considering the subset of legal actions makes more sense to me. I'm having a hard time finding any reliable sources addressing this topic. Any advice/direction would be greatly appreciated.

you only choose the max over valid actions for that state. This is intuitive for the tabular case, there would only be the choice of valid actions as there would be no state-action tuple in the lookup table for invalid action choices.

this might help :  https://ai.stackexchange.com/search?q=mask+action

do we also want to consider the subset of invalid actions for the $\max\limits_{a}Q(s_{t+1},a)$

No.
Doing so would go against the theory behind the Bellman equation from which the update derives. The value of $r_{t+1} + \gamma \max\limits_{a'}Q(s_{t+1},a')$ needs to match to a realisable trajectory, otherwise the eventual expected values may be estimates for a different MDP than the one being learned.
For intuition, you can construct a valid MDP which would clearly give the wrong update if the update maximised over non-valid actions. For instance, in a maze game where as well as the classic N,E,S,W moves, the agent can see and pick up treasure in each location. The "pick up" (P) action is only allowed in a location that has treasure and scores $+10$ reward when successfully used, all other outcomes grant $-1$ reward to encourage the agent to act efficiently and escape the maze.
It is worth considering two broad types of Q-learning here - an approximate learner that uses a neural network (e.g. DQN), and a tabular learner that stores action values in a table.

The approximate learner will associate successful uses of the P action with a higher expected return, and generalise this to other states. If the P action is included in updates where there is no treature to pick up in state $s_{t+1}$, it will incorrectly increase the expected return from all actions. Actions that move away from treasure would score similarly as actions that move towards it.

The tabular learner's behaviour will depend on how Q values are initialised. The table entries for non-valid actions would never be updated, so they would remain at the initialised value. If that was e.g. $0$, then it may sometimes be better than valid actions that lead to longer stretches without treasure. Which in turn means that in some locations, multiple valid actions going in different directions would look the same to the agent - future expected returns would be capped at minimum $-1$ (the immediate reward) irrespective of whether they go towards the exit or not. Some actions, that head towards real treasure, may score higher. However, when there is no treasure or exit nearby, all actions would seem the same.

In both cases, the behaviour of the agent will be compromised. It may ignore nearby treasure, or travel into a dead end or loop.
I have deliberately constructed an MDP where maximising over all actions including non-valid ones will cause a problem. Some MDPs will not cause a problem - for instance the same maze MDP with one difference, where the P action is allowed even when there is no treasure, and scores -1 reward for wasting time. However, having a valid environment where the agent fails, demonstrates that an agent built to maximise over non-valid actions will not be reliable in all MDPs, it would have a designed in "bug" and fail for some use cases.

An excellent reply!  Thank you for spending the time.  In hindsight its kind of funny that I would even consider using infeasible actions.


text:
If the training data are linearly separable, which of the following $L(w)$ has less optimum answer for $w$, when $y = w^Tx$?
I'm studying machine learning and I came into a challenging question.

The answer is 2. But based on my ML notes, all of them are true. Where are the wrong points?

Can you please tell us in which context does this question/problem pop up? Is this a homework question? Can you provide the link to the exam/homework/challenge? This seems to be related to support vector machines, but I suggest that you edit your post to provide more context. Moreover, what do your notes actually tell? Do you understand the question? Because I'm not fully sure I really understand this question or, specificallt, what they mean by "less optimum answer".

@nbro the question is about less optimal hyperplanes. if we know the w vector and bias term, we can draw the discriminant hyperplane. As pointed in the question,  the training data are linearly separable. So we may have a lot of discriminant hyperplane. In 2 there is a regularization term, what is its role? It makes the loss function to find less optimal w vectors?

This really seems to be related to SVMs. Please, provide a link to the resource where you're reading this to have more context. So, as far as I recall, equation 2 is an equation that allows for some tolerance and that some points may not be correctly classified. Check the soft-margin here: https://en.wikipedia.org/wiki/Support-vector_machine for a similar formula.

Would you clarify a bit more? what will happen when C goes to infinity?

@DuttaA what is the difference between option 2 and others?

@DuttaA I want to know is there any difference between when C goes to infinity and when it has a good value.

@DuttaA I mean the interpretation in view of machine learning  when C goes to infinity and when it doesn't, not mathematically

@DuttaA there is no deadline ever. this is very challenging question. I'm wait forever to learn new things. thanks

@Emiliiii As I asked before, if you provide the link to the context/website/whatever lists this challenge, then maybe we can have more context to answer it properly.

@DuttaA The question doesn't mention about bias term.

Sorry @nbro I didn't see your comment. there is no link here. this is hard-copy material gives us by TA.

Since the data is linearly separable linear model $y = w^Tx$ will be able to perfectly classify all the examples. That means that loss functions $L_1(w), L_3(w)$ and $L_4(w)$ will have a value of 0 (since all examples are correctly classified). For the loss $L_2(w)$ second term will be 0 if all examples are correctly classified. The first term of $L_2(w)$
\begin{equation}
\frac{1}{2}||w||^2_2
\end{equation}
will be not be minimized to 0 (unless optimal $w = \mathbf{0}$) so $L_2(w) > 0$. Optimizer will try to minimize $L_2(w)$ to be 0, so because of the penalty to $w$ it will try to find the best tradeoff between minimizing $w$ and loss due to misclassification so the resulting $w$ may not be the best to achieve optimal classification.


text:
Is Machine learning a good approach for this geometric challenge?
For months I have been working on a conventional algorithm that positions circles with different sized radii and either positive or negative charge within an outer radius in such a way that the charge balances out as much as possible. You can think of it like differently charged magnets on an infinitely slippery oil film, arranging themselves until they reach a global minimum of forces.

So far, the results are not good and, especially for complex setups with many circuits, very poor. Also, the calculation takes a very long time.
Now I would like to understand if this is a good use case for a ML model:
I can create any number of learning data sets consisting of the following:

features: The xy coordinates, the radius and the charge (+/-) of the circles, where the number of circles can vary from 1 to 100.
target: for each setup a measure of the "homogeneity" of the charges.

With this, you could then possibly train an ML model, which you then only have to give the radii and charges of the circles, and it gives you back the ideal xy coordinates for each circle.
Would something like that be conceivable? Especially because the features are so "multi-dimensional"? And what would be a good start?

Any ideas if this would work?


text:
Transformer Language Model generating meaningless text
I currently learning on Transformers, so check my understanding I tried implementing a small transformer-based language model and compare it to RNN based language model. Here's the code for transformer. I'm using PyTorch inbuilt layer for Transformer Encoder
class TransformerLM_1(nn.Module):

    def __init__(self, head, vocab_size, embedding_size, dropout = 0.1, device = 'cpu',
                 pad_idx = 0, start_idx = 1, end_idx = 2, unk_idx = 3):

        super(TransformerLM_1, self).__init__()

        self.head = head
        self.embedding_size = embedding_size
        self.vocab_size = vocab_size
        self.device = device
        self.embed = WordEmbedding(self.vocab_size, self.embedding_size, pad_idx)
        self.postional_encoding = PostionalEncoding(embedding_size, device)
        self.decoder = nn.TransformerEncoderLayer(self.embedding_size, self.head)
        self.out_linear = nn.Linear(self.embedding_size, vocab_size)
        self.dropout = dropout
        self.pad_idx = pad_idx
        self.start_idx = start_idx
        self.end_idx = end_idx
        self.unk_idx = unk_idx
        self.device = device

    def make_src_mask(self, src_sz):
        mask = (torch.triu(torch.ones(src_sz, src_sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, 10e-20).masked_fill(mask == 1, float(0.0))
        mask = mask.to(self.device)
        return mask

    def forward(self, x):
        dec_in = x.clone()[:, :-1]
        src_mask = self.make_src_mask(dec_in.size()[1])
        src = self.embed(dec_in)
        src = self.postional_encoding(src)
        src = src.transpose(0,1)
        transformer_out = self.decoder(src, src_mask)
        out = self.out_linear(transformer_out)
        return out

I'm using teacher forcing to make it converge faster. From what I saw from the results, the text generated by the RNN model is better than transformer's.
Here is sample generated text with the expected
Expected: you had to have been blind not to see the scenario there for what it was and is and will continue to be for months and even years a part of south carolina that has sustained a blow that the red cross expects will cost that organization alone some $ n million <eos>
Predicted: some <unk> been the been <unk> not be $ the total has was the may has <unk> the that that be to the <unk> the

Expected: citicorp and chase are attempting to put together a new lower bid <eos>
Predicted: a are <unk> carries n't to the together with <unk> jersey than

Expected: it ' s amazing the amount of money that goes up their nose out to the dog track or to the tables in las vegas mr . katz says <eos>
Predicted: <unk> ' s <unk> comeback money of the in mr to their <unk> and of <unk> <unk> or or <unk> the money

Expected: moreover while asian and middle eastern investors <unk> gold and help <unk> its price silver does n't have the same <unk> dealers say <eos>
Predicted: the production the routes <unk> of its

Expected: a board of control spokesman said the board had not seen the claim and declined to comment <eos>
Predicted: the board said declined of said

Expected: property capital trust said it dropped its plan to liquidate because it was n't able to realize the value it had expected <eos>
Predicted: the claims markets said its was n <unk> to sell insolvent of was n't disclosed to sell its plan

Expected: similarly honda motor co . ' s sales are so brisk that workers <unk> they have n't had a saturday off in years despite the government ' s encouragement of more leisure activity <eos>
Predicted: the honda ' credit . s s <unk>

Expected: we expect a big market in the future so in the long term it will be profitable <eos>
Predicted: it can it <unk> board

Expected: u . k . composite or <unk> insurers which some equity analysts said might be heavily hit by the earthquake disaster helped support the london market by showing only narrow losses in early trading <eos>
Predicted: the . s . s trading sell said which <unk> traders market said the be able in the the earthquake

Expected: this will require us to define and <unk> what is necessary or appropriate care <eos>
Predicted: <unk> is be the $ <unk> <unk> <unk> <unk> is the to <unk> and or

As you can see Transformer fails to grasp grammar compared to RNN. Is there anything wrong with my understanding?
EDIT
This is one example that caught my eye
Expected: also the big board met with angry stock specialists <eos>
Predicted: also met specialists board met the stock big with after

Most of the words predicted have is from the expected but in a different order. I have read that transformers are permutation invariant which is the reason why we include positional encoding with the word embedding.

I may be asking very rudimentary question. Is the model getting pretrained with masked tokens prediction task ?  If so, then do you think, it can be used for sentence generation ?

No, it has not been pretrained with masked tokens. I'm following the tutorial given in pytorch documentation. https://pytorch.org/tutorials/beginner/transformer_tutorial.html. Also, I have tried masking random tokens during training, but the results are of the same nature. The sentences generated doesn't make any sense.

This is probably an issue of complete underfitting. How many training data do you use? What is your vocab size? What is your batch size and how many epochs did you train? Transformers always need more data than RNNs to reach good text quality.


text:
How are these equations of SGD with momentum equivalent?
I know this question may be so silly, but I can not prove it.
In Stanford slide (page 17), they define the formula of SGD with momentum like this:
$$
v_{t}=\rho v_{t-1}+\nabla f(x_{t-1})
\\
x_{t}=x_{t-1}-\alpha v_{t},
$$
where:

$v_{t+1}$ is the momentum value
$\rho$ is a friction, let say it's equal 0.9
$\nabla f(x_{t-1})$ is the gradient of the objective function at iteration $t-1$
$x_t$ are the parameters
$\alpha$ is the learning rate

However, in this paper and many other documents, they define the equation like this:
$$
v_{t}=\rho v_{t-1}+\alpha \nabla f(x_{t-1})
\\
x_{t}=x_{t-1}- v_{t},
$$
where $\rho$ and $\alpha$ still have the same value as in the previous formula.
I think it should be
$$v_{t}=\alpha \rho v_{t-1}+\alpha \nabla f(x_{t-1})$$
if we want to multiply the learning rate inside the equation.
In some other document (this) or normal form of momentum, they define like this:
$$
v_{t}= \rho v_{t-1}+ (1- \rho) \nabla f(x_{t-1})
\\
x_{t}=x_{t-1}-\alpha v_{t}
$$
I can not understand how can they prove those equations are similar. Can someone help me?

@DuttaA I am not sure I understand you correctly, but stochastic gradient descent is one of the basic methods in convex optimization, right? And I don't understand the part "NNs are very bad functions", can you explain more about it?

I see, I understand that maybe you feel annoyed by those unclear assumptions. However, as an amateur, I know that NNs is not CO problem, but the performance of SGD (with or without momentum) is really good to optimize the parameters, so I just want to understand the similarity between those equations (for later or maybe the interview). Anyway, happy new year!

The first two equations are equivalent. The last equation can be equivalent if you scale $\alpha$ appropriately.
Equation 1
Consider the equation from the Stanford slide:
$$
v_{t}=\rho v_{t-1}+\nabla f(x_{t-1})
\\
x_{t}=x_{t-1}-\alpha v_{t},
$$
Let's evaluate the first few $v_t$ so that we can arrive at a closed form solution:
$v_0 = 0 \\
v_1 =  \rho v_0 + \nabla f(x_0) = \nabla f(x_0)\\
v_2 = \rho v_1 + \nabla f(x_1) = \rho \nabla f(x_0) + \nabla f(x_1)\\
v_3 = \rho v_2 + \nabla f(x_2) = \rho^2 \nabla f(x_0) + \rho \nabla f(x_1) + \nabla f(x_2)\\
\dots \\
v_t = \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} \nabla f(x_i)
$
So the closed form update is:
$$x_t = x_{t-1} - \alpha \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} \nabla f(x_i)$$
Equation 2
Now consider the equation from the paper:
$$
v_{t}=\rho v_{t-1}+\alpha \nabla f(x_{t-1})
\\
x_{t}=x_{t-1}- v_{t},
$$
We again evaluate the first few $v_t$ to arrive at a closed form solution:
$v_0 = 0 \\
v_1 =  \rho v_0 + \alpha \nabla f(x_0) = \alpha \nabla f(x_0)\\
v_2 = \rho v_1 + \alpha \nabla f(x_1) = \rho \alpha \nabla f(x_0) + \alpha \nabla f(x_1)\\
v_3 = \rho v_2 + \alpha \nabla f(x_2) = \rho^2 \alpha \nabla f(x_0) + \rho \alpha \nabla f(x_1) + \alpha \nabla f(x_2)\\
\dots \\
v_t = \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} \alpha \nabla f(x_i)
$
So the closed from update is:
$$x_t = x_{t-1} - \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} \alpha \nabla f(x_i)$$
As you can see, this is equivalent to the previous closed form update. The only difference is if $\alpha$ is inside or outside the summation, but since it is a constant, it doesn't really matter anyways.
Equation 3
As for the last equation
$$
v_{t}= \rho v_{t-1}+ (1- \rho) \nabla f(x_{t-1})
\\
x_{t}=x_{t-1}-\alpha v_{t}
$$
Let's do the same thing:
$v_0 = 0 \\
v_1 =  \rho v_0 + (1-\rho) \nabla f(x_0) = (1-\rho) \nabla f(x_0)\\
v_2 = \rho v_1 + (1-\rho) \nabla f(x_1) = \rho (1-\rho) \nabla f(x_0) + (1-\rho) \nabla f(x_1)\\
v_3 = \rho v_2 + (1-\rho) \nabla f(x_2) = \rho^2 (1-\rho) \nabla f(x_0) + \rho (1-\rho) \nabla f(x_1) + (1-\rho) \nabla f(x_2)\\
\dots \\
v_t = \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} (1-\rho) \nabla f(x_i)
$
And so the closed form update is:
$$x_t = x_{t-1} - \alpha \displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} (1-\rho) \nabla f(x_i)$$
This equation is equivalent to the other two as long as you scale $\alpha$ by a factor of $\displaystyle \frac{1}{1-\rho}$.

Great, this is exactly what I want to hear. Thank you for a very detailed answer

@user3667125 So the sum of weights $\displaystyle \sum_{i=0}^{t-1} \rho^{t-1-i} (1-\rho)$ doesn't sum to 1 unless $t \to \infty$ (convergence of geometric series)?


text:
How to use validation dataset in my logistic regression model?
I am new to machine learning and recently I joined a course where I was given a logistic regression assignment in which I had to split 20% of the training dataset for the validation dataset and then use the validation dataset to capture the minimum possible loss and then use the test dataset to find the accuracy of the model.
Below is my code for implementing logistic regression
class LogReg(LinReg):
    def __init__(self, n_dim, bias=True):
        if bias:
            n_dim = n_dim + 1
        super(LogReg, self).__init__(n_dim)
        self.bias = bias

  def __call__(self, x):
      return x.mm(self.theta).sigmoid()

  def compute_loss(self, x, y, lambda_reg):
      # The function has a generic implementation, and can also work for the neural nets!
      predictions = self(x)
      loss = -(y * torch.log(predictions) + (1-y) * torch.log(1 - predictions)).mean()
      regularizer = self.theta.transpose(0, 1).mm(self.theta)
      return loss + regularizer.mul(lambda_reg)
  @staticmethod
  def add_bias(x):
      ones = torch.ones((x.size(0), 1), dtype=torch.float32)
      x_hat = torch.cat((ones, x), dim=-1)
      return x_hat

  def fit(self, x, y, num_iter=10, mb_size=32, lr=1e-1, lambda_reg=1e-2, reset=True):
      N = x.size(0)
      losses = []
      x_hat = x
      # Adding a bias term if needed
      if self.bias:
          x_hat = self.add_bias(x)
      if reset:
          self.reset() # Very important if you want to call fit multiple times
      num_batches = x.size(0) // mb_size
      # The outer loop goes over `epochs`
      # The inner loop goes over the whole training data
      for it in range(num_iter):
          loss_per_epoch = 0
          for batch_it in range(num_batches):
              # has been implemented for the linear model
              self.zero_grad()

              ind = torch.randint(0, N, (mb_size, 1)).squeeze()
              x_mb, y_mb = x_hat[ind, :], y[ind, :]

              loss = self.compute_loss(x_mb, y_mb, lambda_reg)

              loss.backward()
              self.theta.data = self.theta.data - lr*self.grad().data
              loss_per_epoch += loss.item()

          loss_per_epoch /= num_batches
          losses.append(loss_per_epoch)

      return losses

How should I use the validation set at the level of epoch to find the best loss?

So generally, when you seperate your training data to 80%-20% then you fit method should get 2 x,y. better to call them x_train,y_train, x_val, y_val or something similar.
Now its important you do the split before entering the fit, and not do it for each epoch or something alike.
Once you do that and the fit method should be something like:
def fit(self, x_train, y_train, x_val, y_val, num_iter=10, mb_size=32, lr=1e-1, lambda_reg=1e-2, reset=True):

Then you should, at the end of each epoch, test the performance of the model on the validation set entirely and calculate the desired metric for evaluation. If you improved it's better to save the current model. This is done repeatedly for each epoch until the end of the training and you will guarantee to have the model who gave you the best results on the validation set rather than on the training set, which might be overfitting it.
I will do it in a separate method with the following flow:

iterate on each sample in the validation set
for each one calculate the loss/metric
append it to some list
return the mean of that list

And if the average result is better from some previously saved one, save the new model

Thank you for your answer.
What is the purpose of writing x_val and y_val inside fit?
In the separate method can I pass the x_val and y_val and then compute the loss?

@DeepakSonker you need to do the separation beforehand, and at the moment you have x,y data but there is no distinction between the training and validation

How to use the best loss to test the test_data? how will I implement that loss in my code? I am really confused. I have not done this previously.

@DeepakSonker So you will save the current best loss/acc and for each time you evaluate the results on the validation set you will examine the new results. if its better save the model if not skip; or something similar

Can you provide me the reading material where I can find the implementation of this?


text:
Model output segmentation maps which are not full
I created a VGG based U-Net in order to perform image segmentation task on yeast cells images obtained by a microscope.
There are a couple of problems with the data:

There is inhomogeneity in the amount of yeast in the images. 1 image can have hundred of yeast cells while others can have less the one hundred.
The GT segmentation map is also incomplete, and some of the cells are not labeled.

All in all the model, given the above problem, is able to learn in some manner. My problem is that the segmentation maps seem incomplete.
For example:

My loss function contains BCE, I was wondering if there is a way to force the model to create a 'fuller' segmentation maps. Something like using Random fields of some sort. Or maybe to enhance my loss function to overcome the above-mentioned problems.
I wish to stay in the domain of simple architectures rather than using more sophisticated ones such as RCNN.
Would appreciate any suggestions


text:
Why would the lookup table (of a table-driven artificial agent) need to store data at pixel precision?
While reading the book AI A modern approach, 4th ed, I came across the section of "Agent program" with following text:

It is instructive to consider why the table-driven approach to agent
construction is doomed to failure. Let $P$ be the set of possible
percepts and let $T$ be the lifetime of the agent (the total number of
percepts it will receive).
The lookup table will contain $\sum_{i=1}^T |P|^T$ entries.
Consider the automated taxi: the visual input from a single camera
(eight cameras is typical) comes in at the rate of roughly 70 mb per
sec. (30 frames per sec, 1080 X 720 pixels, with 24 bits of color
information).
This gives a lookup table with over $10^{600,000,000,000}$ for an
hour's driving.

Could someone please explain how the lookup table number is derived? (or what the author's point is which I am missing). If I were to multiply all of the numbers $30 × 1080 × 720 × 24 × 8 × 3600$, then I get $1.6124314e+13$ which comes very close I think, but can't get what would be the reason to build a table (even though a theoretic one) in such a way - something which is obviously intractable
edit:
My core question is this:
Assuming $10^{600,000,000,000}$ is derived from $30 × 1080 × 720 × 24 × 8 × 3600$, what is the purpose of storing data in the look up table at pixel precision? Wouldn't storing higher level of details be enough to solve these kind of problems (ie, autonomous driving)? Coming more from standard software database systems, I am missing that point. Thanks

OK, I missed something. The author has correctly identicied all history of percepts as important to making a control decision (I initially thought this was being ignored). That is why you have a guesstimate of roughly $10^{13}$ compared to author's much higher $10^{600.000.000.000}$ - it doesn't really matter since both are infeasible.

A tabular system for agent decisions is a direct and simple map of percept to control choice. For each percept received, the agent looks up the percept and cross-references it to the action it should take. In order to construct this, you need to list all percepts in full detail, with the associated control choice.
Clearly that is not going to be feasible for the automated taxi example. No-one would think to build such a table to handle natural image inputs. That is the author's point.
However, a tabular structure is a reasonable theoretical construct for mapping an arbitrary discrete function, and also is practical for simple environments.
To answer your extended question:

Assuming $10^{600,000,000,000}$ is derived from $30 × 1080 × 720 × 24 × 8 × 3600$, what is the purpose of storing data in the look up table at pixel precision?

It is the only way to get a map from percept to control using a tabular system.
By proposing any kind of summarisation or approximation of the input-to-output function to solve this, you have gone beyond the capability of a tabular system. That again is the author's point.

Wouldn't storing higher level of details be enough to solve these kind of problems (ie, autonomous driving)?

If it is really obvious to you that this is the solution, then that's a good thing as you are thinking ahead. However, you should also consider what that means in terms of what you might be giving up, from a theoretical perspective. For instance, a tabular system can make radically different decisions based on very minor differences between percepts, whilst any form of processing of the inputs to make them easier to handle is necessarily going to remove information that might be important.


text:
Why neural networks tend to be trained to recognize multiple things instead of just one?
I was watching this series: https://www.youtube.com/watch?v=aircAruvnKk
The series demonstrates neural networks by building a simple number recognizing network.
It got me thinking: Why neural networks try to recognize multiple labels instead of just one? In the above example, the network tries to recognize numbers from 0 to 9. What is the benefit of trying to recognize so many things simultaneously? Wouldn't it make it easier to reason about if there would be 10 different neural networks which would specialize to recognize only one number at a time?

You could train 10 different models in this case. Model one would then be trained to learn is this a zero or not a zero, and so forth. This would increase the number of negative training samples for each model. But what if then there comes a case where you want to use this in practice? A: you would have to perform 10 forward passes instead of just one. B: the case were the input is a sloppy written 4 that looks similar to a 9, if you have one model then it could learn those small slight variations, if you have 10 different, the difference between a 4 and a 9 would not be learned.

Thank you for the answer! But wouldn't single purpose networks make each individual network simpler? So you could have less neurons and so on. Which (I believe) would make them more performant. Is it 10x more performant than a bigger network (so that it would even out running the picture trough 10 trained networks)? dunno.

The point about difference between 4 and 9 kind of cases is good, but on the other hand, couldn't both neurons still result similar probability for 9s and 4s even if they would live in the same network? And when you would train e.g number 4, you would still feed 9's to it for negative results, so the learning would still happen I believe. Would it be as well learnt? Not sure and would be nice to understand which way it is. I'm trying to understand the practical differences and pros and cons of these approaches.

Multiple specialized networks are called Ensemble Networks. It may indeed be worth training it that way if you are teaching a CNN various representations of the same number. It'd just be computationally more expensive to train and run. If you want to compare, then create programs as a single and ensemble network and try. Learn by doing.

In practice you never want to classify just a single digit rather than series. In such case you have to pass a patch of image to multiple network, which would make it inconvenient. If you built different accurate models, training parameters will not significantly reduced. For example sloppy written 6, in a single model the probability of being 6 and 0 would be close, not same if you consider likelihood you may get closest answer. While with different models the probability may vary in a greater scale and you may not have good generalization as you have in single model. At the end everything boils down to generalization and in my experience neural networks trained with multiple things have good generalization property that a single.

Your question seems to be talking about two slightly different topics:

Pros and cons of 'one vs rest' approach in multi-class classification
Use of Neural Networks in single-output vs multi-class classification problems

One vs Rest in Multi-Class Classification
Recognising digits is an example of multi-class classification. The approach you outline is the kind of approach summarised in the "One vs Rest" section of the Wikipedia page on multi-class classification. The page notes the following issues with this approach:

Firstly, the scale of the confidence values may differ between the binary classifiers. Second, even if the class distribution is balanced in the training set, the binary classification learners see unbalanced distributions because typically the set of negatives they see is much larger than the set of positives.

You might also like to look into another approach called One vs One ('One vs Rest' vs 'One vs One') which sets up the classification problem as a set of binary alternatives. In the digit recognition case you'd end up with a classifier for "1 or 2?", "1 or 3?", "1 or 4?" etc. This might help with the "4 vs 9" problem but it does mean an enormous amount of classifiers, that might be better represented in some kind of network. Perhaps even a network inspired by brain neurons.
Use of Neural Networks in single output vs multi-class classification
There is nothing magical about a neural network that means it has to be used for multi-class classification. Nor is there anything magical about it that makes it the only option for multi-class classification.
For example:

Using a Neural Network for sentiment analysis outputs one single answer about how positive/negative a piece of text is.
Digit recognition using SVM uses something that isn't a neural network for multi-class classification

Conclusions
A 10-class neural network is used to identify digits because this has turned out to be an efficient way of doing so when compared with one vs rest and one vs all approaches.
A bit off-topic, perhaps, but if you think about this in the context of T5, there does seem to be a trend of moving towards larger more multi-purpose models rather than lots of small specialised models.

Imagine a small kid who has no idea about the world around it. You teach the kid how to write the number "6" and that is the only thing that it knows.
Now, No matter what other number you show the kid , it's gonna always respond with "6" because that is the only thing it knows or it has learned.
You teach the kid how to write the number "9", so now it knows how to differentiate a "6" from a "9" and no matter what other number you show the kid, there is a 50 % chance of it responding with a "6" or a "9" because it knows only that much.
The purpose of a neural network is to understand underlying distribution in data that can help it in classifying different numbers. It's important to have a classifier that understands general characteristics of numbers and help us with our task.
If you have 10 neural networks trained on 10 different digits, and you show each of these networks the number "10", each network will output the number on which it was trained because that is all it knows(similar to the naive kid above).
I hope this answers your question!

I guess the kid would know "6 or not 6" in this case? So in case of 10 different neural networks, only one of them would answer "yes 10" on well trained networks.

Your question was on why does a neural network need to be trained on multiple labels instead of one right? In "6" vs "not 6", the model is already trained on 2 labels

Nope, there would be a confidence between 0 and 1 wether it is the number 6 or not. Only 1 label needed.

I think the question should have been rephrased to recognising multiple digits over labels because labels are local to a model, for a single model to identify whether it is 10 or not, it still needs 2 labels(10 or not 10) whereas it is recognising one digit. Anyways to answer your question on having 10 models recognise 10 digits vs 1 model recognising 10 digits, it's just convenience! Would you have 10 people recognise 10 digits for you or 1 person with the ability to do the work of 10 people? There is nothing wrong with either! It's just convenience I guess!

authors: Аксель Фоулли


text:
Evolved networks fail to solve XOR
My implementation of NEAT consistently fails to solve XOR completely. The species converge on different sub-optimal networks which map all input examples but one correctly (most commonly (1,1,0)). Do you have any ideas as to why that is?
Some information which might be relevant:

I use a plain logistic activation function in each non-input node 1/(1 + exp(-x)).
Some of the weights seem to grow quite large in magnitude after a large number of epochs.
I use the sum squared error as the fitness function.
Anything over 0.5 is considered a 1 (for comparing the output with the expected)

Here is one example of an evolved network. Node 0 is a bias node, the other red node is the output, the green are inputs and the blue "hidden". Disregard the labels on the connections.

EDIT: following the XOR suggestions on the NEAT users page of steepening the gain of the sigmoid function, a network that solved XOR was found for the first time after ca 50 epochs. But it still fails most of the time. Here is the network which successfully solved XOR:

The problem was due to the following issues in my implementation:

The offspring generated in the crossover was not mutated (!)
The mutations did not occur with the expected frequencies (too few links and weight mutations)
The sigmoid activation had to be steepened

Another thing that previously caused issues was the network.activate function. Make sure that you wait for the network to stabilize when doing classification tasks, so all signals have time to propagate through the network.


text:
SeqGAN - Policy gradient objective function interpretation
Could someone clear my doubt on the loss function used in SeqGAN paper . The paper uses policy gradient method to train the generator which is a recurrent neural network here.

Have I interpreted the terms correctly?
What are we summing over? The entire vocabulary of words?

Loss function - my interpretation:

Try to ask a more specific question and simplify the post. In that way, I think you will attract more people and potentially someone that attempts to answer your question. If you have multiple questions, I suggest that you create 1 post for each of the questions (even if they are related) for the simple reason that simplicity/clarity often attracts more people.

@nbro Thanks for the suggestion. I removed the excess questions because my core doubt is what the term under the summation indicates


text:
What is the purpose of Decoder mask (triangular mask) in Transformer?
I'm trying to implement transformer model using this tutorial.  In the decoder block of the Transformer model, a mask is passed to "pad and mask future tokens in the input received by the decoder". This mask is added to attention weights.
import tensorflow as tf

def create_look_ahead_mask(size):
    mask = 1 - tf.linalg.band_part(tf.ones((size, size)), -1, 0)
    return mask

Now my question is, how is doing this step (adding mask to the attention weights) equivalent to revealing the words to model one by one? I simply can't grasp the intuition of it's role. Most tutorials won't even mention this step like it's very obvious. Please help me understand. Thanks.

The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens.
So in order to predict next token, you have to make sure that only previous token are attended. (If not, this would be a cheating because model already knows whats next).
So attention mask would be like this
[0, 1, 1, 1, 1]
[0, 0, 1, 1, 1]
[0, 0, 0, 1, 1]
[0, 0, 0, 0, 1]
[0, 0, 0, 0, 0]
For example: If you are translating English to Spanish
Input: How are you ?
Target: < start > Como estas ? < end >
Then decoder will predict something like this
< start > (it will be given to decoder as initial token)
< start > Como
< start > Como estas
< start > Como estas ?
< start > Como estas ? < end >
Now compare this step by step prediction sequences to attention mask given above, It would make sense now to you

In spanish double question marks are used ¿?

@Leo, shouldn't the mask contain negative infinities, not zeros? Are inputs to decoder a ground truth, or its previous outputs?

@Kari, Yes, Negative infinity mask is used, but after the softmax operation, the values are extremely close to zero. That's why in theory you will always see 0 instead of negative infinity

the mask is needed to prevent the decoder from "peeking ahead" at ground truth during training, when using its Attention mechanism.
Encoder:

Both runtime or training:
the encoder will always happen in a single iteration, because it will  process all embeddings separately, but in parallel. This helps us save time.

Decoder:

runtime:
Here the decoder will run in several non-parallel iterations, generating one "output" embedding at each iteration. Its output can then be used as input at the next iteration.

training:
Here the decoder can do all of it in a single iteration, because it simply receives "ground truth" from us. Because we know these "truth" embeddings beforehand, they can be stored into a matrix as rows, so that they can be then submitted to decoder to be processed separately, but in parallel.
As you can see during training, actual predictions by the decoder are not used to build up the target sequence (like LSTM would). Instead, what essentially is used here is a standard procedure called "teacher forcing".
As others said, the mask is needed to prevent the decoder from "peeking ahead" at ground truth during training, when using its Attention mechanism.

As a reminder, in transformer, embeddings are never concatenated during input. Instead, each word flows through encoder and decoder separately, but simultaneously.
Also, notice that the mask contains negative infinities, not zeros. This is due to how the Softmax works in Attention.
We always first run the encoder, which always takes 1 iteration. The encoder then sits patiently on the side, as the decoder uses its values as needed.

We give the target input into the transformer decoder while training the model. So it is easy for the model to "peek ahead" and learn what the next word would be. To ensure that this doesn't happen we apply an additive mask after the dot product between Query and Key. In the original paper "Attention is all you need", the triangular matrix had 0's in the lower triangle and -10e9 (You can see negative infinity used in recent examples) in the upper triangle. So when the mask is added to the attention score the attention scores in the upper triangle would be really low. When this matrix is passed through the softmax function, these really low values become close to 0, which essentially means not to attend to the words after timestep t. To put in matrix format,
[8.1, 0.04, 5.2, 4.2]
[0.5, 9.2, 2.33, 0.7]
[0.2, 0.4, 6.11, 1.0]
[3.1, 2.1. 2.19, 8.1]

Let the above matrix A the result of the dot product between query and key. The A[0][0] contains the attention score of the first-word query to the first word of the key, A[0][1] contains the attention score of the first word of the query to the second of the key, and so on. So as you can see the after adding the mask and performing softmax on A, the result would be,
[8.1, 0.0, 0.0, 0.0]
[0.5, 9.2, 0.0, 0.0]
[0.2, 0.4, 6.11, 0.0]
[3.1, 2.1. 2.19, 8.1]

This forces the transformer only to attend to words that are before it. You can check out the Transformer lecture available in CS224n for full detail.


text:
Are there any meaningful books entirely written by an artificial intelligence?
Are there any meaningful books entirely written by an artificial intelligence? I mean something with meaning, unlike random words or empty books.
Something that can be charactersed as fiction literature.
If yes, then I think it is also interesting to know if any of those books is available for sale. Is there a specific name for such books? Like "robot books"?

As of now, I don't think that any meaningful book has been written by an artificial intelligence. Maybe the closest thing to what you are looking for are texts generated by models like GPT-3, but I could also be wrong, given that I am not really an expert in this area. For example, have a look at this blog post.

AFAIK, no there is none. But you should define what is a "meaningful book". In France, see writings by Oulipo - in 1960 or 1970 (also this...)

There are none.

How about short novels like 10 or 20 pages?

I found this website - http://booksby.ai - it claims to contain only books written by AI - like for example this one book - Dinner Depression — Julia Joy Raffel –

James Ryan has done a lot of 'archaeological' work on this; you can find references to his work on his website.
Story generation has been a dream for a long time (in computing terms), and various genres have been explored, with not that much success. There have been episodes of a Western written by a computer (and actually filmed and acted out by human actors, see summary here), and various books, but the technology is nowhere near good enough to produce something worthwhile reading or watching without heavy editing.
So far it's only good for curiosity value.
There is NaNoGenMo, where since 2013 people work on programs generating novels. But most of them — again — are more interesting for curiosity. They either take an existing work and modify it procedurally, or generate random templated text (eg travel reports through a ficticious, auto-generated landscape). I don't think anyone has met the target of creating a 50k novel by computer yet.
Modern approaches with deep learning generators can produce reams of more or less well-formed text, but that doesn't make a compelling story. It's the meaning that is currently the problem, and as AI researchers keep finding out, it's hard.


text:
Determining if an entity in free text is 'present' or 'absent'; what is this called in NLP?
I'm processing a semi-structured scientific document and trying to extract some specific concepts. I've actually made quite good progress without machine-learning so far, but I got to a block of true free text and I'm wondering whether a very narrow sense NLP/learning algorithm can help.
Specifically, there are concepts I know to be important that are discussed in this section, but I'll need some NLP to get the 'sentiment'. I thought this might 'entity sentiment' analysis, however, I'm not trying to capture the writer's emotion about a concept. It's literally whether the writer of the text thinks the entity is present, absent, or is uncertain about an entity.
Simple example. "First, there are horns. And second, the sheer size of this enormous fossil record argues for an herbivore or omnivore. The jaws are large, but this is not a carnivore."
And say my entities are horns (presence or absence), and type of dinosaur (herbivore, omnivore, carnivore). Desired output:
Horns (present)
Carnivore (absent)
Herbivore (possible/present) -- fine if it thinks 'present'
Omnivore (possible/present) -- fine if it thinks 'present'

What is the class of NLP analysis that takes an explicit input entity (or list of entities) and tries to assess based on context whether that entity is present or absent according to the writer of the input text? It's actually fine if this isn't a learning algorithm (maybe better). Bonus if you have suggestions for python packages that could be used in this narrow sense. I've looked casually through NLTK and spacy packages but they're both vast, wasn't obvious which class of model or functions I'd need to solve this problem.

If I understood you correctly, you are looking for: Aspect Based Sentiment Analysis (ABSA), but in your case you replace the sentiment with your "present, absent, possible/present".

Right. I think many of the sentiment based methods seem to 'hardcode' the sentiment part though, with dictionaries of positive and negative emotions for example. I guess I need a dictionary of presence/absence/equivocal terms. Does that exist, and if so where?

I would not use dictionaries due to the amount of manual work, and not using the power of embeddings... What about annotating some data and building a multi-label classifier for this purpose? You should focus on generating ambiguous examples, since those will be the hardest!


text:
Would performance of atomic models matter in ensemble methods?
Suppose I have two fitted ensemble models $F_1 := (f_1, f_2, f_3, \cdots f_n)$ and $G_1 := (g_1, g_2, g_3, \cdots g_n)$.
And they were using the same ensemble methods (boosting or bagging).
And I am using some measurement for model performance $M: f_i \to \mathbb{R}^+$, higher the better.
And I know beforehand $M(f_i) \gt M(g_i), \forall i \in [1,n]$, can I conclude $M(F) \gt M(G) $ ?

Do you know $M(f_i) \gt M(g_i), \forall i \in [1,n]$ or is $M(f_i) \gt M(g_i)$ only for a specific $i$?

All $i$ in this case

authors: Егор Андреевич Малютин


text:
Measuring novel configuration of points
I am trying to implement Novelty search; I understand why it can work better than the standard Genetic Algorithm based solution which just rewards according to the objective.
I am working on a problem which requires to generate a fixed number of points in a 2d box centered at the origin.
In this problem, how can I identify which is a novel configuration of points?
Note: I have thought of one way of doing this: We call the mean of one configuration of points to be the mean of all points in that configuration (let's say this tuple is $(m_x, m_y)$, we store the mean of all configurations generated till now, now for a new configuration it's novelty can be defined as the distance of the mean of this new configuration with $(m_x, m_y)$.
But I think it will not work greatly as some very different configuration of points can also have the same mean.

You can define different measures in this way:

Maximum distance of the new point with all points of the configuration ($M$)
Minimum distance of the new point with all points of the configuration ($N$)
$\frac{M}{\text{Maximum distance between two points of the configuration(}D)}$: normalize (1) measure
$\frac{N}{D}$:normalize (2) measure

You can get more ideas from distance measures in hierarchical clustering methods. To select a proper one, you need to elaborate on the context of these points.

I think in the first or third one, Novelty will be directly proportional to the measure, but not so sure about the second and fourth one, because I think these two won't be very good in differentiating between novel and non novel configuration.


text:
Linear output layer back propagation
So I'm stack to something that it's probably very easy but I can't get my head around it. I'm building a Neural Network that will consist of many layers with non-linear activation functions (probably ReLUs) and the last output layer will be linear because we are trying to catch a specific number and not a probability. I've done the forward propagation calculations but I'm stuck at the back propagation ones.
Let's say that I'm gonna use the cross entropy loss function: (we will implement the MSE as well)   $-(y \log (a)+(1-y) \log (1-a))$. (I understand that this is not a good option for a regression problem)
So we can easily find the dJ/dA $d A=\frac{\partial J}{\partial A}=-\left(\frac{y}{A}-\frac{1-y}{1-A}\right)$ of the last layer and we can start going backwards finding the $\frac{\partial J}{\partial Z^{[L]}}$ which we can calculate from the equation: $d Z^{[L]}=d A*g^{\prime}\left(Z^{[L]}\right)$ The problem lies at the second part of this equation where the derivative of g is.
What will be the outcome since we have a linear activation function which derivative is equal with 1? (Activation function: f(x) = x, f'(x) = 1)
Will it be an identity matrix with the shape of Z[L] or a matrix full of ones with the same shape again? I'm asking about the term $g^{\prime}\left(Z^{[L]}\right)$.
Many thanks.

You don't want to implement cross entropy loss with linear activation. The cross entropy loss function is poorly defined when inputs are less than 0 or greated than 1. It can be done though, so if you are still interested in the theory, can you please specify in more detail what the loss function is - give a formula, because there are a couple of variations that are called "cross entropy".

You can use LaTex between $ signs e.g. $\nabla_{Z^{[L]}} = \frac{\partial J}{\partial Z^{[L]}}$ becomes $\nabla_{Z^{[L]}} J = \frac{\partial J}{\partial Z^{[L]}}$. I suggest you do this in the question to make it easier to read. Use [edit] link to alter the question. Also, can you be more specific about what "the second part of this equation" refers to - say which term you mean? Also be clear about "outcome" - are you asking about the overall loss value, or just the value of the term yo uare having trouble with?

I just edited the post. You can have a second read now. Many thanks.

Can you please put your main question in the title? Right now, your title is not a question and it's not very descriptive.


text:
Role of autoencoder in Hierarchical Extreme Learning Machine
I want to build HELM neural network that consists of autoencoder (AE) and one class classification (OC).
HELM with AE and OC have following shape:

That is, hidden layer output of AE is input of OC.
Training of HELM consists of training AE and OC separately. In order to train each neural network in HELM, first there are generated random weights and biases between input and hidden layers, and then based of them and activation function (for example sigmoid), only weights between hidden and output layers are trained. But then what's the point in training weights between hidden and output layers in AE, since only output of its hidden layer is provided as input into one-class classifier? What is point in use AE in HELM if weights between input and hidden layers of AE are basicaly random?
Following paper (page 6):
https://arxiv.org/pdf/1810.05550.pdf
confirms that output of hidden layer of AE is input for OC, but also on the contrary in Algorithms 2 and 3 (pages 6, 7), there is shown that input of OC is AE input vector multiplied by matrix of weights between hidden and output layer, what sounds weird for me.


text:
Getting bounding box/boundaries from segmentations in UNet Nuclei Segmentation
From my understanding, in a tissue where nuclei are present and need to be detected, we need to predict bounding boxes (either rectangular/circular or in the shape of the nucleus, i.e. as in instance segmentation). However, a lot of research papers start with semantic segmentation. Again, what I understood is semantic segmentation won't give the location, bounding box or count of nuclei. It will just tell that some stuff is probably nuclei and rest is probably background.
So, what is the bridging that I am missing when trying to detect nuclei from semantic segmentation. I have personally done semantic segmentation but I can't seem to count/predict bounding boxes because I can't understand how to do that (for example if semantic segmentation gave a probable region for nuclei which is actually a mixture of 3 nuclei overlapping). Semantic segmentation (in the example) just stops right there.

Thresholding algorithm like Watershed might not work in some cases as demonstrated in [Nuclei Detection][1] at 23:30 onwards.
Edge detection between segmented nuclei and background would not separate overlapping nuclei.
Finding local maxima and putting a dot there might give rise to false positives.
Finding IoU but what if the output of segmentation is not a region of classification (1s and 0s) but a continuous probability map from values between 0 to 1.
Isn't finding contours and getting bounding boxes from masks using opencv a parametric method? What I mean is, it being an image processing technique, there are chances it will work for some images and won't work for some.


text:
In reinforcement learning, how to craft observation space when environment is made of multiple blocks?
In reinforcement learning problems like cartpole, usually the environment is a single system that takes an input and gives an output. For example, in cartpole, given positions, velocities as input we can tell if pole is going to fall. Hence, we can craft observation space from positions and velocities.
But if environment is made of sequential blocks of systems {A, B, C}, where input is given to A that passes to B then to C then to output, then how do we craft observations?
Options:

We use a separate block D (linear or non-linear) to create numerical observations for input which is then fed to A. Problem with this: Observation crafted out of D is not correlated to output at C. For example, in cartpole, I would say that I would pass the velocities and positions thru a block D to get amount of fuel left in cart. And I would feed the amount of fuel left as observation to actual system which determines if the pole falls.
We use block A to create numerical observations for input. Better than option 1 but observations crafted out of A are not reflective of the entire process (A -> B -> C).

So, how do we craft observations for such an RL problem?
Edit: An example scenario. I have a system A that takes a song and gives N similar songs as output. I have a system B that takes a song and determines if the song is liked by more than 10 million people. Then I have a system C that takes a song liked by more than 10 million people and says if the song is of genre 'pop' or 'edm'.
I have 10 'edm' songs (liked by more than 10 million people) to start with, and I want to apply Reinforcement Learning to expand the 10 'edm' 'liked by more than 10 million people' songs to more such songs. By optimally, I mean we could always brute force for each song but the system A is non-deterministic. It can give N similar songs (N ranges from 0 to 100). In X amount of time, I want to maximize creations of such songs. Say, song X and song Y are the initial inputs. After we pass them thru A->B->C we could get 100 such similar ('edm' & 'liked by more than 10 million people') songs for X but only 2 for Y. Brute-force would waste time a lot in places that would not yield many such songs.
Now, an RL agent can decide which song to pick to get maximum number of similar such songs in X amount of time. But the environment is A -> B (is it liked by 10 million people, output is 1 or 0) -> C (is it 'edm' or 'pop', output is 1 or 0). Songs are represented as signals (array of continuous values between -inf to +inf) by passing them thru a system D. The system D takes the song name, downloads it and gets the signals (array representation of the songs). So, these numerical representations can be used instead of song names.
How do I craft observations for this combined environment?
As far as I know observation space should be such that given 2 observations we can distinguish which one gives better results, like in cartpole, 2 sets of observations (positions1, velocities1) and (positions2, velocities2) the system can identify which observation is relatively better or worse. But in our system (A -> B -> C) how do we go about crafting an observation that correlated with output?

you could model this as a graph and use a graph neural network to process the state. each block would be a node, and edges would represent relationships between the nodes. this has been covered somewhat in the literature (relational RL)

I am not sure I understand your problem. Could you give a concrete example of a system which consists of these separate blocks that you are concerned about, and explain why you think there is some issue (are you not able to receive or measure state-related information from some of the blocks, if not then why not?)

Thanks for the update, it helps understand what you mean by "block". I don't have any idea for an answer though, sorry. I do think the "block" thing is not the real problem here - your state can just be (a summary of) the songs selected so far, because everything else is either random or deterministic functions that will give you measurable results once you have chosen your songs. So RL would be learning the output of (likely a very complex) return-prediction function based on the song characteristics.  I think the real problem is trying to learn this function at all through trial and error.

Wont RL learn to map the system D (the machine that takes input song name and produces signal [array/numerical representation]) output to the final output (at C). So, if I train RL agent on 10 songs. And I try to predict on 10 unseen songs, won't the agent perform poorly? Agent learnt actions, values based on numerical representation of system D on 10 train songs. 10 test unseen songs may have num representations not similar/closer to training ones. Agent will perform poorly. It hasn't seen those num representations. Do we then train on large number of songs with varied representations?


text:
When should we use separable convolution?
I was reading the "Deep Learning with Python" by François Chollet. He mentioned separable convolution as following

This is equivalent to separating the learning of spatial features and
the learning of channel-wise features, which makes a lot of sense if
you assume that spatial locations in the input are highly correlated,
but different channels are fairly independent.

But I could not understand what he meant by saying "correlated spatial locations". Can some explain what he means or the purpose of separable convolutions? (except performance-related part).
Edit: Separable convolution means that first depthwise convolution is applied then pointwise convolution is applied.

To explain this part "spatial locations in the input are highly correlated", i.e. that pixels in some neighbourhood of a certain channel are likely to be correlated, imagine that in some neighbourhood of the channel you have the face of a person, then it should be clear that all pixels that belong to this face are correlated, in the sense that they will have very similar intensity value (for the respective channel). I can't answer why channels would be independent without more context. Maybe they would be independent because they are not encoding the RGB values but something else.

@nbro It does not have to be RGB, lets say we apply separable convolution as second layer. In this context channel number comes from number of filter we applied as you probably know? The example you gave did not clarify the problem.

Why doesn't my example clarify what the authors mean by "spatial correlation"?

Context of the question
This is a link to the text cited in the question.
It refers to the usage of SeparableConv2D (tf, keras name). A related question on StackOverflow is "What is the difference between SeparableConv2D and Conv2D layers". This answer points to this excellent article by Chi-Feng Wang:
A Basic Introduction to Separable Convolutions
Answer to the question
In image processing, a separable convolution converts a NxM convolution to two convolutions with kernels Nx1 and 1xM. Using this idea, in NN a SeparableConv2D converts a WxHxD convolution (width x height x depth, where depth means number of incoming features ) to two convolutions with kernels WxHx1 and 1x1xD.
Note the first kernel doesn't handles information across features, thus, it is "learning of spatial features". The 1x1xD kernel doesn't handles different points, it is "learning of channel-wise features".
About the phrase "spatial locations in the input are highly correlated", my understanding of what the author means is: Assume we have a channel (feature) image that each pixel measures the "distance to the background". When we pass from one pixel to a neighbors one, it is expected some continuity in the value (except for edge pixels): correlation. Instead, if we have a channel that measures "brightness" and another one that measures "distance to background" the two values for one specific pixel has little correlation.
Finally, about title question "When should we use separable convolution?" : if the final output must depend of some features of one pixel and some other features of neighbors pixels in a very unpredictable way, a complete WxHxD convolution must be used. However if, as is more usual, you can handle first spatial dependencies (neighborhood) to extract pixel features and next handle pixel-by-pixel these features to get the output, better use a WxHx1 followed by 1x1xD, saving lots of network parameters, thus, saving training time.

"a complete WxHxD convolution must be used" Why? A DSC has the same receptive field as a normal convolution so should, in theory, always be able to replace it.


text:
Choosing and Designing Decay Types for Epsilon-Greedy Exploration in Reinforcement Learning
I am working on a reinforcement learning project that involves epsilon-greedy exploration. I have two questions regarding the choice between linear and exponential decay for epsilon, and the appropriate design of the decay constant in the exponential case.

How do we determine whether to use linear decay or exponential decay for the epsilon-greedy exploration mechanism in reinforcement learning?
When using exponential decay, how should we design the decay constant? Different decay constants determine the shape of the curve. In linear decay, we can use the epsilon_decay parameter to determine the shape, but in exponential decay, the relationship between epsilon_decay and the shape is harder to imagine.

I understand that there may not be a single answer for these questions, and I welcome a wide range of discussions on this topic.
Here is the code for the two types of decay:
import math
import matplotlib.pyplot as plt

def exponential_epsilon_decay(step_idx, epsilon_start=1, epsilon_end=0.01, epsilon_decay=100_000):
    """
    Calculates the value of epsilon for a given step index using exponential decay and the specified parameters.

    Parameters:
    step_idx (int): The index of the current step.
    epsilon_start (float): The starting value of epsilon.
    epsilon_end (float): The minimum value of epsilon.
    epsilon_decay (float): The rate at which epsilon decays.

    Returns:
    float: The value of epsilon for the given step index.
    """
    return epsilon_end + (epsilon_start - epsilon_end) * math.exp(-1. * step_idx / epsilon_decay)

def linear_epsilon_decay(step_idx, epsilon_start=1, epsilon_end=0.01, epsilon_decay=100_000):
    """
    Calculates the value of epsilon for a given step index using linear decay and the specified parameters.

    Parameters:
    step_idx (int): The index of the current step.
    epsilon_start (float): The starting value of epsilon.
    epsilon_end (float): The minimum value of epsilon.
    epsilon_decay (float): The total number of steps over which epsilon will decay from epsilon_start to epsilon_end.

    Returns:
    float: The value of epsilon for the given step index.
    """
    return epsilon_end + (epsilon_start - epsilon_end) * max((1 - step_idx / epsilon_decay), 0)

n_steps = 1_000_000
epsilon_decay_rates = [20_000, 100_000, 500_000]

# Calculate epsilon values for each step
exp_epsilon_values = [[exponential_epsilon_decay(step, epsilon_decay=epsilon_decay_rate) for step in range(n_steps)] for epsilon_decay_rate in epsilon_decay_rates]
lin_epsilon_values = [[linear_epsilon_decay(step, epsilon_decay=epsilon_decay_rate) for step in range(n_steps)] for epsilon_decay_rate in epsilon_decay_rates]

# Plot the epsilon decay
plt.plot(exp_epsilon_values[0], 'r-', label='Exp decay_rate=20_000')
plt.plot(exp_epsilon_values[1], 'g-', label='Exp decay_rate=100_000')
plt.plot(exp_epsilon_values[2], 'b-', label='Exp decay_rate=500_000')
plt.plot(lin_epsilon_values[0], 'r--', label='Lin decay_rate=20_000')
plt.plot(lin_epsilon_values[1], 'g-.', label='Lin decay_rate=100_000')
plt.plot(lin_epsilon_values[2], 'b:', label='Lin decay_rate=500_000')
plt.title('Epsilon Decay')
plt.xlabel('Step')
plt.ylabel('Epsilon')
plt.legend()
plt.show()

In linear_epsilon_decay, the epsilon_decay parameter corresponds to the moment when the decay reaches epsilon_end, but in exponential_epsilon_decay, the epsilon_decay parameter has a less direct relationship with when it reaches epsilon_end.
Feel free to point out any issues in the code, and I appreciate any insights or suggestions you can provide.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

I'm not an expert on this area, but i'm currently developing a project in RL and the knowledge that i have, tell me that you can (regarding the 1st question) find the best epsilon-decay strategy at the training phase of your RL agent.
IMO, you need to observe the agent's performance throughout the episode and simply analyze. If your performance is currently aceptable you might consider decrease the decay rate to achive the same performance in less time (be more sample efficient), in case you want more performance, you may need to explore more than exploit, hence have a decay rate more slow. Hence, linear and exponential decay might be choosen depending if you want a uniform decay of exploration to exploitation, or a fast transition from exploration to exploitation.
Regarding the 2nd question, you should have a way to correspond a decay rate to the number of episodes actually needed to train, with this instead of changing directly the decay rate, you lead with the episodes needed to train. To accomplish this you can make:
For Linear Decay:
def cptEpsilonLin():
    DECAY_FACTOR = (eps_start - eps_end) / (TRAIN_EPSIDOES)
    eps = max(eps - DECAY_FACTOR, 0)

For Exponential decay:
def cptEpsilonExp():
    DECAY_FACTOR = (eps_end/eps_start)**(1/TRAIN_EPSIDOES)
    eps = max(eps * DECAY_FACTOR, eps_end)

Adding a minor detail to @Vitor Martins' answer, if performing linear decay you usually want to reach eps_end after a set amount of episodes so you can start seeing the performance of your agent and remove randomness.
Something like
DECAY_FACTOR = 2 * (eps_start - eps_end) / (TRAIN_EPISODES)
will make sure that you reach eps_end after half of TRAIN_EPISODES is complete.


text:
How to define Agar.io state and action space?
I am trying to implement an AI bot for my Agar.io clone using deep neural network.
However, I am struggling with the state and action space of the AI bot.
Because the bot can take real number for position and velocity, can I say the state space is continuous?
For the action space, I am thinking something like (velocityX, velocityY, "split to half", "eject mass").
What should be the number of input nodes in the input layer for my Neural network? And what are those input(observations, rewards)?
As the number of players and AI bots are changing, how can I train a dynamic network with changing input node number?
For the outputs, how can I get a continuous action output like velocity?
As a reference, you can learn about the game rules from this short youtube video:
20 Rules and Game Mechanism of Agar (How to Play Agar.io)

The state space is certainly continuous, assuming that you can somehow feed that AI exact coordinates. You may have to resort to CNNs if you do not have access to this information. For the action space, you should consider how the game actually plays. Since you use a mouse to simply show the direction, you could use (x,y) positions of the mouse as an action, or even just the angle $\theta$ of the mouse cursor in a circle around the agent. If you are playing on the site, then your observations would have to be from a CNN, it should be possible to use the score as your reward, as well as the possibility for eating things or distance from opponents that are bigger or smaller as intermediate rewards. The number of nodes in your network is something that you must find experimentally, you may like to research what kind of architectures other people have used in this field. You shouldn't need to account for different numbers of players. Nothing special needs to be done to get a single node giving a continuous output to represent the angle $\theta$, or alternatively two nodes representing x and y. You can then use tanh or sigmoid to limit the output node values for eject and split actions.


text:
Is my dataset unlearnable, or is my LSTM model not smart enough?
I have time-series data obtained from a video. The data is composed of bitrate and corresponding label pairs for each timestamp:

The distribution over the first 30 seconds is as follows:

I have built an LSTM model for this dataset to be able to classify the labels based on the bitrate. However, it seems that my model is not able to learn. Validation accuracy starts from approximately 0.3 (makes sense, since I have 2 classes (log2 = 0.3)) and it does not improve.
Do you have any idea about this, is it normal considering this sample data distribution, or is something might be wrong with my model? Thanks!

well, what is your model and how are you training it?


text:
Why is tree search/planning used in reinforcement learning?
In AlphaGo Zero, MCTS is used along with policy networks. Some sources say MCTS (or planning in general) increases the sample efficiency.
Assumed the transition model is known and the computational cost of interacting through planning is the same as interacting with the environment, I do not see the difference between playing many games versus playing a single game, but plan at each step.
Furthermore, given a problem with a known transition model, how do we know combining learning and tree search will likely be better than pure learning?

This post is not very clear and it's not clear to me what your question really is. In the title, you have a very general question, but in the body you have "given a problem with the known transition model, how do we know combining learning and tree search will likely be better than pure learning?", which seems to be more specific, but it's not clear what you mean by "learning". Moreover, you also say  "I do not see the difference between playing many games versus playing a single game, but plan at each step.", which is not entirely clear. Do you know what "transition model" is used for?

Question for you: is the transition model known in go? Do you know what planning means in the context of RL? Try to answer these questions and then think about what your question really is. So, please, edit your post to  ask only one question (and make the question in the title the same as the question in the body!).

Some sources say MCTS (or planning in general) increases the sample efficiency.

If we're thinking purely about experiments run in simulations, then I'd estimate there may be cases where a combination of pure learning + MCTS (or some other form of planning / model-based aspect) may be more efficient, and there may be different cases where only a single one of those techniques on its own may perform better. So then I wouldn't say this is always necessarily true.
Often though, when saying "sample efficiency", we would only count the steps actually taken in a "real" environment, but maybe not count steps taken in a lookahead search or planning algorithm. In pure simulations this may seem like a weird distinction to make, but it's more sensible when you consider that we often use simulations just because they're convenient ways to evaluate and compare new algorithms, but often not the end goal. Often, the end goal would be to apply something in the "real world", for example on a robot or something. In such a situation, collecting pure learning experiences can be very expensive (time-consuming, maybe also risky because the robot may fall over and break, etc.). But you may be able to also provide that robot with a learned model, or simulator, and have the robot use that to also run its own search or planning algorithms on an approximated version of the real world. This is a clear case where performing such is much faster and cheaper and less risky than collecting true experiences in the real world for pure RL.

Assumed the transition model is known and the computational cost of interacting through planning is the same as interacting with the environment, I do not see the difference between playing many games versus playing a single game, but plan at each step.

Jumping back to the case where we actually are working purely in simulation, and where there's no difference in computational cost between steps taken for pure learning vs. steps taken in search/planning, there absolutely are still some differences. If you run search, you use extra time and have temporary extra memory usage (which frees up again after completing your search algorithm) to make one really good decision in the "main" environment. You could view the steps taken in MCTS simulations as a form of learning as well, but they have a different purpose from the steps taken in a pure RL setting; these steps are taken with the sole goal of learning how to act well in the root state. All search time and memory usage is dedicated to that single decision. This can enable smarter decision-making than if you're going purely off of what you learned through pure RL, due to 1) focusing more effort on just a single decision, and 2) not being constrained by the capacity of your learning algorithm to actually learn a good decision (simple function approximators may simply not be capable of accurately representing strong policies, and more complex function approximators will take a huge amount of time to learn). This smarter decision-making thanks to search can in turn also improve the quality of the experience used by your pure RL component.

Finally, since you started out by mentioning AlphaGo Zero, I'd like to emphasise that AlphaGo Zero and similar approaches are typically used in multi-agent adversarial domains (like zero-sum games). Pure RL approaches for such multi-agent settings do exist, but there has been significantly less research towards them than pure RL approaches for single-agent settings. Most of these pure RL approaches are really only applicable to single-agent settings, and can easily have poor performance when applied to these kinds of multi-agent settings. Search algorithms like MCTS on the other hand are very well-established techniques in these adversarial domains like games, and the combination of them with learning approaches appears to allow even for learning approaches to be used which are not explicitly "aware" of the fact that they're operating in such a multi-agent domain.

Nice explanation. Thank you. Just a question: assume known transition model, are there circumstances in which pure RL have advantages over RL + search (other than the cases where pure RL already meets the expectation)? Wouldn't RL + search always have equal/better performances as it can put more effort on each single decision?

@SmoothKen Sure. There's cases where you want to work with a pre-built simulator for instance, and that simulator may not have been designed in such a way that it easily exposes some of the functions you'd want to have for tree search (like copying/resetting states). This is the case with StarCraft / DOTA research for instance. There's also problems with continuous state and/or action spaces, for which there are pure RL techniques, but tree search is very difficult. And probably lots of other things that don't immediately come to mind right now for me


text:
How is the latent vector transforming to a feature map in DCGAN (Generator structure)?
I'm working on the code trying to generate new images using DCGAN model. The structure of my code is from the PyTorch tutorial here.
I'm a bit confused trying to find and understand how the latent vector is transforming to the feature map from the Generator part (this line of code is what I'm interested in) :
nn.ConvTranspose2d(  nz, ngf * 8, 4, 1, 0, bias=False)

It means the latent vector (nz) of shape 100x1 transforming to the 512 matrices of 4x4 size (ngf=64). How  does it happen? Hence, I can't even clear to myself how the length of the latent vector influence to the generated image.
P.S. The left part of the Genarator structure is clear.
The only idea that I got is :

E.g. there is a latent vector of size 100 as input (100 random values).
We interact each value of the input latent vector with a 4x4 kernel.
In this way we get 100 different 4x4 matrices (each matrix is for one value from latent vector).
Then we summarize all these 100 matrices and get one final matrix - one feature map.
We get necessary number of feature maps taking different kernels.

Is this right? Or does it happen in other way?

Noise vector (batch_size, 100, 1, 1) is deconvoloved with filter_1 (100, 4, 4). Result is feature_map_1 (1, 4, 4). And since there are 512 filters, so there will be 512 feature maps. Output shape will be (batch_size, 512, 4, 4).
I think you need better undersanting for convolutional calculations in general. In this stack it was explained very well.

So, yeah, it seems that it works in the way like I suggested in my question. Thx.


text:
Purpose of the hidden variables in a Restricted Boltzmann Machine
From the part titled Introducing Latent Variables under subsection 2.2 in this tutorial:

Introducing Latent Variables. Suppose we want to model an $m$-dimensional
unknown probability distribution $q$ (e.g., each component of a sample corresponds to one of m pixels of an image). Typically, not all variables $\mathbf{X} = (X_v)_{v \in V}$ in an MRF need to correspond to some observed component, and the number of nodes is larger than $m$. We split $\mathbf{X}$ into visible (or observed) variables $\mathbf{V} = (V_1,...,V_m)$ corresponding to the components of the observations and latent (or hidden) variables $\mathbf{H} = (H_1,...,H_n)$ given by the remaining $n = |\mathbf{V}| − m$ variables. Using latent variables allows to describe complex distributions over the visible variables by means of simple (conditional) distributions. In this case the Gibbs distribution of an MRF describes the joint probability distribution of $(\mathbf{V},\mathbf{H})$ and one is usually interested in the marginal distribution of $\mathbf{V}$ which is given by:
$$p(\mathbf{v}) = \sum_{\mathbf{h}} p(\mathbf{v},\mathbf{h}) = \frac{1}{Z} \sum_{\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})}$$
where $Z = \sum_{\mathbf{v},\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})}$. While the visible variables correspond to the components of an observation, the latent variables introduce dependencies between the visible variables (e.g., between pixels of an input image).

I have a question about this part:

While the visible variables correspond to the components of an observation, the latent variables introduce dependencies between the visible variables (e.g., between pixels of an input image).

Given a set of nodes $\mathbf{X}$ in a Markov Random Field $G$, the joint distribution of all the nodes is given by:
$$p(\mathbf{X}) = \frac{1}{Z} \prod_{c \in C} \phi(c)$$
Where $Z$ is the partition function and $C$ is the set of cliques in $G$. To ensure that the joint distribution is positive, the following factors can be used:
$$\phi(c) = e^{-E(c)}$$
Such that:
$$p(\mathbf{X}) = \frac{1}{Z} e^{-\sum_{c \in C} E(c)}$$
Where $E$ is the energy function.
I am not sure why there is a need to introduce hidden variables and express $p(\mathbf{v})$ as a marginalization of $p(\mathbf{v},\mathbf{h})$ over $\mathbf{h}$. Why can't $p(\mathbf{v})$ be expressed as:
$$p(\mathbf{v}) = \frac{1}{Z} e^{-\sum_{v \in \mathbf{v}} E(v)}$$
directly? I think it may be because the factors only encode dependencies between variables in cliques, and so may not be able to encode dependencies between variables that are in two separate cliques. The purpose of the hidden variables are then to encode these "long-range" dependencies between visible variables not in cliques. However, I am not sure about this reasoning.
Any help would be greatly appreciated.
By the way, I am aware of this question, but I think the answer is not specific enough.

Welcome to SE:AI!

@DuttaA thanks for the advice! Which chapter in the deep learning book are you referring to?


text:
How can I reconstruct sparse one-hot encodings using an RBM?
I am currently working with a categorical-binary RBM, where there are 50 categorical visible units and 25 binary hidden units. The categorical visible units are expressed in one-hot encoding format, such that if there is 5 categories, then the visible units are expressed as a $50 \times 5$ array, where each row is the one-hot encoding of a category from 1 to 5.
Ideally, the RBM should be able to reconstruct the visible units. However, since the visible units are in one-hot encoding, then the visible units array contains a lot of zeros. This means the RBM quickly learns to guess all zeros for the entire array to minimize the reconstruction loss. How can I force the RBM to not do this and to instead guess 1's where the category occurs and 0's otherwise?
Note that I would still have this problem with a regular autoencoder.

I don't think the problem you are experiencing is related to overfitting (as the title suggests), but due to sparsity of output in the target population for the generator. Perhaps you could [edit] the title? Interesting problem though

Yes you are correct. Here is an article discussing this exact problem. However, this article discusses autoencoders, not RBMs. Got any ideas?


text:
How am I supposed to code equation 4.57 from the book "Machine Learning: An Algorithmic Perspective"?
Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained
$$\delta_o(\kappa) = (y_\kappa - t_\kappa)y_\kappa(\delta_{\kappa K} - y_K),$$
which is derived from equation 4.55 (p. 107)
$$y_{\kappa}(1 - y_{\kappa}),$$
which is to compute the diagonal of the Jacobian, and equation 4.56  (p. 107)
$$-y_{\kappa}y_K$$
In the book, it is not explained how they go from 4.55 and 4.56 to 4.57, it is just given, but I cannot follow how it is derived.
Moreover, in equation 4.57, the Kronecker's delta function is used, but how would one handle cases $i=j$, then we must have some for loop? Does having an $i$ and $j$ imply we need a nested for loop?
Also, I have tried to just compute the derivative of softmax according to the $i=j$ case only, and my model was faster (since we're not computing the jacobian) and accurate, but this assumes the error function is logarithmic, which I would like to code the general case.

Let's first clarify that we are trying to differentiate the softmax function with respect to $h_K$. What is $h_K$? $h_\kappa$ is the output of neuron $\kappa$, but what is capital $K$?