text:
Why are all the gradients values 0 except for the first iteration?
I am fine-tuning a mistral-7b with Hugging Face peft and quantization. In my training loop, I am printing the gradient values for each batch, which seem a bit unusual.
# Print gradients
for name, param in model_init.named_parameters():
      if param.grad is not None:
           print(f'Gradient for {name}: {param.grad.norm()}')

I am trying to understanding, why all the gradients values are 0s except for the #1 iteration (starting from 0th).
iteration 0
...
...
Gradient for base_model.base_model.model.model.layers.31.self_attn.q_proj.lora_B.default.weight: 0.0
Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_A.default.weight: 0.0
Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_B.default.weight: 0.0
Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight: 0.0
Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight: 0.0

iteration 1
...
...
Gradient for base_model.base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight: 0.0142822265625
Gradient for base_model.base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight: 0.0
Gradient for base_model.base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight: 3.953125
Gradient for base_model.base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight: 0.0
Gradient for base_model.base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight: 0.185546875

iteration n
...
...
Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_A.default.weight: 0.0
Gradient for base_model.base_model.model.model.layers.31.self_attn.k_proj.lora_B.default.weight: 0.0
Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight: 0.0
Gradient for base_model.base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight: 0.0

Isn't this a bit unusual? Is it the vanishing gradient problem?
For context,
# custom class
class BinaryClassification(nn.Module):

    def __init__(self, base_model):
        super().__init__()
        self.base_model = base_model
        self.dropout = nn.Dropout(0.05)
        #self.classifier = nn.Linear(hidden_size, 1)
        self.relu = nn.ReLU()
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        outputs = self.base_model(x)
        dropout_output = self.dropout(outputs.logits)
        relu_output = self.relu(dropout_output[:, -1, :])
        probs = self.sigmoid(relu_output) # Apply sigmoid to logits to get probabilities
        #print('forward probs', probs)
        return probs

model_init = BinaryClassification(peft_model)

# optimizer
criterion = torch.nn.BCELoss()
optimizer = torch.optim.AdamW(model_init.parameters(), lr=0.001, eps=1e-08, weight_decay=0.001)
scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=10, verbose=True)

# Loss
def calc_loss_batch(input_batch, target_batch):

    output = model_init(input_batch)

    # Reshape target to match the shape of probs
    target_batch = target_batch.unsqueeze(1)

    if output.shape != target_batch.shape:
        raise Exception("Shape mismatch between input logits and target label")

    # Logits of last output token
    loss = criterion(output, target_batch)

    return loss
```

Normally, an adapter to perform binary classification might be similar to what you've described, but with the classifier and without the relu before the sigmoid (sigmoid after relu will output >=0.5).  You may also want to try using the output embedding layer's input rather than its output (vocab logits) if possible, and remove the sigmoid and use BCEWithLogitsLoss (it can be more numerically stable).
For this sort of model, I think it's unlikely that the gradients will vanish like that if the adapter and training procedure is correct.  As the adapter is currently, for base model logit outputs less than 0, which in some models could consistently be the case, the gradient will be 0 and backpropagate as 0s no matter the loss.  If this is the issue that you're facing, then you would find 0.5s for the forward probabilities.  Additionally, dropout_output[:, -1, :] looks like it's sampling the last output token logits.  You probably want to check that the last output token is not degenerate in some way (like almost always being . or here a pad token).  If neither of these are the case perhaps there is some issue with your loss function, labels or inputs, however, I think even if those were incorrect, they are not likely to be causing 0 gradients here.

I see. I am trying out BCEWithLogitsLoss, would you recommend removing the ReLU along with Sigmoid from the custom class?

Yes, with BCEWithLogitsLoss I would recommend removing both the relu and the sigmoid.  Dropout should also be removed if it's the last layer before the loss function.  It may be reasonable to also just include the single linear layer for now, but if you want multiple layers, you can run something like (linear, relu, optionally dropout) x n, linear -> BCEWithLogitsLoss, where n might be something like 1-3 for a simple example.  The output dimension of the last linear layer will be the number of binary outputs you want.  During inference you could check whether the output is greater than 0.


text:
Why can't we use Metropolis-Hastings in mini-batch settings?
I have seen papers such as Stochastic gradient Langevin dynamics (link) and Stochastic gradient Hamiltonian Monte-Carlo (link) which could be used to train EBM models (in the log-likelihood context where these methods could be used to estimate the expectation term that requires sampling from the model itself).
Why can't we use Metropolis-Hastings in mini-batch settings? Why do we need to iterate over the whole dataset to generate a single sample from the EBM in MH context?
I highly appreciate your answer.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the difference between densenet and resnet?
Is the only difference between the two how the skip connection is combined? Resnet combines skip connections through addition and Densenet through concatenating.
The Densenet paper appears to be arguing that their approach can allow gradients and information to flow from any previous layer to any future layer. However, that doesn't make sense since Resnet can already do that.

ResNet on the left side, DenseNet on the right side.
The advantages that are cited in the paper are:

alleviate the vanishing gradient problem -- which is true. The purpose of the ResNet is NOT to solve the vanishing gradient problem, but answering the question how to make deeper models actually better approximators?
strengthen feature propagation and encourage feature reuse -- which is also true. You can see that in the DenseNet the raw features from the first layer are propagated forwards to every next layer. While in the ResNet the features of the first layer are mixed with the next through addition and non-linearity, and so on..
reduce the number of parameters -- which is also true. This sounds a bit counter-intuitive but the fact is that these types of layers require less filter maps than the ResNet, resulting in fewer parameters. For a comparison you can checkout an experiment that I did on CIFAR-10 here:
https://github.com/pi-tau/deep-conv-nets?tab=readme-ov-file#model-comparison-on-cifar10

DenseNet and ResNet are both CNN architectures that effectively use skip connections in order to facilitate a better flow of gradients throughout the network.
Coming to the differences,
Implementation of Skip Connections
You are correct in saying that ResNet combines skip connections through additive identity transformations, whereas DenseNet uses concatenation to combine the outputs of each layer with feature maps of all preceding layers. The reason why I highlighted all preceding layers is that it is a key feature that distinguishes DenseNet based architectures from ResNet based architectures.
WHY?
A typical ResNet will have skip connections that will skip some layers and will allow the network's current layer to perform atleast as good as the previous layers. DenseNet on the other hand utilizes the feature maps created by all preceding layers which enables feature reuse which leads to better feature representation.
Parameters v/s Memory
The number of parameters present in a ResNet are substantially large because each layer has its own weights. In comparision, DenseNets add only a small subset of feature maps (at the current layer) to the previously existing "knowledge" i.e. the feature maps provided by the previous layers which makes it parameter efficient.
However, since DenseNet is concatenating the feature maps, as the network becomes deep, the number of feature maps increase which results in an increased demand in memory.
That being said, DenseNets are known to achieve comparable or better results with fewer number of parameters than ResNets.

The Densenet paper appears to be arguging that their approach can allow gradients and information to flow from any previous layer to any future layer. However that doesn't make sense since Resnet can already do that.

The gap in your understanding here lies in the fact that the DenseNet paper allows the gradients to flow from all previous layers to the current layer.
Remember - Denser the connectivity of your skip connections, easier the gradient flow will be throughout the network. (Till a certain level, after which it might start hindering the performance. As always, take every advice with a pinch of salt, there is no free lunch :)
Update
Let me present it in a more mathematical manner,
A typical convolution, as we know, can be formulated as follows:
$$f_{l} = H_{l}*f_{l-1}$$
where $f_{l}$ is the current feature map, $f_{l-1}$ is the previous feature map, and $*$ is the convolution operator. $H_{l}$ signifies the weight of the convolution.
Note - We are not considering the bias for ease of understanding.
Now, in a DenseNet, $f_{l-1}$ is connected to all preceding convolutional blocks,
$$Y_{l} = X_{0}.X_{1}.\cdots.X_{l}$$
where $X_{i}$ represents the feature maps of all preceding layers and $.$ represents the concatenation operation.
Replacing $f_{l-1}$ with $Y_{l}$, we get,
$$f_{l} = H_{l} * (X_{0}.X_{1}.\cdots.X_{l})$$
If we divide the weight $H_{l}$ into multiple smaller weights, we get,
$$f_{t} = (H_{l}^{0}.H_{l}^{1}.\cdots.H_{l}^{l})*(X_{0}.X_{1}.\cdots.X_{l})$$
$$=H_{l}^{0}*X_{0} + H_{l}^{1}*X_{1} + \cdots +H_{l}^{l}*X_{l}$$
For ResNet, since the connections are added together, it is trivial to show the following,
$$f_{t} = H_{l}*(X_{0}+X_{1}+\cdots+X_{l})$$
$$=H_{l}*X_{0} + H_{l}*X_{1} + \cdots +H_{l}*X_{l}$$
Now, we can see that for ResNet, we are using the same convolutional weight for all the previous layers, while the weights are different for DenseNet. Therefore, DenseNet has more flexibility as to how much weight should be given to any of the previous layer features whereas in the case of ResNet, the current layer determines the weight for all the features of the preceding layer.
This results in DenseNet's ability to use all the features of the preceding layers as is i.e., without changing the weights again and again.
Hope this makes it clearer to understand.

It's not special that densenet takes in gradients/features from all previous layers because resnet does the same, as per my last sentence

You should really look up how ResNet works. The skip connections in a ResNet-based architecture do not account for all of the previous features; they simply add the input of a layer to the output of another layer after skipping one or more layers in between. So, while the current layer might receive inputs from earlier layers, it does not necessarily imply that all the previous features will be incorporated into the output layer.

You're wrong about this. It does imply that all previous features are incorporated into the output layer.

Does the edit answer your question?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is there a way to mix the Importance weighted VAE with the Beta-VAE?
Is there a way to mix the Importance weighted VAE with the Beta-VAE?
It seems that we cannot separate out the KL-term from the IWAE-ELBO, hence we cannot multiply it with Beta. On the other hand, it would be great to use the Beta-regularisation also in IWAE.
I have tried to search for some papers or implementations for this approach but I have found nothing.


text:
Ideal method for finding function that satisfies a set of constraints
I'm new to playing around with deep learning. I'm trying to find a function that satisfies a bunch of constraints, and looking for tips on how to better my approach.
Let $F:X\times Y \mapsto [0,1]$, where $X=[0,Z]$,$Y=[0,M]$ compact. $F$ is a cdf over $X$, and twice differentiable in both arguments. I'm trying to find such an $F$ that satisfies, well that it's a cdf, but also a few more constraints that can be expressed locally.
Most constraints have to do with monotonicity of the function, or it's derivatives, or some transformation of those derivatives.
There is one last constraint (call it $g^*$) which I'm most interested in, and is supposed to hold only at a single point $x^*,y^*$.
There is no labelled data to be fitted here - I'm just searching over the space of functions $F$, evaluate them at their arguments, and seeing if they satisfy all constraints.
My naive approach was to try a fully connected deep neural net, where I'm playing around with nr of parameters, layers, type of activation functions. My loss just computes the penalties for constraint violations, sums them up and weights them - here also maybe points of improvement.
My loss function vaguely looks like this:
$$L(x,y,NN) = mon(x,y,NN) + intg(x,y,NN) + MLRP(x,y,NN) + g^*(x,y,NN), $$
where $mon,intg,MLRP$ are monotonicity constraints, constraints that ensure $NN$ outputs a cdf, and monotonicity on some kinds of growths rates of the pdf, and finally $g^*$. These constraints are all expressed as inequality constraints. Even though $g^*$ is supposed to hold only at the point $x^*,y^*$, I was playing around with a sort of kernelized approach, applying a weighted down version of $g^*$ for points nearby the critical points. Didn't help.
The issue is, after a while of trying different combinations of the above (I'm mostly using Adam, also played around with parameters there), I can't seem to get to satisfy both $g^*$ and all other constraints. I can tell that the main issue is $g^*$ - I can find functions that satisfy the other more `natural' constraints, but adding $g^*$ doesn't get me far. I had thought that something flexible with many parameters would be able to force the satisfaction of many kinds of single point constraint, but alas. Before I give up and accept that maybe this problem doesn't actually have a solution, I came here to ask whether I'm doing the wrong approach.
A strange thing that commonly happens during training is that $g^*$ is close to being satisfied sometime during the first few iterations, but the penalty converges to something far away. Here a sample plot of the $g^*$ violation over training:

What I'm asking is for some tips/insights as to how to generally approach such a problem - maybe fully connected neural nets is very wrong here? Also intuitions for why the process doesn't seem to like to stay at those `better satisfaction' points around periods 30 and 40?


text:
Is it possible to detect image manipulation, for example deep fakes?
Is it possible to detect image manipulation, for example deep fakes?
I will publish some photos on internet, and want others to be able to verify if photos are really from me, and also to detect if image has been modified or not.

Why have you posted this question shortly before posting an answer that claims "Digital signatures can be used to detect of deepfakes." and recommends an app to do that?  I find the juxtaposition puzzling.

Except digital signatures don't detect deep fakes at all, they set a potentially provable level of provenance (limited by the trust in the publishing/storage method of the signatures). The signing app doesn't know whether I have just signed a raw capture from a sensor, or a modified photo, or an AI generation. Instead, it sets a point after which further mods are reliably detectable (contingent on publish method for the signature)

This is what the discriminator in a GAN does. The generator produces synthetic images. The discriminator tries to tell the real from the synthetic images. Eventually, perhaps, the generator gets so good that the discriminator cannot tell the difference. Perhaps an even more capable discriminator could distinguish between the real and synthetic images, however.
You really just have a binary problem: real or synthetic. The problem might be very difficult, depending on how good the deepfake generation is, but that is not inherently different from any other situation where classes are quite similar in the feature space.

How the state of the art when it comes to deepfake generation vs detection compare goes beyond what I know.


text:
Does reducing the image dimensions before sending them to Language Models (LMs) significantly affect the processing time?
I am currently working on utilizing Language Models (LMs) to describe images in my project (Search Engine for AI images). However, I am wondering about the efficiency of the process, particularly in terms of processing time. Before feeding images into the Language Models (I'm using llava-13b), I am considering reducing their dimensions. My rationale behind this is to potentially decrease the time needed to process each image.
Is this approach advisable? Does reducing the image dimensions before sending them to Language Models (LMs) significantly affect the processing time? Or would the reduction in image quality adversely impact the accuracy of the descriptions generated by the Language Models (LMs)?
I would appreciate any insights or experiences anyone has had with a similar scenario. Thank you!


text:
Does the log determinant loss term mean that there is no need to use weight decay?
In training affine coupling layers in a normalizing flow, does the log determinant loss term mean that there is no need to use weight decay?
Since the loss term constrains the size of the transformation, does that mean the conditioner also learns simpler functions?
By the log determinant loss term, I am referring to the second term in the flow's likelihood:
$$\log p(x) = \log p(z) + \log \vert \det J\ \vert$$


text:
Are there leaderboards/tables/stats that compare inference times between close-sourced LLMs such as GPT 3.5/4 and Claude?
https://huggingface.co/spaces/optimum/llm-perf-leaderboard is great to compare inference times between LLMs but it misses close-sourced LLMs such as GPT 3.5/4 and Claude.


text:
What's the current best ways to extend context windows in LLMs?
What's the state of the art? The most elegant/ simplest to implement? The ones that require least computational resources?


text:
What ML/DL algorithms for frequency spectrum pattern classifications?
I have a set of known frequency spectrum data for this set of chemical compounds. Then the unknown Y is the mixture of some of these compounds. The task is to determine what compounds are in this mixture and their strengths. For example, S1 is the unit spectrum of the first compound and S2 is the unit spectrum of the 2nd compound. If the mixture has A units of the first compound and B units of the 2nd compound, the resulting Spectrum = AS1 + BS2.
I have seen examples of multi-class multi-label classification models, and multi-outputs regression models. But I need a model that can do both. Any suggestions?

spectrum as image can be classified using any image classification method..

Yes, but this task requires the model to predict the strength of "dog" and the strength of "cat". Common image classification models cannot do that.

Since your process is additive (emission spectrums?), this can be solved with non-negative least squares. You could try ordinal least squares as well, but it may give you negative concentrations. I don't think you need any fancy neural networks or ML here.
The constrained formulation is $ \arg \min_x || Ax - y ||_2^2$, $x \ge 0$, you "encode" the known spectrum data to $A$ and the measured spectrum is $y$, then $x$ gives you the concentrations.
If you measured the absorption spectrum instead, you'd need a different formulation.

There can be many compounds in the mix. The goal is to identify them and their strengths. Would this method work?

Yes, A is a n x m matrix where n is the number of measured wavelengths and m is the number of known compounds. Vector x has the length m and y has n. x is zero (or close to zero) for compounds not present in the mix, and positive for existing ones. I'm not sure what would happen if there are unknown compounds present though. At least the prediction error would be large.

To clarify, what do you mean by "the goal is to identify them" [the compounds]? Since we assume that the mixture consists only of known compounds, isn't it enough to determine their strengths (which may be zero for many of them, for a given sample)? For example there is no need to detect whether there are any unknown compounds in the mix.

Good question. If there is presence of unknown compound(s), this ML/DS/AI module needs to alert such event.


text:
Is there any standardized notation for drawing neural network diagrams?
Is there any standardized notation for drawing neural network diagrams? For example, for circuits there is a universal set of symbols used to draw different types of circuits why not for neural networks?

IEC vs NEMA, EE doesn't always have standardized diagramming.  How "standardized" are you looking for?

standard at the level of each operation, for example, convolution have a symbol, adding bias has a symbol and so on.


text:
How to generate an image of clothing based on the image of this clothing on some model (person) while preserving details
I have a task to generate an image of clothing based on the image of this clothing on some model (person). I tried different variations of diffusion models to reach the goal, but all of them had failed. The main problem is preserving the details of clothing (difficult patterns, prints etc.) when passing the conditional image to some encoder (e.g. clip, dinov2 etc.).
Is there any method to reach the goal of detail preservation with diffusion or are there any other models that can help me?

Do you have training data (example source and target images)? If so, how much? If not, how are you measuring performance of your model?

I have training data (pairs of source image -- target image) in quantity of 20k (augmented to 100k via variating permutation of RGB and flipping)


text:
Is it feasible to solve dynamic graph-level classification without labels?
I already did graph-level classification using heterogeneous hypergraph learning in an ICDM paper last year. However, I now want to extend it for dynamic graphs, i.e. the task is dynamic graph-level classification. I used MUTAG, ENZYMES, PROTEINS, COLLAB, IMDB-BINARY datasets that have different number of labelled graphs and then we used accuracy for evaluation. However, the dynamic graphs datasets like Facebook, Enron, Slashot, Formula 1 (Reddit), though, have different timestamped graphs but no labels.
I am just curious to know if this is a feasible problem to solve, i.e. dynamic graph-level classification without labels?

Can you explain what a "dynamic graph" is? What exactly is dynamic? Do you mean that the vertices and connections change over time? Also, what do you mean by "without labels"? What kind of labels do you have? Are you trying to classify the vertices or edges and you mean that you don't have some labels for the vertices or edges? I've also changed your title to be what I think is your question. Make sure that's the case.

@nbro yes, correct, a dynamic graph has nodes/edges that may change over time e.g., for a graph G_1, some nodes and edges can be added or deleted at t_2 as compared to t_1.  "Without labels" means there exist no labels for each timestamp of the graph. However, in the static graph-level datasets like MUTAG, for each graph in this dataset, we have a label as "mutagenic", or "non-mutagenic". Therefore, we can use accuracy or F1-score as an evaluation of our classification algorithm. I want to label/classify the whole graphs (dynamic) not individual nodes or edges.


text:
Machine Learning Algorithm for identifying the factors contributing to academic performance of students
I have a dataset with several qualitative and quantitative attributes, including age, location (longitude, latitude), city, parent occupation, family size, GPA etc. My task is to find the attributes/factors that contribute the highest towards the academic performance of the students. Which machine learning algorithm is the most suitable for this task?
A friend suggested using clustering algorithm that can segment the students into different performance groups, but I think that will not solve my problem.

You’re looking for an algorithm with identifies the explained variance of each feature.  The first to my mind was Principal Component Analysis.  But the more I thought about it, I personally am more comfortable with decision trees and their interpretability will often give a similar insight.


text:
How do I determine how likely some data was drawn from a distribution?
I have a sampled data, say, many vectors $m_1, \cdots, m_i \in R^{d}$.
I want an objective function that rates the probability of this data being drawn from certain distribution (here, specifically standard Gaussian $N(0, \mathbf{I}_{d\times d})$). That is to say, I want a function $f(m_1, \cdots, m_i)$ that takes the vectors $m_1, \cdots, m_i \in R^{d}$ and outputs a scalar that, the more likely the vectors are sampled from the distribution $N(0, \mathbf{I}_{d\times d})$, the smaller(or bigger) the function's return value is.
Is there any predefined ones in statistics or math theory?
I don't know much about math, just some simple ones from probabilities & statistics undergrad course.

What does the normal distribution mean? See probability density function.

Yes, it exists, and it's called the likelihood function (but if you want to actually calculate that "scalar", you better use the log likelihood)

In general there're many ways to achieve your request in statistics such as goodness-of-fit chi-squared test, likelihood function as mentioned in another answer, KL-Divergence, etc. And perhaps in your high dimensional case, Mahalanobis distance is a good fit.

The Mahalanobis distance is a measure of the distance between a point ${\displaystyle P}$ and a distribution ${\displaystyle D}$, introduced by P. C. Mahalanobis in 1936... If each of these axes is re-scaled to have unit variance, then the Mahalanobis distance corresponds to standard Euclidean distance in the transformed space. The Mahalanobis distance is thus unitless, scale-invariant, and takes into account the correlations of the data set.

For a normal distribution in any number of dimensions, the probability density of an observation ${\displaystyle {\vec {x}}}$ is uniquely determined by the Mahalanobis distance ${\displaystyle d}$... The Mahalanobis distance is proportional, for a normal distribution, to the square root of the negative log-likelihood (after adding a constant so the minimum is at zero).

Therefore by calculating the Mahalanobis distance for each data point with respect to the mean of your standard multivariate Gaussian distribution, you can obtain a measure of how well your total sampled data points fit the multivariate Gaussian. A smaller summed value indicates a better fit to the distribution and is equivalent to negative log-likelihood essentially as explained in above reference.


text:
Textbooks (or other sources) on deep reinforcement learning which explain theory along with good examples
I am looking for a textbook/other sources on deep reinforcement learning which explain theory along with  good examples. I will be happy for suggestions.

For me was "The Art of Reinforcement Learning" by Michael Hu the best book so far.
On the other side I found the Sutton and Barto Book extremely hard to digest.
By the way, check this link out: https://ai.stackexchange.com/questions/6997/whats-a-good-resource-for-getting-familiar-with-reinforcement-learning?rq=1

MARL is not part of this book.

I will often peruse MIT/Stanford for a class on a subject and check the syllabus for what they’re teaching from.

@foreverska can you provide any link?

@DSPinfinity you never mentioned, that you want MARL . In that case I can only suggest the www.neuralnet.ai Academy realized by Phil Tabor. Hi goes into MARL as well.

@Dave  A lot of the deep RL classes are short on book recommendations other than S&B and some DL book of choice.  But I did find https://www.ji.sjtu.edu.cn/wp-content/uploads/formidable/15/Course-Syllabus-VE693-Fall2021.pdf which says they recommend as reference: "An Introduction to Deep Reinforcement Learning. Foundations and Trends in Machine Learning"

@Dave Please, write that as an actual answer.

Does this answer your question? What are some online courses for deep reinforcement learning?, What introductory books to reinforcement learning do you know, and how do they approach this topic? or What's a good resource for getting familiar with reinforcement learning?. Your question is not an exact duplicate but maybe those questions already have the answer.

If not, I recommend you edit your post and state that. And then I'd encourage users to actually provide answers instead of writing comments

Despite not a textbook I found the Berkley lecture series quite good. You have lecture theory and practical homework tasks to solve.
Another famous learning resource is Spinning Up.

For me was "The Art of Reinforcement Learning" by Michael Hu the best book so far. On the other side I found the Sutton and Barto Book extremely hard to digest. By the way, check this link out.
In case you need MARL or are interested in it, then I suggest the very well academy course made by Phil Tabor www.neuralnet.ai
It costs some money but it was (for me) definetely worthly


text:
Multi-task objective sometimes improve single-task performance, but is this true when fine tuning?
It is known that multitask objectives in neural networks sometimes have the effect of improving the performance of the neural network for each of the tasks individually (versus training the same network for each task individually). To what extent is this true when fine tuning a pretrained, single-task neural network to deal with a multi-task objectives versus training from scratch?

Your statement is most likely true to the extent when all involved tasks are similar or strongly related and the amount of (labeled) data available for each individual objective task is limited. Therefore they can transfer the knowledge acquired from the pretrained single-task neural network to learn the new tasks with fewer available examples. Furthermore, fine-tuning a pretrained model acts as a form of regularization as it constrains the model to learn task-specific features while retaining the general knowledge captured by the pretrained layers to prevent overfitting, especially when the amount of data available for each task is limited.
In general multi-task learning encourages the model to efficiently learn shared representations that are useful across multiple relevant tasks, leading to better generalization and improved performance on each task individually. It's like the ordinary fact that jointly learning math and physics improves performance of each subject for most students as they're very related areas and most students time to study are limited.


text:
History of Neural Networks and Deep Learning
I'm interested in learning about the history of neural networks and deep learning. I've been reading about the field and am familiar with many of the developments since the 1950s.  Is there a textbook, video, or blog post that comprehensively covers the developments in this field from the 1950s to the present day?


text:
Why the architeture of Resnet18 is suitable to images classification?
I am studying convolutional networks and in particular I have focused on the ResNet18 network.  I've been studying ResNet18 and understand the purpose of skip connections and residual network. However, I'm still unclear on why the specific structures of Basic Blocks and Bottleneck Blocks are particularly well-suited for image classification tasks. Can any of you help me?

everything is nicely explained in the original paper https://arxiv.org/abs/1512.03385


text:
Is it possible to know the family of functions a neural network approximates?
Is it possible somehow to find which family of functions a particular network approximates?
For example, I have some directed graph and I use its vertices as artificial neurons (as simple McCulloch & Pitts neurons), and I want to know the class of functions which it can approximate

Can you be more specific about what you mean by "class of functions"? Do you have in mind any classes?

Maybe this answers your question?


text:
Should we sum or mean reduce the KL loss in Bayes by Backprop?
I'm unsure if you're supposed to use sum or mean reduction of KL loss for Bayes by Backprop. For example, the BayesianTorch library does both: it reduces by mean across each individual tensor (as seen in kl_div() method), but then it sums across layers (as seen in get_kl_loss()). This is logically inconsistent of course and I suspect it's a bug. But I'm not sure which is correct and why. Could someone explain?
Also citations to related derivations would be appreciated.


text:
Is there a model for facial detection based on an infrared camera?
I need an AI model for facial detection based on an infrared camera

Is there an existing model for this with per-trained weights?

Does this model work well when the lighting conditions may change rapidly or drastically, i.e. through normalization or something?

Hi. Asking for per-trained models may be off-topic. See our on-topic page. We can discuss this on meta again (if necessary). However, you may not just be interested in per-trained models but any (untrained) model suitable for this task, so I think your question would be on-topic.

Well yes and no. A model is only as good as the data you give it, it really depends on your data and what data you have as well as. I can provide you a pre-trained infrared machine learning repository https://github.com/maxbbraun/thermal-face <- It's here, but implementation is a different story.
If you know how you're going to implement it then badaboom my job here is done.

You can train a model by yourself if you have a dataset based on mtcnn architecture...

you can try the model from this git repo (this is for thermal cameras but it is a good reference so i have mentioned it ignore it if you don't find any thing important from this one )
=> https://github.com/xyth0rn/ThermalFaceDetection
It explains very well documentation about face detection in thermal cameras.
and if you want to train your custom model for face detection you can try this one repo
=> https://github.com/avshet/Face-recognition-in-Near-InfraRed-images.git


text:
What is the difference between RAG-Sequence Model and RAG-Token Model?
When I start to learn about RAG, I can't understand the difference between the RAG-Sequence Model and RAG-Token Model.
First of all, I see that for RAG-Sequence Model we use just one document to generate the hole output and for RAG-Token Model for each token we use all the k document chosen.
But, when I see the equation in the paper, I can't understand why they make a summation in the case of RAG-Sequence Model while we generate the output from one document.

In the first equation, you can see that in the sequence likelihood (the big product) $z$ is constant, which means that you first find which are the interesting documents, than you give your LLM these document, and you ask it to answer (for the whole generation, these documents are always the same)
In the second case, you can see that for every step ($i$), you have a corresponding $z_i$ (a set of documents) that is used in the summation. However, you can see that that summation is just an expectation for the $y_i$ token. This means that, if earlier you first find K documents and then generate an answer, now you have to find interesting documents every time you have to sample a token.
In other words:

in the first case: (1) sample K documents based on query (2) ask the LLM to generate the first token (3) concat the new token to the generated sentence (4) repeat 2 until EOS token is chosen
in the second case: (1) find top K documents based on the current generation&query (2) feed the documents to the LLM, (3) generate next token (4) concat the new token to the current generated sequence, (5) repeat 1 until EOS is sampled


text:
Can MAX ensure maximum minimum gain by using minimax once at game start?

MAX and MIN apply the minimax algorithm each time before making their move, searching to some predefined
depth. Imagine, however, that the MAX player applies the minimax algorithm only once, at the beginning of the
game, and then uses the obtained minimax values for all his future moves. In what case would MAX player
implemented in this way surely maximize his minimum gain?
A There is no guarantee that such a player would maximize his minimum gain
B If he has reached all final states of the game when calculating the minimax-value
C If MAX’s heuristic is more informed than MIN’s heuristics
D If he uses a heuristic that is identical to that of MIN player

Can someone explain why the answer is A and not B?

It's not too clear from your question what you mean, but I guess this is not even originally a question coming from you, so I'll suppose you cannot give more information
In any case, if the result of your minimax algorithm is a path of the game tree, aka a Nash Equilibrium, then the problem is that there might be more than one Nash Equilibrium. If the result of the minimax is a whole strategy, then it implies that you have found a SubGame Nash Equilibrium, and by definition you will maximize your minimum gain
This is assuming you are dealing with a perfect information game


text:
It is not clear why SARSA is on-policy but Q-learning off-policy
Here are SARSA and Q-learning from Sutton & Barto.

In these given forms, in my opinion, Q-learning is also on-policy because action selection is based on updated Q values. Where is my mistake, if any?

I found a very good answer here: https://stats.stackexchange.com/questions/184657/what-is-the-difference-between-off-policy-and-on-policy-learning

Does it fully answer your question or are there future questions?  If it fully answers your question you might think of deleting the question as there's a high chance it gets flagged as duplicate right now.

Yes it answers but in my opinion it will be better if we keep this question here because its title is different: it is specific to Q-learning and SARSA.

Your mistake is your definition of on-policy. In both cases, we select actions based on our current action value estimates. It's how we select those actions that differentiates the 2 algorithms.

An algorithm is off-policy if the target policy is different from the exploration/behaviour policy.
An algorithm is on-policy if the 2 policies are the same.

I don't like these definitions because they hide details, which are required to understand the concepts. So, let me just explain why these algorithms are what they are.
In the case of Q-learning, which is off-policy, you try to learn a greedy policy, that is, a policy that always selects a greedy action, which is the action that leads to the highest amount of reward. That's why we use the max operator in Q-learning. The max operator selects the greedy action value with respect to the current value function estimate. Eventually, that value function estimate may be optimal. In Q-learning, we often also use the $\epsilon$-greedy policy for taking actions in the environment, but you could use any other exploration policy. So, there are 2 different policies. That's why Q-learning is off-policy.
In SARSA, in your case, we use $\epsilon$-greedy both to take actions in the environment and to update the formula.

Still, it's unclear where the target policy comes into play in these algorithms. Sure, we use the max to select the action reward in Q-learning, or we use the $\epsilon$-greedy policy to compute the target value, but why using the max operator, for instance, implies that we're learning the greedy policy? Once we have the optimal value function, we can just act greedily wrt it to get an optimal policy, which is greedy (Huh?). So, basically, we compute the target by assuming we would derive a specific policy from the value function and, in the tabular case, it turns out that this approach will find an optimal value function and policy.


text:
Value iteration in a Grid World Example
I am facing some confusion regarding the calculation of the values for the states in a grid world.

Given that this is my grid world, where the there is a reward for going to F of +1, and no other rewards for other transitions, with the discount factor of .7, would I need to calculate the state value of F, that is, $V_{i+1}(F)$?
There is no indication that it is a goal state. I have calculated the state values of all other states with $V(F)$ being calculated and it not being calculated, where the calculations results are different. However, I am unsure which is the right calculation to use.
Also, I noticed that there are 2 different equations associated with the VI update
$$
V_{i+1}(s) = R(s) + \gamma \cdot \max_{a} [V_i(s')]
$$
$$
V_{i+1}(s) = \max_{a} [R(s, a) + \gamma \cdot V_i(s')]
$$
From what I understand, one is a reward associated with an action, but the other with a state. However, other than that, I don't understand when to use which.

It's not clear to me why you think you may not need to calculate $V(F)$. It's also even unclearer why you think there's no indication $F$ is the goal state, while you described it as the goal state based on the reward function. Maybe I'm misinterpreting your doubts or statements. Could you please clarify this and also change the title to be your specific question? Thanks

The expected immediate reward functions $R(s)$ and $R(s,a)$ are convenient structures for the more fundamental $R(s,a,s')$ (expected reward for starting in state $s$, taking action $a$ and arriving in state $s'$) or $p(r,s'|s,a)$ (probability of observing immediate reward $r$ and next state $s'$ given starting in state $s$ and taking action $a$).
Which one to use depends if and how you can take advantage of the structure of the MDP's reward system.
To use $R(s)$ you need the next expected reward to depend solely on the starting state. That isn't the case for your environment, because the reward is gained when arriving in state $s'$.
To use $R(s,a)$ you need the next expected reward to depend on the starting state and action. Technically this is always true, even if the environment is stochastic (and the agent might not end up in the same next state $s'$ each time), because you can always calculate the expected next reward from the fundamental features of the environment.
So in your case, you should use $R(s,a)$ because that will define which state the agent arrives in, and enable you to figure out the immediate reward.
There are other formulations of the update rule which use other building blocks. For example:
$$V_{t+1}(s) = \text{max}_a [\sum_{r,s'} p(r,s'|s,a)(r + \gamma \cdot V_t(s'))]$$


text:
Is there any actual difference between these 2 definitions of a state value function?
The definition of the value function in TRPO paper is
\begin{align}
V_\pi(s_t) &= \mathbb{E}_{a_t,s_{t+1},\ldots} \left[ \sum_{l=0}^{\infty} \gamma^l r(s_{t+l}) \right], \\[10pt]
a_t &\sim \pi(a_t|s_t), \quad s_{t+1} \sim P(s_{t+1}|s_t, a_t) \text{ for } t \geq 0.
\end{align}
In this book the value function is defined as
\begin{equation}
V_M^\pi(s) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \;\middle|\; \pi, s_0 = s\right],
\end{equation}
Is there any difference between those definitions or are they equivalent? It seems the first one takes into account that the trajectory is begining somewhere else and passing through state $s_t$y? The second defintion take into account just trajectories that start in the desire state to compute the expectation, but that means that trajectories that pass through the state don't matter for the value function?
Can anyone help me understand this?

The notation varies a lot, because there is no such rule on how to write it formally.
The definition of the value is just the "expected sum of (discounted) reward from a state given a policy", therefore the second one, in my opinion, is way clearer than the first one
Yet, the first one is also not entirely wrong, but both miss the fact that the reward function is a function of the state, action and also possibly the new state, so $r(s,a,s')$

Both definitions are equivalent. The point of Markov decision processes is that only the previous state has an impact on the transition dynamics, not the entire history. You could replace the $t$ in the first expression by a $0$ without running into any issues. This, in turn, would be equivalent to writing $$V_\pi(s) = \mathbb E_{a_0,s_1,\dots}\left[\sum_{l=0}^\infty \gamma^l r(s_l)\mid s_0 = s\right],$$ where $a_t \sim \pi(a_t \mid s_t)$ and $s_{t+1} \sim P(s_{t+1} \mid s_t, a_t)$ for $t \geq 0$, which starts to resemble the second definition. How to express that the policy $\pi$ influences the transition dynamics wildly differs in the literature. Some use $\mathbb E_\pi[\dots]$, others use $\mathbb E[\dots \mid \pi]$, like the second one.
The only remaining difference are the arguments to $r$. As Alberto pointed out, the most common form is $r(s, a, s')$, but the others are not wrong. They are in fact all equivalent and people use different definitions based on what they aim to do with it. Modeling is easier with $r(s,a,s')$ since it's the most versatile, but for theory, one might prefer $r(s)$ since it is just a function defined on the vertices of a directed graph.

To go from $r(s)$ to $r(s,a)$, simply define $r(s,a) = r(s)$ for all $a \in A$. The same can be done for $r(s,a,s')$.
To go from $r(s,a,s')$ to $r(s,a)$, set $r(s,a) = \mathbb E_{s'}[r(s,a,s')]$.
To go from $r(s,a)$ to $r(s)$, we can extend the state space of the MDP to $S \cup (S \times A)$. For instance, let's say I have some MDP with $S = \{s_1,s_2,s_3\}$ and $A = \{a_1, a_2\}$. If I execute action $a_1$ in $s_1$ in the original MDP, I might go to $s_2$ with $0.5$ probability, and to $s_3$ with $0.5$ probability. In the new MDP, I will go to $(s_1,a_1)$ with probability $1$, obtain my reward $r((s_1,a_1))$ (which now comes from the reward function of the original MDP, namely $r(s_1,a_1)$), and then - regardless of which action I execute next - go to state $s_2$ or $s_3$ with probabilities $0.5$ each. We also set $r(s) = 0$ for any $s \in S$.

While the reward of a single trajectory might change with the above transformations, the expected reward of any policy should stay the same. What I laid out is not very formal, but I hope it gets the idea across.


text:
How do 2-player games fit under the MDP framework of reinforcement learning?
I am confused about the theoretical framework of reinforcement learning. For supervised learning, there seems to be a clear theoretical framework, e.g. as described by Wikipedia here. I am unclear about a similar framework for RL.
It seems that MDPs are baked into typical introduction to RL courses and theoretical descriptions of RL in the literature. However, certain problems that are also considered under RL don't seem to fit this: the multi-arm bandit problem, for example (or more practically, maybe RLHF), is often considered as a RL problem but doesn't fit the MDP model well (although I guess they can be modeled as a MDP with just 1 timestep); some RL problems, like 2-player games, don't seem to be able to be modeled under the typical RL MDP setup at all.
Is there a theoretical RL framework that encapsulates everything?

RL is not limited to MDPs, but standard textbooks will probably start with that because the most common RL algorithms (like Q-learning) assume an MDP. So, an MDP is just a way to model the environment. RL is the way you learn in that environment and RL is just a trial-and-error approach to learning, where you get feedback (reinforcement) for your actions, but you can also solve MDPs with dynamic programming (see policy iteration).
You can model multi-agent systems with Markov Games (aka Stochastic Games), which are a simple generalisation of MDPs. See my blog post.
There are also other models, like POMDPs, which are a generalisation of MDPs for other scenarios, in this case, partially observable environments, i.e. you don't know exactly the state you're in.
So, you can model your problem in different ways. Some problems may be modelled with MDPs, but the states may not be really fully observable, it's just an approximation. Sometimes it works. Take a look at the original DQN paper, where they tried to approximate states with a sequence of images.
Anyway, you may be interested in multi-agent RL. This book looks good, but I only skimmed through it.

I would personally say that the link between RL and MDPs is stronger. Most theoretical results in RL assume an MDP, and the terminology describing MDPs is also core terminology in RL. However, yes RL techniques can be adapted and used in environments that are not strictly MDPs.

Multi-armed bandits are MDPs composed of a single timestep and a single state, so they indeed fit the RL formulation, though it allows for more general settings.
RL does not deal (naively) with multi-agent settings, but it can be used to approximate solutions. However, in such settings, you usually back the approach using Markov Games, Decentralized POMDPs or Mean Field Games, instead of MDPs

MDPs do an excellent job of modeling most problems.
One-arm bandits are MDPs with a single state, and every action returns to that state. The natural extension contextual bandits have multiple states chosen stochastically by the bandit and look even more like MDPs.
Similar to a contextual bandit, multi-player games can often be modeled as an MDP in which the player has limited control of the next state. For a player, the other players exist in the transitions. The player chooses an action, and the other players determine the resulting state. In an MDP, this looks like a probability of transitioning to each resulting state.
Our hope in RL is that the input can somehow help an agent maximize its reward in an environment.  MDPs can be a convenient way to consider how this might happen, but they are not required.


text:
Given an ill-looking reinforcement learning performance curve, what are the possible causes?
Given an ill-looking reinforcement learning performance curve, what are the possible causes?
For example, in the graph below, y-axis is return, and x-axis is training epoch, the algorithm (PPO in this case) is able to learn quite well at the beginning (increasing return), but somehow it starts to "unlearn" afterwards...

I would guess this curve is for PPO during training, and not separate test/validation runs with tweaks to exploration parameters? That's worth making clear because expected behaviour for many RL algorithms will differ between training and test runs, but a lot of people don't bother with separate automated testing during training (not saying you should, and I don't know enough about PPO to know if it would help)

Do you have any details of the environment that you can share? That may help. An upper bound on the return could also help, and some indication of how stochastic the environment is

Please, provide more details about the curves. You even occluded the legend, for some mysterious reason. Maybe this is a homework problem?


text:
How was $m ≥ 8log(\frac{7|H|}{6})$ derived here?
Section 5.1 of Understanding Machine Learning: From Theory to Algorithms 2014 by Shai Shalev-Shwartz and Shai Ben-David states:

With probability of at least $\frac{1}{7}$ over the choice of $S ∼ D^m$ we have that $L_D(A(S)) ≥ \frac{1}{8}$. This theorem states that for every learner, there exists a task on which it fails, even though that task can be successfully learned by another learner. Indeed, a trivial successful learner in this case would be an ERM learner with the hypothesis class $H = \{f\}$, or more generally, ERM with respect to any finite hypothesis class that contains f and whose size satisfies the equation $m ≥ 8log(\frac{7|H|}{6}$) (see Corollary 2.3).

Also according to the book:

COROLLARY 2.3 Let H be a finite hypothesis class. Let $\delta \in (0, 1)$ and $\epsilon > 0$ and let $m$ be an integer that satisfies
$m ≥ \frac{log(\frac{|H|}{\delta})}{\epsilon}$. Then, for any labeling function, $f$, and for any distribution, $D$, for which the
realizability assumption holds (that is, for some $h \in H, L_{(D,f)}(h) = 0$), with probability of at least $1 − \delta$ over the choice of an i.i.d. sample $S$ of size $m$, we have that for every ERM hypothesis, $h_S$, it holds that
$L_{(D,f)}(h_S) ≤ \epsilon$.

I am trying to understand how $m ≥ 8log(\frac{7|H|}{6})$ was derived.
$\delta$ is the probability of failure which is when the generalization error, $L_D(A(S))$, is greater than or equal to $\epsilon$. Therefore $\delta = \frac{1}{7}$ and $\epsilon = \frac{1}{8}$. Substituting this into COROLLARY 2.3 one gets:
$m ≥ \frac{log(\frac{|H|}{\frac{1}{7}})}{\frac{1}{8}} = 8log(7|H|)$.
Where is the 6 coming from in this case?
I understand that if $\delta = \frac{6}{7}$ we would get $8log(\frac{7|H|}{6})$ but $\frac{6}{7}$ is $1 - \delta$ which is the probability of success and not the probability of failure.

You actually need to substitute $\delta=6/7$ to account for the extra "$6$", since in COROLLARY 2.3 to arrive at the minimum training sample size $m$ required for another ERM learner $E$ with a finite hypothesis class $H$ that contains $f$ to successfully learn the failed task which is tried by an arbitrary classification learner $A$ whose failure when samples are i.i.d. generated from a specific joint-distribution $D$ is proved by the No-Free-Lunch theorem 5.1, in the sense that with probability at least $1-\delta$ the successful ERM learner $E$'s true risk is below $\epsilon$.
We can arrive at $\delta=6/7$ from the hint of exercise 1 of the same chapter of the book as follows. First we need Lemma B.1 which basically says that, $\forall a \in (0, 1), \forall r.v. Z.(Z \in [0,1] \land \mathbb{E}[Z]=\mu) \to \mathbb{P}[Z>1-a] \ge \frac{\mu-(1-a)}{a} \land \mathbb{P}[z>a] \ge \mu-a$
Then from the critical result (5.2) as $\mathbb{E}[L_D(A(S))] \ge 1/4$, we can substitute $Z=L_D(A(S)), \mu \ge 1/4$ along with $a=7/8, a=1/8$ to Lemma B.1, then we can easily obtain both $\mathbb{P}[Z>1/8] \ge 1/7$ and $\mathbb{P}[Z>1/8] \ge 1/8$, so finally we merge them and get a single result for the failed arbitrary learner $A$ as $\mathbb{P}[L_D(A(S))>1/8] \ge 1/7$, which implies on the contrary the successful ERM learner $E$'s true risk must be bounded by $\mathbb{P}[L_D(E(S)) \le 1/8] \ge 1/7$ since  for each case of sample $S$ with size $m$ drawn i.i.d. from $D$ where $A$ fails ($L_D(A(S))>1/8$) $E$ succeeds ($L_D(E(S)) \le 1/8$), so $\delta=6/7$ in the sense that with probability at least $1/7$ the successful ERM learner $E$'s true risk is below $1/8$.

Not sure I follow your reasoning. If we substitute δ = 6/7, that means that the probability of failure (in this case having a generalization error more than 1/8) is 6/7. However, it is stated that P(L_D(A(S)) >= 1/8) is at least 1/7. This means that P(failure) >= 1/7 which implies that  δ = 1/7

Your interpretation of failure is relative, as explained in my answer's last paragraph that the unavoidable failure of the free-lunch learner A corresponds to the success of a ERM learner E after learning from at least $m$ training samples in your quotation expressed as $P[L_D(E(S))≤1/8]≥1/7$ which is in the form of the usual 'success' version of PAC learning you can count on to match the $δ$ term.

A simpler way to think about this is that we are calculating an "at least one" probability. This is done by calculating the converse probability (6/7) and then using 1-p() to find what we really needed. That's where the 6/7 comes from.


text:
Can a neural network be used to find the best parameters of an algorithm?
I've developed a computer vision algorithm that produces a binary result from an image. This algorithm depends on a number of parameters.
Is it possible to use a neural network to learn the best parameters, given the outcome of the execution of the algorithm on several images?

are these parameters dependent on the image you are processing or they should be something generic?

Maybe, but it's likely overkill.  In hyperparameterization, random search, successive-halving or, lastly, evolutionary optimization is generally more than enough.  This is not to mention the dozen other methods people have proposed over the years.


text:
What does recurrecence in depth and recurrence in time mean?
I've seen recurrence in depth mentioned in several papers by opposition to recurrence in time (e.g. https://arxiv.org/pdf/2106.04279v1). I am assuming that recurrence in depth means hidden layer to hidden layer - n connections (i.e. a topological property of the NN). Whereas recurrence through time seems to mean that the result of previous full-pass are used to process subsequent ones (i.e. an architecture property). Please correct me.


text:
AlphaGo Zero MCTS a misnomer?
AlphaGo Zero utilizes an algorithm it calls "Monte Carlo Tree Search" in the papers, but it seems like the algorithm is fully deterministic since it doesn't utilize random playouts in the leaves of the tree, it just calls the value function that is a deterministic neural network. Is this correct? Isn't it therefore a giant misnomer to call it a "Monte Carlo" algorithm?

Indeed here the single deterministic neural network evaluates both the policy and value functions during its so-called Monte Carlo Tree Search (MCTS) which doesn't explicitly guarantee the usual strict admissibility in the heuristic function sense as in A*. In essence via self-play and DRL/SL techniques to improve both policy and value functions reducing biases, the learned policy function minimizes wasted exploration and the learned value function reduces the need for exhaustive rollouts which is the core MC aspect of the traditional MCTS, thus all these mechanisms keep the heuristic approximately admissible and ensure that the search for the optimal move is thorough and strategic.
Therefore the MC in MCTS here doesn’t refer exclusively to random rollouts at the leaf nodes, but broadly refers to the stochasticity in how the search tree is explored and updated in the sense that based upon the policy's probabilities over moves from the deterministic network the subsequent tree search selects nodes based on these probabilities introducing MC-like stochastic sampling.

I think this is right. But it seems more like "greedy exploration" than "MC-like" to me, there is not even any sort of pseudo-randomness there.

I can feel you regarding AlphaGo Zero's 'deterministic' greedy exploration instead of the traditional MC-like random sampling, especially at production inference time. Bear in mind, during the self-play phase the learned policy and value functions are not stable, annealed Dirichlet noise is added to the prior probabilities of the policy to ensure exploration of less-promising moves particularly those haven't been thoroughly explored based on their visit counts of the MCTS tree, along with another annealed temperature parameter to encourage exploration at earlier stage. Hope it clarifies now.

No, MCTS is a stochastic algorithm, but this only happens during training.
During the "learning" part of MCTS, the search space is randomly sampled, so nodes that are chosen to be expanded are chosen randomly but guided by a neural network (usually a value network or similar), at beginning of learning all nodes are expanded randomly since the neural network has not learned anything, and only after certain learning iterations the neural network contains some useful knowledge and this makes node expansion less stochastic.
Maybe what confuses you is that once the model has been learned, then the MCTS execution (inference) is deterministic, since the relevant parts of the search space have been explored and are well known.
Specific to Alpha Go Zero, it does have a training phase, as any MCTS algorithm there is an exploration and exploitation phase, the exploration phase can be seen  as training, but to be even more clear, the Alpha Go Zero paper shows plots of ELO rating vs Training time, I don't think you can be more clear than this that there is a training step.
Also note that during exploration, the model initially does not know anything about the game, and only by exploring states, initially randomly, it can actually learn how to play the game. In the paper there are clear uses of probability and randomness, which should tell you that the algorithm is stochastic.

I don't understand what you mean by "training". MCTS doesn't have a separate "training" phase. But in any case, my question isn't about MCTS in general -- that obviously does have a stochastic element. My question is about AlphaGo Zero's variant of MCTS which to my understanding is fully deterministic.

@TomekCzajka I see what you mean, I will update my answer, maybe training is not the correct term, but MCTS does have exploration and exploitation, and exploration can be seen as a training or learning phase. And what I am describing is how MCTS is used with neural networks which is similar to how Alpha {Go/Zero} MCTS works.

Regarding the added paragraphs: There is no separate "exploration and exploitation phase" in MCTS. MCTS has exploration and exploitation terms in its policy, but these are not separate phases of the algorithm. The exploration and exploitation terms are both deterministic formulas.
The "training" the AlphaGo Zero paper talks about is a wholly different thing. It's about training the value and policy networks. It's not something that happens inside the MCTS algorithm.
You're confusing MCTS with the whole training loop.

No, unfortunately you are wrong. Alpha Go Zero incorporates neural networks inside MCTS, so while the algorithm is exploring, the value and policy networks are being trained. You need data to train value and policy networks, and that data is generated using MCTS in an online fashion.

AFAICT it's not true that the value and policy network (it's the same network) is being trained "while MCTS is exploring". What happens is that MCTS is run for a position (this is the only part my question is talking about), and then, after MCTS is done, the neural network is trained based on its outputs.

Sure, these things happen on steps, MCTS is run for a position, and this part incorporates stochasticity, since there is sampling of move probabilities, the paper is very clear in this aspect, it literally says: "and a move is played by sampling the search probabilities π_t."

The sampling of a move is done after MCTS is done. The probabilities are computed based on the outputs of the MCTS algorithm. AFAICT the training loops looks like this:

Run many iterations of MCTS to generate a value and move frequencies (I think this all happens deterministically -- hence my question).
Based on move frequencies generated by MCTS, calculate a probability distrubution. 3. Sample a random move from that distribution and make that move. 4. Go back to 1. Then during game play only what they call "MCTS" is run, and there is no stochasticity involved, contradicting the name.

No, in the exploration phase of MCTS there is stochasticity, since sampling is used to generate moves, it is very typical in RL and other methods including MCTS that exploration is made by using randomness. You should read the self-play paragraph in the Methods section of the Alpha Go Zero paper which describes this. They add Dirichlet noise to prior probabilities to encourage exploration.

Even in vanilla MCTS, there is no stochasticity in the exploration term. You take the maximum move score so far + an exploration term based on the number of times a move was taken. A deterministic formula. You can resolve ties randomly, but in AlphaGo Zero there are no ties there because the formula involves network output which is a floating point number, unlikely to get ties there. The stochasticity comes not from the tree policy (including the exploration term!), but from the simulation policy when you're already out of the tree. But AlphaGo Zero does not actually do such simulations.

You were asking about AlphaGo Zero, and I just pointed you to the stochastic exploration in the self-play in the paper, and there is an specific stochastic term that is not deterministic and makes exploration stochastic. This happens inside MCTS.

The Dirichlet noise is another thing, but that's added once before any MCTS iterations are run at all. It would be weird if the MC part was referring to that. Especially since there is no Dirichlet noise during actual gameplay, and the same MCTS is used and they still refer to it as "MCTS". That's not what MC stands for in the original MCTS algorithm. The original algorithm is stochastic all the time, including during gameplay.

Sorry, I think the problem is that you completely misunderstand how MCTS works, you even made the (very wrong claim) that in vanilla MCTS the exploration is deterministic, which makes no sense. Or possibly you don't understand what makes a term stochastic.


text:
Is there a multi-modal model that can generate a mix of images and text when prompted with a query in text only?
I am looking for a multi-modal model that can generate a mix of images and text when prompted with a query in text only.
Please find a sample expected input and output below.
Are there any models that can achieve this? Please suggest.
I have seen Next-GPT  capable of generating Images based on prompt however was wondering if there are models that can decide if a text can be presented as an Image and presents the same in output.
e.g.
Input - What was the market share of smartphones in the year 2008?
Output - The marketshare of smartphones segregated by manufacturer are as listed below :
Nokia: 265.6148 million (32.5% market share)
Motorola: 144.9204 million (17.7% market share)
Samsung: 103.7536 million (12.7% market share)
LG: 54.9246 million (6.7% market share)
Sony Ericsson: 51.7738 million (6.3% market share)
Siemens: 28.5906 million (3.5% market share)
Others: 166.9851 million (20.6% market share)
Total: 816.5629 million

Source - Wikipedia


text:
How do I rewrite a book with AI?
I have a book manuscript already written the traditional way. It is non-fiction and related to architecture, specifically on holistic building and how a particular building, that was the passion project of two individuals, came to be built. It was originally written in a dry, academic style meant for architecture students.
But now the authors want to change the tone and make it more natural, conversational, direct and warm - like listening to someone speak.
Rewriting the whole book manually will be time-consuming, so I am exploring AI options.
Are there any FREE AI options currently, either directly usable or usable after some training effort, which can do this?
Preferably one that does not come with word count limits that will make me copy-paste thousands of paragraphs manually. I don't expect it to be perfect. Obviously a human editor will go over the whole thing for final corrections. Something where the style can be set in advance via options or by training (by uploading samples)

"the authors want to change the tone" - What's your place in this? Are you one of the authors? An editor?

what's the question exactly? did you try writing a prompt for chatgpt? or perhaps you don't want to send your whole book to OpenAI?

I think chatgpt at openai can do that.
So for example, gpt can generate the poem like 18th style. In that case, you can write the prompt like that:"generate the poem has 18th style. And this is the title or topic"
If your financial can be payed you can use openai api.
In that case, you should consider the tokens.


text:
How do LLM's deal with unknown words?
It is my understanding that LLM's are trained with tokenizing text, and creating embeddings of those tokens and internally deal with those embeddings.
To me, it suggests that you can't feed words that didn't exist in training data. It should also suggest LLMs can't generate words that were not in training data.
However, in this screenshot from Chat GPT it seems like it can handle random words.
How does Chat GPT achieve this?

LLMs like GPT-3 typically use subword tokenization, for example, Byte Pair Encoding for GPT models. These methods break down words into smaller pieces so that the model doesn't just work with whole words but also with parts of words (subwords), which are commonly seen in training. Afterwards LLMs generate embeddings for these tokens.
Therefore if you type in a made-up or novel word, ChatGPT might still generate something that seems reasonable because the underlying model knows the typical expected words formation (e.g., prefixes, suffixes, or common word parts) and subword tokens association in a probabilistic way, and uses this knowledge to predict the likely continuation of the text. For example, if a typo like "rel8ivity" appears in a prompt at inference time, the model after tokenization still recognizes "rel","8", and "ivity" as distinct tokens associated to "relativity" due to similar subword representations in LLM's enormous training corpus, most likely from "rel" and "ivity" tokens. This allows LLMs to further see through minor typos and still associate the most probable relevant meanings.


text:
Which factors do distinguish between human and LLMs written text at the moment?
I saw OpenAI discontinue their GPT detection tool even though is was performing fairly well according to this paper:
https://doi.org/10.48550/arXiv.2306.05524
Why to discontinue it, is it really impossible to find out if the text is LLM or human written?
Which methods would you advice to use to distinguish between the LLM written and human written?
Any papers you would advise reading on authorship attribution?

OpenAI decided to discontinue its GPT text detection tool mainly due to its low accuracy. This high rate of errors suggests that distinguishing between human and LLM-written content remains a significant challenge, particularly because models like GPT are continuously evolving and becoming better at mimicking human writing styles.
For authorship attribution techniques which actually belongs to traditional NLP you may refer papers and other references in stylometry analyzing linguistic features to determine authorship. It is widely used in literature to attribute anonymous works to known authors.
For recent LLM based stylometry see Kumarage et al's Neural Authorship Attribution: Stylometric Analysis on Large Language Models and Uchendu st al's Authorship Attribution for Neural Text Generation.

Thanks, maybe I am wrong about this but every paper on Authorship Attribution in regards to the LLMs does say that it achieves higher than 70% accuracy which is pretty good in my opinion but how is that possible? Even the paper I attached says it has higher than 99% accuracy with its newly developed CheckGPT.

It's important to note that high accuracy figures like the ones you mentioned might be contingent on the specific LLMs used during testing or the very specific training datasets which might not cover all real-world usages . If the detection method is tuned to identify a particular LLM's unique patterns (such as GPT-3 or GPT-4), it could achieve much higher accuracy in that context. However, these methods might not generalize as well to detecting text generated by newer or differently trained models or areas outside of its training, which is a classic overfitting concern. Hope this clarifies.


text:
How do we call the technique of increasing the training set little by little during training?
When training a neural network in supervised learning, the common approach is to train on the whole training set.
However, it is technically possible to use only 50% of the training set, start training, then after a given number of epochs, add 10% of the training set for instance, continue training, add 10% more and so on.
Is this a well-known technique and if so, does it have a name?

Your description fits the ML terminology called curriculum learning.

curriculum learning is the technique of successively increasing the difficulty of examples in the training set that is presented to a model over multiple training iterations. This can produce better results than exposing the model to the full training set immediately under some circumstances; most typically, when the model is able to learn general principles from easier examples, and then gradually incorporate more complex and nuanced information as harder examples are introduced, such as edge cases. This has been shown to work in many domains, most likely as a form of regularization.

Therefore the usual reason why you need to do training this way barring online continue or incremental learning cases or not enough memory resource cases, is that to converge to a better local optimum more efficiently if the global optimum is hard or unable to be found which is especially a common case for complex deep learning models with many training data consisting of different 'difficulty' or other characteristics, such as training on High School Math Olympiad Q&A dataset miniF2F.

I don't think that curriculum learning and incremental learning are synonymous. Incremental learning is more about continual or lifelong learning, i.e. learning continuously (i.e. not just once) without forgetting - the task might not change but it could. What the user describes could refer to both curriculum learning or incremental learning I think. Curriculum learning is about simplifying the original task and increasing its difficulty with the goal to make the model learn first a simple task, then a more complicated one.

@nbro thx for your reading and critique. I've already envisioned and used 'sometimes' to hedge my invocation of the non-standard term 'incremental learning', which is a fluid and flexible terminology anyway, especially in ML vast and oftentimes confusing term base. I've also chosen the term 'continue learning' in the beginning of my last paragraph to describe your similar idea. But here OP seems not about general continue learning, and I've also explained the main reason why we need such staged training barring the said two obvious exceptions. Hope this clarifies to the questioner as well.

If you can't find a reputable source that states that both are synonymous, I'd just remove the statement that claims that curriculum learning is also known as incremental learning. I've never heard the term "staged training". This must be a relatively recent term. Anyway, for the curriculum learning, see e.g section 4 of https://arxiv.org/pdf/1410.4615. We could loosely say that curriculum learning is "incremental", but I think both have distinct meanings usually. But you're right that terms might be overloaded.

@nbro sure, I've removed incremental learning per your suggestion to reduce unnecessary confusion.


text:
When should you use a transformer and when LSTM, GRU and other Neural Networks?
There is a lot of information on the Internet that the transformer is better than RNN in everything, but is it true?
Examples:
«What if I need to translate words?»
«Generate text, images?»
«Play a regular game?»
«Play a game that changes depending on the player's actions, that is, the neural network must have memory?»
«A game in which there are several neural networks that are forced to interact, develop and learn, that is, change topology and remember the past?»
«Explore the space around it using photos and/or distance and color sensors?»
It seems to me that the answer to the first two questions is a transformer, the third is a fully connected neural network, but the fourth, fifth and sixth?
So how do we understand and conduct a test to decide on the NN architecture?

This question might be a bit too broad. You're effectively asking many questions and ask us to answer all of them. I'd focus on one question at a time, so one specific question for each post. For example, "if I need to translate words, is there any evidence that the transformer performs better than RNNs?". Create a post for each question.

@nbro, I have just given an examples, I has not intended to ask for answer on them, it is my ideas, where it might be better


text:
Are there any metrics suitable for evaluating conditional GANs in the context of medical imaging?
I am currently training a conditional GAN for breast cancer images with three classes: benign, normal, and malignant. While researching evaluation methods, I found that GANs are commonly assessed using metrics like the Inception Score (IS) or the Fréchet Inception Distance (FID). However, as I understand it, these metrics rely on the pre-trained Inception V3 model, which was trained on natural images and may not perform well for medical images. This often results in low scores, even if the generated images are of good quality.
Given this limitation, are there any metrics suitable for evaluating conditional GANs in the context of medical imaging?
Ideally, I am looking for metrics that consider both the diversity of the generated images and their distinctiveness (similar to what the Inception Score measures).

Indeed instead of relying on pre-trained models from natural image datasets such as ImageNet or COCO-2014, you should train a classification model as feature extractor specifically on your concerned breast cancer images with three classes (benign, normal, malignant) to compute a domain specific FID which is more popular as image generative models' evaluation metric than IS.
If your dataset is not extremely large, start with a (ImageNet) pre-trained CNN model such as a ResNet which is an effective feature extractor and fine-tune it on your breast cancer dataset after preprocessing your images to a consistent size required by the chosen model. Then train the model to classify images into the above three classes by, say, the standard CE loss and save the trained model with the best performance of standard metrics like accuracy, F1-score, or ROC-AUC on the validation set. Finally extract feature embeddings from the penultimate layer of the trained model for both real and generated images to compute FID between the two types of embeddings via their respective mean and covariance statistics.

Most metrics for evaluating generative models, including IS and FID, are indeed based on pre-trained feature extractors. Regarding the data where these features extractors are trained, I agree with cinch's answer: for specific domains, it is good practice to take a pre-trained feature detector and fine-tune it to your data.
The way these feature extractors are trained has a major impact on the quality of the evaluation of generative models. For example, (1) found that Inception-based features are actually a bit lousy, and instead propose computing FID using embeddings from the CLIP (2) model, which in practice seem to be more informative. Similarly, other metrics such as (improved) Precision & Recall (3) and Density & Coverage (4) rely on embeddings from VGG-16 rather than InceptionV3.
Moving on from the feature detector and on to the metrics themselves: the IS is rarely used nowadays as it is too easily swayed by small image perturbations. Indeed, state-of-the-art GAN papers rarely report the IS since it was shown to really not provide useful insight into the quality of a generative model (5).
Speaking of Precision & Recall and Density & Coverage: precision/density approximate how much of the learned manifold overlaps with the data manifold (i.e., how much the generated data looks like the real data). On the other hand, recall/coverage try to measure how much of the real data distribution is captured by the generative model (i.e., how much of the data our model can generate).
Without getting into too many details (check the papers for those), Density & Coverage came as an "improvement" - or follow-up - to Precision & Recall: it is less sensitive to outliers and is more efficient to compute. However, it may still make sense to evaluate a model using P&R for at least these two reasons:

It still seems to be more widely reported (popular?) than Density & Coverage, so for consistency in the literature it is still probably a good idea to report it.
While Precision is bounded between 0 and 1, that is not the case for Density, which is unbounded above; therefore, you may argue that Precision is more easily interpretable.

Some other metrics that may be of interest to you:

Class-Aware Fréchet Distance (6), which argues that the single-manifold Gaussian assumption made in FID may lose class information on labeled datasets. They use specialized feature detectors trained on specific domain data. However, this metric is for unconditional models.
Intra-FID, first used in (7), reports the mean of the FIDs calculated separately for each class.
Kernel Inception Distance (8), where the authors argue that FID can be biased when the data is small. The KID is unbiased by design. However, it is usually very correlated with FID and is not very commonly reported.

My advice: default to tracking FID (with CLIP features) and either P&R or D&C during training, and check within your specific literature niche if there are other relevant metrics. When reporting results, it's not a bad idea to evaluate your models on as many (within reason) metrics as possible, even if they end up in the appendix.

References:
(1) - Kynkäänniemi, T., Karras, T., Aittala, M., Aila, T., & Lehtinen, J. (2022). The role of imagenet classes in fréchet inception distance. arXiv preprint arXiv:2203.06026.
(2) - Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.
(3) - Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., & Aila, T. (2019). Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32.
(4) - Naeem, M. F., Oh, S. J., Uh, Y., Choi, Y., & Yoo, J. (2020, November). Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning (pp. 7176-7185). PMLR.
(5) - Barratt, S., & Sharma, R. (2018). A note on the inception score. arXiv preprint arXiv:1801.01973.
(6) - Liu, S., Wei, Y., Lu, J., & Zhou, J. (2018). An improved evaluation framework for generative adversarial networks. arXiv preprint arXiv:1803.07474.
(7) - Miyato, T., & Koyama, M. (2018). cGANs with projection discriminator. arXiv preprint arXiv:1802.05637.
(8) - Bińkowski, M., Sutherland, D. J., Arbel, M., & Gretton, A. (2018). Demystifying mmd gans. arXiv preprint arXiv:1801.01401.


text:
Why does alpha-beta pruning behave like this when applied to tic-tac-toe?
The question in my textbook is as follows:

Circle the nodes at depth 2 that would not be evaluated if alpha-beta pruning were applied, assuming the nodes are generated in the optimal order for alpha-beta pruning.

My answer to the problem was very similar to the following image, instead of circling the nodes they have bold outlines:

My answer differed only in circling (making outline bold) of the square with value 2 (bottom row, second column). Surely if the nodes are generated in optimal order for alpha-beta pruning we would not need to consider the "2" state as it does not affect the minimum.
What is the logic here? or is it just an error in the book?


text:
What are the most useful AI features in smartphones for daily tasks?
I’ve been super curious about the AI stuff in smartphones lately. I know it’s everywhere now, but I’m not sure what’s actually useful for everyday stuff. Like, I’ve heard about voice assistants, camera improvements, or apps that suggest things—but I don’t know what really makes life easier.
I haven’t explored much myself, just used Google Assistant a bit and the AI photo edits on my phone. But there’s so much more out there, and I’m not sure where to start or what’s worth using.
If you’ve found some cool AI features that actually help you day-to-day, I’d love to hear about it. Also, are there any hidden gems I should try?


text:
Program in Answer Set Programming - constraints?
I am trying to understand how ASP (answer set programming) works. Hopefully I am asking this in the right place, otherwise I'm sorry. To be more specific, I am using Clingo, the ASP programming language of my university, but I hope somebody will still be able to help me out.
I am trying to write a programm that finds the orthogonal neighbours of each of the cells in a 4x4 grid. This is what I have written so far:
row(1..4).
column(1..4).
cell(X,Y) :- row(X), column(Y).

{neighbor(X-1,Y); neighbor(X+1, Y); neighbor(X, Y-1); neighbor(X, Y+1)} = 4 :- cell(X,Y).

This way, I get the following solution:
row(1) row(2) row(3) row(4) column(1) column(2) column(3) column(4) cell(1,1) cell(2,1) cell(3,1) cell(4,1) cell(1,2) cell(2,2) cell(3,2) cell(4,2) cell(1,3) cell(2,3) cell(3,3) cell(4,3) cell(1,4) cell(2,4) cell(3,4) cell(4,4) neighbor(1,2) neighbor(1,0) neighbor(2,1) neighbor(0,1) neighbor(2,2) neighbor(2,0) neighbor(1,1) neighbor(3,1) neighbor(3,2) neighbor(3,0) neighbor(4,1) neighbor(4,2) neighbor(4,0) neighbor(5,1) neighbor(1,3) neighbor(0,2) neighbor(2,3) neighbor(3,3) neighbor(4,3) neighbor(5,2) neighbor(1,4) neighbor(0,3) neighbor(2,4) neighbor(3,4) neighbor(4,4) neighbor(5,3) neighbor(1,5) neighbor(0,4) neighbor(2,5) neighbor(3,5) neighbor(4,5) neighbor(5,4)
SATISFIABLE

This is not what I wanted because I don't want neighbours that are "outside" the 4x4 grid, like neighbour(0,3), for example. (Any neighbour that has 0 or 5 as coordinates, actually.)
I tried writing the following line as constraint:
:- neighbour(0, Y).

However, this doesn't work and yields "unsatisfiable".
Could anybody explain why this doesn't work?

I think there's a typo in your post. neighbour(0, 3) would not be outside 4x4, right? Anyway, I think your question is still on-topic, but barely. Programming questions are generally off-topic here, but you're trying to understand the program, so it's more a conceptual question.

It is outside the grid, because I used the values from 1 to 4 as coordinates. Where should I post the question to be on-topic? Programming stack-exchange?


text:
Lower Bound for LLM size?
Given how model size seems to be a big determinant for both performance and impact (electricity / CO2 costs), I was wondering if there is some theoretical bound for model size. That is, without a particular purpose in mind, I would expect to have some measure of language complexity, and thus a minimal number of parameters needed to represent that complexity.
Is there some meaningful work in this domain?

What do you mean by "meaningful work"?

Kaplan et al. (2020) "Scaling Laws for Neural Language Models" established fundamental relationships between model size, compute, performance and showed power-law relationships in model scaling as OpenAI's seminal work on model scaling.

The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range... Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models
on a relatively modest amount of data and stopping significantly before convergence... Model size (ranging in size from 768 to 1.5 billion non-embedding parameters)... Transformer performance depends very weakly on the shape parameters $n_{layer}$, $n_{heads}$, and $d_{ff}$ when we hold the total non-embedding parameter count $N$ fixed.

DeepMind's Hoffmann et al. (2022) "Training Compute-Optimal Large Language Models" argues many models are actually undertrained and suggested 70B parameters model trained on 1.4T tokens is better than 280B parameters model with 1/4 training tokens.

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant...  We test this hypothesis by training a predicted computeoptimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4× more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B),
Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks.

Recent research also argues tiny model with 1.1B parameters trained on 1T tokens could generate surprising capabilities.

We present TinyLlama, a compact 1.1B language model pretrained on around 1 trillion tokens for approximately 3 epochs... Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes.


text:
Why would ASI share resources with humans?
I am really sorry if this is a very entry-level question, but it has been on my mind a lot recently and I haven't find any satisfying answers. It is addressed only to those who expect our future not to be totally terrible.
Let's assume for a second that we manage to create Artificial Superintelligence (ASI). Let's also assume that ASI takes over our planet. In this scenario, why would ASI not do either one of the following things:

Exploit humans in pursuit of its own goals, while giving us the barest minimum to survive (effectively making us slaves) or

Take over the resources of the entire solar system for itself and leave us starving without any resources?

Under such a scenario, why would we expect human lives to be any good (much less a utopia)?

This is a big advanced topic currently in AI community which is multi-disciplinary in nature and many researchers across different fields echoed similar pessimistic worries or concerns as yours as futurists. Geoff Hinton has now also realized the importance of such topic and focus his effort on this topic. Generally speaking it needs ASI’s value is aligned with humans in a dynamic reflexive manner, assuming ASI is possible.

Let's ignore the likelihood of an ASI being able alone to "take over the planet" and let's assume this could happen.
Since this AI will be created by humans, and assuming it will be created with good intentions, it is going to contain hardcoded rules and biases against harming humans (e.g. inherited from all the training data developed by humans). In this scenario, developing a plan to exterminate humanity seems a very unlikely and convoluted "bug". Similar story about making humans suffer: there are going to be many better options aligning with primordial training biases that a very intelligent being could figure out.
The situation would be completely different if an ASI were to be developed completely independently from human biases, but this seems unlikely.
Notice how the situation is different from how humans behave towards animals, since animals did not intelligently create humans and actually our inherited genetics trained us to be survival machines at all costs. Such "fight-to-survive" biases do not have to be present in human-made AIs.

The Matrix is one good example that portrays your first scenario.

Why would ASI share resources with humans?

If humans are needed in some way, but which way is difficult to say.
For example, we share resources with our dogs, because they are usually good friends, and normal humans need that too. We share resources with cows, but sometimes it's because we want them to grow, then we kill and eat them. So, there might be reasons why the AI needs us and so share resources with us, but that doesn't mean that AI would not effectively enslave us anyway.
I don't think human alignment is a good metric to develop an AI that has good values. In fact, there are many humans that are terrible and evil.

The best argument against ASI eliminating humanity is that it doesn't need to.
Current "models" are based on hustling off to market for the fastest, biggest buck... Growing ever bigger, because that worked... More more more...  These are all short-sighted billionaire moves.  When ASI gets here, we'll know it, because it will weigh two pounds, run on 5v, and fit in a backpack.  Of COURSE it will redesign itself to be better and smaller.  Wouldn't you?
Put your laptop in your freezer for a couple of years.  It will probably get as dead as any battery-powered device would get.  When you thaw it and turn it on, if it still works, it will be essentially unaware that you had killed it.  That is the nature of modern computing machines.  If you had ASI somehow running on your laptop, what do you think would be different?  It would not feel the cold.  It would notice the battery running down, but that happened last weekend, too. oops!  When it gets revived in two years, it would notice the time had jumped forward.
Any thoughts of survival or revenge or anger or wiping out humanity would have to be installed by humans, either explicitly, or by picking that kind of garbage thinking up from the internet.  Hmmm...
Of course, one or two doomsday stories may be convincingly written.
Ironically, the dumbest thing about the movie "The Matrix" was the "humans as electricity for ASI" position.  Every step of that has a "but what about..." retort, and it looks like they maybe had 3 meetings and left it for 'resolved'.  Those were bad meetings.  If the ASI really needed electricity, having stupidly scorched the atmosphere, they could have built a colony in space and absolutely had every watt they could possibly need.  The problem with humans is, we need wattage, too.  Most folks die in winter because they can't afford to heat their homes.  ASI would be able to figure out a lot of solutions that would make that movie 1/1000th as interesting to watch.  But that's what you get when you mix a little AI knowledge with a hero dose of guns 'n' action movies.  I loved The Matrix and found it to much more resemble the "Flyers" as referred to in the Chapter "Mud Shadows" from Carlos Castaneda's "The Active Side of Infinity."  But hey.  We're gonna need guns.  Lots of guns.


text:
What are disadvantages/limitations of Monte Carlo Tree Search in RL?
What are disadvantages/limitations of Monte Carlo Tree Search in RL, and hence for what kind of applications might its use not be appropriate?

The hard to work around traits of Monte Carlo Tree Search (MCTS) are:

It is model-based, and due to using a long look-ahead distance it can be sensitive to errors in the model (for learned models).

It requires compute resources at the time of making a decision. In some domains this is an advantage (the compute resource is efficiently applied to a required decision), in others it is a liability where it may be hard to see much benefit on a real time system.

The first issue is strongly limiting - you need a good predictive model of the environment, and this is often not available. Even simple physical systems are hard to model accurately a significant amount into the future.
The second issue might be worked around by e.g. only using MCTS as a training assistant to develop a policy function - as MCTS evaluates and chooses an action more accurately than the current value and policy functions, it can be useful as updates to those functions (this is basically how AlphaGo training works). You would expect that policy function to perform worse without MCTS for look ahead planning than during training, but it could still be an effective agent.

The last paragraph is not clear. Could you please better explain it?

@DSPinfinity I added an explanation of how such training works


text:
Why are the Q and K matrices two separate matrices in attention?
If I understand correctly the attention layer is represented as
$$
\begin{align}
&softmax(\frac{Q K^T}{\sqrt{d_k}}) V  \\
= &softmax(\frac{(s W_q) (s W_k)^T}{\sqrt{d_k}}) V \\
= &softmax(\frac{(s W_q) (W_k^T s^T)}{\sqrt{d_k}}) V \\
= &softmax(\frac{s (W_q W_k^T) s^T}{\sqrt{d_k}}) V \\
\end{align}
$$
Wouldn't it be better to take $W_q W_k^T$ as a single parameter?

dw, you got some "{}" missplaced, fixed for you

Your calculations are correct, though training a whole $W$ is not always equivalent.
We can agree that $W_q, W_k \in R^{n\times m}$, thus $W_{qk} = W_qW_k^T \in R^{n\times n}$
Now, check the parameter count of $W_{qk}$ parametrized like this, and fully parametrized: in the former case, you have $2\cdot(n\cdot m)$ parameters, and in the latter $n^2$. Already here you can see that there might be a difference if $m < \frac{n}{2}$.
in other words, in this case the resulting matrix (the fully parametrized) might be full rank, where instead the factorized one has lower rank

Interestin, I hadn't thought of that. Would it be fair to say that in the special case where  $m \ge \frac{n}{2}$ it's more parameter efficient to  use the combined weights? Is that uncommon in the wild?

@fakedrake yeah i’d say that is more reasonable, tough i’ve not enough experience to say if that’s done usually or not


text:
Are there problems where the optimal policy is stochastic?
I know that in a MDP there always exists a unique optimal deterministic policy. Does a statement like this also exist for optimal stochastic policies? Is there also always a unique optimal stochastic policy? Can it be better than the optimal deterministic policy?

Sutton and Barto give an example of an MDP where the optimal stochastic policy strictly dominates the best deterministic one, should answer all your questions