text:
Does changing deep learning model loss function affect the occurence of plateau?
If we use two loss functions that are in the same "units" (say MAE and RMSE). Should training the same model with these two different loss functions result in plateau happening at a different place?
Say in this particular case our dataset does not have many outliers so there's not much penalizing the outliers.

If you train the same model with two different loss functions, such as MAE and RMSE, the model may reach a plateau at different places for each loss function. This is because these loss functions have different sensitivities to errors and are optimized differently during training.
In your specific case, where there are not many outliers in the dataset, the differences in the training dynamics between MAE and RMSE might be less pronounced. However, it is still possible for the models to reach a plateau at different points, because the two loss functions handle errors differently. The model with RMSE may converge slightly later than the one with MAE because it focuses more on reducing larger errors.


text:
Can models like chatGPT learn functions with infinite domain or range
Lets assume two types of prompt:
A fixed prompt for which reasonable responses can be infinite. For example:

> output a random number
> output a palindrome

A prompt that can slightly varied to get different output each time which reasonable mapping. Quotes mark what will be varied.
> what is the double of "2"
> echo "xyz"

Now that the network is trained on finite data and has finite parameters, does it inherently   prevent it from perfectly answering these questions each time.
If yes, can a tiny network be handcrafted to model an infinite domain function and its only the "learning from data" part that prevents it from learning such functions.

ChatGPT cannot even count the words in sentence.

It depends on the design of the model, of course.
The overall design of a GPT model is like this: the last certain number of tokens (e.g. 2048) are passed into a neural network which outputs probabilities for the next token. Then a next token is chosen according to the probabilities. This repeats until something. So it can't see more than a certain number of tokens back - and that's the fault of how it's designed. If you ask it to generate a longer palindrome, it will eventually forget the beginning of the palindrome and also forget the instruction to generate a palindrome. However, if you ask it to generate random numbers, as long as the inputs are random numbers it will probably keep generating more random numbers.
Other kinds of models exist, like RNNs, which process one token at a time and remember some data between tokens. Theoretically, an RNN could keep passing information down the chain indefinitely. In practice, they do not. Also, nobody has made one work as well as GPT.
Transformer models, the precursor to GPT, worked like GPT but they did have separate prompt and output strings. The input is the last certain number of output tokens, plus all the input tokens. This kind of model would not forget the instruction to generate a palindrome, but it would forget the beginning of the palindrome after a while.

An encoder-decoder transformer isn't a precursor to GPT; it is for machine translation, whilst GPT, the transformer decoder, is for causal language modelling. Also, there is nothing in the design of transformers or GPT that fixes the input size (which is how recent GPT models are able to have very long contexts); they are as recurrent as an RNN, except with better long-range dependencies since there is no bottleneck that moves through the context.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why do we not calculate value for the done state in PPO?
In PPO a common pattern I see in calcualting advantages is:
$$ delta = reward[t] + (gamma * valueNewState[t] * done[t]) - valueOldState[t]$$
Such as in this article. I am wondering why we multiply by done[t] indicating for the last step in the epoch we take reward only and not the value of the current state.

done[t] indicates whether the state at timestep t was terminal or not.
It is used to say when or not to bootstrap (i.e. estimate) the value V(s) of a state: we bootstrap when done[t] = False, providing an estimate for the future discounted reward that the agent should get in expectation (this is also useful when the episode is artificially truncated but not really terminated.) If true, the state at time t is terminal and so its value V(state[t]) should be zero, because the agent landed in a terminal state that "absorbs" the future rewards.

This question and corresponding answers (https://ai.stackexchange.com/q/17890/37607) add some detail to your last sentence.

Thank you both this clears it up for me :)

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How can I limit the number of things Yolov7 can Identify?
So Yolov7/v8 are able to classify numerous distinct objects. One of those objects is cups for example. If I wanted to only look of cups how would I change my setup without having to do custom object identification?

It's straightforward in theory, although in practice might depend on your codebase. There is a classification head or network that takes the bounding boxes and gives you class probabilities (e.g. see here). What you do is to keep the bounding boxes with high class probability on the class you're interested in. If you want to detect cups, you need to find what's the cups class and then filter according to that class (e.g. pred_cls2 == 5 if 5 is the "cups" index).


text:
Reproducing AlphaZero/MuZero: Failed to beat initial model in arena
I am trying to reproduce AlphaZero's algorithm on the board game Carcassonne. Since I want to use the final game score differences (i.e. victory point of player 1 - victory point of player 2) as the final and only reward, AlphaZero's UCB score can no longer work since it assumed that the game reward is either 1 or -1. Therefore, I am using the trick from MuZero, which, when computing UCB score, normalizes the child value estimate to the range $[0, 1]$ using the minimum and maximum value estimates in the entire tree.
However, after implementing this, my algorithm almost never get past the first iteration, since the win-rate against last iteration is almost never higher than 50% (I am using an arena threshold of 0.55). If I don't normalize the reward, then the win rate seems to be higher.
My hypothesis is that, in the first few MCTS searches, there are not many estimates in the tree, and a few of the node values will be normalized to 0 or 1, which are very extreme, despite their original model-predicted values being very small in magnitude. This may cause the tree to overfit to these values instead of exploring other nodes.

Is this a common issue with self-play, or is it indicative of a bug in my implementation?
What are some common ways to overcome the issue (e.g. increasing data, MCTS iterations, etc)? If the algorithm does not get past the first arena, then it will never progress since it always gets reset to the initial model.

I've tried to organize my answer as best as possible, first answering your specific questions and then giving some higher level thoughts.
Using score as the value
PUCT as used in AlphaZero doesn't actually require that $Q \in [0, 1]$. The term $Q + c_{puct} U$ is only used to compare nodes to each other, so any constant offset is irrelevant. Additionally, scaling Q can be compensated by scaling $c_{puct}$. So you can scale down $Q$ like they did for MuZero, but just tuning $c_{puct}$ should work just as well.
You're trying to use the in-game score as the value instead of just the expected winrate. I'm not sure if this is a good idea, I would start with the winrate: it's constrained to a small range which makes both PUCT and neural network training simpler and more stable.
Debugging AlphaZero
AlphaZero has a lot of moving parts, which can make it hard to debug. How I approach debugging AlphaZero:

See if tree search can find the right moves in some states that are very close to the end of a game, ensuring that MCTS is implemented correctly.

you can already tell if the value scale is a big problem here

Generate a lot of randomly played games (1 million?) and train your NN value head on those. This should already provide some useful signal.

See if the value head predicts the right values in positions that are completely winning for one side.
Run some games using MCTS with this NN, it should already be better than a randomly initialized one.

Only if all of this works start running the full selfplay <-> train loop.

Look closely at plots of the train/test loss, average game length, ... to figure out if anything interesting is happening.
Inspect some games to see if the quality of play is increasing.

Stochastic games
Carcassonne is a stochastic game, you don't know what tile you're going to draw in the next turn. This is a big obstacle when doing any tree search, how are you handling this right now? Typically this is represented using chance nodes, for example in the Stochastic MuZero paper. Unfortunately for Carcassonne the branching factor of these nodes is very high: you can draw every remaining tile, so up to 84. This seems like a big obstacle to using AlphaZero-like methods here, I'm not sure if there's a good solution to this problem.
Other thoughts

Newer AlphaZero papers dropped the arena idea and just always use the newest network for training, simplifying things somewhat and perhaps making it easier to get out of local minima.
Sometimes AlphaZero just doesn't work because you haven't reached the required scale yet. You didn't give much details here, but how many games did you play, how large is the NN, how much training steps are you taking? As a rough guideline for at least 100k games before A0 starts working at all.

Fantastic answer! Some follow-up comments: "The term $Q + c_{puct} U$ is only used to compare nodes to each other, so any constant offset is irrelevant" I thought that if $Q$ is always in the range $[10, 20]$ for example, then for small $c_{puct}$, UCB score will always prefer $Q$ and ignore $U$ if there's at least one sample (since $U$ is always in $[0, 1]$).

"Sometimes AlphaZero just doesn't work because you haven't reached the required scale yet." I am actually following this implementation here, which successfully uses 100 games per iteration and a 2000 game buffer to train Othello. Could you elaborate on the guideline of 100k games?

"Newer AlphaZero paper dropped the arena idea" Right, I noticed that MuZero doesn't have arena. Why doesn't this creates training instability where model could gradually become worse and get stuck with degenerate strategies?

Maybe wait with accepting answers for a bit, a better one might come along! Your comments addressed in-order:
For UCB I'm saying the offset doesn't matter because it's relative and then separately the size of the range doesn't really matter since you could just change $c_{puct}$ to compensate. This may be a more "stable" solution than relying on per-tree normalization.
The 100k games guideline is from my own experiments on chess, where eg. the policy head only started to learn interesting things after 160k games (40k on the x-axis, after the stagnation): https://i.imgur.com/wzhtiWc.png ...

... This "required" scale is not related to buffer size or generation length, just the raw number of games played. But maybe Othello is simpler, or maybe I'm doing something wrong that's slowing me down. Just don't be discouraged too quickly, A0 can take a while to start showing interesting behavior.
The arena isn't necessary because some combination of a large enough buffer + high enough move selection temperature + low enough learning rate + high enough Dirichlet epsilon + ... seems to work well enough against catastrophic forgetting, local minima, ...


text:
Self Supervised Learning Application of trained model... A bit confused
I am trying to apply a self supervised task as stated in this github repo.The Self-Supervised Sketch Recognition
In this work, authors are using 345.000 image samples to train the model and the dataset is constructed by rotating the images in 0/90/180/270 degrees. So the number of classes is 4.
When I train the model, I can get the best model recorded as the best epoch with the above parameters. (Alexnet is used in training with *.png files)
Then I need to apply this model and weights to downstream tasks, where there are 345 different classes. (But the trained model is with 4 classes) I know they do not need to be the same but I am confused how to make transfer learning..
Should I train the model again with the weights I received in pretext tasks? Or what should I do? Any reference or sample project is greatly welcome.
Thanks in advance...

For those who happened to come across my question, I solved my problem.
After training the model, I initiated the network with the trained parameters (weights). Then as I have a loaded model, I removed the last output neurons, which was 4 ea. and I added 345 neurons.
Afterwards, I froze the all neurons's weights except the last last layer of 345 and trained the model again to adjust last added weights.
Then voila!


text:
What does "cognitive" mean when saying cognitive neural network?
I sometimes hear the term cognitive neural networks thrown around.
What does the word "cognitive" specify when discussing a neural network?
From https://en.wikipedia.org/wiki/Neural_network

Since neural systems are intimately related to cognitive processes and behaviour, the field is closely related to cognitive and behavioural modeling.

Is "cognitive" basically a buzzword add-on to neural network?

It means the processes of thought in the human brain. The quoted sentence is saying that the study of neural networks is related to the study of human brains and animal behaviours.
For example, neural network research has generated some possible explanations for why humans need to sleep - because some types of neural networks also need to sleep. Many neural network designs (indeed, the whole concept of a neural network) are also based on animal brains work.


text:
Batch Normalization as the Final Layer
In this community implementation of Inception-ResNet, Batch Normalization is used as the final layer.
This is unlikely to be a mistake, but also seems counterintuitive. The inceptionv2 paper suggests that Batch Normalization acts as a regularizer. But wouldn't it be too strong of a regularizer as the very last layer?

The regularization effect of Batch Normalization (BN) is a side effect, since it introduces noise during training because it uses batch statistics (first and second moments) to normalize the data.
I guess that, in the implementation that you have seen, BN is used as the last layer just to provide an output that approximately follows a Gaussian distribution (that can be 0-mean and unitary variance, for example), and so on for the purpose of regularization.

Why does it cause the output to approximately follow a Gaussian distribution?

Why is it useful for the output to approximately follow a Gaussian distribution?

Because BN standardizes the batch, by subtracting mean and dividing by variance. So it's like transforming the data to be Gaussian-like. Then, it scales and shifts them to provide more flexibility as a 0-mean 1-variance normal can be too restrictive. 2) It can be due to the data be normalized in the same way, but also to ensure better convergence given a specific loss function, or even to comply to some theoretical property of the problem (but that is beyond Inception I guess.)


text:
Creating a Dataset from Time Series Data
Context
I'd like to build a regression model for this data to predict a user's test scores given their study habits.
Basically, the variables are in two separate csv tables similar to the ones below. Eventually, I'd like to create a forecasting model.
Training variables:

Time series data of study habits (multidimensional: location, duration, date, user_id)

Target variables:

Test scores (scores, date, user_id)

Problem

I'm not sure how to actually put together instances regarding time. Do I just say that every training instance before the next test date gets that test result as the target variable?
Is there a term for temporally aligning data in a time series or techniques?

What you are trying to accomplish is called sequence prediction. It takes in a sequence of data, and spits out a score (target variable).
Time-series data is usually formatted as 3D arrays with shape (batch size, sample length, data). So in your case, the data would be (location, duration, date, user_id). It's preferable to have all sequences be of the same length by adding padding at the end of sequences that are too short. The padding can be dealt with in the model training itself later on.
I'd keep the date variable in the data itself, but maybe format it as an offset, so make the first time studying equal to 0, the second instance equal to the number of hours later (like 24 for the next day) and have the prediction variable as the last offset.

If the answer has sufficiently answered your question, make sure to select it as the correct answer such that future visitors can better find what they are looking for.

It does, but I'm not sure about data leakage with sequence prediction. I have a large gap between sampling and tests, like samples are daily, whereas, tests roughly every 90 days, does this matter?

Data leakage is, to my knowledge, an unrelated topic. However, whether the gap is too large or too varied to predict is heavily dependent on the data. If the correlations of the data with the target variable are solid enough and the model is trained properly, i don't see why it would not work. However, having a large variability in the gaps might induce a large amount of unnecessary/unlearnable noise. However, handling the specifics of induced noise etc, might require a separate question on this site. Because that would require a lot more information about the distributions of the data etc.

Thanks! If you have any related keywords, I'd be grateful for the orientation.

The type of time-series data you are using is called 'irregular time-series data'. Not sure if that might help in your future searches, but it might.


text:
Can I implement a sklearn model inside a Pytorch nn.Module?
I am making a custom Pytorch model that at some point, clusters a latent space that was created by another, previous routine of the model (Autoencoder).
In a bit more detail, my model is a regular Autoencoder, but in every training step, I want to perform a clustering of the representations in the latent space Z, and save the clustering algorithm inside the model to use after training, ONLY for inference (to make certain kind of predictions).
I thought about defining an sklearn model as an instance of the nn.Module as
class Model(Module):
    def __init__(self, hparams):
        super().__init__()
        self.clustering_model = sklearn.clustering_model()

This way, I could call the clustering model to fit the latent space Z, after calculating the reconstruction loss as a regular autoencoder.
After training, I would then have in the variable model.clustering_model a trained clustering model that I could use for inference.
However, this doesn't seem to work out, probably because of incompatibilities between sklearn and Pytorch. Basically, it seems that training the clustering_model and saving it inside the pytorch model didn't actually save any trained weights.
I'm right now unsure whether it didn't save weights because it actually wasn't trained (by an error on my side) or if it is that PyTorch just can't save a sklearn trained model as a variable in a nn.Module.
Did anyone try before to implement sklearn inside a nn.Module?

I've not used Pytorch before, but skorch might be a helpful library for you.

@LittleLulatsch I've checked! Sadly the library is made to transport Pytorch to sklearn, and not the other way around.

It is indeed possible, yet It may be trickier than just add the clustering code to the forward method of autoencoder. Are there any particular reasons to not do it? forward method also allows to return several values.

@Ciodar Do you mean that they should be saved if they are added to the forward method instead of just the class? I'm unsure about which is the criteria of Pytorch to save information about a model instance. It seems that Pytorch will only save weights that were trained (i.e. that were part of the loss), but my clustering algorithm wouldn't enter into the pytorch loss calculation.

Maybe I misunderstood the first time I read that. Do you want to compute a latent representation $z_k$ for each of your input samples, and then cluster all latent representations into a latent subspace? In this case, I think the simpler solution is to split these two problems, storing all output latent spaces and performing a clustering of those.

Once you have computed the centroids, you can save your clustering params and restore them. I'm kinda wary to actually introduce Kmeans parameters as module parameters, since they should not be added into the computational graph (Kmeans is not typically trained with GD)

One solution to have them in your custom Pytorch model is to register a buffer where you store your clustering parameters, and define a procedure to save/restore the clustering weights from the buffer. Then you can call the clustering algorithm (maybe only during prediction) using a custom method or also in your forward. (I personally would call a custom method)

@Ciodar Indeed a nn.parameters is what I need to tell Pytorch to save those weights. Thanks!

I would use buffers instead, as specified in the link above and also here. Anyways, I will transform these comments in an answer if they fit your needs.

Programming questions (which includes questions related to specific libraries) are off-topic here. Please, ask them at Stack Overflow or Data Science SE. See our on-topic page. We focus on the theoretical aspects of AI.

Yes, you can define a totally custom Model, maybe with a clustering method you can call after forward only during inference.
Clustering parameters (e.g centroids in Kmeans) can be stored inside a buffer (see here and here)
Clustering is not typically optimized with Gradient Descent, so clustering parameters should not require gradient, and you need to pay attention to not include the clustering step into your computational graph.
I would implement this pipeline:

Autoencoder training: Train your AutoEncoder as usual, and save the model's checkpoint.
Feature extraction: Extract all latent codes using your trained autoencoder, then save them onto disk into a numpy array or a tensor.
Clustering: Load all latent codes and apply a clustering algorithm (e.g Kmeans) and save all relevant parameters (centroids) onto a buffer .
Inference: Load trained autoencoder and centroids. Both parameters and buffers are into model's state_dict, so they can be loaded with the load_state_dict method. Pass new data into the encoder, then cluster the latent representation onto the closest centroid

By using a modular approach, you can experiment different clustering methods without losing all the features (saving additional training), and then put all together once you're satisfied with the model.


text:
How would I use backpropagation with a changing cost function
I have a neural network that is being trained with a changing cost function. Could I use backpropagation at all? If yes, how would I do this?

Hi, I think we need more detail to answer this question. What do you mean by "changing cost function"? Are some of your NN parameters influenced by an external stochastic variable? Then you'd have to look in to Monte Carlo methods for gradient estimation, but it's really hard to say whether that is what you are looking for.

There is nothing wrong with changing your cost/loss function after every step while training a neural network. For example, this paper looks at a weight decay scheduler (which changes the weight regularization at every step): https://arxiv.org/abs/2006.08643.


text:
Keras Subclassing : TypeError: 'KerasTensor' object is not callable Call arguments received by layer
Implementing UNet but getting an error: type error 'KerasTensor' object is not callable
class UNet(keras.Model):
  """
  argument: input_shape=(572, 572, 1) => default
  """
  def __init__(self, input_shape=(572, 572, 1), **kwargs):
    super().__init__(**kwargs)
    self.inputs = keras.layers.Input(shape=input_shape)
    self.maxpool = keras.layers.MaxPool2D(pool_size=2, strides=2)
    self.concat = keras.layers.Concatenate() # concats through depth
    self.conv_sz1 = keras.layers.Conv2D(filters=2, kernel_size=1, padding="same")

  class CONV2_BLOCK(keras.layers.Layer):
    def __init__(self, filters, **kwargs):
      super().__init__(**kwargs)
      self.filters = filters

    def call(self, inputs):
      x = keras.layers.Conv2D(filters=self.filters, kernel_size=3, use_bias=False)(inputs)
      x = keras.layers.BatchNormalization()(x)
      x = keras.layers.Activation(keras.activations.relu)(x)
      x = keras.layers.Conv2D(filters=self.filters, kernel_size=3, use_bias=False)(x)
      x = keras.layers.BatchNormalization()(x)
      outputs = keras.layers.Activation(keras.activations.relu)(x)
      return outputs

  class CONV_T(keras.layers.Layer):
    def __init__(self, filters, **kwargs):
      super().__init__(**kwargs)
      self.filters = filters

    def call(self, inputs):
      outputs = keras.layers.Conv2DTranspose(filters=self.filters, kernel_size=2, stride=2)(inputs)
      return outputs

  class CROP(keras.layers.Layer):
    def __init__(self, cropping, **kwargs):
      super().__init__(**kwargs)
      self.cropping = cropping

    def call(self, inputs):
      outputs = keras.layers.Cropping2D(cropping=self.cropping)(inputs)
      return outputs

  def call(self, inputs):
    # self.conv_arr = [64, 128, 256, 512, 1024]
    # self.crop_arr = [4, 17, 40, 88] down to up
    x1 = self.CONV2_BLOCK(filters=64)(self.inputs(inputs))
    x = self.maxpool(x1)

    x2 = self.CONV2_BLOCK(filters=128)(x)
    x = self.maxpool(x2)

    x3 = self.CONV2_BLOCK(filters=256)(x)
    x = self.maxpool(x3)

    x4 = self.CONV2_BLOCK(filters=512)(x)
    x = self.maxpool(x4)

    x = self.CONV2_BLOCK(filters=1024)(x)

    x = self.concat([self.CONV_T(filters=512)(x), self.CROP(cropping=4)(x4)])
    x = self.CONV2_BLOCK(filters=512)(x)

    x = self.concat([self.CONV_T(filters=216)(x), self.CROP(cropping=17)(x3)])
    x = self.CONV2_BLOCK(filters=256)(x)

    x = self.concat([self.CONV_T(filters=128)(x), self.CROP(cropping=40)(x2)])
    x = self.CONV2_BLOCK(filters=128)(x)

    x = self.concat([self.CONV_T(filters=64)(x), self.CROP(cropping=88)(x1)])
    x = self.CONV2_BLOCK(filters=64)(x)

    outputs = self.conv_sz1(x)

    return outputs

test_model = UNet(input_shape=(572, 572, 3))
print(test_model(tf.random.uniform(shape=(2, 572, 572, 3))).shape)

but getting error as
---> 48     x1 = self.CONV2_BLOCK(filters=64)(self.inputs(inputs))
     49     x = self.maxpool(x1)
     38

TypeError: Exception encountered when calling layer 'u_net_1' (type UNet).
'KerasTensor' object is not callable
Call arguments received by layer 'u_net' (type UNet):
• inputs=tf.Tensor(shape=(2, 572, 572, 3), dtype=float32)

The issue is when calling self.inputs(inputs),(you can't use input layer in keras subclassing, only for sequential and functional API) but in general you're defining the layers and model wrongly. Here is one way to define a custom keras.Model (I've also cleaned up the code a bit and removed duplicated lines):
import tensorflow as tf
import tensorflow.keras.layers as tfkl
from functools import partial

class ConvBlock(tfkl.Layer):
    def __init__(self, filters: int, **kwargs):
        super().__init__(**kwargs)

        self.conv1 = tfkl.Conv2D(filters=int(filters), kernel_size=3, use_bias=False)
        self.bn1 = tfkl.BatchNormalization()
        self.bn2 = tfkl.BatchNormalization()
        self.activation = tf.nn.relu
        self.conv2 = tfkl.Conv2D(filters=int(filters), kernel_size=3, use_bias=False)

    def call(self, inputs, **kwargs):
        x = self.conv1(inputs)
        x = self.bn1(x)
        x = self.activation(x)

        x = self.conv2(x)
        x = self.bn2(x)

        outputs = self.activation(x)
        return outputs

class UNet(tf.keras.Model):
    def __init__(self, input_shape=(572, 572, 1), filter_list=(64, 128, 256, 512),
                 crop_values=(4, 17, 40, 88), **kwargs):
        assert len(filter_list) == len(crop_values)

        # pre-define some layers
        max_pool = tfkl.MaxPool2D(pool_size=2, strides=2)
        concat = tfkl.Concatenate()

        ConvT = partial(tfkl.Conv2DTranspose, kernel_size=2, strides=2)
        Crop = partial(tfkl.Cropping2D)

        # model architecture
        inputs = tfkl.Input(shape=input_shape, name='image')
        x = inputs

        layers = []

        # conv_block -> max-pool
        for filters in filter_list:
            x = ConvBlock(filters=filters)(x)
            layers.append(x)
            x = max_pool(x)

        x = ConvBlock(filters=1024)(x)

        # concat(conv_t, crop) -> conv_block
        for filters, crop, layer in zip(reversed(filter_list), crop_values, reversed(layers)):
            print(filters, crop, layer.shape)
            x = concat([ConvT(filters=filters)(x), Crop(cropping=crop)(layer)])
            x = ConvBlock(filters=filters)(x)

        outputs = tfkl.Conv2D(filters=2, kernel_size=1, padding='same')(x)

        super().__init__(inputs=inputs, outputs=outputs, **kwargs)

But to make it have matching shapes, I had to change a bit the values you use for the crop layer:
unet = UNet(input_shape=(572, 572, 3), crop_values=[4, 16, 40, 88])
print(unet(tf.random.uniform((2, 572, 572, 3))).shape)  # -> TensorShape([2, 388, 388, 2])

Lastly, I got an output shape of (388, 388, 2) is that right for your problem?

bn1 and bn2 are two different instantiations of a batch-normalization layer, they are two different objects and you want to do that because each internally keeps statistics of the inputs. Same for ConvBlock they are the same set of layers but different instantiations (also with different parameters, i.e. filters for example). Basically, the syntax ConvBlock(filters=...) creates the instance of type ConvBlock, then the second call (x)  applies that to the layer x returning a symbolic tensor. Have a look at the Keras Functional API.

Programming questions are off-topic here. Please, flag them as off-topic and encourage the user to ask them at Stack Overflow or Data Science SE. See our on-topic page for more info.


text:
Is it feasible to perform facial recognition on hundreds of thousands of individuals?
I came across a video with the title "you can buy things with your face in China". in the video, a woman scanned her face into a vending machine to buy a drink with only her face and without anything else. The description claimed that each face is a digital ID linked to China's CBDC. (link to the video)
But I can't understand how this is possible from a technical viewpoint. For this, the model should classify hundreds of thousands of faces and this doesn't sound feasible to me. The ImageNet dataset has only 1000 classes and the Google JFT dataset has only 30K labels!
Is it feasible to perform facial recognition on hundreds of thousands of individuals?

What makes you think there is an AI with hundreds of thousands classes behind this technology? finger print detection seems to work to a pretty good extend, why shouldn't face recognition?

Whilst identities can be viewed strictly as a classification problem, typically at scale they are solved as a regression problem to a description vector, followed by lookup of nearest vectors outside of the AI (e.g. with a database search optimised to find nearest multi-dimensional vectors). To help with this approach, the regression system is trained to minimise distance between pictures of objects with the same identity and maximimse distance between pictures of objects with different identity.
Here's a reference to the triplet loss training approach used to do this.
In brief, the training database requires at least 2 images of each subject. Pairs of images are selected, some identical, some different, and with a bias towards scoring "difficult" pairs. Each image is converted to a description vector by the neural network, and the difference between pairs is used as the loss function (so unlike many supervised learning schemes, the loss is not per function call, but calculated over sets of three function calls).
Prior to use of triplet loss function, it was quite common to have the neural networks learn to predict actual biometrics (e.g. distance between eyes) from images. This had the advantage that the description vector was meaningful, but the disadvantages of needing a lot of difficult data collection, and no guarantee that the chosen biometrics would be good at separating identities in practice, depending on the population.


text:
Fluctuations in loss during in epoch evaluation of GRU
I am training a one-layer unidirectional vanilla GRU on a next item prediction task with regard to the last 10 interacted items. In my original experiment, where I trained on approx. 5.5M samples and validated on around 1M samples, I saw periodic fluctuations in my loss/accuracy curves. Now I made a few experiments with a far smaller dataset (train: 250K, val: 10K) and compared different learning rates. I still saw this effect and I would like to know what causes it, just to understand if it is an issue or not.
I am evaluating every 25% per epoch, so 4 times per epoch. I accumulate the loss and the hit-ratio for the processed training batches and when it is time for evaluation, based on these I compute train metrics and then compute the validation metrics on the entire validation set.
I am using Adam as optimizer.

The plots show HR@10 (=accuracy for top-10 predictions) and CrossEntropyLoss for train and val. One step is referring to 25% of an epoch, so step 0 = 25% of first epoch, step 4 = 25% of second epoch done. You can see that the phenomenon gets more extreme for higher learning rates. The spike in training HR/loss always happens at the first evaluation during an epoch (so 25%). It is paired with a drop in validation accuracy, which gets better for the next three steps (50%, 75% 100% of epoch). Then falls again when spike in training accuracy occurs.
Things I excluded already:

data gets fed randomly, it is not always the same order
checked whether gradient is accidentally not reset before calling backward()
possible wrong calculation of loss/accuracy due to wrong number of batches I divide by

I would appreciate any help to understand what is happening with the model. It still seems to work but I just don't understand what causes this behavior.

Oscillating loss is a symptom of divergent training: it can be due to large gradient updates, and/or numerical instability. Moreover, you said that it gets worse with higher LR: that's a confirm.
To fix that you should try:

A decaying lr schedule, that basically after a given amount of optimization steps (or epochs) your lr is reduced by a factor $\gamma$ ($\gamma < 1$) until a minimum lr is reached. The decay can be linear (so $\gamma$ is proportional to the number of steps) or exponential for example.
Layer normalization: this can help to stabilize the gradient, for a faster training.
Gradient clipping: it can scale the gradients individually (per-layer) or globally to not be above a given norm. Basically it reduces the magnitude of the gradients to control the size of the update.

I suggest you to also monitor the (euclidean) norm of both gradients and weights. So that you can understand if gradients are big or exploding/vanishing, and also have a look at possible overfitting.
Update: numerical instability of the loss computation can lead to large gradients, and even NaNs. In particular, the suggested usage of the cross-entropy loss is from logits (i.e. unnormalized log-probabilities of the classes) since it can be implemented in a numerically stable way, conversely when directly applied to a softmax output it can be unstable.

Thanks a lot for your detailed response. I think your suggestions and explanations make total sense. So it could indeed be a learning/gradient related problem due to too strong/big gradients leading to overshooting and alternation. Okay let's assume this is the case, why is it always the same pattern, so better train accuracy/loss in the first 25% of the epoch and worse for the rest? That's the only thing still worrying me.

Maybe the model after that starts to fit the noise in the data (i.e. starts to overfit, that explains also the lower HR on validation, for blue curve - lr=0.01), the gradients begins unstable and so the loss fluctuates. It can be also a numerical instability in the cross-entropy loss: try computing that from logits (if not doing so). Lastly, you can even try a different (or simpler) optimizer or to tune a bit the Adam's parameters like the betas and epsilon too.

Thank you, that makes a lot of sense. I will try and play around with your suggestions!


text:
Has anyone tried to train a GPT model predicting the next N tokens instead of the next one token?
I have been thinking about how learning via text works on humans: we read words, and often we need to read ahead a few words to understand more clearly the ideas that we read before. Most of the time, just reading the next word in a sentence is not enough for clear understanding.
Has anyone tried (and if no, what is your opinion) to modify GPT to predict the next N tokens instead of the next single token? My intuition says that the loss function would decay faster that way.

Sure, I think there have been studies on modifying language models, to predict multiple tokens at once instead of just one. This technique is known as n-gram language modeling, where the model is trained to predict the next n words in a sequence.
The basic idea of this approach is to better capture the dependencies and interactions between multiple words in a sequence, which can improve the model's ability to generate coherent and fluent sentences. However, this technique comes with some challenges, such as increased (or even impossible to handle) computational cost.
Regarding your intuition about the loss function decaying faster with multiple tokens, it is possible that predicting multiple tokens at once could lead to faster convergence during training. However, this would depend on the specific details of the model and training regime. Ultimately, the effectiveness of n-gram based modeling would need to be evaluated empirically on various tasks and datasets.
https://web.stanford.edu/~jurafsky/slp3/3.pdf


text:
Regression Model diverging after adding a new feature with higher variance and magnitude
In a time series regression problem I'm predicting "change" rather than the actual intended value i.e
Instead of:
time, feature 1, feature 2, ....
2020, 1, 3.3
2021, 1.5, 5.2
2022, 1.3, 6.1
now, y_pred_1, y_pred_2, ....

I'm feeding the model and predicting percent changes:
2020 0, 0
2021 0.5, 0.57 # 5.2 / 3.3 - 1 = 0.57 percent change
2022, -0.13, 0.17

Now a problem that I'm facing is that when I introduce a new feature of a much larger magnitude and variance my model diverges during training:
# new feature
2020, 3000
2021 4
2022 600

# Which is transformed to percent changes
2020, 0
2021, 0.99
2022, 149 # oof

I tried mean normalizing that particular feature before transforming it to the percent change domain but it still diverges.
Any ideas what else I can try to introduce this new feature to the inputs? What are some best practices for such contrasted inputs?
To get a general idea of the model if it has something to do with this:
The model consists of 5 parallel flows that get added together at the end. Each flow has two pairs of CNNs with different kernel sizes followed by 1-3 LSTMs. The output shape of each flow is the same so they can be added together. Then after adding it's followed by two fully connected layers predicting the change for the given features.

The problem as you already pointed out is in the high change variance of the new feature. Normalising using the mean does not impact the variance since it's merely a linear transformation, still good to keep all feature in the same range but not impacting the variance at all.
What you need is a non linear transformation that change the magnitude in change variance without introducing artefacts. The most simple thing to try would be to just take the log of the new feature. Since you're working with time series another possibility could be applying a moving average, exploring different time windows to see how much the feature will be smoothed out.

Thanks! If I used log as a normalization would it not generally "suppress" the sensitivity of the model to the changes of that feature? Or is that the tradeoff I have to accept?

Also is there any point for choosing a specific base of the log function given the variance or just better off with natural log?


text:
How a begginer should start studies in ML?
People, I am a chemistry undergrad that wants to learn ML for application in my field(chemistry), however, I had never studied programming or anything related to code in my life, for this reason, I am building a study plan for this goal.
Can you give me any advice on this endeavor? I would appreciate any suggestions to know if I am on the correct path.
Cycle 1: Learning the basic concepts of Programming
• Algorithm and Programming Logic with Python: https://www.youtube.com/playlist?list=PLQKlKz1J9WWTUUMnfa2PWaPU3qPND7GJN
• Kaggle: https://www.kaggle.com/learn
Cycle 2: Learning how to Program in Python
• Python Course for Beginners: https://www.youtube.com/playlist?list=PLyqOvdQmGdTSEPnO0DKgHlkXb8x3cyglD
• Kaggle: https://www.kaggle.com/learn
• CodingBat: https://codingbat.com/python
• Project Euler: https://projecteuler.net/
Cycle 3: Learning how to use Python for Data Analysis and learning Statistics
• Python Course for Machine Learning and Data Analysis: https://www.youtube.com/playlist?list=PLyqOvdQmGdTR46HUxDA6Ymv4DGsIjvTQ-
• Khan Academy: https://pt.khanacademy.org/math/em-mat-estatistica
• Pandas, Matplotlib and NumPy:
https://www.youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS
https://www.youtube.com/playlist?list=PL-osiE80TeTvipOqomVEeZ1HRrcEvtZB_
https://www.youtube.com/playlist?list=PLzgPDYo_3xukqLLjNeuCxj4CwvkJin03Z
• How to go from ZERO to Data Science in Just One Lesson: https://www.youtube.com/watch?v=WJE4spsP-Xk
Cycle 4: Learning basic concepts about Machine Learning and Probability
• Course Introduction to Machine Learning and Machine Learning Algorithms:
https://www.youtube.com/playlist?list=PLyqOvdQmGdTSqkutrKDaVJlEv-ui1MyK4
https://www.youtube.com/playlist?list=PLyqOvdQmGdTS4PcZsAKhXgkdUyLjXnptS
• Machine Learning with Python and Scikit-learn: https://www.youtube.com/playlist?list=PLeFetwYAi-F9Z0N3ZSjPL2i24PO5iiXaY
• Kaggle: https://www.kaggle.com/learn
• Khan Academy: https://pt.khanacademy.org/math/em-mat-probabilidade
• Google Machine Learning Crash Course: https://developers.google.com/machine-learning/crash-course
Cycle 5: Learning Applications of Machine Learning in Chemistry
• EMMSB LNCC: https://www.youtube.com/playlist?list=PLIJhet1J-_l82wpFzn2ZOyCIKy-FotuND
• The Computational Toolkit: https://www.youtube.com/playlist?list=PLgCwaJhZsSlMQky78U5JLyMOgO16o6ubE
• Learn how to use RDKit or DeepChem, a collection of cheminformatics and machine learning tools for Python
Cicle 6: Learning Deep Learning Fundamentals
•   Deep Learning by Andrew Ng on Coursera: https://www.coursera.org/specializations/deep-learning
•   Kaggle:
https://www.kaggle.com/learn

I graduated with a university degree in Artificial Intelligence, I only covered Artificial Intelligence a little bit on it, surprisingly, but this brings me to the actual point of my answer. I have started a bit here and there to try and learn myself I mainly use the site https://datacamp.com a lot if I am honest, as for Python when I learned this I tried to stick to one resource and learn algorithms and problems later. This is a good resource on Python: https://roadmap.sh/python and also this course is good which is free: https://futurecoder.io/
I mean your list is good, but what I would say try and find overlaps more in what you are learning because especially I find myself with programming say doing a Python programming course at beginner level, then doing another at beginner level but 'data science' you will find a lot of overlap.


text:
OpenAI: What is the difference between model "gpt-3.5-turbo" and "gpt-3.5-turbo-0301"?
I have performed an API call to OpenAI's endpoint https://api.openai.com/v1/models .
The endpoint lists the currently available engines, and provides basic information about each one such as the owner and availability.
As a logged-in user, I get a JSON response of 63 models. These are the most recent ones (currently) , formatted, shown with release date.
59: "11/28/2022, 2:40:35 AM : text-davinci-003"
60: "12/16/2022, 8:01:39 PM : text-embedding-ada-002"
61: "2/27/2023, 10:13:04 PM : whisper-1"
62: "2/28/2023,  7:56:42 PM : gpt-3.5-turbo"
63: "3/1/2023,   6:52:43 AM : gpt-3.5-turbo-0301"

I notice that there are 2 very similar models , "gpt-3.5-turbo" and "gpt-3.5-turbo-0301", with gpt-3.5-turbo-0301 released only 11 hours after gpt-3.5-turbo.
What is the difference between these two model versions? It does not seem to be a glitch or a misnaming error. Why did OpenAI bother to include both of them, and why didn't take the inferior version?
(I haven't experimented with these two models in any way yet. I might do this very soon. However I though I might as well ask here. Informing others in this forum might have some benefit.)

Taken from here: https://platform.openai.com/docs/models/gpt-3-5

I think its literally an update but the specifics of what that updates are I do not know I looked through the documentation but this was all I could find.

Thanks. I think the decisive keyword from the documentation you posted is "snapshot". I think OpenAI received some outrage recently, when they announced their intention to remove the previous iteration,  because many 3rd-party researchers and reviewers would have lost their "baseline model" which they refer to in their manuscripts pending review and publication.

Ah interesting, I did not know that, thanks for the info. It is very hard to keep up with AI at the moment. I wanted to answer your question as soon as I seen it because I am trying myself to understand the docs at the moment haha.


text:
GREED - preservation theoretical properties in the GED(graph edit distance) pridiction
In this paper "GREED: A Neural Framework for Learning Graph Distance Functions", function F is defined to satisfy metric property and triangle inequality property.
I wonder how can I prove that these properties are maintained in actual forecasts? Or does defining the F function like that make it a natural fact without having to prove it?
These are the architecture in the paper.
I'm not sure.. If I want to make a model using deep learning, is it enough for satisfying properties at the real prediction value to define loss function that is always satisfying properties which i want?

If I understand correctly, the concept is to, first, embed the given graphs using the graph isomorphism network (GIN) [Xu et al.] and then, for the GED, feed any two embeddings $\mathbf{x}_{\mathcal{G}_1}$ and $\mathbf{x}_{\mathcal{G}_2}$ of graphs $\mathcal{G}_1$ and $\mathcal{G}_2$ to the euclidean norm, i.e. $\text{GED}(\mathcal{G}_1, \mathcal{G}_2) = \parallel \mathbf{x}_{\mathcal{G}_1} - \mathbf{x}_{\mathcal{G}_2} \parallel_2$. This architecture is then trained to predict the graph edit distance.
To my understanding, the metric properties of this approach rely entirely on the expressive power of the GIN. If the GIN generates the exact same embedding for two dissimilar graphs, then $\text{GED}(\mathcal{G}_1, \mathcal{G}_2) = 0$ even though $\mathcal{G}_1 \neq \mathcal{G}_2$ which violates the properties of a metric. As GINs expressive power is upper bounded by that of the Weisfeiler Lehman test (WL-test) (cf. Xu et al.), you can easily find such pairs of graphs. For more on this, you can read this nice blog post.
At first glance, this seems to be contradicting the claim of the authors. However, as far as I understand, the authors only claim that $F$ preserves metric properties. Since $F$ is the p-norm, i.e. the GIN is excluded from the definition of $F$, they are correct. The authors don't claim that the full architecture including the GIN preserves metric properties.
To answer your questions:

I wonder how can I prove that these properties are maintained in actual forecasts?

You can only disproof that, because GIN cannot distinguish graphs that the WL-test cannot distinguish.

is it enough for satisfying properties at the real prediction value to define loss function that is always satisfying properties which i want?

Sadly no, this is not enough - neural networks are only function approximators, there is usually no guarantee that an NN will always satisfy the properties promoted by the loss function.
The bottom line is that if you truly rely on absolutely correct measurements, then you should not use this approach. However, exact measurements take a lot of time to compute (as the authors say it's an NP-hard problem). Therefore, this method presents a tradeoff between accuracy and time complexity. If the accuracy suffices, then it can be orders of magnitude faster than explicitly computing GED.


text:
Do different ngrams share embedding in Fasttext?
As per Section 3.2 in the original paper on Fasttext, the authors state:

In order to bound the memory requirements of our model, we use a
hashing function that maps n-grams to integers in 1 to K

Does this mean the model computes only K embeddings regardless of the number of distinct ngrams extracted from the training corpus, and if 2 different ngrams collide when hashed, they share the same embedding?
Thanks.

Yes, your interpretation is correct.
This trick reflects that there are many possible character n-grams, but only relatively few of them contribute to the word's meaning. Instead of resolving what n-grams are the most salient ones, they just opt for a stochastic solution.


text:
Do different models using early stopping have the same validation set to check model training performance?
I, i have a doubt about making validation using early stopping given two NN models.
Suppose I have two models M1 and M2 and a Training set TS and Test set TS.
Take the TS and consider TS_80% and TS_20%, the first as training and the second as validation in the hold out procedure
Train M1 with early stopping on TS_80, choosing 10% of TS_80 as validation reference set for the early stopping procedure, we could call this TS_80_es_1
Then train M2 with early stopping on TS_80, choosing 10% of TS_80 as validation reference set for the early stopping procedure, calling it TS_80_es_2.
NOW, must TS_80_es_1 be identical to TS_80_es_2? I mean, the validation set we use for the early stopping must be the same or can it be chosen as 10% randomly in each individual training for each model and so we can consider different validation sets in the early stopping procedure?
In other words, every model we train using early stopping has the same validation set for early stopping procedure, or the validation set can be different for each training phase?

Do you use the same early stop criterion for both models?

@LucaAnzalone Yes, the criterion is the same (a very superficial one , |validation_loss_epoch_x - validation_epoch_time_x+5| < threshold, if there is no significant increase in 5 epochs, then stop)

Short answer: Yes, the validation set should be the same otherwise you risk that a "lucky" set of validation samples is responsible for better performance.
Long answer: A fair comparison of 2 or more models can only occur if these are both trained and evaluated in exactly the same conditions. That is:

Same training data, shuffled and batched in the same order just because of the stochastic nature of the optimizers.
Same initialization of weights, or at least, in case of models with different architectures/layers, from the same pseudo-random generator's (PRNG) state.
Validation should occur in the same way: data should be the same, as well as the model selection criterion.

In general, all the stochastic operations (SGD, weight init, train-test splits, samplings, etc) should be reproducible, for two reasons: 1) the performance of your model should not depend on random numbers that change every run, and 2) you want to avoid having both a lucky or unlucky training just due to the PRNG. To (almost) solve this, one has to set the same random seed for all the random generators. This may be not enough, due to some inherently non-deterministic operations (e.g. some calculations on GPU due to CUDA optimizations, parallelism, etc) but more importantly the (sequence) of PRNG states should be the same across runs and comparisons: Python or numpy's PRNG are stateful, meaning that each time you ask for random numbers they change their internal state. In particular the numpy's PRNG consumes entropy, being more and more deterministic as it is used. One should do as JAX forces: use stateless PRNGs, and manually keep track of the state.
Regarding statement 2, ou have to consider that a single run of a model gives a punctual estimation of the real performance of the model: what you want to know is an expected value, instead, i.e. something associated with a standard deviation for example. This is for a more broad discussion, but to properly compare two or more models (methods, etc) a single number (e.g. AUC) is not enough because you cannot decide whether M1 is really better than M2 because of the superior architecture or better hyper-params, or because it was just a lucky random seed or lucky data: since your training and validation sets are not infinite, model performance are subject to inherent fluctuations. Instead, training a model 100 times, for example, considering different initial situations (but different across the runs, and not along compared models), allows you to aggregare performance obtaining a mean and associated std. For example, M1 can achieve 95% ($\pm 3$) AUC while M2 achieves 95% ($\pm 1$). So what model you choose? both achieve the same performance on average, but M1 has more variability; there are approaches that perform statistical tests to say that M1 has a probability $p$ to beat M2, and M2 has a prob $p_1$ to be better. According to such probability you can make a decision, for example.
To conclude, this is an open question (also because 100 runs are not practical).


text:
What sort of computer would be necessary to run queries on a LLM?
I've heard that to train a model like GPT 4.0 you need a very powerful computer and ~$10M of computing power, but once you've produced the trained ~570GB model, what sort of computing power is necessary to execute specific queries with it?

Executing specific queries, in the context of large language models, is referred to as inference. The hardware that runs GPT-4 has not been disclosed. However, Meta's LLaMA can be run on consumer hardware. llama.cpp can run the 7B model on an M1 Pro MacBook – a decent, but not top of the line, computer:

As the models are currently fully loaded into memory, you will need adequate disk space to save them and sufficient RAM to load them. At the moment, memory and disk requirements are the same.
Model     Original size   Quantized size (4-bit)
7B        13 GB           3.9 GB
13B       24 GB           7.8 GB
30B       60 GB           19.5 GB
65B       120 GB          38.5 GB

LLaMA-65B can be run on a CPU with 128GB of RAM, although this is unlikely to be efficient compared to renting data centre GPUs. GPT-4 is reported to be a much larger model than LLaMA-65B, with support for a 32K context window. Since the amount of fast GPU memory required scales with the size of the model, and inference cost scales quadratically with input sequence length, GPT-4 inference cannot be performed on consumer hardware.

Likely one or multiple A100 based servers (i.e. servers with 8xA100 cards from Nvidia).
Posssibly the same for H100 cards, but you could likely reduce memory usage with quantification and get a lot more throughput from the same hardware, so can do more queries per time. Memory would be more efficient on quantificatin, otherwise be quite similar to A100 - I hear that 3.5 runs on around 770 or so gb vram. One such node is around 300.000 USD.
4 bit quantification should result in serious savings, possibly giving good performance on high end servers, with - as research shows - POSSIBLY very limited impact (there is a loss of quality that is small and gets smaller on larger models).
The real hardware is not disclosed - you have to wait for large open source models to become available.


text:
What is a neuron in large language models?
I'm reading OpenAI's new paper "Language models can explain neurons in language models" And I can't fully understand the concept of neurons here.
Can you please explain it? Is it related to the attention mechanism?

It seems that in this paper neuron means the perceptrons in the MLP layers of the transformer.


text:
gradient descent optimization to make orthogonal vectors
I am trying to use gradient descent to optimize 3 3-d vectors to be perpendicular. To do this I am summing the dot product of each pair of vectors. When they are all perpendicular the result should be 0.
This is the code I have written
import torch

N = 3  # number of vectors
d = 3  # dimensionality of each vector

# initialize a random tensor of shape (N, d)
vecs = torch.randn(N, d, requires_grad=True)

def pairwise_dot_sum(vecs):
    """
    Computes the sum of dot products of all pairs of vectors in vecs.
    """
    dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
    return torch.sum(dot_products.triu(diagonal=1))  # upper triangular part only

optimizer = torch.optim.Adam([vecs], lr=0.01)  # choose an optimizer

with torch.no_grad():
    vecs /= torch.norm(vecs, dim=1, keepdim=True)

# train for some number of iterations
for i in range(1000):
    loss = pairwise_dot_sum(vecs)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    with torch.no_grad():
        vecs /= torch.norm(vecs, dim=1, keepdim=True)

# check that the vectors are approximately orthogonal
dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
print(torch.max(torch.abs(dot_products - torch.eye(N, N))))
print(vecs)

#vecs = torch.tensor([[1,0,0],[0,1,0],[0,0,1]])

dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
print(dot_products.triu(diagonal=1))

The output is this:
tensor(0.5543, grad_fn=<MaxBackward1>)
tensor([[ 0.7315, -0.5695,  0.3749],
        [ 0.0595,  0.4699, -0.8807],
        [-0.8440,  0.0857,  0.5294]], requires_grad=True)
tensor([[ 0.0000, -0.5543, -0.4677],
        [ 0.0000,  0.0000, -0.4762],
        [ 0.0000,  0.0000,  0.0000]], grad_fn=<TriuBackward0>)

So you can see that the loss is 0.5, instead of 0.
and the vectors seem to form an equilateral triangle centered around the origin.
What mistake have I made? How can I get this fix this to correctly find orthogonal vectors?

There's no need to reinvent the wheel -- PyTorch allows you to implement orthogonal constraints on parameters. See: https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.orthogonal.html

Your issue is that your function pairwise_dot_sum
is minimized when the dot products are negative, not when they're zero. This line:
print(torch.max(torch.abs(dot_products - torch.eye(N, N))))

is not actually what you're optimizing. The loss is approximately -1.5, (the sum of the elements of dot_products.triu(diagonal=1)), not 0.5.
The fix is to make sure that your the function is minimized when the vectors are orthogonal, for example:
def pairwise_dot_norm(vecs):
    dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
    return torch.linalg.matrix_norm(dot_products.triu(diagonal=1))  # upper triangular part only

p.s. There is a known, deterministic way to generate a set of N orthogonal vectors that doesn't require gradient descent: https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process


text:
Loss function not able to capture the maxima of probability distribution
I am trying to predict noise (random gaussian) with the help of a neural network. I am implementing a L2 loss (torch.nn.function.mse_loss) for computing the loss function between the prediction distribution and input distribution (random gaussian). Any suggestions on what I may do to correct this?
1. For epoch 1 : Train loss 1.0933226346969604

2. For epoch 200 : Train loss 1.000555157661438

3. For epoch 400 : Train loss: 1.174836277961731

Gaussian noise is usually 0-mean, meaning that it cancels out on average especially with a squared loss (e.g. MSE): so your approach won't work.
I think that you should instead predict the mean and variance of such noise distribution: in such a way you could handle noises that are not 0-centered too!
In practice, I guess you can get inspiration from variational autoencoders. I mean, you set-up a neural net to predict the mean and log-variance. Then the loss function will be the KL divergence between the predicted Gaussian distribution (the one parametrized by mean and exp(log_var)) and the Gaussian prior, which corresponds to the distribution that describes the noise: $\mathcal{N}(0, 1)$, in your case.
In principle, with such approach you can learn also non-Gaussian noise distributions.


text:
Possible to use GPT for specific set of documents?
I have a 100+ PDF documents regarding a company's policies, procedures and guidelines etc.
Is there an AI tool that was trained in general understanding of language, that I can feed all those PDFs and make my own model where I can ask questions about the company's rules and procedures specifics, find relevant pieces of information and get answers based on the documents provided?
I know I can feed a PDF to an AI and summarize it, but I'm looking for a permanent solution, kind of like a database but where instead of standard search I could have a conversation with AI?

You can fine-tune a model with your own documents: https://platform.openai.com/docs/guides/fine-tuning
You can also fine tune other models, like if you needed one that ran locally.
Alternatively, you could do it in stages of prompts. Fed the model the documents and have it create a list of tags for topics in each document, then include that in your initial prompt. So when you ask a question the first time, the prompt context includes the generated document tags, and you're actually asking "what documents would be useful to answer this question?", then take that response, get the documents, send another prompt "with these documents, answer this question", then return that answer to the user.


text:
What is the impact of the initialization of weights in the performance of a neural network in machine learning?
In my own experience, weight initialization matters for model convergence.

Theoretically, can different weight initialization methods eventually converge to the same optimal solution? Are their weight initialization methods theoretically proven to be worse than others?
Is there research into what weight initialization methods work best across the board for different kinds of architectures?

Progress about how to best initialize the weights, is what has made neural networks to be popular again.
Initially (around the 80s I think), NNs were initialized from Normal distributions like $\mathcal{N}(0, I)$, but that caused unstable training prone to divergence. Also, initializing the weights (excluding the biases) to be constant is something you would NEVER do because the weights with same values are updated in the exactly the same way: so these will be just redundant, not helping the network to learn. Therefore, you want to pick random weights, but how?

Are there weight initialization methods theoretically proven to be worse than others?

Is there research into what weight initialization methods work best across the board for different kinds of architectures?

Consider that the choice of which weight initialization scheme to utilize should be made according to what activation function to use in your model, and in some cases is also related to the network architecture. So, weight init and activation are strictly related.

This paper introduces what is now called the Glorot (or Xavier) initialization, also motivating why NNs initialized in the classical (thus wrong) way fail. In few words, the Glorot initialization is designed to work well with saturating activations like the sigmoid and tanh, thus preventing them to saturate (which leads to vanishing gradients) early in training.
This more recent paper, instead, introduces the He initialization strategy that is designed to work with the rectified activations (ReLU, leaky ReLU, etc), thus reducing the chances of dead units. Indeed, this weight init is also demonstrated along the ResNet architectures.

Indeed, this does not necessarily mean that e.g. using Glorot with ReLU is always bad. But these two papers provide an analysis of the variance of the weights with different initialization strategies, being designed to keep that in control.
A note about pre-training (also related to transfer learning): strictly speaking this is not a weight initialization strategy. Indeed, you can pick a popular architecture and download the weights pre-trained on ImageNet (for example); that would give a nice initial point, useful to speed-up convergence, down-steams tasks, and even when having few training data. I consider pre-training to be a second step, because the first time you have to pre-train the model yourself and so starting from a proper weight initialization is beneficial also for pre-training alone.

For the constant init, see also https://ai.stackexchange.com/q/6789/2444. Btw, I think it's "Xavier" (I suspect this is just a typo).

Weight initialization can and often does matter, hence why pre-training language models are useful for downstream tasks.
A randomly initialized model is not guaranteed to converge, especially if the model is training on some downstream task with limited data. Furthermore, there is no best practice initialization for weights, but people tend to fall into two camps. Initialize from zero or from some random distribution. However, de-noising objectives can help improve initial weights for downstream tasks.
Example of benefits of pre-training:
I pre-trained a model on 97M examples of molecules formatted in a particular molecular language called SMILES, and then trained the model on reaction prediction. Without pre-training, accuracy after just one epoch was .284%; with pre-training, it was 51%. This demonstrates that non-random initialized weights are preferable (in this case), to random ones.


text:
What would the receptive field of a neuron with a 2x2 max pool, stride of 3?
Bit confused about the concept of a receptive field. I understand it to be the size of the region which influences the output of the neuron of interest. If I have an input of 16x16, and immediately pool afterwards (for simplicity) with a 2x2 window and a stride of 3, does the receptive field of an output neuron end up being 3x3? Or does it remain 2x2, since we do not draw upon any useful info with the gaps between the windows?

The receptive field of a neuron at a given depth (layer) is the number of input pixels that affect its output.
For example if you apply a $K\times K$ kernel at each layer, the receptive field of an element in the $L$-th activation would have a size of
$$(1 + L(K-1))\times (1+L(K-1))$$
Now if you introduce a stride $S$ each time, the size of the receptive field is approximately divided by $S$ (you need to take into account padding, and rounding...)
In your specific example, you have a $2\times 2$ kernel (it doesn't matter whether the layer is a pooling or a conv) with $S=3$ as stride. This means that you discard $5$ pixels at each slide of the kernel: basically you have $3\times 3=9$ pixels each time but just consider $2\times 2=4$ of them, so five got lost.
So, at the input level, the receptive field of the pooling will still be $2\times 2$ but the large stride would have the effect of reducing the receptive field of the next layer(s).

If I have a pooling layer with 2x2, stride=3, and then I have a conv layer with a 3x3 kernel, and I ask what is the receptive field of the output neurons, shouldn't it be larger? I suppose I get confused because I am also thinking about dilation, and if I understand correctly, that also can expand the receptive field but throws away information. Can you elaborate further on the part where you say "the large stride would have the effect of reducing the receptive field of the next layer(s)."

Well the conv after the 2x2 pool won't never "see" the pixels that were discarded due to the stride of 3, so its output neurons aren't influenced by those and so you don't include them in the receptive field. Dilation is something different from the stride: it expands the kernel (enlarging the receptive field), instead the stride shifts it entirely by some amount.


text:
Is it effective to use images without objects in object detection?
I am currently using faster_rcnn to train a set of images with only one category, in fact, there are only good images and images with defect in the whole dataset,and I use roboflow to labeling this datasets,
My problem is that when I encounter a good image, I just mark this image as null, but I don't know if it is effective to put it into the model for training.
Should I frame the entire picture into a new category(two categories in total), or remaining mark those image as null?

Hi @Kekai and welcome to AI Stack Exchange! It would be helpful to add more details to this question to help identify the details of your problem. Thank you for posting, and we look forward to more of your questions on this site!

Marking these images as null is the correct thing to do. Object detection models will learn not to predict unlabeled areas (background area) through their loss function.
It can sometimes be useful to add negative annotations if there’s a particularly subtle difference. For example “screw” and “bolt” — this will give the model more information to go off of, let the “objectness” part of the model so it’s thing, and the classification part do its thing also.
By the way, be sure not to add too many null examples to your dataset or it might learn the optimal strategy to optimize its score is “predict nothing”. If this is happening you might want to use the “filter  null” preprocessing step in Roboflow to adjust the ratio.


text:
Why use ReLU over Leaky ReLU?
From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive.
With just this info to go off of, it would seem that the leaky ReLU is just an overall improvement to the standard ReLU, yet ReLU still seems to be the gold standard of activation functions. Why is that?
Does the additional sparsity outweigh the value gained from negative local gradients? If so, is tanh just too sparse?

Can I ask what you mean by sparsity?

@JobHunter69 A sparse activation function is more strict when it comes to letting non-zero activations through than a non-sparse function. This is beneficial b/c it leads to a more generalizable model that "focuses" on only the most distinguishing features of the data. Read Deep Sparse Rectifier Neural Networks by Bengio et al. for a more thorough explanation of the topic.

Your understanding or Leaky ReLU is correct, and, yes, it has been proposed to mitigate the dying neurons issue in ReLU: when these are negative, they got zeroed.
Regarding the answer of @Regresslt:

I would not say that leaky relu slows done training due to its computation overhead, since it's quite simple to implement. Instead, ELU (exponential linear unit) is quite expensive and that can increase training time.
Also it's true that the slope parameter (usually called $\alpha$), if mistuned can lead to worst performance, but one can try the PReLU that learns it.

Does the additional sparsity outweigh the value gained from negative local gradients?

That's an interesting. I mean the dying ReLU phenomenon is usually observed on a fraction of the units, not the totality. In my personal experience I've noticed (dense) layers with ReLU to be sparse as much as $50$ to $90\%$. In general this causes an under utilization of the capacity of the model, which may lead to under-fitting. But from another perspective can be seen as implicit $l_1$-regularization applied on each layer with ReLU, that is optimized along the model which may lead to better generalization performance: simply because the model learned to achieve the right capacity by itself. Also, if such large sparsity is achieved the use of Dropout on top can be totally non-beneficial, because it would drop also the remaining "active" units/neurons.

Is tanh just too sparse?

First, $\tanh$ is not sparse since the output values can be either $-1$ or $1$ when saturation is reached, which is its main issue that shares with sigmoid (that instead can lead to sparsity.)  When tanh or sigmoid saturate you get vanishing gradients, and the training stops. Instead, the sparsity induced by ReLU does not reduce the gradient, since it will only be multiplied by some zero elements (corresponding to the died units.): the gradient magnitude depends on the overall output of the layer. In practice (but I'd say it's hard to happen), only if all units of all layers are died your network won't learn anymore, and the gradient would vanish because each layers would only output zeroes.
To conclude, ReLU is usually the first activation to try - it seems to always work sufficiently well - but remember also to pair the activation fn with the right choice of weight initialization, in order to prevent issues (dying units, saturation, etc), speed-up convergence, and even increase performance.


text:
Is there a correct order of "conv2d", "batchnorm2d", "ReLU/LeakyReLU", "MaxPool2d" for UNet like architectures?
Context
I'm investigating the UNet architecture for a little while now. After investigating the structure of the official UNet architecture as proposed in the official paper I noticed a recurrent pattern of Conv2d->BatchNorm2d->ReLU(->MaxPool2d)->Conv2d->BatchNorm2d->ReLU(->MaxPool2d) for the encoder part but I have also came across other implementations of a custom UNet where this order is different like Conv2d->BatchNorm2d->ReLU(->MaxPool2d)->BatchNorm2d->Conv2d->ReLU(->MaxPool2D) where the operation in the paranthesis is considered optional.
I've tried training both models and they both work reasonably well but I still want to understand the following:
Question

Is there a "correct" order for these operations for the encoder/decoder part of the UNet architecture?
What is the intuition behind shifting the order of these operations in the model?

I suggest to follow the official U-NET implementation.
To me, the second option Conv2d -> BatchNorm2d -> ReLU (-> MaxPool2d) -> BatchNorm2d -> Conv2d -> ReLU (-> MaxPool2D) seems more a mistake that an alternative:

The part -> BatchNorm2d -> ReLU (-> MaxPool2d) -> BatchNorm2d -> doesn't make much sense: the first BN already centers the activation then max-pool takes only the max that doesn't introduce any substantial change in distribution, so the second BN in my opinion is not that much necessary.
The following part -> Conv2d -> ReLU (-> MaxPool2D), now, lacks the BN operation which should be put in between conv2d and ReLU.

So I suggest you to follow the Conv2d -> BN -> ReLU (-> MaxPool) design. At the best you can swap BN with the activation (i.e. conv -> relu -> BN): there is a debate about this regarding BN, some people think is better some other that's equivalent..

Is there a "correct" order for these operations for the encoder/decoder part of the UNet architecture?

The correct order, is the first option you specified.

What is the intuition behind shifting the order of these operations in the model?

At most it can be motivate by re-normalizing again (by means of BN) the activation or even the max-pool's output. But if so, there are two missing BNs: after the second ReLU and max-pool.


text:
Which face filter algorithms can work on CPU or integrated GPU?
I see many realtime face swap filters and appearance enhancement filters on smartphone apps. Even apps that can make you look like a granny or show you having a frown, no matter what your actual facial expression is.
On searching for open source algorithms/code to apply such effects to my desktop PC's webcam, all I find are hugely resource-heavy programs that require a dedicated GPU or TPU.
May I know how smartphone apps process face filters? If they send the data to a server for processing, how does it work in realtime? Or do they use scaled-down CNN models or GAN that can run on the phone's processor? If so, are there lightweight models or algorithms that can run on the desktop PC's CPU or integrated GPU (APU)?

Face filters works by first detecting and localizing the face, then predicting the so called facial landmarks (a set of points that depict the geometry of the face, like its contour, shape of eyes, nose, mouth, ecc), and lastly applying the filter potentially yield by some generative model.
These are all heavy work. For example, you can have a pipeline that first detects a face, if no faces you stop the processing. Otherwise you can have a multi-task model that yields the face bounding-box and landmarks directly, so improving the runtime because you reuse most of the model for both predictions. However, the part related for the filters is probably always separated, moreover you have to deal with computer graphics to render the filter and also apply some nice FX.
What I want to say is that both solutions are possible, and even a combination of them.

Many camera software (especially on mobile) have a very fast implementation of face detection (probably an improvement of Viola-Jones method). So once you detect a face, you can send that frame on the cloud for fast processing. Once you have the landmarks and, I guess, also the correct geometry of the filter, to enable real-time effects usually tracking is used. There exist very efficient tracking algorithms that rely un classical computer vision, that can either track objects and even landmarks. Basically you run the expensive part (or cloud processing) one frame every $N$, and for the other you use only tracking.

Scaled-down CNN models or GAN, are also possible. There exist optimization frameworks like tf-lite that allow you to compress (by quantization and pruning), optimize, and even deploy a DL model. Indeed, you need to care about specific optimizations for the CPU and GPU.

If you plan to implement something similar on your on PC, you can use OpenCV (either with Python or C++) for, e.g., tracking, then compress a DL model with tf-lite in case you use tensorflow and/or Keras: you can take a pre-trained model, and compress with such tool. In that way, you can target optimized CPU and GPU models. For the graphics part you should find some optimized library for GPU, or write the code yourself like in OpenGL or similar. You can also have a look at Dlib: it's a nice lib, that has many image processing, computer vision, ML implementations, and even pre-trained models.

Thank you Luca. By "one frame every N", I assume you mean "one frame every N number of seconds"? So the tracking algorithms run on the phone to track facial features. Any such algorithm names you remember?

To achieve real-time processing you should process at least 24 image frames in a second (say up to 60 for best smoothness). It's not necessary to either gather more of them, so you should capture camera images on a fixed interval basis. Assuming that, you do all the heavy work at frame 1, then for the next $N$ frames (e.g. 10) you do tracking ($N$ should be not too large otherwise you loose accuracy). At frame $N+1$ you do the heavy work again, and so on. For tracking you can start from the Lucas-Kanade method.


text:
Is it true that a multi-armed-bandit selects an action based on the future reward?
During my machine learning lecture I saw the following statement on the slides: "Multi-armed-bandit selects an action based on the future reward"
Is that statement true?
In my opinion it is false because the selection of the action is not based on the reward directly as it is not known but on some strategy to maximize the reward in the long run.
Could maybe someone provide some references if it true or not, as I could not find any.

If you changed this:

"Multi-armed-bandit selects an action based on the future reward"

To this

"Multi-armed-bandit selects an action based on the expected future reward"

The description would be better in my opinion. Perhaps expected is just missing on the slide for brevity. I would rate the original quote as mostly true and give the author the benefit of the doubt here.
However, the quote is out of context, and how a multi-armed bandit agent selects an action can be more complex than simply considering the expected future reward. In bandit agents, the agent is usually considered to be learning "live", and has to balance exploring for better options versus exploiting a known best option so far. This is commonly referred to as minimising regret - which technically is a measure of how the agent performs in expectation (using actual action choices, but mean rewards for those actions as opposed to observed rewards) against an imaginary version of itself that always chose the best action in hindsight (or by revealing hidden knowledge of true values in the end for test cases).
If you include this training behaviour in the description, then you could also say:

"Multi-armed-bandit selects an action based on minimising regret"

Although this is not technically excluding basing on expected future reward, since usually the estimates of action values are used to make a decision, and "based on" is not the same as "only considering".


text:
Which preprocessing is the correct way to forecast time-series data using LSTM?
I just started to study time-series forecasting using RNN.
I have a few months of time series data that was an hour unit.
The data is a kind of percentage value of my little experiment and I would like to forecast the future condition of this.
Many tutorials and web info introduced direct training and forecasting the time series data without any data pre-processing.
But for the RNN (or ML and DL), I think we should consider the data's condition that is stationary or not.
My data is totally random condition which is stationary data (no seasonality, no trend).
For example, the US stock prediction tutorial showed super great accuracy forecasting performance according to many LSTM tutorials.
[If this really works and is true, then all ML developers will be rich.]
And, Some of them didn't emphasize and note a kind of the data pre-processing such as non-stationary to stationary something like that.
According to my short knowledge, I think the non-stationary data such as stock price (will have trend) should be converted as a stationary format through differencing or some other steps. and I think this is a correct prediction as a view of theoretical sense even if the accuracy is not high.
So my point is, I'm a bit confused about whether that really is no need for any preprocessing to treat stationary or not.
For my case, I applied differencing step ($t_n - t_{n-1}$)to my time-series data in order to remove the trend or some periodic situation.
Is my understanding not correct?
Why do time-series forecasting tutorials not introduced data stationarity?

You stated that your data is 'totally random'. How do you know it is 'totally random'? Truly random time series data can not be forecasted. Stock price data is not random although it has some degree of randomness.

You mentioned stock data and 'future condition'. That is very unclear. Are you trying to predict the future price of a stock given historical stock price data?

You mentioned that a tutorial shows high forecasting accuracy. Please cite the tutorial. You pointed out that ML developers would be wealthy if high accuracy was possible. The fact is, practically all tutorials and papers do not report a valid time series forecasting metric! RMSE and other metrics that lose the directional accuracy of the forecast are useless for forecasting direction error. Another important fact is that LSTMs alone do not provide an accuracy suitable for forecasting stock prices for profit for long time regimes.

Sorry for the unclear post. Actually, My data is a CPU Usage history from my lab top. This was a kind of practice work. According to my data, I found It seems no periodic situation or trend. So that is the reason why I said 'My data is totally random' but I think I misunderstood the random data such as stationary. the stock is one example to explain my question.

Yes. Basically, I would like to predict future CPU Usage using CPU usage history as same as stock price prediction. In this step, not only 1 unit prediction a period prediction I want to do.

I saw a few tutorial blogs and webpage that introduced 'LSTM for the time-series data'. most of the page is not written English (I'm sorry) and here is one of a page that I saw before. https://towardsdatascience.com/aifortrading-2edd6fac689d

A standard method for pre-processing time series data for neural network architectures, such as an LSTM, is to normalize the data. Good tutorials will include this step. There are several variations to normalizing time series data. Jason Brownlee writes a lot of good stuff on machine learning. Please see his article on How to Scale Data for Long Short-Term Memory Networks in Python.
Also, see the replies/answers to: https://stackoverflow.com/questions/43467597/should-i-normalize-my-features-before-throwing-them-into-rnn
The following is a tutorial on predicting stock prices, employing normalization, using an LSTM: Time-Series Forecasting: Predicting Stock Prices Using An LSTM Model

Thank you for your reply. It was a great helps to understand the LSTM using more reasonable data and model structure. I will carefully read the article that you recommended.


text:
At what text-based tasks are "dumb humans" still better than the best language models?
I ran into this AI-SE question from 5 years ago and I believe that an updated version could be interesting to discuss nowadays: Is the smartest robot more clever than the stupidest human?
Today's best LLMs are displaying a lot of human-like abilities: proficiency in natural languages, ability to code, logical reasoning, role playing and so on. They can even solve CAPTCHAs, design games, answer questions about stories or write new ones: these were the "shortcomings of robots" in 2018, according to the answers to the question I linked.
Question
How do the best LLMs of today compare to a "dumb human"? In what tasks are all normal humans still better than AIs? Is there any test that every able-bodied human would pass, but top LLMs would still fail?
Definitions and clarifications
A "dumb human" is a person without recognized disabilities or obvious problems, who doesn't have particular skills and who is considered not very intelligent (low IQ).
Of course the LLMs available to the public have a number of objective limitations: they can only process text-to-text, they work with tokens rather than characters, context length is just few kilo-tokens, and they have no long-term memory. However a number of open source projects have shown various solutions to these problems, and the non-public version of the commercial LLMs already support much larger context windows, image input and similar features. Observations like "LLMs can't move arms as they don't have it", "LLMs fail to count characters because they're token based", "LLMs can't speak nor listen to speech" are not interesting.

Looks like a duplicate of https://ai.stackexchange.com/questions/7021/is-the-smartest-robot-more-clever-than-the-stupidest-human?noredirect=1&lq=1. If it isn't please explain why not.

@BruceAdams: that question is about general capacities of robots in 2018 and the answers say those robots aren't able to solve CAPTCHAs and write stories. This one is about text-based tasks of LLMs in 2023.

LLMs seem to be limited at "compositional tasks." Have a look at this paper, in which the authors

investigate the limits of these models across three representative compositional tasks—multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer.

I don't know if a "dumb human" can do dynamic programming problems, but

humans can solve 3-digit by 3-digit multiplication arithmetic after learning basic calculation rules. Yet, off-the-shelf ChatGPT and GPT4 achieve only 55% and 59% accuracies on this task, respectively.

I gave ChatGPT two tries (what is "311 times 877") and ("what is 513 times 799"), and it got them both wrong.
Another example from that paper is a "zebra" or "einstein" puzzle. Look at page 18 in the paper for an example. (I'll copy it below) It took me a couple minutes to solve. ChatGPT tries its best, but gets it quite wrong.
the puzzle
There are 3 houses (numbered 1 on the left, 3 on the right). Each has a different person in them. They have different characteristics:

Each person has a unique name: peter, eric, arnold
People have different favorite sports: soccer, tennis, basketball
People own different car models: tesla, ford, camry

clues:

The person who owns a Ford is the person who loves tennis.
Arnold is in the third house.
The person who owns a Camry is directly left of the person who owns a Ford.
Eric is the person who owns a Camry.
The person who loves basketball is Eric.
The person who loves tennis and the person who loves soccer are next to each other.

Match the person, favorite sports, and car models to each house.
ChatGPTs output (for me just now)
GPT's full output
<omitting the output where it "thinks">

Therefore, the matching of people, favorite sports, and car models to each house is as follows:

House 1:

    Person: Peter
    Favorite sport: Tennis
    Car model: Tesla

House 2:

    Person: Eric
    Favorite sport: Basketball
    Car model: Ford

House 3:

    Person: Arnold
    Favorite sport: Soccer
    Car model: Camry

and its easy to see that some of the constraints are violated.  (e.g. the first one:  Ford goes with Tennis)

Interesting to note that you've only found weaknesses in LMMs where traditional AI would have no problem.

Here is a paper on one of several ongoing attempts to marry the two https://arxiv.org/abs/2305.12295

@bogovicj: GPT-4 was able to compute "311 times 877" on the first attempt: https://chat.openai.com/share/bca563aa-09ae-44e4-aea2-1463f7d3ce77 .
It failed to compute "513 times 799", but it succeeds on other attempts or if you ask it to break the computation in even smaller steps.
Similarly, it can solve the puzzle, if you ask it to come up with a formal way to model the problem, then solve it step by step.
Would a "dumb human" be able to solve these problems more reliably than GPT-4? I tend to believe that today's top LLMs are better than low IQ individual at these tasks.

Actually, I looked again at the computation of 311877. There's a mistake in there ("273200 is 3118", when it should be "248800"). In spite of that the result is surprisingly correct (how?!). Nevertheless, it should be able to solve these problems if you let it "reason" on it enough, through its output.

@BruceAdams that looks like an interesting paper, thanks for sharing.  Just want to clarify some small things: 1) I didn't those weaknesses, its the papers' authors that study them. 2) Those are the "only" ones in that paper, which is not an exhaustive list, of course.  3) One doesn't need "AI" to do multiplication or satisfy constraints; regular old comp sci can do it.

@BlueNebula, The point is not that gpt-4 can't do multiplication.  The point is that its not good at multiplication and being wrong 40% of the time seems "not good" to me. I'm not surprised that GPT will give different results on different runs since it is not deterministic, so of course it will be right sometimes. GPTs job is text completion, not multiplication, and you can do multiplication using text completion. Your example suggests to me that it does not know the algorithm for multiplication and happened on the right answer that time in some other (wrong / unreliable) way.

Contraint satisfaction like many problems before and since it were considered AI problems until they became relabelled engineering. Multiplication was not.

Humans (any human) are still better at one of the most important task (whether it is a text-based task or not).... Humans can give you their own opinion or "sentiment" about a text. This means any human can tell you if he/she likes the text or not. An LLM don´t have any kind of own "personality", "opinion" or "feelings" so it can not give you its own opinion but a "general" sentiment (which is generally mentioned as Sentiment Analysis in NLP tasks) based on all the training data used by the bot.

This is tricky. You can definitely tell an LLM to roleplay as a certain personality or character, and it's opinion as observed from the output would meet criteria of having a specific self-consistent view. You can then get into the weeds about whether such output is a real measure of something internal, or more like text puppetry. IMO this gets into "it has no arms" territory - so what if the raw model has no personality, if it can realistically assume a personality on demand? At least when we're talking about input/output capability

@RaulAlvarez: can you come up with a test to verify what you said? LLMs are able to share a sentiment or opinionated impressions about a text. To get commercial LLMs that have been through RLHF to do this, you need to ask them to role play. But LLMs pre-RLHF do it spontaneously.

Maybe you can ask any LLM if they like a chocolate ice cream or a strawberry ice cream. Same about a text. You can ask him if he prefers Real MAdrid to Barcelona, red or blue color.... he will not give you his opinion... nor his preference.

@RaulAlvarez The LLM will give an answer though, and it will typically be a popular one, plus if challenged on why it made the choice, it will be able to explain. When it does this, you have to perform extra tests, or make assumptions about the internals of the model in order to back up your claim that the model doesn't really have a preference or an opinion. IMO this comes under the OP's request not to reference things that the model does not have . I.e. not having a true opinion is the same as not having arms to wave


text:
Yolov8 with key points detection without object detection
I have a project where the data is annotated with key points without any bounding boxes. If I want only the key points without the bounding box around the object, is that possible or each object should have both bbox and key points. Can I train Yolov8 with key points detection without object detection (bounding boxes)?
If the answer is yes, how the annotation file should look like, please give me an example.

According to this github issue, it's not yet possible to train the model without any bounding box information unless you make changes to the architecture, and evaluation metrics. So the simple answer is "No, you can't train the model without box info".

that was me who asked the question :)


text:
How does Conv2Plus1D reduces the number of paramateres?
Based on this tutorial, and the

An advantage of this approach is that factorizing the convolutions into spatial and temporal dimensions saves parameters.

statement, the Conv2Plus1D must have fewer parameters than a Conv3D layer. I compared the number of parameters between a Conv2Plus1D and a Conv3D layer and the Conv3D layer had much fewer parameters (around 1/3 of the Conv2Plus1D) than a Conv3D layer.
Am I missing something? Or am I misunderstanding something?
Please clarify if you can.

By reading the paper that proposed the idea of (2+1)Convolutions, it seems that the Conv2Plus1D implementation of the tutorial is partially incorrect.
By citing the paper (section 3.5):

[...] full 3D convolutions may be more conveniently approximated by a 2D convolution followed by a 1D convolution, decomposing spatial and temporal modeling into two separate steps.

So they assume a 3D conv, with an $N\times t\times d\times d$ kernel (where $t$ is the timesteps, and $d$ the depth), can be "factorized" into space and time components by two convolutions: one is $N\times 1\times d \times d$ with $M$ filters, and the other has a $M\times t\times \times 1$ kernel with $N$ filters. The former is the spatial convolution, and the latter is the temporal one.
Now:

The hyperparameter $M$ determines the dimensionality of the intermediate subspace.

They provide a formula to compute $M$:
$$M_i = \frac{td^2 N_{i-1}N_i}{d^2 N_{i-1} + tN_i},$$
where $i$ is the layer index. Basically, with such a formula they choose $M$ so that:

[...] the number of parameters in the (2+1)D block is approximately equal to that implementing full 3D convolution.

In short, the first conv in Conv2Plus1D should have $M_i$ filters, and the second conv $N_i$ filters (i.e., the number you would use for the regular 3D conv.). Indeed, can be that $M_i = N_i$ as in the tutorial but that won't follow the paper. Now, if you compute $M_i$ according to the eq. you'll get approximately the same overall number of parameters compared to a full 3D conv. But you can set it as you wish, thus also reducing the #parameters if you want to do so.

Appreciate the time. I also thought the same thing as I read the paper but because the tutorial was on TensorFlow's own website, I got the sense that I might be missing something. Anyways, thanks for the help.