text:
Which type of neural network to use to classify data by which equation most likely generated it?
Problem Summary: Identify which equation a set of data was most likely generated from
Problem Description: Let's say I have two different equations that are functions of variables X and Y and parameters A and B.

Class 1: $Z = f(X,Y,A,B)$
Class 2: $Z = g(X,Y,A,B)$

Now lets say I solve each equation 10k times using provided X and Y values, but randomized parameter values. This will generate two matrices with columns X, Y, and Z and 10k rows which I will now refer to as "snapshots". Below is an example of a visualization with the color representing the Z value.
NOTE: I'm not interested in just building a CNN to classifying the snapshot visualizations below because I hope to generalize this problem to equations with many more variables so they cant really be visualized easily.

My goal is to train a neural net using these snapshots (raw data, not visualizations) that are labeled with the equation that generated them.
However, I'm not really sure which type of neural net I should use. I could perhaps take a simple approach with a few fully-connected layers and the input would be the stretched out snapshot (so 30k long vector). Or maybe I need to use a CNN?
My primary concern is that unlike with an image, the features of these snapshots are kind of meaningless. For example, in a normal image each feature refers a specific pixel. But with these snapshots, each row is just a random simulation of the equation.

My Question:
Does anyone have a recommendation on which type of neural net to use for this problem? Perhaps recommendations on useful abstractions/transformations of the raw data I could use as features? Any online resources (github/kaggle notebooks, academic papers) that have investigated a similar question would be extremely helpful.

EDIT:
Some additional thoughts. Perhaps I need to sort the snapshot based on the Z values so now the rows have meaning (largest output, 2nd largest output, etc).

Any neural network might be able to find some pattern (if there is one), provided adequate data. But you can always optimize with right assumptions.
For instance, there might not be always a relation between visual representation of the snapshot and the function family (many families can span similar space).
So, using a CNN might not be a good idea. The data has no temporal relation, so RNNs are out as well.
An adequately large FNN and data will give you good results. You can later prune it and optimize.

according to the question you have 2 classes. the samples from the first class are generated according to f and the samples from the second class are generated according to function g which is different from f, i.e the points are simulated according to different dynamics.
Now we need to train a classifier to return the conditional probability of $P(Class|X,Y,Z,A,B)$.
depending on how non-linear this classification problem is, You can simply start with a simple fully connected network (lets say with 3 to 5 layers)
and obviously the input layer has 5 node and hidden layers arbitrary size (shown with h_i) and the output layer should have two nodes (because you want to assign a score to 2 classes).
here is the number and size of the layers
Layers: [5,h_1,h_2,...,h_last,2]
Then you can train your network with CrossEntropy loss (https://medium.com/swlh/cross-entropy-loss-in-pytorch-c010faf97bab)
Once you got the first result, I would play with the the number of hidden layers (m) and size of each layer (h_i). If still the result is not good enough then you can think of more complex models such transformers -the backbone of large language models- (https://arxiv.org/pdf/1706.03762) which is more powerfull.
I hope I gave you some ideas how to proceed.


text:
Why might my policy gradient agent appear to maximize the absolute value of rewards?
I have a toy policy gradient RL algorithm using REINFORCE (aka monte carlo policy gradients) that involves bots moving on a grid attempting to "acquire" targets in Pytorch. The bots receive +1 for moving closer to targets, -1 for moving farther away/invalid actions, some larger amount for acquiring a target (a rare action), and a smaller penalty for inaction.
What I have observed is that the agents swiftly learn to attempt moving off the grid. In doing so, the bot has learned to ironically maximize the negative reward.
So I tried swapping the sign of all the rewards, and yet the behavior persists, hence my belief that it's almost learning on the absolute value of the reward. It's also possible the network is just randomly converging to one action, but it's unclear how this is the case if so.
Why might this be the case?
My code is long and the environment self-defined, but I believe the relevant parts should be my model action choice and the reinforce part. Rewards and movement themselves seem fine as, if I comment out the reinforcement updated, rewards and movement are applied as expected to the random outcomes of the network.
Here's the model output part:
    def forward(self, x):

      # Run through all of our layers defined above
      x = #...removed for brevity
      x = F.softmax(self.linear_final(x), dim=1)

      return x

    def decide_action(self, x):

      # Create the prob distribution from output and return the action/logprob
      probs = self.forward(x)
      prob_dist = Categorical(probs)
      action = prob_dist.sample()
      action_item = action.item()
      logp = prob_dist.log_prob(action)

      return action_item, logp

Here's the reinforce part:
# Function to calculate loss and update bot network
def reinforce_bot(b, debug):

    # Setup lists and vars to work off
    discounted_reward = 0
    l_returns = []
    l_policy_loss = []

    # Work through the bot's episode rewards backwards
    # The net effect of this will be such that we built rewards for only actions and their following rewards
    # (i.e. action for step n only gets rewards for steps > n, never steps < n)
    # Additionally we'll build in our reward discounting (where future steps contribute less to overall reward)
    for reward in b.l_episode_rewards[::-1]:
        discounted_reward = reward + gamma * discounted_reward
        l_returns.insert(0, discounted_reward) # but insert back at the beginning to get correct order

    # Now turn the rewards into a tensor for working with gradient
    t_returns = torch.tensor(l_returns)
    # But standardize the rewards to stabilize training
    t_returns = (t_returns - t_returns.mean()) / (t_returns.std())

    # Now build up our actual policy loss by multiplying it by our logprobs
    for logp, discounted_reward in zip(b.l_episode_log_probs, l_returns):
        l_policy_loss.append(-logp * discounted_reward)

    # Zero our gradient to get ready for backprop
    b.optimizer.zero_grad()

    # Technically our l_policy_loss is a list of tensors, so smoosh those together
    # Then sum to get the total loss
    policy_loss = torch.cat(l_policy_loss).sum()

    # Now run our optimizer
    policy_loss.backward()
    b.optimizer.step()

    # Text to print some helpful debugging
    if debug:
        print(f'{b.team} had awards array of {b.l_episode_rewards}')
        print(f'{b.team} had l_returns of {l_returns}')
        #print(f'{b.team} had l_policy_loss of {l_policy_loss}')
        print(f'{b.team} had a policy loss of  {policy_loss}')

    # And cleanup our epsiode tracking lists now since we don't need them
    del b.l_episode_rewards[:]
    del b.l_episode_log_probs[:]

Perhaps the implementation of REINFORCE is not properly handling the mix of positive and negative rewards?

I've found at least part of the problem -- quite simply the learning rate was too high at 0.01. My guess is that, since rewards are fairly common, the agent would occasionally find cases where heading in a certain direction constantly gives it a reward, the gradient explodes, and the agent is unable to recover. The seen behavior then is eventually an agent that, in a new context, is repeating a move that reaps negative rewards (but is unable to properly update it's way out of it).
I also implemented a different version of reward discounting that looks more like reward * gamma ** i which causes the rewards to exponentially decay over the array. Convergence with this was much quicker and more stable, but it's still unclear to me if Pytorch's official example has issues or what is really appropriate for rewards processing.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Should the concept of discounted rewards result in multiple arrays per episode in RL?
Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below:
Say you have a reward array of length n and a discount hyperparam gamma. You might calculate the discount factors like:
gamma^i for i in range(n)
ex: [0.99^0, 0.99^1, ... 0.99^n]
Then get the discounted rewards by multiplying each discount factor by the corresponding reward and doing a cumulative sum.
However, this results in an array of the same length as rewards, but this seems incorrect to me?
I understand discounted rewards as a way to deal with the uncertainty of past actions on future awards. It would seem to me that uncertainty should result in multiple different contexts and then multiple different arrays with their own discounts.
For instance, say I have 4 states with 4 rewards that looks like [2, 3, 1, 3]. It would seem to me I should then have 4 reward arrays:
[2, 3, 1, 3]
[3, 1, 3]
[1, 3]
[3]
and I should have 4 discount factor arrays also:
[0.99^0, 0.99^1, 0.99^2, 0.99^3]
[0.99^0, 0.99^1, 0.99^2]
[0.99^0, 0.99^1]
[0.99^0]
because of the different contexts. For instance, in the first state reward value 2 is certain and reward value 3 slightly less so. But once we're in the second state, reward value 3 is certain and reward value 1 slightly less so.
If we don't do something like this, isn't our agent being updated on a system that always keys on the first state?

The discount factor is not used to take into account uncertainty, but to encourage the agent to have a longer term view when it takes an action. If the discount factor is close to 0, the agent is encouraged to maximise the immediate reward it gets, whereas if the discount is close to 1, the agent is encouraged to take the action which will lead to high rewards in the future too.
To see why, let's denote $G_k$ the cumulative discounted reward starting from timestep $k$ in an episode of length $T$. Then, by definition, $G_k = \sum_{k=0}^T \gamma^k r_k$.
Then, if all rewards are 1, $G_k \approx \frac{1}{1-\gamma}$. If $\gamma$ is close to 1, then $G_k$ will be large, because it will take into account all future rewards almost at their full value, whereas if $\gamma$ is close to 0, $G_k$ will be close to 1, i.e. it will take into account just the first reward
Now, in REINFORCE (according to the implementation OP mentions in their comment to this answer) at the end of each episode of length $T$, the parameter update looks like $$\theta \leftarrow \theta + \alpha \nabla_{\theta}\sum_{k=0}^T\ln (\pi(a_k | s_k, \theta))G_k$$.
In your example, at timestep 0, $G_0 = 0.99 ^0 * 2 + 0.99 * 3 + 0.99^ 2 * 1 + 0.99^3 * 3$.
Then at timestep 1, $G_1 = 0.99 ^0 * 3 + 0.99 * 1 + 0.99^ 2 * 3$.
And so on.
L71-72 of the algorithm is constructing the list $$[\ln (\pi(a_0 | s_0, \theta))G_0, \ln (\pi(a_1 | s_1, \theta))G_1, ..., \ln (\pi(a_T | s_T, \theta))G_T]$$, which then gets summed to construct the fimal loss.

I do not believe the REINFORCE algorithm updates parameters at each timestep though but rather at the end of the episode. I take most of my opinion from PyTorch's implementation here: https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py . I view the "long term" take as synonymous with "uncertainty" in that at each timestep, we want long term rewards to tie back to our preceding action, but the discount factor allows some uncertainty for the extent. My concern or confusion then comes in with REINFORCE doing one update with an array of length n timesteps.

I see, instead of updating at every time step, they calculate the loss as the sum over timesteps in the episode. I've updated my answer.

I appreciate the edit, but I feel it doesn't exclude my worry. My contention is that what is a "far future" reward at timestep 1 is not necessarily a far future reward at timestep n - 1. As I read the implementation, it does not rerun the discount factor from the perspective of teach timestep as I suggest and is recapitulated by the G_0 and G_1 notes above. By working with the rewards across the whole episode as a single array as the implementation I believe does, it must seem to me that the reward (and thereby reward * -logprobs) takes the reference frame of the first action only.

The code is a bit difficult to interpret but it's doing the thing describe in the answer.
Make an empty array $returns$ and initialise R to 0. Loop in reverse order over the rewards, set $R = r_t + \gamma R$ and prepend it to $returns$. So the first iteration of the loop does $R = r_T = G_T$ and $returns = [G_T]$, the second iteration does $R = r_{T-1} + \gamma R = r_{T-1} + \gamma r_T = G_{T-1}$ and $returns = [G_{T-1}, G_T]$, and so on, so at the end you have $returns = [G_0, G_1, ..., G_T]$.

I see it now. The implementation is indeed doing what I felt like should be the case, but I missed it in expecting a double loop (that clearly isn't necessary).

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Are there limitations on network output architecture and action mapping in reinforcement learning?
I'm easing my way into a toy reinforcement learning problem where my objects can move and take different actions on a simple grid, but I'm having trouble understanding what constraints might exist in how I build my output layer and map it to the environent. I understand that, since vanilla backpropagation doesn't make sense in this context, we use methods like Policy Updating or Q-Learning to be able to differentiate with respect to the gradient. However, it is unclear to me if these methods impose constraints on the "form" of the output layer and its mappings to actions?
For instance, in the examples I read, typically there is one output node for each of the available actions in the environment. However, in a complex environment I could see one wishing for a variety of output forms. Perhaps, in my case, I desire to have one series of nodes determine the action to take and another node (i.e. a sin output for the angle) or two (i.e. determining the x/y) to determine the direction (i.e. "grab" action at "angle/position"). Is this viable, and indeed are any arbitrary mappings from the output layer to the environment valid? Or does current reinforcement learning techniques constrain the output and mapping (such as each node must be a possible action choice)?

Answering my own question due to little traction here--- the answer is yes, there are constraints, often of the form of one output node for each action.
In order to get a gradient differentiable with the network, the network generally must output one node for each of the available actions. Broadly speaking, reinforcement learning will take the perspective of either Q-learning or Policy Gradients as the algorithm used to update the network, both of these effecting the structure and interpretation of the output. In Q-learning, each output node will essentially represent the q-value or expected value return on that action. In the Policy Gradient approach, this output layer is generally capped by a softmax function essentially converting each node to the probability of an action, making the output layer a probability distribution over the available actions. The actual action taken in this case is through sampling from that distribution (so there can be some stochasticity). There is also the possibility of flipping this idea of input = state, output = action around in some architectures so that the input is the state after a given action, and the output is a single node (representing, for instance, the q-value).
However, in either case the network output is indeed constrained in order match the design needs created by the overall algorithm type (i.e. q-learning or policy gradient or similar such flavors).


text:
What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?
I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, especially PyTorch's implementation (found here). It is clear to me now that there are superior methods to train RL agents (PPO for instance), but as I read, these feel beyond my current intellectual or time resources. As such, I'd like to eek out as much power through modifications of REINFORCE as possible before determining how I might move on.
As such, are there modifications to the REINFORCE training algorithm that might yield benefits without straying far into new algorithm territory? Or perhaps, what is the "SOTA" version of REINFORCE?
For instance, perhaps a simple gradient clip in some way approximates some of PPO's benefits? Or maybe setting a baseline reward based on a rolling reward set of previous episodes?

If a specific context is useful, I'm applying this to a self-made simple grid environment where agents receive points namely for moving closer to and acquiring "targets." There are other rewards of lesser significance, but key is the environment is set for a sort of "continuous play" such that agents are very frequently receiving rewards (due to closeness) and occasionally receiving reward spikes (due to getting a target), but there is no true episode definition other than an arbitrary timestep length. I am not using batches (perhaps that is useful?), as I have found that gradually stepping up the episode length allows the agents quicker access to simpler rewards and appears a sort of scaffolding to more complex behavior. Agents might reasonably gather the "large" rewards in as few as 10-25 steps. Generally things are working fine, and I am most interested in how to extract as much value from the agent updating mechanism (REINFORCE) as possible.

One simple improvement over the REINFORCE algorithm you've linked to is to use the advantage function instead of the normalised cumulative discounted return. The implementation is can be found in the same folder in actor_critic.py.
In your implementation of REINFORCE, the gradient of the loss is calculated as:
$$
L(\theta) = -\sum_{t=0}^T\hat{G}_t\log(\pi_\theta(a_t|s_t))
$$
with
$$\hat{G}_t = \frac{\sum_{k=0}^t r_k - \hat{r}}{\hat{\sigma_r} + \epsilon}$$
where $\hat{r}$ is the empirical mean over the trajectory, $\hat{\sigma_r}$ the empirical standard deviation, and $\epsilon$ a constant to ensure the denominator is not too close to zero.
Now, actor_critic.py replaces $\hat{G}_t$ in the loss with $\hat{G}_t - V_\phi(s_t)$, called the advantage function, where $V_\phi$ is the state value function, which is a neural network with parameters $\phi$. $V_\phi$ is learnt, so the actual loss you optimize is:
$$J(\theta,\phi) = L(\theta) + \sum_{t=0}^T\left(V_\phi(s_t) - \hat{G}_t\right)^2$$
This method is superior to your plain REINFORCE because the variance of the gradient updates is lower (which means learning is more steady and you obtain a better estimate of the optimal policy $\pi^*$).

Am I correct in understanding that, even though the actor and the critic are the same network except the last layer, that change is sufficient enough for the network to provide the state value? And event though the policy is providing the action at step t (which creates a new state t+1), the critic is providing the value for the new state (t+1)?

Yes, it's common for the value and the policy network to share most of their weights, except the final layer(s). You can play around with what's shared and what's not though, there are no rules. I'm not sure why you say that the critic is providing the value for the new state: the idea of the advantage is to tell you how much better the return you get by taking action $a_t$ is compared to the return of the "average" action (what the value function tells you)


text:
Using pre-trained models on image dataset that is totally different for object detection?
I have been trying out various tutorials on object detection machine learning. All the tutorials so far have been to use a pre-trained model for practical reasons when detecting objects that the pre-trained model learnt (e.g cats & dogs). However, will this pre-trained model work if I input a few hundred images of a particular car engine part and predict this class, which the pre-trained model did not train on? Is it recommended to make a model from scratch in this case?
I am further confused by this in TensorFlow documentation (Images -> Transfer learning and fine-tuning), the summary states:

Using a pre-trained model for feature extraction: When working with a small dataset, it is a common practice to take advantage of features learned by a model trained on a larger dataset in the same domain

By that meaning, if I need to predict a particular car engine part then this statement seems to suggest I create a model from scratch?
TLDR: Will pre-trained model be able to work on image dataset that it has never learn before or better to work on a model from scratch?

you might want to look at: https://deepai.org/machine-learning-model/text2img - this is what is produced from typing "car engine"  https://i.sstatic.net/mN8K2.jpg

Well, I think you forget about "fine tuning" stage here. What they mean in these tutorials is that you take such model that was pretrained on large dataset and you usually freeze from training all layers except the last one or few last ones and you train these last layer/layers on your smaller specific dataset. This is called "transfer learning".
So in theory model should learn more general features during that first trainin and use them without changes in the second training (fine tuning), when it learns features more specific to your problem. Only then you can use it for detection objects from your dataset. If the model has never seen a labeled part from your dataset during its training it will never come up what this is by itself that's why you need this fine tuning process. But fortunately it requires far less data and thanks to that is much faster too.
Usually you let fine tune only last layer. But if you feel that your problem is noticeably different from the images the model been trained on then you can try tuning two or three last layers.
It's not alaways easy to find a model pretrained on dataset from every possible domain so usually you just start with a model pretrained on datasets containing images of very general classes of objects such as ImageNet or COCO.
If you work on a model from scratch it would probably require a lot of data. It's easy to overfit deep learning models with just few hundreds samples.

Thanks for your reply. I tried to find information regarding freezing the last layer you mentioned but TensorFlow objection detection API does not seem to have these layers in the pipeline.config (I am using Tensorflow 2 Model Zoo). Do you have a website which clearly show this? I see there are example in Keras but I am not using Keras.

You don't freeze the last one but all previous ones except last one. I haven't used this object detection API for a long time but I am pretty sure these weights are already freezed by default. I followed these tutorials and it was working for me all fine: https://youtu.be/yqkISICHH-U and https://youtube.com/playlist?list=PLQVvvaa0QuDcNK5GeCQnxYnSSaar2tpku

@SunnyBoiz  Also if your goal is to make a good object detection model and you dont care what models you use I would recommend Ultralytic's implementation of YOLOv5 instead of Tensorflow object detection API. Much smoother and easier to setup and better results.

Besides the answer from @GKozinski, you can also fine tune all the layers of the pretrained model.
Yes, it is incredibly commonplace to do transfer learning using pretrained models from different domains.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Could you provide more reading articles on fine tuning all the layers? In object detection API of TensorFlow, I do not really see any layers.


text:
Mapping an image to a well defined object using OpenCV
I am completely new to computer vision and I am working on a small hobby project.
The goal is to use camera footage of a foosball table to map the image to already well defined object geometry with few moving parts.
To illustrate the exact problem the input could be this:

I am looking to identify all the features of the image and map them to an exact model that could be rendered like so:

It shows the position of the ball and player rods.
I would already have an exact definition of what the image contains like:

Exact size of the playing field
Exact positions of player rods relative to the length of the field
The player sizes and positions on the bar
Every detail of the table is described already in the model and can not deviate from it

Some limitations:

The image would always be at an angle. So part of the playing field is cropped and would have to be extrapolated.
The lighting can vary from one set-up to another.

There are only a few moving parts that need to pinned pointed- the position of the rods (can for now disregard the rotation) and the position of the ball.
It sounds quite complicated but given that I already can tell exactly what I am looking for maybe it can make the task easier.
What would be the most straight-forward approach to solving this problem? Any suggestions or further reading would be greatly appreciated.
My current ideas would be to maybe identify a few anchor points from the image. For example if I would identify the top edge of the playing field - where the green playing pitch ends. And If I were to then be able to identify the position of the top player rod - the goalie rod. Measuring the distance between these lines would allow me to calculate the height of the camera angle because I know the exact spatial relationship between these two features in the pre-defined table model/geometry. With this data I would be able to calculate the positions where all the other features in the image could be located. If I could narrow down areas in the picture that need to be looked at for special features of the table I could maybe add additional constraints on the things that need to be looked at and simplify the task further.
Another idea I had was using the table surface image to identify the table orientation. For instance given the above input picture I might also have a hardcoded reference picture like this:

Can this reference picture be used to find it in the input image in a skewed / cropped way and infer the table position from this overlaying of the two images?

I would first think of some more details concerning the data: Do you already have labeled data samples?  If so how many? Would camera position or the angle change?  Would the design of the table change (like figures, grass or ball colors)?

@GKozinski, thanks for the comment as I am new to this it helps me formulate the question better. As for data - nothing. I was hoping to be able to identify some anchor points of the image and then use geometry to transform everything to the 2D model. For example: if I can detect the very top green edge of the table and also the position of the top goalie rod. I know exactly their relative distances from the pre-defined table definitions. So I should be able to calculate the view angle from those relative values.
Nothing can change as far the table set-up. All is fixed and well defined.

Will there always be this one table design or it can change in future, I mean for example will the players figures always have the same shape and yellow/black colors, or you plan using your model on different tables? Will camera stay always at exactly same position? You can edit the  question to deliver those details so it will be easier to understand your entire question

@GKozinski, the camera can change. For simplicity I would assume it will always be centered but the height can change in small variations. For simplicity it can be assumed that the players and everything will also always remain the same.


text:
Multi-layer network only predicts linear trends
I have made a neural network from scratch (in java), which is refusing to switch out of linear regression. I have pushed up the layer sizes (it now has 2 hidden layers, both with 5 neurons), and yet when given harsh sloping polynomials to train on, it still predicts values that follow a gradient, even though this returns high cost.
The network is working in that the predictions do somewhat follow the polynomial as best as a line could, but why wont it actually give me predictions that follow a polynomial like the one it trains on?
I have checked all aspects of training, SGD is working as it should, as is the cost function (MSE), and yet the network just isn't able find a way to minimise cost, it can't seem to break free of linear regression.

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

What kind of activation fuctions do you use?

Ah, currently I don't use any. And now I have just found a resource going through why I need to: https://stackoverflow.com/questions/51442459/why-is-relu-used-in-regression-with-neural-networks. Thank you.

Neural network is basically a composition of matrix multiplications (linear combination) and non-linear activation. In classical libraries, when an activation function is not specified then everything stays linear (with the linear activation function y=x).
These non-linear activation functions have several properties : hyperbolic tangent and sigmoid are double saturated, often used in classification task or combined with Normalization layers.
Cost function has nothing to do with non-linearity, it is just a way to measure how far we currently are from the goal.


text:
Can MSE be used for NN categorical classification problems
I currently have a neural network that can manage to perform polynomial (single output) regression problems. I now want to upscale to classification problems (eg: image recognition). Can I do this with MSE loss, or do I need to switch to a more complicated categorical cross-entropy loss function...?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

You sound confused; SGD is not a loss, it is an optimization method that can be used for minimizing various functions usually used as loss ones in NN (both for classification and regression).

Yeah sorry I meant to say MSE.

There are at least two reasons, why cross-entropy loss is preferred over mean squared error in classification problems.
A theoretical reason
Both aforementioned losses are negative logarithmic likelihoods for following parametric statistical models.
MSE corresponds to the parametric regression model $Y = f(\theta, X) + \epsilon$, where $\epsilon$ is a normally distributed zero-mean random error, independent of $X$ (and if you make $\epsilon$ not normally distributed, but Laplace-distributed, you will get another well known loss — mean average error).
Cross-entropy loss corresponds to a classification model with two classes $C_0$ and $C_1$ where $P(C_1|x) = f(\theta, x)$.
Minimizing those losses means getting a maximum likelihood estimator for the regression/classification parameter $\theta$.
A practical reason
Cross-entropy loss is just plain better in distinguishing “good” classification from a “bad” one.
Consider a following example:
Suppose we have $101$ object from class 1 and two experts deciding whether they are from class 0 or from class 1. The first expert made one blatant error, assigning score (probability of belonging to class 1) 0.00001 to one of the objects and 0.99 to the others. The second expert is always correct but a bit less confident — they assigned score 0.9 to all objects. Based on common sense, predictions of the second expert are obviously better than those of the first one. Now let’s look at the losses.
First expert has MSE $\sim 0.0099998$ and Cross-Entropy loss $\sim 12.52$.
Second expert has MSE $0.01$ and Cross-Entropy loss $\sim 10.64$.

+1 The same idea applies to Frank Harrell’s comments here.


text:
Why is my validation accuracy fluctuating between two inverse values?
I am currently going through the FastAI course and to practise, I wanted to code a neural network that classifies the FashionMNIST dataset from scratch.
Lately, I've been running into an issue where I get a consistent validation accuracy score of 0.9 sometimes (afer reinitiating my weights and biases), but occasionally it's stuck at 0.1, and does not go anywhere from there.

I tried to start again from zero multiple times, but can't figure out the issue. I would very much appreciate any help :)
Below is my code:
#hide
!pip install -Uqq fastbook
import fastbook
fastbook.setup_book()

#hide
from fastai.vision.all import *
from fastbook import *

matplotlib.rc('image', cmap='Greys')

import torchvision.transforms as transforms
from torchvision.transforms import ToTensor, Lambda
from torch.utils.data import DataLoader
from torchvision import datasets
import torch
import matplotlib.pyplot as plt
from torch.nn import CrossEntropyLoss

tfms = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1037,), (0.3081))])

training_data = datasets.FashionMNIST(
    root = 'data',
    train = True,
    download = True,
    transform = tfms,
)

train_x = training_data.data.view(-1, 28*28).float()/255

df = pd.DataFrame(train_x[0].view(28,28))
df.style.set_properties(**{'font-size': '6pt'}).background_gradient('Greys')

target_transform = Lambda(lambda y: torch.zeros(10, dtype=float).scatter_(0, y, value = 1))
train_y = torch.stack([target_transform(y) for y in training_data.targets])

dset = list(zip(train_x, train_y))
dl = DataLoader(dset, batch_size=1024, shuffle=True)

testing_data = datasets.FashionMNIST(
    root='data',
    train= False,
    download= True,
    transform = tfms
)

valid_x = testing_data.data.view(-1, 28*28).float()/255
valid_x.shape

valid_y = torch.stack([target_transform(y) for y in testing_data.targets])
valid_y.shape

valid_dset = list(zip(valid_x, valid_y))
valid_dl = DataLoader(dset, batch_size=1024, shuffle = True)

# Init params

def init_params(size, std=1.0):
  return (torch.randn(size)*std).requires_grad_()

weights = init_params((28*28, 10))
biases = init_params(10, 1)

weights.shape, biases.shape, train_x[0].shape

# Predict

def linear1(xb):
  return xb@weights + biases

# Calculate loss

def cross_entropy_loss(predictions, targets):
  predictions = predictions.softmax(dim=1)
  cross = -targets*torch.log(predictions)
  return cross.sum(1).mean()

cross_entropy_loss(linear1(train_x[0:40]), train_y[0:40])

# Calc grad

def calc_grad(model, xb, yb):
  preds = model(xb)
  loss = cross_entropy_loss(preds, yb)
  loss.backward()

# Train epoch

lr = 10
params = weights, biases

def train_epoch(model, params, lr):
  for xb, yb in dl:
    calc_grad(model, xb, yb)
    for p in params:
      p.data -= p.grad * lr
      p.grad.zero_()

def batch_accuracy(xb, yb):
  preds = xb.softmax(dim=1).round()
  correct = preds.max(1).indices == yb.max(1).indices
  return correct.float().mean()

def validate_epoch(model):
  accs = [batch_accuracy(model(xb), yb) for xb, yb in valid_dl]
  return round(torch.stack(accs).mean().item(), 4)

train_epoch(linear1, params, lr)

validate_epoch(linear1)

for i in range(5):
  train_epoch(linear1, params, lr)
  print(validate_epoch(linear1), end= ' ')

print(linear1(valid_x[0:15]).softmax(dim=1).round().max(1).indices == valid_y[0:15].max(1).indices)


text:
Model Suggestion for graph data
I am trying to figure out the right model/algorithm for a graph dataset to develop a machine learning pipeline. I have looked into Graph Neural Network(GNN) but all of the tutorials I found, trained the model on a single large graph where my dataset consists of thousands of smaller graphs like a typical machine learning dataset. Is GNN the right model for this kind of dataset? If not what should I look for?
To give an idea about the dataset, I am not still sure about the representation but the dataset will describe the AST of different code snippets.
Edit:

graphs will have different numbers of nodes and edges as they are produced from different code blocks. Also, the node types are not homogenous i.e., they have different features.
There are multiple use cases for the model, but primarily I am interested to develop a model that predicts link or edge between two similar node.

It depends mainly on your end use case and the nature of the data. The questions to ask yourself are: Do all those graphs have the same number of nodes and edges between nodes? If so, you can approach the problem as a more traditional problem since you would have the possibility to convert it into a tabular one. On the other hand, what do you want to do with that model? It is not the same to make a model to classify or to do regression on each of the nodes of the network knowing a partial part of it than to do a classification or regression of the whole network.

I have looked into Graph Neural Network(GNN) but all of the tutorials I found, trained the model on a single large graph

In that case, the tutorials were probably working on transductive prediction tasks. These are tasks were you have information on only a few nodes in a graph and want to infer all the missing pieces. An example for this kind of data is the cora dataset.

my dataset consists of thousands of smaller graphs like a typical machine learning dataset.

This is then more related to inductive prediction tasks, as common for chemistry datasets were the task is to predict properties of different molecules. An example is the Mol-HIV dataset.

Is GNN the right model for this kind of dataset? If not what should I look for?

Yes, a GNN can be used for this - if you build your pipeline for inductive tasks you should have no problem applying GNNs to your kind of data.

graphs will have different numbers of nodes and edges as they are produced from different code blocks. Also, the node types are not homogenous i.e., they have different features.

This is exactly what GNNs good at. So, in order to work with your data you should lookup how to prepare your data for inductive tasks (Here is a keras guide on molecular data) and how to setup a GNN for link prediction (like in this DGL guide) and you should be good to go.

pardon my ignorance, the example you gave for the Mol-HIV dataset, the link shows it is in the category of Graph Property Prediction or graph classification. What I want to do is link prediction, which means I will train the model on graph A,B,C,D,E,F,G.... and test on other graphs to check if the model can predict if there should be an edge between two similar nodes (similar by the property)
Any idea how I should approach this?

Sure, what I mean is that you have to solve two independent problems: 1) Data loading when you have multiple Graphs (this has nothing to do with your model - its just your processing pipeline. That's where looking at something like MolHIV helps) and 2) Doing the link prediction. This has something to do with your model and how you use it. Here you can look at the cora link prediction example. I hope it's more clear now?


text:
Which algorithm for hierarchical and ordered classification?
I have developed the "Pyrates" application which is a serious game to learn programming in Python.
In each level of the game, you have to pick up a key and open a chest. To do this, you need to use the control functions available (walk, jump, etc.). However, they are not sufficient. To complete the level, you must implement an algorithmic notion (loop, conditional, variable, etc.). This notion is not made explicit, it is the game situation that makes it necessary. Here is a screenshot to make it more meaningful:

I am currently developing an automatic help system (given by the parrot -see below screenshot-). For each level, when a student clicks on the "Help" button, the application should propose one of the five existing helps or no-help ("I can't help you at this point. Keep searching!").

The five helps types (Level objective, Control functions, Notion involved, etc.) are ordered (1,2,3,etc.). That means that once a help is given, it is not relevant to give a lower grade help.
Here is the tree that schematize these different aids:

Concerning the data, I will soon have a set with the following observations.
For a given student and a given level, each time he clicked on the "help" button during the data collection :

Features: cumulative activity of the student before help asking since the beginning of the level
Label: type of help chosen by the teacher (1,2,3,4,5 or no help).

My goal is to train a multi-class classifier on this data so that it can predict what help to give when a student clicks the "help" button, based on the accumulation of their activity since the beginning of the level.
My questions is : What type of machine learning algorithm should be used for this type of task?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.


text:
Is there a difference between 'inference step time' and 'inference time'?
I was looking at the table presented in Keras Applications and there is a column that says Time (ms) per inference step (CPU) and another for GPU. I did a quick research on the internet and I came across some explanations of what inference time is as we can see 1-here  and also 2-here.
1-The inference time is how long is takes for a forward propagation. To get the number of Frames per Second, we divide 1/inference time.

2-In deep learning, inference time is the amount of time it takes for a machine learning model to process new data and make a prediction.

But the explanations are talking about inference time and not Time per inference step, so I was wondering if there is some difference to it? Is this time related to steps_per_epoch or validation_steps from fit function?
They also mention these two lines in the Keras website :
Time per inference step is the average of 30 batches and 10 repetitions.
Batch size: 32

Does this mean that the Time per inference step is the average time it takes the model to make predictions for each batch of 32 samples as opposed to just one sample?
Thanks in advance!

Yes, your interpretation is correct, it's also stressed out in the documentation: "Time per inference step is the average of 30 batches and 10 repetitions.".
Be aware that the relation between batch size and inference time is not linear, so you can't halve of double the time reported on that table to estimate inference time for different batch size. And it's also very specific to the GPU used, so variations with different hardware must be expected.
Also, the table in keras documentation refers to the first definition of inference time you linked, which is more precise but less useful for real use cases. So if timing is crucial for you be sure to time also every preprocessing and post processing step that you're applying to your images and predictions.


text:
How to improve the performance when no shuffling of dataloader is needed?
I'm currently doing some researches on video recognition. What I'm trying to do is like this paper.
The idea is that: for processing a specific input video clip (shape: [T, C, H, W]), it needs features of the video clip from last timestamp, where we are trying to build a long-time feature memory.
As a result, we have to sequentially read consecutive video clips, and the dataloader will output the data like this: (the number represents the timestamp of a video clip, and each row represents a batch)
[1,  9, 18, 27, 36]
[2, 10, 19, 28, 37]
[3, 11, 20, 29, 38]
[4, 12, 21, 30, 39]
...

The thing is, due to the sequential reading, I couldn't do shuffling for the dataloader, which unsurprisingly results in a pretty bad performance.
For such streaming loading problem, are there any tips to improve the bad performance caused by no shuffling?
Noted: I have done the following experiments to verify that the problem is really caused by no shuffling:

original dataloader with shuffling: good accuracy
original dataloader without shuffling: worst accuracy
the dataloader I mentioned in the question: medium accuracy but still bad

There are no ways to make the model or training pipeline adapt to the sequential loading of data, since the model pipeline is pretty much independent of the data pipeline due to minibatching.
There are however some ways I can think of to handle random shuffle of video segments.

First, is to convert all the videos into uint8 4d numpy array of (T, H, W, C) dimension, then you could use memmap to read a segment from it during dataloading, which would be very trivial in cost. However, the downside is obviously you need a lot of disk memory to do this. In particular, from mp4 to npy would almost give you an 100x memory increase.

You can pre-slice the videos into segments of length that your model expects. This would allow you to shuffle the segments easily.

Use a very efficient video segment processor such as opencv, ffmpeg, or the recent PyTorch video loading API. However, this is still bound to be slow, as you are randomly accessing the middle of a video segment.

Sorry, my question may be misunderstanding. What I want to do is to read consecutive video clips (segments) not frames, so I couldn't shuffle the clips. But still thanks for your reply!


text:
How many parameters of a RL environment?
I’m working at a Reinforcement Learning model, using PPO algorithm, in which the agent has 4 possible actions, acting in a stochastic  environment defined by 3 parameters.
Given its stochasticity, I was thinking it would be better to store also the parameters of the previous step, ending up in 6 parameters.
My question is: when these parameters are too many? If I store the previous 5 results, I end up with 5*3=15 parameters, thus it could be more complicated to reach a local maximum.
When do you think I should stop? What’s the reasoning behind it?


text:
Deep reinforcement learning the board game "Battle Sheep" - too large action space?
I was recently introduced to this simple board game called "Battle Sheep". In this game, two to four players try to acquire as many hex tiles from a hex grid as possible.
You can find the complete rules here.
I decided to program a game engine for this game. You can find it here. I also created a SFML-based graphical interface for the game, which can be found from the same repo - here.
The reason I created the game engine in the first place was a challenge to myself - I wanted to create an AI which would have superhuman skills in this game, since I thought that the game is pretty simple and it would be easy for a neural network to master it. I have been studying the work about deep reinforcement learning networks, especially Leela chess zero (Alphazero) and AlphaGo. I thought that a DQN should be possible to master the BattleSheep game, but I have stumbled across a problem:
Let's consider a game with 2 players. Since the game has two phases (map creation and the gameplay itself), the action space of the game is unknown at the start because the two players can place their pasture boards as they wish as long as it follows the rules (see the link above). In my game engine, I have solved this problem by creating large-enough map so that any kind of piece placement combination in the map creation phase will fit inside of its bounds. During the map creation phase, the chosen hexes are set "active" (defined HEX_FREE in my engine). "inactive" hexes cannot be used, or could be considered "out of bounds" during gameplay (defined HEX_OOB in my engine). For this code, see the constructor of my Map class.
Now the question itself: Deep Q-learning requires a fixed-size action space, which is hard to define in BattleSheep, since the map is different in every game. One possibility would be to create an action space that consists of the complete map (including these "inactive" hexes in my game engine), and define every possible move in that map. However, if we consider a map with for example 40 width and 40 height, the action space would be
40 x 40 x 6 x (40 x?) 16

40 x 40 - from which hex the player will start the move
6 - to which direction the player will move
40? - how far the pieces will move (is this required since you are always forced to move as far as possible?)
16 - how many pieces the player will move
Without the extra x40, the action space will yield 153600 different moves, and with the extra 40 (move length), the action space will have 6.1 million different possible moves. This is surely too large. How should I tackle this problem?

I can't write a full answer yet, but checkout parametrized reinforcement learning or hierarchical reinforcement learning. You can either treat those extra details as the parameter of your action or as the sub-action of your action (hierarchical).

Thank you for the insight @Sanyou . I will surely check it out. I guess by "extra details" you mean the state of the hex tile on the hex map (active/oob)?

you could also just make the entire map the action space, but mask or filter the move that is infeasible for each player, then the action space will just be Area of Map * num of players


text:
How is the formula for off policy lambda returns derived (formula inside)
In particular, I am speaking about the formula $(12.26)$ in Suttons Book : Introduction to Reinforcement Learning:
$G_{t}^{\lambda a} = R_{t+1} + \gamma_{t+1}(\bar{V}(S_{t+1}) + \gamma_{t+1}\rho_{t+1}[G_{t+1}^{\lambda a} - \hat{q}(S_{t+1}, A_{t+1})]) $
which combines off policy traces with control variates. What I currently understand is this:
We can rewrite the $\lambda$-Return, which is generally defined as
$G_t = (1-\lambda)\sum_{n=1}^{\infty} \lambda^{n-1}G_{t:t+n}$
recursively as
$G_{t}^{\lambda} = R_{t+1} + \gamma (\lambda G_{t+1}^{\lambda} + (1-\lambda)V(S_{t+1})) $
Moreover, the $n$-step Return, which is generally defined as (note, that the n-step Return appears in the $\lambda$-Return formula above)
$G_{t:h} = R_{t+1} + \gamma G_{t+1:h}$
can be extended for off-policy methods, by introducing importance sampling weights $\rho$ and using control variates (equation $(7.13)$ in Sutton book)
$G_{t:h} = \rho_{t}(R_{t+1} + \gamma G_{t+1:h}) + (1-p_{t})V_{h-1}(S_{t}) $
And some how, combining this definiton of the $n$-step Return, with the recursive definition of the $\lambda$-Return, should give the formula $(12.26)$, but unfortunately I have no clue how.


text:
How is reinforcement learning applied in the real industry?
I'm a newbie to reinforcement learning.
While studying reinforcement learning, a question arose about how to apply reinforcement learning in the real world.
Assuming that a reinforcement learning agent trained in simulation is used in the real world, do they usually only use the optimal policy learned in the simulator? Or are they need additional exploration (learning) in the real world?
In my view, the latter approach is quite impractical in real-world environments where failure by agents' actions is fatal. However, it also seems very difficult for the simulation environment to reflect the real world.
What approach is being adopted by the industry field?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

RL is not used much in the real industry, as you said because of safety concerns. There are perhaps three different ways of how this would be possible

Use safe exploration to learn a model of the environment and apply model predictive control algorithms on the learned model. This seems to be the approach taken for data center cooling using model-predictive control.

Design an accurate simulation model as a digital twin and use it to train policies there before deploying into the real world. This was the approach taken in the recent breakthrough in nuclear fusion: Magnetic control of tokamak plasmas through deep reinforcement learning

Use existing controllers to generate data sets and train RL agents with offline RL (or perhaps imitation learning) and use it as a prior before finetuning it in the real world.

Now, as you mention there is the sim-to-real problem if you train in simulation. The real world usually comes with various uncertainties that can not be simulated well. This may be not extremely relevant for controlled industrial environments but is important in robotics. We can divide the training process into multiple phases, where the agent has less and less information available. The idea is to train a belief model, which aims to assess all the uncertainties and safely adapt to them. An example is:
Learning robust perceptive locomotion for quadrupedal robots in the wild


text:
How to filter out class for which the model has not been trained in ml web app?
I have developed an python based ml web app. It gives details of the book from a image of book cover.
Problem: When I upload the book cover image then it works but when i click image of any random object, it still gives the result related to some book in the dataset. How can i detect or filter out such images and tell the user that the image doesn't contain book cover?
Somewhere I read that we need to add some garbage classes (other classes I am expecting users will upload like person, objects etc.) in the training process. How can we do this?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

This is a common problem with all ML models. At the end of the day, they are just math functions. They take the inputs and compute the outputs based on the weights they learned. You can upload random noise to a well trained model and it will give you confident predictions. Ultimately the model can only be relied on to perform well on inputs that fit on the training data manifold.
You can try create a "book/not-book" binary classifier, but creating a good dataset for the "not book" class is much harder than it may seem. Should you include pictures of cars? Of sandwiches? Of random noise? Of TV screenshots?
It's much easier to define the space of "this specific class" as opposed to "all things that are not this specific class".
One ML approach might be to train a book cover object detection model. This would both identify positive examples of book covers and give you a bounding box to crop the input image around just the book cover.
Alternatively, you could look at cases where users are uploading non-book covers and try to change the UI/UX of the app to incentivize correct behavior.

One approach would be to train a classifier that will take an image and return a binary prediction of whether it is a book cover or not. This can be done in a supervised fashion. You will use this classifier on every given image input and proceed to the next step of returning details of the cover if and only if the classifier predicted that the image is indeed a book with some probability $p>threshold$, which can be predefined.
As this process is generally an overhead to the actual goal of your app, I would try using basic classifiers first, while also providing a low-dimensional input - I suppose that much can be learned from edges only


text:
As a intermediate in Deep Learning, from which research paper should I start reading from?
I have done Andrew Ng's ML and DL courses, and some projects and implemented some important ML algorithms from scratch. Now reading the deep learning book. <=(Edited)
I want to start from the beginning (in terms of reading research papers), i.e, deep feedforward networks, regularization techniques,{then maybe conv nets and others}etc, etc and some tips on how to tackle the difficulty in understanding it. Thank You.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

For your question, I strongly suggest you start from this book: https://www.deeplearningbook.org/ instead of papers.

The best research paper for you to read will depend on your specific interests and background knowledge in deep learning. However, a good place to start might be the CS230 Deep Learning, which covers a wide range of topics in deep learning and provides links to further reading on each topic.[https://cs230.stanford.edu/]

@Faizy http://cs230.stanford.edu

At your stage, I don't think jumping straight into reading research papers would be efficient. Generally, reading textbooks/review-articles, or simply watch a couple introductory youtube courses would do a better job at getting you up to speed with the background knowledge. Of course, you can always find a project that interests you and try to incorporate some elements of ML into it, which allows you to naturally learn ML at the same time.
Some standard introductory textbooks/courses are:

The deep learning textbook, more theoretical driven
Andrew Ng's courses on youtube, more application driven

which should cover the topics you mentioned.
If you want to focus on a specific topic (e.g. ConvNets, transformers, recurrent networks, etc.), it's generally helpful to find a recent review article on this topic and read through it. This is just to understand the current state of the field, and you can then read specific papers that interests you with this contextual knowledge in mind. Note these fields are moving so fast that certain seminal papers are no longer hugely relevant (e.g. many network architectures and training methods proposed in the classic  AlexNet paper are outdated.)

I have done Andrew Ng's ML and DL courses, and some projects and implemented some important ML algorithms from scratch. Now actually reading the deep learning book.

Also, wiki is free XD.

There is no "beginning" with research papers. Papers are published as they are ready, in no particular order with respect to complexity or topic. I think you just have to jump in.
Pick papers that match your interests. Look things up as you read to understand. You might need to brush up on Math.
Here is a list of resources where you can find research papers to start. I am sure you can find others as you learn what interests you.

Not sure what to recommend, since you say "from the beginning" in the text but "intermediate" in the title...
Anyway, for the "then maybe conv nets" part, there is a tutorial from 2021 that relates convolutional networks with the matched filter, a well grounded technique in signal processing. I find this a great idea; depending on your background, it may be interesting for you as well.
https://arxiv.org/abs/2108.11663
Is this the kind of papers you are looking for?

In terms of reading research papers, I meant the beginning.

The good news is that there are many freely available educational resources online and at your local library.  Here are some that I used to get me started:

Kaggle:  Intro to Deep Learning
Kaggle:  Computer Vision
Pattern Recognition & Machine Learning (Chapter 5)
Machine Learning (Chapter 4)

Good luck and happy learning!
P.S.  I know these are not research papers, but I would encourage you to start with these anyway.

Thanks, some of the links were helpful, but I want an answer specific to my question.


text:
How do Transformers compute the words embeddings at inference time since the embeddings are dynamic?
In Word2Vec, the embeddings don't depend on the context.
But in Transformers, the embeddings depend on the context.
So how are the words' embeddings set at inference time?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

Transformers do still use pre-trained embeddings but these are then given context by the positional encoding and self-attention blocks of the architecture.
I will explain in the context of a sequence-to-sequence transformer used for translating a sentence from one language to another (e.g. that used in Attention is All You Need):
At inference time the input sequence process (i.e. the encoder) is the same as during training time: each word in the input sequence is mapped to the embedding vector for that word, and 'given context' by the positional encoding and encoder self-attention blocks.
In the decoder, at inference time we are predicting the translation of each word in the input sequence, one at a time. So instead of feeding in the target translated sentence to the decoder, we feed in an empty sequence with a start-of-sentence token. In much the same way as in the encoder, each token in this sentence is given context by the positional encoding and self-attention blocks. The next word in the output sequence is then predicted by the rest of the decoder layers and this is appended to the output sentence. This output sentence (now consisting of the start-of-sentence token and the first word(s)) is then fed into the decoder to predict the next word, until the end-of-sentence token is predicted.
So in answer to your question, the blocks of the transformer that give the word embeddings context are the same during training and inference.

So to sum up, the word embeddings are the same as in training time, the positional encoding differs, that's right?

Yes - the word embeddings are the same in training and inference. The word embedding stage is just a lookup from word to pre-trained embedding representation for that word.
As you say, these are then transformed by the positional encoding (depending on where the word is in the sentence) and by the attention layers (i.e. weights of the key, value and query matrices that were learnt during training).

it might also help to point out that that self-attention isn't technically an embedding and isn't learned but computed on the fly for each input (whether at training or at inference).

Embedding doesn't provide context like self-attention does. Embedding is what you do to represent your training data (words or pictures) into numerical vectors.
Word embedding is learned and will be the same at training and inference. For example, "Transformers are cool" will be embedded as [122, 5, 43], while "How are you" will be embedded as [10, 5, 2].
Positional embedding is a deterministic function. It is not only the same at training and inference, but it is also the same on every sample in your training data. Both "Transformers are cool" and "How are you" will have positional embedding [1, 2, 3]. (Positional embedding could also be a learned transformation, but in the original paper, they didn't find much improvement in using one).
Now, self-attention is what gives inputs context (for example, "are" and "you" are related and conjugated accordingly in the sentence "How are you"). Self-attention is a matrix of weights which is indeed dynamic since it is computed on the fly during training. But you can also compute self-attention on your new predictor on the fly during inference.


text:
How to deal with delay in reinforcement learning, an unclear case
According to the question in How to deal with the time delay in reinforcement learning?, we can tell the delay in the reinforcement learning can be observation delay, action delay and reward delay.
I have a special case of the delay but I am not sure what kind of delay is it, and how to deal with it.
For example, at a state St0, my agent takes action A1, but we need to wait for a while to gain the reward R1. Meanwhile, my agent keeps taking action A2 and A3. The trick part is A2 and A3 both influence the environment and may affect the R1.
So the timeline is agent plays action A1, A2, and A3, all of them being effective in the environment immediately but we need to wait for a while to see the reward R1, R2 and R3.
Shall we model this question as an observation delay or reward delay?
When my agent receives the R1 but not R2 and R3, can I update my Q-table by eligibility trace or any other kind of method?

Are you in an episodic setting? If so, you could pretend that you get the sum of reward as the terminal reward, i.e. $R_T = \sum_{k=0}^T R_k$. Then you can use any RL algorithm you want (rather than trying to assign partial credit to actions during the episode).

that is the tricky part, my  setting is for the continues job, so it may not provide a clear bound as the episodic end.

Instead of trying to find the "correct" mathematical approach, I wonder if you can get good results simply by applying a sufficient delay, i.e. you sum all rewards you got over a certain period of time T and make as if you received that summed reward at timestep $t+T$. Then you can play around with the value of $T$.

it might work. I was considering a similar approach which whenever I got the reward, I use the reward to update all the (State,action) in the waiting buffer. This means each pair might be updated multiple times. For example, when I got R1, I updaate for A1,A2, A3 with same number; when I got R2, I update A2 and A3. But it does not make any sense in the theory.

Should I add my suggestion as an answer?

sure and thanks for the answer. we had a good discussion

You ask what kind of delay you are facing, and claim you have a "special case", but don't explain anything about the nature of the environment that causes this delayed reckoning. A lot of time you don't need to change any reinforcement learning maths, but adjust your state representation. I cannot tell if that is the case here, but might be able to help if you explain more about the environment and how you have modelled it for RL.

In the meantime, I would be interested to know if this helps:  https://ai.stackexchange.com/questions/30502/delayed-state-observation-or-caching-action-in-openai-gym-can-it-still-learn

Given that you are not in an episodic setting (based on the comments), I would suggest the following: accumulate rewards for a period of time $T$, that is, at time $t+T$ do as if you had received from the environment a reward $R = \sum_{k=0}^{T-1} R_{t+k}$ .
Then, treat $T$ as a hyperparameter and see which value gives the best agent.


text:
Are there existing approaches to train AI for consciousness
I am somewhat of an amateur to this topic so I just want to put forward an idea and I would be happy to be educated what the current state of the art in that direction is.
One of the big questions is how to get an AI that is conscious. It seems to me the surest way to get that would be to train for it, rather then let it emerge by chance. To do that I would imagine to use problems that need one or more properties associated with a "self" to solve it.
E.g. catching a ball that was thrown. To do that the self needs to know its position in space relative to the ball thrown. Maybe an intermediate layer could be trained for the current position, while the output could be trained for the velocity and direction to go to catch the ball. This should help to establish the concept of ones own position in space for the AI. Of course there should be many different problems that help solidify the same property.
Many different properties that can be associated with a self should be trained for. The nodes that correspond to those properties should probably be closely connected, to establish some kind of self core where that information is compressed into.
One could start with problems that require one dimension of the self core to solve, respectively. Then move on to problems that require multiple ones. I suspect the higher the number of dimensions of the "self core" the closer it would get to a state we would recognize as consciousness.
My question would be, whether some kind of approach like that was ever investigated? Or alternatively, why this approach would not be feasable or would not work?

Well, let's say you train some robot to catch a ball (like in your example). Your training data would be, for example, a stream of images (captured from a camera), and the (relative) x-y coordinates for the "self" part.
Even if you do so, and the robot is able to correctly catch the ball, you still haven't achieved consciousness: making it know its position in space, is not the same as knowing that the code that runs on it (the robot), has actually a body that can move, with sensors, and that it can reason about possible movements (e.g. would such trajectory miss the ball? or would I damage myself? or even harm someone?)
Consciousness is something way more powerful than training a control policy. It involves self-awareness and reasoning at very least. And I guess we should first understand how such works in the human brain, before trying to mimic it.
Even if you equip the robot with the largest LLM (e.g. ChatGPT), and you ask it if it's conscious or even if it has a body, well, probably it would respond: "yes, I've a body because I'm a robot, I'm designed to etc". But that either does not mean anything, unless it won't be a creative copy paste of some piece of text encoded in billions of neural networks weights.

Catching the ball was just an example and the idea was to train a net that can reason about itself in many dimensions, and not even exclusively physical ones. Like you mentioned, not only position, but rotations, limb positions, maybe limits of the vision field, but also maybe mood, recent rest quality (or loading status), and so on. I suspect the higher the dimension of the "self" core the closer it would get to something we recognize as consciousness.
Maybe a main requirement to solve the problem would be to actually give it a physical body.

Neural nets solve problems without us understanding how it works. We can try to understand it in hindsight, but it is not necessary beforehand. So maybe through neural nets we would be closer to creating an AI rather then understanding it and the understanding could come later.


text:
Classify sequence of flags
I am not able to find an answer to how I should classify a varying number of sequence of binary flags + other features. My data looks like this (these are events, so the order is important and I may have other features in addition to sequence):

ID
Flag 1
Flag 2
Flag 3
Other Feature

A
1
1
0
0.1

A
0
1
0
0.3

A
0
0
1
3.1

B
0
1
0
1.1

B
1
1
0
0.0

Notice that ID:B does not have the same number of entries (only 2). Any suggestion on how I should organize this data and what should I use to classify? How do I better capture the sequence of Flags? During inference, I will provide the sequence of flags [[1,1,0],[0,1,0],[0,0,1]] OR [[0,1,0],[1,1,0]] and "other feature" to get the label since the order of sequence makes up the positive or negative label.

Can you please put your specific question in the title? Thanks. "Classify sequence of flags" is not really a question.

You have a classification problem over a time-series, basically. You need to group all the events with the same ID under a single sequence, like [A, A, A] (is one sample) and [B,B] (is another).
Your targets ($y$) will be the class label, for each sequence and each element of the sequence. Then the $x$ will comprise the three flags and "other feature(s)" you have.
You can tackle the problem with three kinds of neural networks: 1D CNNs (convolutional nets), RNNs (recurrent nets), and even transformers (i.e. attention-based models).

Since you have sequences of the kind $(B,T,N)$ - where $B$ is the batch size, $T$ the num of events with same ID (you may need to zero-pad), and $N$ the number of features - and have a target for each $t\in T$, you need to predict the class label for each timestep $T$, and each sequence in $B$.
So you're output won't be a single value for one input sequence but a sequence of values of size $T$.

Thanks for your answer - much appreciated. So, let me elaborate this a bit. If I take data as a sequence and because I don't know the number of entries ahead of time, I will always zero pad and truncate to get a fixed length of sequence (for the above example, sample "A" would look like: [1,1,0,0.1,   0,1,0,0.3,   0,0,1,3.1] and "B" would look like: [1,1,0,0.1,   0,1,0,0.3,   0,0,0,0.0]). Is this correct? In that case, I can also use something simple like XGBoost etc.?

You can do that, but I was thinking about representing A as [[1, 1, 0, 0.1], [0, 1, 0, 0.3], [0,0,1,3.1]] so as a (1,3, 4) tensor, and B (with zero-pad) as [[1,1,0,0.1], [0,1,0,0.3], [0,0,0,0.0]]. You transform your dataset accordingly, then train a RNN by sampling $N$ of such sequences to build a mini-batch, and repeat that until convergence. Actually, if you provide masks to the RNN is as if having varying-length sequences. For example, the mask for B should be 1,1,0. To not consider the last timestep.


text:
What type of fine-tuning does the OpenAI API use?
They have instructions on how to fine-tune. But I'm not clear on the type of fine-tuning that is done. Maybe this is mentioned somewhere. I would imagine it's some type of PEFT to avoid catastrophic forgetting. Plus fine-tuning the whole model for individual customers seems unlikely also.


text:
If I freeze pre-trained model weights and than train a classifier on top of its embeddings does that called fine-tunning?
In the context of machine learning.  If I freeze pre-trained model weights (for example, BERT) and then train a classifier on top of its embeddings, does that called fine-tuning?

Wikipedia defines "fine-tuning" as:

an approach to transfer learning in which the weights of a pre-trained model are trained on new data. Fine-tuning can be done on the entire neural network, or on only a subset of its layers, in which case the layers that are not being fine-tuned are "frozen"

You didn't update any weights of the original model, so, at least according to Wikipedia, that wasn't "fine-tuning".
On the other hand, one can argue that nowadays the common use of "fine-tuning" covers much more than just directly updating the weights of the original model.
A popular Parameter-Efficient Fine-Tuning package by huggingface lists (soft)prompt tuning and Low-Rank Adaptation (LoRA) as its "fine-tuning" methods. It is possible to argue that these approaches do not completely satisfy the Wikipedia definition. Still, both approaches modify the original model weights, albeit indirectly: soft prompts modify the embedding vectors at the model inputs, and LoRA adds the low-rank restored matrices to the original model weights.
Finally, if somebody told me that he "fine-tuned BERT" and later I've discovered that he did what you've described -- I would think that I was misled. To me, the ultimate criterion than disqualifies something from being "fine-tuning" is the fact that for a fixed input you can just save the pre-trained BERT outputs and never return to running BERT inference again.

Yes, that would still be counted as a fine-tuning model. Unless untouching the pre-trained model, any modification that helps for your downstream task or to suit your data more IS an act of fine-tuning your model.

I'd like to add another angle to @kostya's excellent answer.
I see the lack of use of additional input as a key distinction for not calling this adaptation fine-tuning. Moreover, as in today's realities, fine-tuning is still an art, so we might consider the amount of professional skill and effort required. The amount of input, skill and effort are all continuous quantities, and there is no clear threshold at which a process of adapting of the trained model is no longer “fine-tuning”. However, when the above qualities are absent or minimal, the use of the term "fine-tuning"  tends to devalue the term and blur its meaning.
It is likely that the term “fine-tuning” and a few others will continue their transformation and become a bit of a catch-all phrase, and being used rather loosely in different contexts. But in a professional setting, it's better to avoid any misunderstanding about the complexity of the work involved.


text:
Is orthogonal initialization still useful when hidden layer sizes vary?
Pytorch's orthogonal initialization cites "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
", Saxe, A. et al. (2013), which gives as reason for the usefulness of orthogonal initialization the fact that for a linear multi-layer network, the singular values of the product of all weight matrices are "isometric" i.e. all close to 1.
However, I can empirically see that if I use this initialization method for hidden layers that have a higher number of neurons than the input and output, then isometry goes away. E.g.
import torch, numpy
def g(r,c):
    w = torch.empty(r, c)
    return torch.nn.init.orthogonal_(w).numpy()
numpy.linalg.svd(g(5,7)@g(7,5))[1]
# Out[1]:
# array([1.0000001 , 1.        , 0.99999994, 0.7571472 , 0.05615549],
#      dtype=float32)

My question: is orthogonal initialization still useful when hidden layer sizes vary? Should I be doing some additional things when dealing with such variations?


text:
What is better: train a model from scratch on your own data vs. fine-tune pretrained model?
Problem: I am interested in building a Q&A engine on top of my private data. I am only interested in asking questions related to my data.
Options:

I train a model from scratch on my own data
I pick a pretrained large language model and fine-tune it on my data

With option 1, I don't expect to train a model with billions of parameters. I understand training from scratch is expensive and time-consuming but I am going to use a much smaller parameter model in that case. For sake of argument, quoting [1]

Falcon-7B and Falcon-40B have been trained on 1.5 trillion and 1 trillion tokens respectively,

so if I have 1 billion tokens in my dataset, I will train a model with 40M parameters.
Is there some objective answer / study done as to which option turns out better?
For completeness, option 2 is something I have tried but I did not get any decrease in my training loss i.e., the fine-tuning was a no-op. I used gpt2 pretrained model with 130M parameters or so and my dataset had about 600 training examples in it. # of tokens = 600 * 1024. any pointers whether this (i.e., fine-tuning made no difference) is to be expected and why would also be appreciated.

I will start by answering questions in the following order, why you shouldn't train your own model, why fine-tuning didn't work, and then what you can do.

As you yourself pointed out, training your own model is expensive in every sense. Not just that, it also involves a lot of complications. Transformers have a simple architecture but they aren't easy to configure.

Fine-tuning didn't help because fine-tuning doesn't add new information to the transformer's 'vocabulary', it only teaches the transformer what kind of a sequence (e.g. text-to-keywords, question-to-answer, word-to-definition, etc.) to output based on the input sequence, but the output is still limited by the data that the transformer was trained on. While an English language transformer can be fine-tuned on an en-to-fr dataset, during inference it will most likely output gibberish as it doesn't inherently understand the abstractions in the French language and the abstractions that come into play when translating from English to French.

A good way will be to create an embedding space (of vectors of all your documents), compare it with the question embedding when required, and use the top n texts that get returned as context to a large language model for it to answer.


text:
Is there an ANN vector-search index that supports incremental ingestion and deletion of elements?
I have looked at a few libraries for ANN search of high dimensional vectors. Although impressive, they come with a huge baggage of fine print. Many of them only support ingesting the vectors in one shot. there is no support for deleting elements from the index. There are a ton of hyperparameters that need to be tuned etc. Is there any index that supports following requirements:

I should be able to add vectors incrementally to the index just like I can do with conventional databases. I will not be having my entire dataset available in advance.
I don't know how many vectors I will be adding. Could be 1000 or could be 1B. I need an index that auto-scales as I add vectors to it. RAM limitation is okay i.e., assume infinite RAM is available for now (bonus if it can handle datasets that cannot fit in RAM).
I don't know how many nearest neighbors I will be querying for at time of building the index. Could be 5 or could be 100. The build step should not ask me to declare in advance the number of NN I will be querying for (looks like ScaNN algorithm asks for this).
I should be able to delete elements from the index. It should be a real deletion leading to freeing up of memory, not marking element with a tombstone.
I don't want to be tuning dozens of hyperparamaters. I can tell the index the length of my embedding. It should derive hyperparameters from it and change them as the index size grows etc.
It should have Python bindings. however, willing to compromise on this somewhat. I can add the bindings myself.

Of course, there are price-performance tradeoffs in these scenarios. It goes without saying that I am not looking for brute-force algorithms and other trivial solutions. I am willing to pay a reasonable price in terms of performance as long as above requirements are met. What is reasonable? lets say I am willing to settle for half of the QPS than what the top-performing algorithm gives but falls short on requirements above.


text:
What is $z|y$ in Conditional Adversarial Nets?
I am currently going through Conditional Adversarial Nets (CGANs) and the modified objective function of the two-player minimax game is stated as follows:
$$\min_G \max_D V(D, G)=\mathbb E_{x\sim p_{data}(x)}[\log D(\textbf x|\textbf y)] + \mathbb E_{z\sim p_z(\textbf z)}[\log(1-D(G(\textbf z | \textbf y)))].$$
I really don't understand what the conditioning does. G and D are not really probability functions, but rather real functions on samples from a distribution (which are real vectors). So what does the conditioning actually do? If the idea is just to include $y$ as an additional input to the generator neural net, then why not just $D(x, y)$ instead of $D(x|y)$?

Yes, they might be abusing a bit the notation, however the meaning is close to what that notation would like to say.
$x|y$ and $z|y$ just mean that $x,z$ might be the same for multiple samples, only differing by conditioning $z$... it's easier to see this on the generator: you can feed the network the same noise, but you expect that depending on what $y$ is, it generates different images.
And yes, ideally you are modeling the joint distribution, thus it would be $G(z,y)$ and $D(x,y)$, but since you have no "generative model for $y$", you are just learning the marginal
For categorical variables it's easy to create such generative model for $p(y)$, but for example consider the height of the person in the picture as conditioning, then you should know the distribution of heights if you really want to model the joint ditribution $p(z,y)$


text:
Are transformer models better than comparable-complexity MLP-based models?
I've watched the outstanding Andrej Karpathy's From Zero to Hero course. In the last lecture, he introduces Transformer decoder architecture, which is able to produce Shakespear-like text. However, there was no direct comparison of the achieved cross-entropy loss (~1.4) with simple MLP models he talked about in the first 5 lectures.
What if one trains an MLP based model with a similar number of parameters/layers, the same context length and also including layer normalization, feed forward and dropout, would the result be substantially worse? Would the training take longer? Are there direct comparisons like that in the literature?

I can't provide you with numbers and results, but I'd expect (for not triavial problems) the MLP-based model to be worse than a Transformer.
The reason is that transformers are designed to handle sequences, whereas MLP are not (even if you flatten the entire sequence into a vector). Trasformers can leverage an inductive bias specifically designed for sequences (likewise CNNs for images), which should account for a better learning, at least, even in case of similar number of parameters and regularization.
Thanks to self-attention transformers are able to learn to focus on relevant parts of the sequence, even far in time, whereas simple MLPs cannot do this. Moreover, positional encoding provides the transformer to use the sequence order information, enabling it to learn to attend even to relative positions. Instead, MLPs just combine information from all the vectors' components.

Not to nitpick, but vector components of course have ordering (it's a list, not a set, in python terminology)

You're correct: I've updated my answer.


text:
Trying to understand the definition of environment in this paper on monte carlo approximation of AIXI
Here is the link to the paper https://www.davidsilver.uk/wp-content/uploads/2020/03/mc_aixi_long.pdf
Definition 2. An environment $\rho$ is a sequence of conditional probability functions $\{ \rho_0, \rho_1, \rho_2, ...\}$, where $\rho_n: A^n \rightarrow Density(X^n)$ that satisfies
$$ \forall a_{i:n} \forall x_{<n} : \rho_{n-1}(x_{<n}|a_{<n}) = \sum_{x_n \in X} \rho_n (x_{1:n} | a_{1:n})$$
In the base case, we have $\rho_0(\epsilon | \epsilon) = 1$
Ok so the right hand side seems to be saying, that $x_{1:n}$ will be made from $x_{<n}$ concatenated with $x_n \in X$, and the sum will be over these different realizations of $x_{1:n}$, so the probability function for the shorter sequence $\rho_{n-1}(x_{<n} | a_{<n})$ depends on the possible longer sequences? This seems to be what the equation is saying but it doesn't make sense that the present probability of a sequence depends on possible future outcomes.

I'm not familiar with this paper, although it's one I've been trying to dedicate some time to read, but, if you read their explanation of how we should intuitively look at the AIXI decision rule, it shouldn't be surprising that something in the present depends on the future (aka planning). The idea that an environment is a sequence of probability distributions that map actions to rewards and observations is fine. But currently that equation is also not clear to me. It looks a bit like the law of total probability.

They call it the "chronological condition" and refer to Hutter 2005. I tried to read that book a few years ago. It's very technical so not easy to read, but quite interesting


text:
How to generate original training videos based on existing videoset?
I am a software engineer who is quickly ramping up on AI tech, but am nevertheless very new to the sector.
A collegue has an extensive collection of training videos, the vertical is wheelchair seating and mobility and the training content are the mechanical hardware skills gained over a 30 year career.  I am telling him that he needs to be the first seating vendor on the block with an AI that has incorporated into its context everything he has included in his video series and is able to generate training videos of its own.
Stated another way, I'm looking for a way for an AI to learn the skills my friend teaches not only through verbal prompts, but through hands-on demonstrations.
Although he is very precise in his presentations and one approach could be to pull text from the audio and normalize it for input into training an OpenAI model, I can't help but think that an AI should have full access to the videos, including the visual portions and should be able to generate the same kind of output.
If we can have AIs that can generate entertainment videos with a textual input, couldn't we have AIs that generate full training videos with textual and video inputs?
Perhaps some kind of time code correlated input with screen grabs from the training videos presented in training alongside with text scraped/recognized from the audio track.
I hear that GPT4 is able to train from correlated text/images but I'm having a hard time finding more information about this.
The owner wants to be able to validate/challenge the AI and correct any errors in the resulting output (audio and/or video).  My OpenAI API reading indicates that at least for text based AIs this is now possible.
Is current technology up to this task?   How would a content creator correct an AI that produces incorrect video output ?
Exemplar:
https://youtu.be/hhgEBm7C2G8
Thanks !

Learn the graphic design and programme language to using AI to create training virtual anime video

You are a a year or three too early. Best options for video generation today are not up to scratch for such a task. Some creatives working in video could make use of today's image, text, voice and video generation for effect, and sometimes for efficiency. But is still a way to go before it could be core to any decent prodtion

Examples of current cutting edge videos using AI gen: https://m.youtube.com/watch?v=saZtmlJYagE

Checking back in, I'm aware of Synthesia and Heygen doing script-driven AI animations (with collateral like background video and pictures added to the scene)>
I'm also aware of AIs that generate small video clips from textual or photo prompts: https://stability.ai/stable-video
What would be the next step to have an AI understand the correlations between an audio track and the vocational skill being shown?
Can anybody see a roadmap here?

I see that GPT4o has been released, which apparently can consume and understand video.  I'm looking at whether this model gets closer to what I need to do as described in this article.

I also see NVIDIA and OpenAI Sora being models that have some understanding of world physics.  Could these be used in conjunction with GPT4o to achieve the stated goal?

Looking even farther forward, I can envision autocreation of training video content and 3D artifacts to populate a 3D assessment space where students must master and demonstrate learned concepts where the results are judged in a virtual space against a gold standard, but otherwise allowing students to emulate a real shop in the virtual space for the particular skills(s) being taught.

This research shows some promise in   better enabling multimodal time series analysis : https://techxplore.com/news/2024-07-brain-artificial-dendritic-neural-circuit.html?utm_source=nwletter&utm_medium=email&utm_campaign=daily-nwletter

Also interesting - adding ambient audio to training processes - gong beyond simple text to speech and paying attention to the sounds made by performing some vocational tasks allows them to be performed more correctly and quickly : https://techxplore.com/news/2024-07-adding-audio-robots-job.html?utm_source=nwletter&utm_medium=email&utm_campaign=daily-nwletter

I’ll check out GPT4o’s ability to accept both video and audio input and see how well it correlates these modalities:  https://cookbook.openai.com/examples/gpt4o/introduction_to_gpt4o

I've made some progress, but it's slow going as gpt4o appears to have trouble reasoning beyond the surface.  What techniques could I use to better help the AI correlate multimodal text and image inputs across time (as not every input is localized, sometimes a newer subtitle helps to unlock the meaning of an older image in a video, and vice versa) ?
More details:
https://community.openai.com/t/having-trouble-in-advanced-multimodal-reasoning-beyond-the-surface/1011106

I'm also aware that fine tuning is now supported on gpt4o, but only textual training data sets are allowed, so I want to train on multimodal inputs I'd have to make sure that images are properly reduced to text descriptions.  This is trickier than it sounds as there are a mountain of corner cases.  I'm trying my hand at manually adding training details to input context to produce better and more nuanced and sufficiently detailed image summaries that are useful for the purpose (distilling training details from the vocational training video) but the human effort mounts and AI performance lacks.

I switched the entire factory from OpenAi GPT4o and am having markedly better results. It was a pleasure to move an “AI on Botox” to an AI that showed a deeper level of nuanced understanding of the problem space. Now to figure out how to fine tune Claude Sonnet 3.5. Apparently I’ll need to invest in some beefy hardware as I don’t think Anthropic has a fine tuning API exposed like OpenAI has.

I've had some good success with Sonnet, please see https://dev.to/mdear/inceptive-techniques-to-pull-summaries-and-e-learning-out-of-vocational-training-video-series-25eo
and
https://gist.github.com/mdear/4cd8ee0bfb807840c64b34a6569a8949
@NeilSlater

I'm now considering using laion/CLIP-ViT-H-14-laion2B-s32B-b79K OpenCLIP for embeddings against Pinecone for natural language semantic search access to the growing corpus of AI summaries and e-learning content generated from this video series as part of a RAG architecture.

I think I may move towards BLIP and pre-translate any non-English search text into English as BLIP and BLIP2 purportedly better aligns objects represented by both text and images into a single aligned embedding, whereas CLIP can only accept either text or images and puts them in a semantically similar space.  I'm looking for a model that can understand the relationships among the modalities to produce a better embedding that incorporates aspects of all modalities including cross-understanding (using text to inform image understanding and vice versa).


text:
Autodistill: Train YOLOv8 with ZERO Annotations
I'm trying to create a distilled model of SAM
https://colab.research.google.com/drive/1PKc5x1XjxOPP-2msFp6uMSULvnm9dorw?usp=sharing
I'm able to label the data and train, but the predict doesn't find anything
Please help
I tried:

remove the whole doc BB by removing the biggest bounding box, it still return empty
Train on JPG/PNG, same result

NOTE that I am able to train with autodistill/GroundedSAM on the same dataset (I use it to find the document boundaries for auto cropping)
The purpose is to find bounded area in a document
SAM can do it well but it's heavy and slow, therefore I'm trying to create a Autodistill YOLOv8 that will find the bounded area in a document
I'm not sure that YOLOv8, because the concept of document is foreign to YOLOv8, is there a better way to create a SAM distilled model?
I can see that the labels are assign correct:

It looks like the problem is that it is also predicting a bounding box around the whole document. Overlapping boxes of the same class will confuse the model training process (I believe only the biggest one will get used).
You may want to try some different prompts in your ontology to see if you can get it to label things better.

I change the code to remove the whole doc BB by removing the biggest bounding box, it still return empty

also, it should have return the whole doc BB, not empty

Looking at the training logs in the Colab it looks like not nearly enough images in the training set (8; try with at least 100) and for such a small dataset you will want to train for many more epochs (it only got to 2% mAP after 50).

I tried with more images (87), I would expect it to at least return something, note that I use the same dataset to train another model with autodistill and GroundedSAM and it was working ok

Is this all in the same Colab environment? Maybe try resetting it and starting from scratch

Yes, I tried to reset it more than a few times