text:
More iterations of Deep Q Learning yields worse results
As a learning exercise, I am trying to implement Q-Learning and Deep Q-learning for the two player game Dots and Boxes. I have implemented Q-Learning, and it is working great. I have one model that plays itself many times. The more games it plays during training, the better it becomes at playing as both player 1 (the one who starts) and player 2. However, when I do the analogous thing for DQL, the model gets worse the more it plays. In particular, after training, when the model plays against a player that chooses actions randomly, it has a certain win-loss-tie record. The more I had trained the model, the worse this record becomes. Through various posts on this site and elsewhere, I do not necessarily expect this self-play algorithm to yield a model that plays optimally, but I was hoping it at least improved as it learned!
For the life of me, I have been unable to find any errors in my implementation, but of course I assume they are there. Or, is this outcome what I should be expecting?
The code for playing the game many times and creating a memory that stores actions and rewards is very similar to my QL code, which works well. The new part is the optimization, and then every 20 games I copy the policy network to the target network.
Below is a sketch of the (presumably flawed) DQL optimization.
current_q_values, target_q_values = [], []
policy.train()  # policy and target are subclasses of torch.nn.Module

for (
    state, # tensor representing a state
    turn, # the player who performed the action
    action, # integer specifying an edge
    new_state, # tensor representing the new state after the action
    reward, # float
    new_turn, # the player who will perform the next action
    game_over, # bool
) in memory_batch:

    if game_over:
        # When the game is over, the target q value is set to the reward
        targetq = torch.FloatTensor([reward])
    elif turn == new_turn:
        # player `turn` performed an action `action` such that
        # it is still their turn
        with torch.no_grad():
            targetq = torch.FloatTensor(
                reward
                + discount * target(new_state).max()
            )
    else:
        # player `turn` performed an action `action` such that
        # it is now the other player's turn
        with torch.no_grad():
            targetq = torch.FloatTensor(
                reward
                - discount * target(flip_state(new_state)).max()
                # flip_state makes it so that the state is now in
                # the perspective of player `new_turn`
            )

    # current Q values
    current_q_values.append(policy(state))

    # Get the target set of Q values
    target_q_value = target(state)
    target_q_value[action] = targetq
    target_q_values.append(target_q_value)

# Loss for the whole batch. loss_fn = torch.nn.MSELoss()
loss = loss_fn(torch.stack(current_q_list), torch.stack(target_q_list))

# Optimize the model
optimizer.zero_grad()
loss.backward()
optimizer.step()

For reference, my Q-Learning implementation (ie the one that works well) of the Bellman equation is below.
if game_over:
    Q[(state, action)] = reward
elif new_turn == turn:
    # player `turn` performed an action `action` such that
    # it is still their turn
    Q[(state, action)] = (1 - lr) * Q[(state, action)]
        + lr * (
            reward
            + discount * max(Q[(new_state, a)] for a in actions)
        )
else:
    # player `turn` performed an action `action` such that
    # it is now the other player's turn
    Q[(state, action)] = (1 - lr) * Q[(state, action)] +
        lr * (
            reward
            - discount * max(Q[(flip_state(new_state), a)] for a in actions)
            # flip_state makes it so that the state is now in
            # the perspective of player `new_turn`
        )

I cannot figure out why the latter works great but the former does not. Any help is greatly appreciated, thanks!


text:
Biological approch for AI consciousness?
To me it is very obvious that consciousness can only result from

Existing in a confined area that allows differentiation between self and environment
having basic desires i.e. staying alive, risk aversion
Cut motivations from a human and he will die. We exclusively do something to satisfy desires.

Are there research groups working on AI that has stuff like that programmed, coupled with the ability to recieve sensory inputs from its environment?

"To me it is very obvious that..." - Explaining why you think that might help you get useful answers. As would defining consciousness.

Consciousness is the ability to reflect about existence. Being aware to exist, requires formation of an ego. This is naturally driven by experiencing cause and effect between the confined body and the external environment. This requires sensory inputs.

Without needs there could not be an ego. The brain would not have a reason to do anything. Every activity is driven by a need.

While I can´t exclude alternative formations of consciousness it seems most promising to take nature as an example.

It's all good as long as you provide the definition you're using here. Different people mean different things when they use that word, so it's good to define it up front so you don't end up talking past each other.

These are fine ideals, but it comes down to how. Despite the enthusiasm, our methods are still crude. Success is often measured statistically, which means that even when everything is the same between runs, solutions might be different (even as much as never learning).
Reinforcement learning is most similar to what you describe. We give an agent a stimulus, and it takes action to maximize the reward. Research is grappling with how to do this on ever-larger scales in more reliable ways. Of course, inspection into what is learned is an ever-growing science, and without it, we cannot determine what notion of “self” an agent might have.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
I have a doubt in inference using variable elimination in Bayesian network

how to calculate number of operations in this question

we can’t tell you until you give us the definition of each random variabile

I added the network can you explain the process

@Curious_knight welcome to AI Stack Exchange! If possible, please discuss the difficulties you are having with this problem or the steps you have taken when trying to find a solution. Thank you for posting, and we hope to see more of your questions here soon!

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Issue with PPO agent taking action, sticking to a fixed value during training after 100K time-step
I'm facing an issue during training where my PPO agent consistently chooses actions close to the upper bound of the action range, sticking to a fixed value after 500K time-steps. The actions are within the range of (1e-12, 1e-3), and I'm using action scaling. What could be the main problem?
class SNLSOptEnv(gym.Env):
def init(self, task_name="Becker_Science2010", log_scale=False):
super(SNLSOptEnv, self).init()
    self.task_name = task_name
    self.log_tolerance = []

    # First action is the relative tolerance, second action is the absolute tolerance
    if log_scale:
        self.action_space = spaces.Box(low=-12, high=-3, shape=(2,), dtype=np.float32)
    else:
        self.action_space = spaces.Box(low=1e-12, high=1e-3, shape=(2,), dtype=np.float32)

    # Initialize variables for previous actions and performance metrics
    self.previous_action = np.zeros(2, dtype=np.float32)  # [NEW]
    self.previous_val = 0.0  # [NEW]
    self.previous_oval = 0.0  # [NEW]

    # Update the observation space to reflect the new state size
    # We'll define the state to include specific features
    num_features = 10  # Adjust this number based on the actual state size
    self.observation_space = spaces.Box(low=-np.inf, high=np.inf, shape=(num_features,), dtype=np.float32)

    self.cur_state = dict()
    self.previous_error = 0
    self.step_count = 0
    self.use_log_scale = log_scale

    self.eng = matlab.engine.start_matlab()
    self._init_matlab_engine()
    self.init_rtol = self.eng.eval("ar.config.rtol")
    self.init_atol = self.eng.eval("ar.config.atol")

    self.noise = NormalActionNoise(mean=np.zeros(2), sigma=0.03 * np.ones(2))

def _init_matlab_engine(self):
    self.eng.addpath("stepwise_snls/")
    self.eng.cd("arFramework3/Examples/" + self.task_name)

    out = StringIO()
    err = StringIO()
    self.eng.eval("arLoadLatest", stdout=out, stderr=err)
    recompile = False
    if "Warning: No results folder exist" in out.getvalue():
        print(f"No results folder found, creating a new one")
        self.eng.mkdir("Results", nargout=0)
        recompile = True

    if "Please compile" in out.getvalue() or recompile:
        print(f"Not found compiled model, now trying to compile the model, it may take a while")
        self.eng.Setup(nargout=0)
        self.eng.arSave('', nargout=0)

    # Save the initial ODE parameter
    self.init_ode_param = self.eng.eval("ar.p", nargout=1)

def parse_snls_optim_output(self, optim_data):
    # This function can be implemented to parse optim_data if needed
    return optim_data

def parse_snls_rl_output(self, rl_data):
    state = {}
    for key, value in rl_data.items():
        if key == 'success':
            state['success'] = 1
        elif key == 'EXITFLAG':
            state['EXITFLAG'] = value if not np.isnan(value) else 0
        else:
            state[key] = value
    return state

def step(self, action):
    self.step_count += 1

    # Add noise to the action for exploration
    noisy_action = action + self.noise()
    print(f"Step {self.step_count}: Noisy Action before clipping: {noisy_action}")

    # Clip the action based on the scale
    if self.use_log_scale:
        noisy_action = np.clip(noisy_action, -12, -3)
    else:
        noisy_action = np.clip(noisy_action, 1e-12, 1e-3)

    print(f"Step {self.step_count}: Clipped Noisy Action: {noisy_action}")

    # Apply the action to the MATLAB environment
    self.apply_action_to_matlab(noisy_action)

    # Execute one step in the SNLS optimization via MATLAB
    optim_out, rl_out = self.eng.arSnlsStep(nargout=2)
    rl_state = self.parse_snls_rl_output(rl_out)
    optim_state = self.parse_snls_optim_output(optim_out)
    self.cur_state = {**rl_state, **optim_state}

    # Extract necessary values from rl_state
    val = rl_state.get('val', 0.0)
    oval = rl_state.get('oval', 0.0)

    # Compute performance deltas
    delta_val = val - self.previous_val  # [NEW]
    delta_oval = oval - self.previous_oval  # [NEW]

    # Update previous metrics
    self.previous_val = val  # [NEW]
    self.previous_oval = oval  # [NEW]

    # Construct the next observation explicitly
    next_obs = np.array([
        val,
        oval,
        rl_state.get('diff', 0.0),
        rl_state.get('runtimeODE', 0.0),
        self.previous_action[0],
        self.previous_action[1],
        delta_val,
        delta_oval,
        rl_state.get('iter', 0.0),
        rl_state.get('ratio', 0.0)
        # Add more features if needed and adjust num_features accordingly
    ], dtype=np.float32)

    # Update previous action
    self.previous_action = noisy_action  # [NEW]

    # Calculate the reward
    reward = self.calculate_reward(rl_state, optim_state, noisy_action)

    # Check if the episode is terminated
    terminated, truncated = self.is_done(rl_state, optim_state)

    return next_obs, reward, terminated, truncated, {"EXITFLAG": rl_state.get("EXITFLAG", 0)}

def is_done(self, rl_state, optim_state):
    # Determine if the episode should terminate
    terminated = rl_state.get("EXITFLAG", 0) > 0
    truncated = False  # You can set conditions for truncation if needed
    return terminated, truncated

def calculate_reward(self, rl_state, optim_state, action):
    val = rl_state.get('val', 0.0)
    oval = rl_state.get('oval', 0.0)
    runtime = max(rl_state.get('runtimeODE', 1.0), 1e-8)

# Improvement calculation, avoiding division by small oval
    improvement = (oval - val) / oval if oval > 1e-8 else 0.0

# Penalty for actions near the boundaries
    action_range = self.action_space.high - self.action_space.low
    normalized_action = (action - self.action_space.low) / action_range
    action_penalty = np.sum(np.abs(normalized_action - 0.5)) * 0.05  # Reduced penalty

# State-dependent reward (success bonus)
    success_bonus = 1.0 if rl_state.get('success', 0) == 1 else 0.0

# Final reward calculation
    reward = improvement - (runtime / 10) - action_penalty + success_bonus
    return np.clip(reward, -10.0, 10.0)

def apply_action_to_matlab(self, action):
    if self.use_log_scale:
        # Convert action from log scale to real scale
        action = 10 ** action  # Assuming log base 10
    # Ensure the action is in the correct format for MATLAB
    action = np.clip(action, 1e-12, 1e-3)

    print(f"Step {self.step_count}: Passing action to MATLAB - rtol = {action[0]}, atol = {action[1]}")

    # Pass the action to MATLAB
    self.eng.eval(f"ar.config.rtol = {action[0]};", nargout=0)
    self.eng.eval(f"ar.config.atol = {action[1]};", nargout=0)

def reset(self, seed=None):
    print("Resetting the environment")
    self.eng.clear("arSnlsStep", nargout=0)

    # Reset ODE Parameter
    if seed is not None:
        self.eng.eval("ar.p = arRandomPars(1, %d);" % seed, nargout=0)
    else:
        seed = np.random.randint(low=0, high=2 ** 32 - 1)
        self.eng.eval("ar.p = arRandomPars(1, %d);" % seed, nargout=0)

    # Reset variables
    self.cur_state = np.zeros(self.observation_space.shape, dtype=np.float32)  # [MODIFIED]
    self.previous_error = None
    self.step_count = 0

    # Initialize previous actions and metrics
    self.previous_action = np.zeros(2, dtype=np.float32)  # [NEW]
    self.previous_val = 0.0  # [NEW]
    self.previous_oval = 0.0  # [NEW]

    return self.cur_state, {}

def render(self, mode='human'):
    raise NotImplementedError

def close(self):
    self.eng.quit()


text:
Is there a suitable supervised learning algorithm to use instead of deep Q-learning?
I have a problem writing a deep q-learning model for a multi-agent game.
The game has very few states. I mean it only has 10 states at most and around 71 actions which some of them have some correlation.
My problem is I have to write the whole model from scratch.
Some identical functions need to be considered twice because the model is confusing and I don't have much time.
If there has been a supervised learning or a backward temporal difference algorithm I would be rescued.
I don't want to waste my time rewriting everything from scratch.
Edit: The set of possible actions in each round is a sampled subset of 71 actions.
Edit 2: I tried to use multi-agent reinforcement learning but the loss function doesn't decrease at all. Using the network for all players isn't helpful. What should I do? The terminal reward depends on the players and It changes based on their behavior. What should I do?

10 states and 70 actions can be handled with tabular RL

A player doesn't know whether the rest of players play which actions, so it basically isn't converging.

It's not clear, but you might not have the problem/need for supervised learning algorithm that you think you have. It may be better to explain your project, what you have tried, what happened and why you are stuck on what to do next.

It's worth explaining why you don't model or account for other player choices when considering each action. That's more of a game theory thing. But how the game works has a huge impact on which RL approaches work

For clarification, a supervised learning approach is possible if you have some way other than random exploration to set the correct action choice. E.g. a full tree search using minimax. Then you could train a model to predict the policy

Terminal reward changes based on the numbers guessed by the players and their behavior in the game. Using a single network for all players isn't useful. The loss only increases. What am I doing wrong?


text:
Specifying different actions in reinforcement learning
I have a problem with my reinforcement learning project. The project is defined as follows.
The agent chooses a number and based on that number it should do a task in every state.
When the number of tasks accomplished by the agent exceeds that predefined number, the agent shouldn't win any other tasks.
In other words, the agent should play for winning and achieving something in each state when the number of acquired tasks is less than that pre-defined number and it should play for losing when the the number of acquired tasks exceeds that pre-defined number.
My solution is to define two actions for every task. One for losing and one for winning. This means the size of output layer of Q-network should be twice of a game that only requires winning every task. When the player needs to win (the number of acquired tasks is less than the pre-defined number) it should choose the winning action and when it needs to lose it should choose the losing action.
Is my understanding of the problem correct? I tried to train the network without separating the actions and my network always played for winning (even when the agent needs to lose in the remaining states) because the data sampled during training is biased towards winning.
How should I define the actions, rewards and transitions?
Could you help me? Sorry for my weak English.
Edit: The problem is that when I define two different actions, I don't know what to do with the terminal reward. Is the terminal state when the player achieves enough tasks or when it reaches the end of episode? In other words should I slice the episode into two episodes or not? I am really confused.

What “rewards” are you using to estimate the Q of losing actions?


text:
Do any Machine Learning Models Use Hardware Differently?
My understanding is that most if not all of the current models are running on top of basically the same hardware operations.
That is to say, typically library calls to pre defined gpu routines that are essentially the same across all current models.
The one exception I imagine would be with tensor chips -- although I'm not too familiar with that.
Are there any current Machine Learning Models that manipulate gpu hardware differently?

The scope of this question seems unclear. Are you familiar with the difference between a FG layer, a LSTM layer and a CNN layer? Transformers? Attention?

@MSalters what I discovered is most models basically are using the same "frameworks" pytorch and tensorflow

True, but that says nothing about the hardware. Those are high-level software interfaces. At a lower level, the software interface is usually NVidia's CUDA. Again, a software abstraction, not hardware.

Differently from what?

They are all the same, which is basically number. Numbers can be represented in many ways, such as with digital technology (current GPU technology) and analog technology, like Mythic AI or IBM analog computing.
If you are referring to whether a neural block (tensor layer) is using specialized hardware, for example, an attention mask must be using specialized hardware, then no. It's basically the same (they are just numbers that can be represented in many ways, such as bits, qubits, or continuous values from voltage).
In digital technology, it's basically using floating point operation from IEEE 754 convention. That's why you commonly see the specs of an AI accelerator using FLOPS (Floating Point Operation Per Second).
While analog technology, the number is not represented digitally from the convention of IEEE 754, it might be using a physical unit such as voltage, or ampere, or even ohm.
But there is a special case for Neuromorphic computing that using real human neurons, which operated on reward and punishment, the reward system is using chemicals addictive like drugs, while the punishment system simply randomly electrifies the neuron. But I'm still not sure the number interpretability for such technology.


text:
Unclear points regarding ROC curve in machine learning
Consider the ROC curve below:

It is not clear to me why "PERFECT CLASSIFIER" line is really perfect because there are points on this line (see the green circles) for which false positive rate is non-zero.

The "PERFECT CLASSIFIER" line is perfect because at the optimal threshold (somewhere near the top left corner, with TPR = 1 and FPR = 0), it makes no classification errors. The non-zero FPR points you see occur when you change the cutoff threshold score (for instance, due to imbalanced dataset or cost sensitive learning ), which includes more negatives in the positive class. The ROC curve is a summary of classifier behavior across all thresholds, but the classifier itself can still be perfect if you set the optimal threshold.

Question: Does the ROC curve represent a classifier with the same parameters under different thresholds ? Are we allowed to change the model parameters of the classifier in the ROC curve?

@DSPinfinity Indeed, see ROC: Given a threshold parameter ${\displaystyle T}$, The ROC curve plots parametrically ${\displaystyle {\mbox{TPR}}(T)}$ versus ${\displaystyle {\mbox{FPR}}(T)}$ with
${\displaystyle T}$ as the varying parameter.

are the model parameters fixed?

@DSPinfinity $T$ is a varying parameter of the model as explicitly described in my above comment.

@DSPinfinity: It's probably useful to consider two classifiers f(x) and g(x)=2*f(x). Clearly, they have diffrent parameters, the same ROC curve, but T_g=2*T_f. And obviously, g is perfect if and only if f is perfect.

An ROC curve, strictly speaking, does not represent "a classifier", but a scoring rule that can be thresholded at different levels to yield many different classifiers. So, the line labeled "perfect classifier" is actually a bit of a misnomer - it's not actually a classifier, it's a scoring rule, that when combined with a threshold, yields a classifier with particular TPR/FPR.
A "perfect classifier" yields a specific set of classifications that have 100% sensitivity and 100% specificity, which would be represented by the single point in the upper-left of the ROC space. Any other point in ROC space represents the existence of classification errors. Any of the other points on the "perfect classifier" line are classifiers that could be implemented with the "perfect" scoring rule, but which would be needlessly sub-optimal as they sacrifice sensitivity or specificity for no reason.
For the other sub-optimal scoring curves, there may be domain reasons why one wishes to trade sensitivity for specificity or vice versa, meaning the whole ROC curve represents potentially useful classifiers. But with the perfect scoring curve, there is no reason to choose a threshold that gives less than 100% sensitivity/specificity, meaning the "perfect ROC curve" represents one classifier you'd actually use in practice (located at (0,1)), and many other classifiers which serve no practical purpose (which are located anywhere else on the ROC curve).

“Perfect classifier” might take it too far, but “perfect separation” is a good description, where “perfect separation” means that there is some threshold where all predictions corresponding to one category are above that threshold while all predictions corresponding to the other category are on below that threshold. Consider, for example, if all predictions corresponding to category $0$ are between $0.1$ and $0.4$, while all predictions corresponding to category $1$ are between $0.6$ and $0.9$.
In that case, the separating threshold gives a decision rule with perfect sensitivity and perfect specificity. However, other decision rules are possible. “Always classify as zero” gives perfect specificity at the expense of no sensitivity. “Always classify as one” gives perfect sensitivity at the expense of no specificity. These could correspond to thresholds of $2$ and $-1$, respectively.
If you use a threshold like $0.2$ or $0.7$, however, you wind up with perfection in either sensitivity or specificity and and imperfect, but not zero, value of the other metric. This explains the points you see in the plot.
Let's look at some plots (code given at the end).

From the box plots, we see that the lowest value of the blue group (1) is greater than the highest value of the red group (0): no overlap. That is, there is perfect separation between the two groups. Interpreting these as predictions for the subjects corresponding to groups 0 and 1, respectively, we can plot a ROC curve based on these.

This shows the thresholds to which each point on the ROC plot corresponds. You can use many more values at the thresholds, as is common in software implementations of ROC curves (even when there is perfect separation).

Finally, let's look at the box plots as we move the threshold from $-\infty$...

...to $0.25$...

...to $0.5$...

...to $0.75$...

...to $+\infty$.

No matter where the threshold is, at least one group has its members predicted perfectly, so we always have at least one of sensitivity = 1 or specificity = 1. However, when we do not use a threshold that is between the highest prediction for group 0 and the lowest prediction for group 1, we do not get sensitivity = 1 and specificity = 1 simultaneously.
R CODE
library(ModelMetrics)
library(ggplot2)
set.seed(2024)
N <- 1000
p0 <- runif(N, 0, 0.4)
p1 <- runif(N, 0.6, 1)
y0 <- rep(0, N)
y1 <- rep(1, N)
p <- c(p0, p1)
y <- c(y0, y1)

d1 <- data.frame(
  y = as.factor(y),
  p = p
)
g1 <- ggplot(d1, aes(y = p, col = y)) +
  geom_boxplot() +
  scale_color_manual(values=c("black","blue")) +
  theme(legend.position = "bottom")

thresholds <- c(-Inf, 0.25, 0.5, 0.75, Inf)

tprs <- fprs <- rep(NA, length(thresholds))

for (i in 1:length(thresholds)){

  classifications <- rep(0, length(p))

  # Determine which p-values are above threshold i
  # Assign those to category 1 and, otherwise, to category 0
  #
  idx <- which(p > thresholds[i])
  classifications[idx] <- 1

  # Calculate TPR and FPR for ROC curve
  #
  tprs[i] <- ModelMetrics::tpr(y, classifications)
  fprs[i] <- 1 - ModelMetrics::tnr(y, classifications)
}

d2 <- data.frame(
  FPR = fprs,
  TPR = tprs,
  Threshold = as.factor(thresholds)
)

g2 <- ggplot(d2, aes(x = FPR, y = TPR, col = Threshold)) +
  geom_point() +
  theme(legend.position = "bottom")

# Redo with thresholds at all values

thresholds <- c(-Inf, p, Inf)
tprs <- fprs <- rep(NA, length(thresholds))

for (i in 1:length(thresholds)){

  classifications <- rep(0, length(p))

  # Determine which p-values are above threshold i
  # Assign those to category 1 and, otherwise, to category 0
  #
  idx <- which(p > thresholds[i])
  classifications[idx] <- 1

  # Calculate TPR and FPR for ROC curve
  #
  tprs[i] <- ModelMetrics::tpr(y, classifications)
  fprs[i] <- 1 - ModelMetrics::tnr(y, classifications)
}

d3 <- data.frame(
  FPR = fprs,
  TPR = tprs,
  Threshold = thresholds
)

g3 <- ggplot(d3, aes(x = FPR, y = TPR, col = Threshold)) +
  geom_point() +
  theme(legend.position = "bottom")

# Create a ROC curve using a standard package to compare to the results here
#
r <- pROC::roc(y, p); r$thresholds`

################################################################################
################################################################################

# Let's look at what happens when we apply the original five thresholds

thresholds <- c(-Inf, 0.25, 0.5, 0.75, Inf)
Ds <- Gs <- list()
for (i in 1:length(thresholds)){

    regions <- data.frame(
    xmin = c(-Inf, -Inf),
    xmax = c(Inf, Inf),
    ymin = c(-Inf, thresholds[i]),
    ymax = c(thresholds[i], Inf),
    Classification = c("Classified as 0", "Classified as 1")
  )
  Ds[[i]] <- regions

  Gs[[i]] <- ggplot() +
    geom_rect(
      data = regions, aes(
        xmin = xmin,
        ymin = ymin,
        xmax = xmax,
        ymax = ymax,
        fill = Classification
      )
    ) +
    geom_boxplot(
      data = d1,
      aes(
        y = p,
        col = y
      )
    ) +
    scale_fill_manual(values=c("gold","lightgreen")) +
    scale_color_manual(values=c("black","blue")) +
    theme(legend.position = "bottom")

}

# Look at the plots
#
g1
g2
g3
Gs[[1]]
Gs[[2]]
Gs[[3]]
Gs[[4]]
Gs[[5]]

Because this means you have already found all true positives. And after you found all true positives you are left with false positives only.
Imagine three classifiers sorting candidates by probability of being positive (from left to right):
Random classifier:  0 1 0 1 0  0 0 1 0 1  0 1 0 1 0  1 0 1 0 1
Better classifier:  1 0 1 1 1  1 1 0 1 1  0 0 1 0 0  0 0 0 1 0
Perfect classifier: 1 1 1 1 1  1 1 1 1 1  0 0 0 0 0  0 0 0 0 0

Now imagine you are searching for fraud in insurance claims and you need one day for a claim to be checked. Using random classifier it would take you 20 days to catch all fraudulent claims. With better classifier you will find 8/10 (80%) positives after 10 days (starting from left) and 19 days for all fraudulent claims to be caught. Using perfect classifier you catch 100% of positives after just 10 days. What is left are false positives only, nothing but zeros.


text:
Why do LLaMa and its variants have non-“round” numbers of parameters?
LLaMa was released in several sizes, with 7B, 13B, 33B, and 65B parameters. These values look a little weird, because they are very close to powers of two (8, 16, 32, 64) which are more conventionally considered “round numbers” in software. Why were these specific numbers chosen?

Can you please provide a reference that explains what these "round numbers" are in "software"? I've been programming for a few years and I've never encountered such an expression, which, of course, doesn't mean that it doesn't exist or doesn't define a value concept.

@nbro round numbers

At first I thought that they were just leaving some headroom by using less-than power-of-two number of parameters, but 33 and 65 actually break this pattern. But I still think it is memory-usage related and they aim to have the full system working at 16, 32, 64 etc. GBs of memory.
There are some hints at their GitHub repository (sunset as of 2024)

Accounting for 14GB of memory for the model weights (7B model), this leaves 16GB available for the decoding cache which stores 2 * 2 * n_layers * max_batch_size * max_seq_len * n_heads * head_dim bytes.
With default parameters, this cache was about 17GB (2 * 2 * 32 * 32 * 1024 * 32 * 128) for the 7B model.

I don't think developers of Llama (and LLMs in general) always work backwards from a chosen model size. They instead choose model hyperparameters such as model dimension, vocabulary size, and number of decoder blocks. For Llama 3, these numbers are 4096, ~128K, and 32.
Model size is then a consequence of chosen hyperparameters.
Llama 3 8B model for example has exactly 8,030,257,152 trainable parameters. See this post for complete derivation.


text:
Is it possible to achieve the transcript accuracy of Whisper with the timestamp accuracy of Vosk in speech-to-text tasks?
I need to transcript some speech files. I've played around with a number of tools and I found that:

OpenAI Whisper transcribes speech with absolute accuracy. It's even more accurate than me.

However the timestamps it gives are very imprecise: sometimes they are several seconds off.

VOSK has a fantastic support for timestamping words: if you crop the audio file at the start and end of any word detected by VOSK, you hear that word precisely.

However it's far less accurate in transcription: it often misunderstand speech.

Is there any way to achieve high accuracy in both tasks?
I have tried a few tools that aim at improving Whisper's timestamp accuracy. Namely WhisperX and whisper-timestamped, but neither of them is remotely close to VOSK.

I'm considering trying to combine Whisper and VOSK: maybe I could transcript some speech using both tools, then map each word detected by VOSK on those detected by Whisper, so that I can use VOSK's timestamps for Whisper's words.
However this task seems quite tricky, since many words are going to differ. I wonder whether anyone has attempted this or found better solutions.

how can vosk be accurate for timestamps but bad at predicting words?

you can take the timestamps of Vosk and the words from Whisper, and should be good to go no? like "vosk tells you the 4th word starts at 4.1s and ends at 5.3s and whisper tells you that the 4th word is blablabla"

@Alberto: try Vosk and you'll see. There's no obvious mapping between the words generated by the two tools. Some words are missing, others are extra, whole phrases differ.

Curious about what you're building. I needed timestamps to create subtitles and followed this tutorial.
https://picovoice.ai/blog/how-to-create-subtitles-for-any-video-with-python/
https://picovoice.ai/docs/leopard/#add-custom-vocabulary
I was happy with the results, because I could add new words and boost them pretty quickly. I couldn't figure out how to customize VOSK though but it could be me. lol
They were gonna add timestamp and diarization features to whisper, but i dont think they did, worth exploring though https://deepgram.com/openai-whisper

Checkout stable-ts (Transcription, forced alignment, and audio indexing with OpenAI's Whisper)


text:
dropout test phase probability calculation
This following slide is from Deep Learning Lecture by Justin Johnson that is accessible on Youtube.
This is part is about how to handle dropout during test time.
I derived the equation just like the green box, but it is different from the original text. What I have done wrong?

You're taking an expectation of a function $f$, which depends on 2 variables (including $z$), with respect to $z$.
The general formula for an expectation as an integral is $E_z[f(z)] = \int p(z) f(z) dz$, where $p$ is the density.
See the law of unconscious statistician.
So, it's not $p(x, y) = p(x | y) p(y)$. That's something different.

Your green box derivation is only valid for computing marginal distribution $f(x)$ from the joint probability distribution $f(x,z)$ via chain rule, assuming $f(x,z)$ is a joint distribution first. However, here for test time drop out to rid of random result from random mask $z$, the neural network mapping $f(x,z)$ is not a joint distribution but some deterministic universal approximating function.


text:
Derivation of Sutton & Barto TD(λ) Weight Update Equation with Eligibility Traces
I'm working through Sutton & Barto's Reinforcement Learning: An Introduction, 2nd edition, and trying to understand the derivation of Equation 12.7 for TD(λ) weight updates in Chapter 12, specifically when using eligibility traces. Here’s the update equation in question:
$$\bf{w}_{t+1} = \bf{w}_t + \alpha \delta_t \bf{z}_t$$
In this equation:

$\bf{w}_{t+1}$ is the updated weight vector.
$\alpha$ is the learning rate.
$\delta_t$ is the TD error defined as:
$\delta_t = R_{t+1} + \gamma \hat{v}(S_{t+1}, \bf{w}_t) - \hat{v}(S_t, \bf{w}_t)$
$\bf{z}_t$ is the eligibility trace vector, which is recursively defined as:
$\bf{z}_{t} = \gamma \lambda \bf{z}_{t-1} + \bf{x}(S_t)$

My Question:
I attempted to derive Equation 12.7 from first principles, but encountered an issue. Using the definitions provided, my derivation included an additional term:
$$\bf{w}_{t+1} = \bf{w}_t + \alpha \delta_t (\bf{z}_{t} - \gamma \lambda \bf{z}_{t-1})$$
Expanding this leads to:
$$\bf{w}_{t+1} = \bf{w}_t + \alpha \delta_t \bf{z}_t - \alpha \delta_t \gamma \lambda \bf{z}_{t-1}$$
This extra term $- \alpha \delta_t \gamma \lambda \bf{z}_{t-1}$ does not appear in Sutton & Barto's formulation. I suspect that the recursion within $\bf{z}_t$ might implicitly account for this term, but I would like to confirm that understanding and clarify why it can be omitted.
Key Points I'm Looking For:

A step-by-step derivation from the definitions, starting from the TD error and eligibility trace definitions to show how the final weight update form is obtained without needing the extra term.
An explanation, if possible, of how $\bf{z}_t$ itself encapsulates the historical contributions from prior time steps.

Any references to additional literature that validate this would also be very helpful. Thank you!

Your concern about the extra term seems due to a symbolic error. When you compute $\bf{z}_t$, it already incorporates the contribution of the previous eligibility trace $\bf{z}_{t-1}$ due to the term $\gamma \lambda \bf{z}_{t-1}$ in your above definition which is (12.5) in S&B book, and the contributions of all previous eligibility traces according to their recency with its obvious recursive nature which can be shown from the below corrected weights update formula:
\begin{align*}
\bf{w}_{t+1} &= \bf{w}_t + \alpha \delta_t (\gamma \lambda \bf{z}_{t-1} + \bf{x}(S_t)) \\
&=\bf{w}_t + \alpha \delta_t (\gamma \lambda (\gamma \lambda \bf{z}_{t-2} + \bf{x}(S_{t-1})) + \bf{x}(S_t)) \\
&= \bf{w}_t + \alpha \delta_t (\gamma^2 \lambda^2 \bf{z}_{t-2} + \gamma \lambda \bf{x}(S_{t-1}) + \bf{x}(S_t)) \\
&=...
\end{align*}
The eligibility trace serves as a dynamic memory of past state contributions automatically decaying their influence over time. Therefore the adjustment for $\bf{z}_{t-1}$ does not need to be included in the approximated value function's weight vector update at time step $t$ as you claimed, as it is already built into $\bf{z}_t$ as shown above corrected formula.

Thank you for the explanation, however, I’m still puzzled why, in a purely symbolic derivation from the definitions provided, this extra term doesn't disappear explicitly. Intuitively, it seems that mathematically it should either cancel out or be shown redundant. Does this behavior suggest an approximation rather than an exact equality, or is there a way to show that $\bf z_t$ fully encapsulates all past influence in a way that aligns symbolically with equation 12.7?
Thank you again for the help

@AJR I've updated per your request, seems you have a symbolic error in your question.


text:
Which model should I apply on sequential data?
I need to predict a binary vector given a sequential dataset meaning the current datapoint depends on its predecessors as well as (known) successors.
So, it looks something like this:
Given the sequence: X = [x_1, x_2, x_3, ..., x_N]
I want to predict: Y = [y_1, y_2, y_3, ..., y_N] with y_i \in {0, 1} a binary label
Now, the sequence X is fully known, that is: future observations in the sequence are completely known at any time.
Therefore, in contrast to normal time series data, I can also harness any x_i+1, x_i+2,... from the full X sequence for predicting y_i at any time and not only x_i-2, x_i-1, x_i etc.
Also the data X is a sequence of R^dxN vectors, i.e. N d-dimensional datapoints containing real numbers.
In my case the dimensionality of the data is d=140.
Now, what I want to predict is the following:
What is y_i given X, e.g. what is y_3 given the observations x_1, x_2, x_3, x_4, x_5?
So, eventually I need something like this:
for i in range(N):
    predict y_i given X = [x_1, x_2, x_3, ..., x_N]

This actually is a many-to-one prediction task.
Now you could use a RNN, but the problem imho is that "future" observations are not taken into account when applying an RNN.
Maybe I am wrong about this assumption.
But therefore I am asking: Which model would you suggest to use for this problem?

Okay, I finally ended up using a Transformer based Encoder/Decoder model.
It is a very similar structure as the NLP model described in the Attention is all you need paper.
However, since my use case is not NLP but rather sequential data in general I used to follow the Transformer Encoder/Decoder method as described in this  paper: Deep Transformer Models for Time Series Forecasting
I slightly modified their model in my implementation to account for my special needs. I used pytorch to build the transformer architecture.
The results are very good.
I think when trying to predict sequential data, a encoder/decoder based transformer model will be a good choice and at least worth a look :)

A more recent model you can try for sequential data is the S4 model, which is a more lightweight and stable recurrent network that was able to perform better than transformer models on certain tasks. For time series forecasting in particular, there are cases where standard recurrent networks (LSTM, GRU, etc) are still preferable compared to transformer models.


text:
Is there any evidence that the bias terms help in the attention mechanism of the transformers?
In the original transformer paper, the attention mechanism uses parameter matrices, but no bias terms. However, in more recent implementations I see people often using a bias term when computing "key", "query", and "value". For example, in Andrej Karpathy's recent implementation of GPT, whether a bias term is used can be determined in the config:
bias: bool = True # True: bias in Linears and LayerNorms, like GPT-2. False: a bit better and faster

This makes me wonder whether there is any evidence that the bias terms help. In particular, if, according to Karpathy, not using bias is "a bit better and faster", why is he using them by default?

I guess that bias terms have the ability to increase the expression power of the model.According to the paper:"BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models".

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
Early divergence of YOLOv7-tiny train and val obj_loss plots
I am training a YOLOv7-tiny model and have the following observations from the training session:

the train and val objectness loss plots diverged pretty early on in the training process
the class and box losses, while not exactly diverging haven't converged either
the P, R and the mAPs seem to be ok.

The batch_size is 8 and the rest of the hyperparameters have default values as defined in the official repo's config file with some changes to the augmentation specific parameters. Below is the command i used to run the training:
$ python3 train.py --workers 4 --device 0 --batch-size 8 --data data/data.yaml --img 1088 1088 --cfg cfg/training/yolov7-tiny.yaml --weights "" --name train_dir --hyp data/hyp.scratch.tiny.yaml --epochs 200

There are 8 object categories; the train set has >30k images, validation set has >5k images.
What you guys think? I need advise with respect to interpreting the plots to identify where this is going wrong and the corrective actions that can be taken.


text:
Captioning an average of image set (instead of individial pictures)
I'm looking for a captioning model that would be able to describe a group of images in a single sentence. Alternatively, I need a way to conceptually average a group of images before feeding that "concept" (presumably a feature vector) to a regular captioning model.
Why?
For training dataset curation. It would be beneficial to test the trained generation model on a prompt that would fit it as a whole instead of selecting captions of single images or trying to find what's common among them. Moreover, this would also allow to produce a single negative prompt to test how model behaves on out of scope prompts.
What I've done so far:
I've modified an existing CLIP+BLIP interrogator to work with image sets. However, while CLIP captioning allows averaging image features before using them to choose the best caption, it's far less accurate than captions produced by BLIP, which only works with single images. I need a model that would take in feature vectors like CLIP so I can preprocess them.

ask on fitting stackexchange >get ignored >ask on SO >get closed

[ai.se] says "Q&A for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment" you could try asking on [stats.se] which is the engineering-minded sibling site, or [datascience.se]. be sure to check out each site's "on-topic" article in the [help/on-topic]. the big index: https://stackexchange.com/sites#questionsperday

This is the fourth time I'm asking this question in different SEs and every time I'm either ignored entirely or my question is closed without any help. I'm disappointed in this site, I've done my research, I built a prototype, everything checks out and yet I'm being given the middle finger.

@Seedmanc Your question has not been closed here, but anyways there are never guarantees of getting an answer. People might simply not know what you are asking.

see also: https://stats.stackexchange.com/questions/657030/captioning-an-average-of-image-set -- OP asked this twice on SO. -- I guess it's a matter of feeling entitled to something the community didn't promise.


text:
How transformers can be used to fix typos?
Can typos be detected only using "decoder only" transformers, or in "encoder" transformers as well (that lack the generative part)? This would assume typos in titles of songs, movies, video games, etc. that are autocorrected in a way similar in how search engines correct typos in these names.

Obviously the decoder-only transformers like GPT or CTRL models are more naturally suited for generative autocorrection tasks using BPE like subword tokenization methods along with their vast training corpus containing the correct titles of most songs and movies, or at least most of them are correct, as further explained here.
Encoder transformers like RoBERTa can also be used for typo detection and correction, alibeit in different ways. One possible way is to first prepare a RAG like vector database with embeddings of correct titles of songs and movies, etc, then use the pretrained encoder's output embedding of your input title with typos to do a similarity search across the vector database to find the closest match. If your vector database is large and sufficient, it should perform quite well for most popular songs or movies.


text:
Is there a fast way to raise the confidence levels of an object detection model?
I'm currently building an Object Detection Model to recognize food ingredients in refrigerator pictures. I use Google Colab to code and train my model, and I use the architecture of RESNET-50. I wiped the already existing classes and I'm training it on a dataset with about 6'000 images (already annotated, 70% train, 15% valid, 15% test) with Google Colab's best GPU. After training it's like my model still hasn't learnt anything. Is it normal to achieve such poor results or is there a way to train faster with better results? I've heard of Google Cloud but I don't know if it'll make a difference. Am I missing some hidden parameter?
Here is the Google-Colab Notebook I coded everything on: https://colab.research.google.com/drive/1m4zbNE8-qVUkep-37Mz2S3URvELdGWX6?usp=sharing
To train my model, I use pytorch_lightning's Trainer. Training needs a really long time though, and even after training for 30 epochs the confidence levels on unseen data remain below 5% when the preds are even right (which is not usually the case).
Edit: At the moment, I'm using a new, more qualitative Dataset containing 20k images divided into 68 classes. I haven't tested the model enough as I saw that the predictions were much too low, if you look at the Notebook you'll see that I have to set a CONFIDENCE_THRESHOLD_2 = 0.03 to get ANY results, but I'll assume that is due to the non-sufficient training.

Can you please provide information regarding the dataset as how many instances  per class you have mentioned preds are right so for if possible can you please share the insight of predictions like on which classes you are getting good results and bad results , and also which loss function and optimization you are using it would be best if you provide the colab file

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.


text:
How to analyze furniture for digital reconstruction?
My end-goal is to take a single photo of a piece of common furniture (couch, chair, table) and create a 3d model from that. I'm a novice with deep learning as I've only done basic CNN's and such with existing datasets (mnist).
To get 3d data from a 2d image what will I need in my training set?
My best guess is that I'll need 3d models & raw textures with corresponding 2d renders from multiple various angles.
Am I on the right track?
EDIT -- adding project overview
My current high-level overview for this project looks like the following:
Prep

Assemble a set of 1000+ couches from various angles with backgrounds removed (opencv)
ShapeNet, ModelNet for common furniture geometry datasets

Segmentation

use Mask R-CC to segment cushions, arms, backrest, etc
DeepLab/HRNet for finer segmentation as needed

Object Detection / Classification

YOLOv5/v8 for detection/labeling
CNN for component detection

Depth Estimation

DenseDepth (mono) or PASMNet (stereo)

3D Reconstruction via Blender Python API

NeRF for 3d mesh/model generation
Meshlab for mesh refinement & optimization

Shape Completion and Occluded Part Estimation

VAE/GAN/DCGAN? for prediction from dataset
DeepSDF for signed distance functions to predict full 3d from partial input

Texture Mapping

MaterialGAN to synthesize textures from photos

A lot is assumed between milestones so constructive criticism is welcome


text:
What Policy/Agent and Observation Spec To Use For TensorFlow Agents For Video Game Platformer?
I'm trying to create a model to beat a video game platformer I made a few months ago.  In the game, the platforms scroll down from the sky and the player has to keep jumping to them to avoid touching the ground.  The platforms scroll down faster and faster as the game progresses.
I've setup the environment and can run a training loop but the player can't seem to figure out how to jump up to the 4th platform and beyond.  They player can make it up the third platform though.
I've adjusted the rewards and let it train for over an hour with no results so I think it may be to policy or agent I'm using to train it with.
I also found when I added the platforms's X and Y to the observation spec, it made results worse. I think this is due to added complexity.
Here is a link to a web version of the game on itch.io


text:
Who attends to whom in Attention Weight Matrix? (Transformers)
If the question is: which token pays most attention to which other token, is that the same asking as which token attends to which other token?
What exactly does that mean? Do I have to look it from the Keys perspective or the Queries perspective?
Suppose I performed: $\text{softmax}(\frac{QK^T}{\sqrt{d_k}})$
Each row will sum up to 1.
If I have the sentence: I am good and attention weight matrix is $3 \times 3$.
If I want to know which token pays most attention to 'good', where do I have to look?
There are exactly two answers and I can't figure out which because I'm not sure if I'm not familiar with the lingo or if I'm confusing myself.
Each row sums up to 1. So we know that if you have the highest value in a row, then that is somewhat meaningful and attention is put on there. For vectors, it just means the value vectors is scaled strongly by those weights and less by others.
What that means for tokens is ambigious. I assume it means that if the model reads that token, it pays attention to the those tokens that have highest values in the row and that can in turn help it to figue out what the next token is. The idea is to use the context surrounding the token to figure out what the next token should be. Attention helps the model to pull this off.
On the other hand, if the question is: Which token pays most attention to token X?. Then it's more about which token does the model have to read to consider token X the most when predicting the next word. I think? So then you look at the columns but again, columns do not sum up to 1, so I could be very off here too. Could someone clarify?
TL;DR:
What does 'attend to' mean and is it related to 'token A pays attention to token B'?
How do I have to read the attention weight matrix to answer these questions?

a TLDR of all the questions would be the following:
1.) which token pays most attention to which other token, is that the same asking as which token attends to which other token?
Not exactly. The operation isn’t scalar but vectorial. When working with vectors, all weights and outputs influence the final output token. A single token doesn’t attend to another token directly; rather, all tokens attend to each other, and all their outputs contribute to the final output. It’s not only about the one with the highest value but about the entire vector's influence. Remember that transformers have various layers and multiple attention heads. You also have to consider the attention  matrix for the different heads to really know the final influence two tokens have with each other.
2.)  What exactly does that mean? Do I have to look it from the Keys perspective or the Queries perspective?
It doesnt mean anything in self-attention.The important part is not to have a Q or K matrix. Becuase we could have the same Linear Projections for Q or K and it woulnt mean that much in terms of Architecture( Graph Attention Models uses the same Projection Matrix, so theres no difference between Q and K in that case).
because at the end they use the same tokens for attention. The most important part is the they use the same tokens for self-attention becuase you want that the final output is a square matrix. all the tokens attends to all the other ones.
3.) Which token pays the most attention to token X?
If we define attention as the operation  $\space \text{Softmax}(\frac{QK^T}{\sqrt{d}})$, then the argmax of the softmax function can indicate the most attention given to a token pair. However, this doesn’t mean the other outputs don’t contribute to the final result; it just indicates which has the highest contribution weight.
Extra Comments:
The challenge with Transformers and these types of questions is that while the model works very well empirically, it lacks an extensive theoretical background . Therefore, I wouldn’t focus too much on the meaning and semantics of each operation,because there isnt a mathematical theory we can rely on.
I wouldn’t worry too much about the meaning of “paying attention” to a token, as this can vary based on interpretation. Instead, it’s best to rely on the mathematical definition of the attention mechanism,that is just an operation that has proven effective for sequential data in a empirical way. Additionally, there’s an interesting field called Mechanistic Interpretability that seeks to make Transformers interpretable across various tasks. It maps which operations contribute to the model’s final output. There are hypotheses, like Induction Heads for In-Context Learning, but in my opinion many results rely heavily on the model architecture, training data, and human pattern recognition, rather than on a deep or intrinsic understanding of how the operations translate to behaviors. Ultimately, the attention mechanism is only one part of the full Transformer model,you still need to interpret the impact of Value weights, concatenation, linear projections, deep layers, positional encoding, and more that also affects on the final 'attention' that one token pays to another.

During inference time in an autoregressive model like GPT or Llama, at each step the attention matrix is a strictly lower triangular. Each row represents the attention distribution for a particular token as query attending to all previous tokens' keys including the said particular token itself since each row sums to $1$ due to the $\text{Softmax}$ function as you correctly claimed. If the question is "Which token pays the most attention to 'good'?", then no previous tokens like "I" or "am," can attend to "good" during prediction because they were generated without any knowledge of it. You need to look at the column for 'good' (the third column in your example) and have to find the latter row generated after the token "good" with the highest value in that column of your attention weight matrix.


text:
What is the difference between SLP and perceptron and do they require non-linearity or not?
I couldn't find the exact answer to my question because the discussions tend go somewhere else, so I want to ask:
What is a single-layer perceptron and what makes it different from 'perceptron'? Are those two synonyms?
To my understanding: $y = W \mathbf{x}$ is a single-layer perceptron, with $y$ being the output layer and the input layer not being counted. There is a single 'linear' transformation.
However, you can in theory just use an activation function that is 'non-linear'
$$y = \sigma(W\mathbf(x))$$
Where $\sigma$ could be ReLu, Sigmoid, tanh or some exotic activation function that is not linear at all.
Can a single layer perceptron use non-linear activation function or is this considered improper?
If yes, is this model able to 'mimick' a non-linear function?
To my understanding, the whole goal of neural networks is that we assume there is a function that can solve our problem but we aren't really able to figure it out using math, so we approximate it instead using adjustable parameters. Is a single layer perceptron unable to ever approximate a non-linear function,  even though a non-linearity is performed on the output?

In modern ML Perceptron usually means a type of linear binary classifier introduced by Frank Rosenblatt in 1958.

In the modern sense, the perceptron is an algorithm for learning a binary classifier called a threshold function: a function that maps its input ${\displaystyle \mathbf {x} }$ (a real-valued vector) to an output value ${\displaystyle f(\mathbf {x} )}$ (a single binary value): ${\displaystyle f(\mathbf {x} )=h(\mathbf {w} \cdot \mathbf {x} +b)}$, where ${\displaystyle h}$ is the Heaviside step-function

And of course there's nothing wrong for a single-layer perceptron refers to a more general neural network with real-valued output via your mentioned nonlinear activation function. So you should always judge the almost synonyms definition of either a perceptron or a single layer of perceptron based on their context.
Finally even with a non-linear activation function, a single-layer perceptron cannot model highly complex non-linear relationships. This limitation is due to the lack of hidden layers which are critical for feature transformation and hierarchical representation of data. For instance, the classic XOR problem (which is not linearly separable) cannot be solved by a single-layer perceptron even with a non-linear activation function.

A single perceptron can learn to classify any half-space. It cannot solve any linearly nonseparable vectors, such as the Boolean exclusive-or problem (the famous "XOR problem").

A perceptron network with one hidden layer can learn to classify any compact subset arbitrarily closely. Similarly, it can also approximate any compactly-supported continuous function arbitrarily closely. This is essentially a special case of the theorems by George Cybenko and Kurt Hornik.

Thanks for the answer. Our learning material was Jurafsky and Martin's book. In Ch7, Section 7.2, it is written in bold that a perceptron does not have a non-linear activation function. As you pointed out and also when you google it, people use Heaviside function for activation. Is the Heaviside function a linear function? I don't think it is because you can't come up with $a,b$ such $f(x) = ax + b$ is the same as Heaviside, but perhaps I'm wrong...

Indeed Heaviside step function is nonlinear too strictly speaking, but its output is binary, which was use for classification purpose. While all other usual nonlinear activation functions you mentioned output is continuous real numbers, which can also be used for regression purposes. Hope this clarifies your lingering confusion for this post.

The confusion is still not quite resolved yet. According to your answer, you need a non-linearity to be called a perceptron (if Heaviside is non-linear as well). According to Jurafsky and Martin, you do not. So someone is wrong or some part is still left unclear.

Maybe you shouldn't nickpick for Jurafsky and Martin's choice of words. When people say "non-linear activation functions" in the context of modern neural networks, they often mean smooth continuous non-linearities to produce real-valued numbers like those you mentioned. And strictly speaking, step function is non-linear, so I believe this minor semantic issue caused by their non-strict use of words in language composition should settle now.


text:
Understanding Backward pass using a simple Computational Graph Implementation
I'm trying to understand backward pass with code by referring to a simple implementation of backward pass. The PyTorch one was too complicated for me as it has a bunch of other properties, so I tried to stick close to the theory. The class as well as add and mul was generated by ChatGPT. I altered it, implemented subtraction and division as well a sigmoid but it still seemed to work properly, so I got curious.
class Node:
    def __init__(self, value, parents=None, op=None):
        self.value = value
        self.grad = 0
        self.parents = parents or []
        self.op = op

    def backward(self, grad=1):
        self.grad += grad

        for parent, local_grad in self.parents:
            parent.backward(grad * local_grad)

def add(a : Node, b : Node ) -> Node:
    value = a.value + b.value
    node = Node(value, parents=[(a, 1), (b, 1)], op="add")
    return node

def mul(a : Node, b : Node) -> Node:
    value = a.value * b.value
    node = Node(value, parents=[(a, b.value), (b, a.value)], op="mul")
    return node

The actual code is longer but I want to focus on the backward function as well as how the gradient seems to be computed in the forward pass already.
Observe how add and mul function create parents and their respective partial derivatives already. If you use this basic implementation to implement a forward pass and you call backward, you will apparently get the correct partial derivative for output with respect to each node.
def comp_graph_ex(a, b):
    node_a = Node(a)
    node_b = Node(b)
    node_1 = Node(1)
    node_c = add(node_a, node_b)
    node_d = add(node_b, node_1)
    node_e = mul(node_c, node_d)

    node_e.backward()
    print(f'∂e / ∂b = {node_b.grad}')

if __name__ == "__main__":
    comp_graph_ex(2, 1)

The graph in code corresponds to this one
This will apparently compute: $\frac{\partial e}{\partial b} = \frac{\partial e}{\partial c}\cdot \frac{\partial c}{\partial b} + \frac{\partial e}{\partial d} \cdot \frac{\partial d}{\partial b} $ using the backward pass. (I'm not sure what this is called, I found 'product rule' but this would also work if $e=c-d$, $e=\frac{c}{d}$, $e=c+d$.)
So does this mean, the backward pass isn't the one that computes derivatives but it's the forward pass?
Specifically, observe how each mathematical operation contains already the derivative, so when I compute $a+b$, I know already that $\frac{\partial a}{\partial b}=\frac{\partial b}{\partial a}=1$, hence the moment I perform that operation, I store those derivatives.
Does the backward pass use the precomputed values from the forward pass to employ chain rule?

Are you asking a question based on the assumption that code generated by an AI is correct?

Yes because I tried it out with different inputs and was able to get the same numbers when I solved for partial derivatives by hand on examples computational graphs. ChatGPT was trained on a bunch of data, so what it outputs is what it 'thinks' will most likely correspond to my prompt, so assuming it had a bunch of PyTorch/TensorFlow docs fed in at some point, it's not too far fetched to assume it's correct. I was just surprised to see derivatives being computed in the forward pass already, which I don't think it hallucinated.

The forward pass computes the actual values and sets up the computation graph, while the backward pass uses the precomputed local partial derivatives (the gradients with respect to each operand, for example, when you perform addition $a + b$, its derivative w.r.t. $a$ and $b$ is $1$ which are stored) from the forward pass to apply the chain rule and compute gradients. Therefore the backward pass is not redoing the derivative calculations from scratch which allows you to compute gradients for complex computations in an efficient manner, leveraging the power of the chain rule and the precomputed derivatives.
You can also further refer automatic differentiation's reverse accumulation mode which corresponds to your case for neural networks.

Reverse accumulation traverses the chain rule from outside to inside, or in the case of the computational graph in Figure 3, from top to bottom... This is only half the work when compared to forward accumulation, but reverse accumulation requires the storage of the intermediate variables $w_i$ as well as the instructions that produced them in a data structure known as a "tape" or a Wengert list, which may consume significant memory if the computational graph is large.

So if we use something like PyTorch or TensorFlow, they implement computational graphs beneath their Matrix Multiplications somewhere? So each 'value' in a Matrix would correspond to a Node in the computational graph? I was able to create a computational graph and solve the XOR problem by just hard coding 8 parameters as nodes (2 x 2 matrix, 2 x 1 matrix and bias). The loss function is then just 'another node' added and we can call .backward from that node to compute gradients, then subtract them from parameter values.

Yes in frameworks like PyTorch and TensorFlow, computational graphs are constructed implicitly and dynamically (e.g., pytorch) under the hood during your defined operations like matrix multiplication. Your description of creating a computational graph by manually defining nodes as parameter values aligns with what these libraries automate. Calling backward() on the loss node traverses this graph in reverse, applying the chain rule to compute the gradients of each parameter node. Reverse mode AD with its graph is key for efficiently computing gradients across neural networks. Hope it clarifies.


text:
Is masking output for partially labelled data an acceptable practice?
Minimal Context
Given the scarcity of labelled data I'm writing a multihead-NN to predict multiple partially labelled properties.
Some these aren't of interest but make the backbone is robust; later on, the NN can be used independently for each head.
In practice, at the output, a mask (binary vector) will zero unavailable label-columns in turn making that loss_i=0.

Comment
This is preferable to just a single output head:

it will be less of a large network to 'export'.
It will also easily allow for separate Losses,

Questions

Is masking output for partially labelled data an acceptable practice?

If this is a reasonable idea, it must be quite common. But I wonder whether there is a name for it, and maybe any 'canonical' paper?

It and may need a way to normalise the gradients?

My idea is to use it to train with Graph Neural Networks, with multi-head property predictor.


text:
Deep RL problem: Loss decreases but agent doesn't learn
I'm implementing a basic Vanilla Policy Gradient algorithm for the CartPole-v1 gymnasium environment, and I don't know what I'm doing wrong. No matter what I try, during the training loop the loss decreases (so the model is actually learning something), but the episode total reward also decreases until it reaches around 9-10 steps (which I imagine is the minimum number of steps needed to make the pole fall). So it's learning to do it bad!
In this algorithm there are a few moving parts that influence each other, and I run out of ideas to see where the problem could be.
For the discounted rewards I'm using
$ Q_{k,t} = \sum_{i=0}{\gamma^{i-t} r_i} $
for $k$ = all episodes and $t$ = all steps in an episode.
And for the loss:
$ L = -\sum_{k,t}Q_{k,t}log\pi_{\theta}(a_t | s_t)$
The code is a mix from Maxim Lapan's Deep RL Hands-On book, Karpathy's Pong example (blog, code), and personal tweaks.
Here's my code:
import gymnasium as gym
import torch
from torch import nn
import torch.nn.functional as F
from torch.nn.init import xavier_uniform_
import numpy as np

GAMMA = 0.99
LEARNING_RATE = 0.001
BATCH_SIZE = 4
DEVICE = torch.device('mps')

class XavierLinear(nn.Linear):
    def __init__(self, in_features: int, out_features: int, bias: bool = True, device=None, dtype=None) -> None:
        super().__init__(in_features, out_features, bias, device, dtype)
        xavier_uniform_(self.weight)

class VPG(nn.Module):
    def __init__(self, input_size, output_size):
        super(VPG, self).__init__()
        self.net = nn.Sequential(
            XavierLinear(input_size, 128),
            nn.ReLU(),
            XavierLinear(128, output_size),
        )

    def forward(self, x):
        return F.softmax(self.net(x), dim=0)

def run_episode(model, env):
    obs = env.reset()[0]
    obs = torch.Tensor(env.reset()[0]).to(DEVICE)
    te = tr = False
    rewards, outputs, actions = [], [], []
    while not (te or tr):
        probs = model(obs)
        action = probs.multinomial(1).item()
        obs, r, te, tr, _ = env.step(action)
        obs = torch.Tensor(obs).to(DEVICE)
        if (te or tr):
            r = 0 # Here I also tried -1 if the pole falls and 0 otherwise
        rewards.append(r)
        outputs.append(probs)
        actions.append(action)
    return torch.Tensor(rewards).to(DEVICE), torch.concatenate(outputs).reshape(len(rewards), 2), actions

def discount_rewards(rewards):
    discounted_r = torch.zeros_like(rewards)
    additive_r = 0
    for idx in range(len(rewards)-1, -1, -1):
        to_add = GAMMA * additive_r
        additive_r = to_add + rewards[idx]
        discounted_r[idx] = additive_r
    return discounted_r.to(DEVICE)

def loss_function(discounted_r, probs, actions):
    logprobs = torch.log(probs)
    selected = logprobs[range(probs.shape[0]), actions]
    # discounted_r = (discounted_r - discounted_r.mean()) / discounted_r.std()
    weighted = selected * discounted_r
    return -weighted.sum()

# The actual training loop:

episode_total_reward = 0
batch_losses = torch.Tensor().to(DEVICE)
batch_actions = []
batch_disc_r = torch.Tensor().to(DEVICE)
batch_probs = torch.Tensor().to(DEVICE)
best_ep_reward = 0
losses, ep_total_lenghts = [], [0]

episodes = 0
TARGET_REWARD = 100

env = gym.make("CartPole-v1")
model = VPG(env.observation_space.shape[0],
            2).to(DEVICE)
optim = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)

while np.array(ep_total_lenghts)[-100:].mean() < TARGET_REWARD:
    rewards, probs, actions = run_episode(model, env)
    discounted_r = discount_rewards(rewards)
    episode_total_reward = rewards.shape[0]
    ep_total_lenghts.append(episode_total_reward)
    episodes += 1
    batch_actions += actions
    batch_disc_r = torch.concatenate([batch_disc_r, discounted_r])
    batch_probs = torch.concatenate([batch_probs, probs])

    if episodes % BATCH_SIZE == 0:
        loss = loss_function(batch_disc_r, batch_probs, batch_actions)
        losses.append(loss.item())
        model.zero_grad()
        loss.backward()
        optim.step()
        batch_actions = []
        batch_disc_r = torch.Tensor().to(DEVICE)
        batch_probs = torch.Tensor().to(DEVICE)
        print(f"Episode {episodes}. Loss: {loss}. Reward: {episode_total_reward}")
print(f"Success in {episodes} episodes. Loss: {loss}. Reward: {episode_total_reward}")

Tried: changing signs in loss functions, changing rewards (non-terminal step = 0 and terminal step = -1), updating manually the weights (adding the gradient or substracting it instead of using Adam optimizer). In each case I get the same: the loss decreases but the agent doens't learn to keep the pole up. Actually, it learns to throw it.
Expectation: Loss decreases and episode total reward (steps played) increases.

I'm pretty sure you should throw a minus sign in front of the selected log probability in your loss function. (this turned out to be wrong and the code works fine: https://colab.research.google.com/drive/1Ecnqb_D_Oe0_kMc2ZnA6stAlVCbAyE9H?usp=sharin)

Doesn't the minus sign in front of weighted.sum() a couple of lines below have the same effect?
I tried that and the output was the same. The loss was decreasing during training, but the agent was perfoming worse every time. It really puzzles me

yeah you're right.
ive tried to run your code just now and I dont get the same result

I have a link to a colab where I ran the code it worked and I turned a copy of it into A2C.
link
https://colab.research.google.com/drive/1Ecnqb_D_Oe0_kMc2ZnA6stAlVCbAyE9H?usp=sharin

Wow, thanks for taking the time Leonhard!
Yeah, this morning I did those changes, and I also changed the reward for -1 if the episode is terminated, and 0 otherwise, and worked perfectly.
I still don't know exactly what was happening before. I was really confused by having a decreasing loss and the agent performing worse and worse each time.
But anyway, it works now.
And thanks for the A2C implementation! I'm studying RL at the moment and that will help me!

No worries show me the next step you take please


text:
How to overcome overfitting when you can't increase the training data due to constraint in time and resources?
I trained a complex cnn but when I look at the training and validation loss, it seems that the model is overfitting. But due to limitation on resources I can't increase the datasize. Is there any other way to overcome the overfitting. I have attached the code for training and the model below.
class CV_CNN_Net(nn.Module):
    def __init__(self,device):
        super(CV_CNN_Net, self).__init__()
        self.device = device
        self.nn = nn.Sequential(
                                ComplexConv2d(1, 128, kernel_size = (3, 3),stride = (2, 2),padding=(1,1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexConv2d(128, 128, kernel_size=(2, 2), stride=(1, 1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexConv2d(128, 128, kernel_size=(2, 2), stride=(1, 1)),
                                ComplexBatchNorm2d(128),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                nn.Flatten(),
                                ComplexLinear(2048,1024),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexDropout(p=0.3,device = self.device),
                                ComplexLinear(1024, 512),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),

                                ComplexDropout(p=0.3,device = self.device),
                                ComplexLinear(512, 73),
                                # ComplexReLU(),
                                C_CSELU(),
                                # ZReLU(),
                                ComplexDropout(p=0.3, device=self.device),
                                )
        self.FNN = nn.Sequential(
            nn.Linear(146, 73, bias=False),
            nn.Sigmoid()
        )
    def forward(self, data_train):
        Outputs = self.nn(data_train)
        Outputs = torch.cat([Outputs.real,Outputs.imag],dim=1)
        Outputs = self.FNN(Outputs)
        return Outputs

The training:
def one_epoch(model,optimizer,lr,criterion,L1_Regularizer,dataloader,device,epoch_now,epoch_all,is_train,is_L1_Regularizer):
    if is_train==True:
        model.train()
    else:
        model.eval()
    mean_loss = torch.zeros(1).to(device)
    dataloader = tqdm(dataloader, desc=f'Epoch {epoch_now + 1}/{epoch_all}', file=sys.stdout)
    for step, data in enumerate(dataloader):
        x_batch, y_batch = data
        ouputs_train = model(x_batch.to(device))
        loss = criterion(ouputs_train, y_batch.to(device).to(torch.float32))
        if is_L1_Regularizer == True:
            for model_param_name, model_param_value in model.named_parameters():
                if model_param_name in ['FNN.0.weight']:
                    loss = L1_Regularizer.regularized_param(param_weights=model_param_value,
                                                                  reg_loss_function=loss)
        mean_loss = (mean_loss * step + loss.detach()) / (step + 1)  # update mean losses
        if is_train==True:
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            #dataloader.desc = "[Training epoch {}] mean loss:{}".format(epoch_now, round(mean_loss.item(), 6))+' lr:{}'.format(lr)
        else:
            dataloader.desc = "[Val epoch {}] mean loss:{}".format(epoch_now,round(mean_loss.item(), 6)) + ' lr:{}'.format(lr)
    return mean_loss.item()
def Pre_Train():
    xTrain, xVal, yTrain, yVal = My_Train_DataLoader(filename_data_Pre_train, Validation_set_size,
                                                     Cov_Matrix_type_Pre_train)
    criterion = nn.BCELoss().to(device)
    L1_Regularizer = L1Regularizer(model, lambda_reg=5e-5)
    optimizer = optim.Adam(model.parameters(), lr=1e-4, betas=(0.85, 0.95), eps=1e-08, weight_decay=0, amsgrad=False)
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.7, patience=5, verbose=True)
    Train_dataloader = DataLoader(dataset=MyDataset(xTrain, yTrain), batch_size=batch_size_Pre_train, shuffle=True)
    Val_dataloader = DataLoader(dataset=MyDataset(xVal, yVal), batch_size=batch_size_Pre_train, shuffle=True)
    Val_loss_min = 99999.0
    for epoch in range(epochs_Pre_train):

        Train_loss = one_epoch(model,optimizer,optimizer.param_groups[0]['lr'],criterion,L1_Regularizer,
                               Train_dataloader,device,epoch,epochs_Pre_train,is_train=True,is_L1_Regularizer=True)

        Val_loss = one_epoch(model,optimizer,optimizer.param_groups[0]['lr'],criterion,L1_Regularizer,
                             Val_dataloader,device,epoch,epochs_Pre_train,is_train=False,is_L1_Regularizer=True)

        scheduler.step(Val_loss)

        writer_Pre_train.add_scalar("Train_loss", Train_loss, epoch)
        writer_Pre_train.add_scalar("Val_loss", Val_loss, epoch)
        writer_Pre_train.add_scalar("lr", optimizer.param_groups[0]['lr'], epoch)

        if (epoch + 1) % 10 == 0:
            torch.save(model, filename_save_model_Pre_train + "/model_{}.pth".format(epoch + 1))
        if Val_loss_min > Val_loss:
            Val_loss_min = Val_loss
            torch.save(model, filename_save_model_Pre_train + "/model_best.pth")

def Fine_tuning():
    xTrain, xVal, yTrain, yVal = My_Train_DataLoader(filename_data_Fine_tuning, Validation_set_size,
                                             Cov_Matrix_type_Fine_tuning)
    # initialize Fine_tuning model with  paremeters of Pre_train model
    checkpoint = torch.load(filename_save_model_Pre_train + "/model_200.pth", map_location=device)
    model.load_state_dict(checkpoint.state_dict())

    criterion = nn.BCELoss().to(device)
    L1_Regularizer = L1Regularizer(model, lambda_reg=1e-5)
    optimizer = optim.Adam(model.parameters(), lr=5e-5, betas=(0.85, 0.95), eps=1e-08, weight_decay=0, amsgrad=False)
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.7, patience=5, verbose=True)
    Train_dataloader = DataLoader(dataset=MyDataset(xTrain, yTrain), batch_size=batch_size_Fine_tuning, shuffle=True)
    Val_dataloader = DataLoader(dataset=MyDataset(xVal, yVal), batch_size=batch_size_Fine_tuning, shuffle=True)

    Val_loss_min = 99999.0
    train_losses, val_losses = [], []

    for epoch in range(epochs_Fine_tuning):

        Train_loss = one_epoch(model, optimizer, optimizer.param_groups[0]['lr'], criterion, L1_Regularizer,
                               Train_dataloader, device, epoch,epochs_Fine_tuning, is_train=True, is_L1_Regularizer=True)

        Val_loss = one_epoch(model, optimizer, optimizer.param_groups[0]['lr'], criterion, L1_Regularizer,
                             Val_dataloader, device,  epoch,epochs_Fine_tuning, is_train=False, is_L1_Regularizer=True)

        scheduler.step(Val_loss)
        train_losses.append(Train_loss)
        val_losses.append(Val_loss)

        writer_Fine_tuning.add_scalar("Train_loss", Train_loss, epoch)
        writer_Fine_tuning.add_scalar("Val_loss", Val_loss, epoch)
        writer_Fine_tuning.add_scalar("lr", optimizer.param_groups[0]['lr'], epoch)

        if (epoch + 1) % 10 == 0:
            torch.save(model, filename_save_model_Fine_tuning + "/model_{}.pth".format(epoch + 1))
        if Val_loss_min > Val_loss:
            Val_loss_min = Val_loss
            torch.save(model, filename_save_model_Fine_tuning + "/model_best.pth")
    plot_losses(train_losses, val_losses, title="Fine-Tuning Loss Over Epochs")
if __name__ == "__main__":
    if os.path.exists(filename_save_model_Pre_train) is False:
        os.makedirs(filename_save_model_Pre_train)
    if os.path.exists(filename_save_model_Fine_tuning) is False:
        os.makedirs(filename_save_model_Fine_tuning)
    Pre_Train()
    Fine_tuning()

Currently the training loss looks as follows:[Loss]

I was thinking if I train the network multiple times on different sets of data, will it help in generalization?

With BatchNorm and Dropout already, have you tried k-fold cross validation and image augmentation trick via random rotating/flipping/cropping/zooming/color shifting, etc?

@cinch, due to constraint on training resources, I can't use huge amount of data as it will take alot of time to train. So, I didn't increase the dataset using data augmentation. Still, I have used around 40,000 training samples and it took almost 20hrs to train.

Your result graph clearly shows overfit to your training set. So most likely a cross validation with your limited data resources is a good way to tune some of your parameters and hyperparameters like number of cnn nodes/layers to reduce your validation error.

@cinch, without k-fold cross validation, for 300 epochs the training is taking around 20hrs, so even if include k = 5 fold, it will take around 3 days. Given that my model isn't that deep, do you think it should take so much time to train?

It’s possible since you’re using complex valued cov2d layers, try use real valued simple layers and increase dropout to 0.5, or reduce filter size to reduce your model’s complexity.


text:
Are these objective and loss functions from Actor-Critic Methods correct?
I'm doing a research about actor-critic methods and I want to make sure that I understand these methods right.
First of all, I understand that as it's a combination of value-based and policy-based methods it will use their functions to estimate the optimal policy, being the loss function the Mean Square Error (MSE), which is the square of the TD Error, the one to learn an accurate estimate of the value function, which is used as feedback for the actor. Being the function: $$L(\phi) = \dfrac{1}{2}\left(V_\phi(s_t) - \left(r_t + \gamma\,V_\phi(s_{t+1})\right)\right)^{2}$$
And expressed with gradient:$$\nabla_\phi L(\phi) = \underset{\tau \sim \pi_\theta}{\mathbb{E}}\left[\nabla_\phi\left(V_\phi(s_t) - \left(r_t + \gamma\,V_\phi(s_{t+1})\right)\right)^2\right]$$
Now for the actor we would use the objective function: $$\nabla_\theta\,J(\theta) = \underset{\tau \sim \pi_\theta}{\mathbb{E}}\left[\nabla_\theta\log\pi_\theta(a\,|\,s)R(\tau)\right]$$
Finally, for Advantage Actor-Critic methods, we would use the Advantage function for both loss and objective functions:$$\nabla_\theta J(\theta) = \underset{\tau \sim \pi_\theta}{\mathbb{E}}\left[\nabla_\theta\log\pi_\theta(a_t\,|\,s_t)\,A(s_t,\,a_t)\right]$$
$$\nabla_\phi L(\phi) = \underset{\tau \sim \pi_\theta}{\mathbb{E}}\left[\nabla_\phi(A(s_t,\,a_t))^{2}\right]$$
Where $\tau$ is an episode of $\pi_\theta$. Am I missing something or is it a good summary of AC methods ? Feel free to make any suggestions or improvements.

Indeed it's a good summary of AC methods and perhaps a few clarifications might enhance your further research.
You didn't mention the advantage of the advantage function for A2C method's actor network which often reduces variance in updates significantly compared to raw TD error and thus stabilizes training, and it's rare for critic's loss to use advantage function unless in some customary or dual critic models. Also many modern AC methods such as A3C and PPO sometimes might add an entropy regularization term $H(\pi_{\theta})$ to the actor's objective to encourage exploration.
Finally your summary seems to only assume an on-policy setup, the update equations at least for the main networks would include experience replay buffer and target network(s) for some popular off-policy methods such as DDPG and SAC which improve sample efficiency as experiences can be reused multiple times for training and improves learnings dynamics as slow soft target updates avoid moving target non-convergence and ensure that data in the replay buffer remaining relevant for a longer period.

Thank you for your feedback. I wanted to make an introduction to AC methods, in which I would mention the intoduction of the loss function to critic (having previously defined value-based and policy-based methods and objective function). Then I was thinking about defining the methods, the first being A2C, and their functions so then I would continue with TRPO, PPO and SAC among others.
However, right now I wanted to focus on A2C. You said that I did not mention the advantage for the actor network but I included it in the fourth equation.

Your plan to introduce A2C in the context of actor-critic methods sounds well-structured and pedagogically sound. In addition to the clarification of my above answer, also you may further ponder about why off-policy SAC only needs one critic target network while DDPG needs target networks for both critic and actor. Hope this clarifies and helpful to your concerned question here.


text:
How do I find the actual vs difference for calculating backpropagation on the first hidden layer in a AI with 2 hidden layers?
I have a simple AI built out in an Excel sheet for testing. It has 4 inputs, 1 hidden layer with 4 Neurons, and 4 outputs. I am trying to add a second hidden layer, with 4 Neurons. I am using backpropagation. I understand the point where one hidden layer's weights and bias are adjusted based off the error from the output sigmoid vs the actual desired output. But how do I determine that for the hidden layer prior in a 2 hidden layer AI? Does the backpropagation need to adjust the weights and bias in the output layer first?
I am new to learning AI and want to understand it from the ground up, so please bare with me! I do not fully understand the math symbols involved in all this, my background is in programming, so some simple guidance in non-calculus language would be really helpful. I am also reverse engineering this in SQL. I started with an excel sheet from this site that is only 2 inputs and outputs, and I am expanding on it, but is essentially the same.
https://www.codeproject.com/Articles/1273000/Machine-Learning-in-Excel
I have used ChatGPT and tried to read and understand from various sites and have gotten this far but this is really stumping me.
Thanks!

I don't understand this question "But how do I determine that for the hidden layer prior in a 2 hidden layer AI?" Are you asking how do you determine the weights for this new layer you want to add? The general answer is: backpropagation. But I suppose you're asking for the formulas and an explanation of how back-propagation would work for a 2-hidden layer NN, right?

There is a video on backpropagation by a Stanford grad who posts about math under the name 3blue1brown. I recommend it. His videos have a lot of helpful visualizations to build intuition around the math.

@nbro Yes thats correct. I understand how it works in a 1 hidden layer network, the idea being you are taking the difference between the output vs desired, and doing some maths and applying it backwards. But in a multi layer network, does this mean I take some output vs desired in between HL 1 and HL 2? Do I run backpropagation on HL 2, then somehow look at what HL 1 is outputting vs a desired in order to activate HL 2 properly? Just confused on this part.

@JimMcGaw thank you I will definitely be taking a look at this! Appreciate it!


text:
the actor loss is getting higher and higher in DDPG

when i train my DDPG ,the actor loss is getting higher and higher which should get lower and lower ,while the critic loss is decreasing to zero. i wonder why and how to prevent this situation. I apply DDPG to power distribution of electrical vehicle,when i use my dataset as power input,the ddpg will go bad,also the sac,but the reward will get higher while the actor loss is increasing.In another situation,the critic loss is stable and near to zero, but the actor loss is still increasing.

Can you provide more details about your problem? What are you applying DDPG to?

yes！i add some pictures to describe the same problem of actor loss, if you'd like ,please help me,thanks!

The actor's loss increasing is often a symptom of poor gradients being propagated from the critic. If the critic provides bad value estimates or is stuck predicting constant Q-values, the actor won't learn useful policies leading to increasing loss. Notice your critic's loss is very unstable even towards zero.
Possibly if the exploration policy is not diverse enough or replay buffer isn't large enough , the critic does not see a wide range of state-action pairs limiting the quality of its Q-value estimates due to overfit a narrow range. In DDPG the actor learns by maximizing the Q-value predicted by the critic, therefore if the Q-value gradients are noisy or flat the actor's updates will become unstable or ineffective. Try to monitor the estimated Q-value.
Also an imbalance in learning rates or update frequencies between the actor and critic can lead to this behavior, similarly if the target networks for both actor and critic are not soft-updated with $\tau$ properly, the learning dynamics can destabilize.

thank you! i add two more pictures in which the critic loss is stable while the actor loss is increasing.i use the action noise and soft update,also i decrease the update frequencies of actor,the learning rate of actor and critic are 1 : 10. the problem  still exist,and i wonder whether this problem is caused by my dataset or reward function.


text:
Loss function on intermediate layers of the networks
Typically in supervised learning, a neural networks' output is compared to the targets through a loss function, and the gradients are backpropagated. Is it a bad idea to also have a loss function on intermediate layers? I feel that restricts the networks ability to actually optimize the final, global loss function. What are the instances in which an additional loss on intermediate layers would work?
----EDIT----
Clarification of my thought process for supervising intermediate layers, via examples:

Merging contrastive loss to a Variational AutoEncoder (VAE) or a Classifier. I would have the original VAE loss function (or classifier's cross entropy loss), but additionally supervise the output of the encoder or some intermediate layer with a contrastive loss (for example the triplet loss, where the targets would be positive and negative samples from the batch)

A more classic example would be teacher-student distillation, supervising student's intermediate layers with the teacher's intermediate layers.

When intermediate layers provide a prediction itself. For instance, let's say we have an object detector where the Mth layer outputs segmentation masks, and the Nth layer (N>M) outputs the bounding boxes.

So the question is, when would these additional losses work, or will they confuse the network just more?

Can you explain how you would compute such a loss? What would be the targets, for example? Maybe your idea makes sense, but right now it's a bit unclear. So, backpropagation is a global update rule. Maybe you have some local update rule in mind?

@nbro yes thank you for the question. I edited my post with to exemplify my idea in mind. Let me know if I should elaborate more.


text:
When we should use NEAT and GA?
When we should use NEAT and GA? Today the problem of “We do not know which answer is correct” was solved by Reinforcement Learning, so when using genetic algorithm is still useful, and do we have RL that changes topology of NN?

'Today the problem of “We do not know which answer is correct” was solved by Reinforcement Learning' - What? That's a very unclear way of phrasing what RL does. Can you clarify what you mean by "why when using genetic algorithm is still useful"? Why or when? It's unclear. This post is so confusing and it's unclear why you're asking the question in the title, which is the only thing that is more or less clear.

That is why = so

Generally speaking, when architecture search is essential or the problem objective is non-differentiable while creativity and diversity in solutions matter most, use NEAT/GA. When rewards are well-defined, environments are complex but problem objective is differentiable, and ML model topology is fixed, use RL.
Having said that, there're RL approaches changing the topology of neural networks during training. One popular framework is NAS/AutoML which employs RL to adapt and optimize ANN topology.

Reinforcement learning (RL) can underpin a NAS search strategy. Barret Zoph and Quoc Viet Le applied NAS with RL targeting the CIFAR-10 dataset and achieved a network architecture that rivals the best manually-designed architecture for accuracy... In the so-called Efficient Neural Architecture Search (ENAS), a controller discovers architectures by learning to search for an optimal subgraph within a large graph. The controller is trained with policy gradient to select a subgraph that maximizes the validation set's expected reward.

In some cases NAS variant also employs evolutionary algorithms including GA to adapt and optimize ANN topology.

An Evolutionary Algorithm for Neural Architecture Search generally performs the following procedure... On CIFAR-10 and ImageNet, evolution and RL performed comparably, while both slightly outperformed random search.

To be sure, by GA we mean crossing and changing the weights of neural networks while leaving the topology unchanged?

Not necessarily, here meant for your concerned hyperparameters optimization of topology of ANN, see GA definition: Genetic algorithms are commonly used to generate high-quality solutions to optimization and search problems via biologically inspired operators such as selection, crossover, and mutation. Some examples of GA applications include optimizing decision trees for better performance, solving sudoku puzzles, hyperparameter optimization, and causal inference. Hope this clarifies your lingering concern and helpful for your question.

Thank you, I understand that GA is a more general concept that includes NEAT?  And I can't remember when a problem is not differentiable for NN, give a couple of examples

If you only restricted to NN with non-differentiable objective, my above answer deals with hyperparameter optimization of ANN topologies for your OP since number of layers, types of activations, adding/removing nodes are all non-differentiable. Also even in RL such as AlphaZero, you still need MCTS heuristics where there's no gradient to backprop. And your loss in some cases could be F1 or BLEU (for language generation) liek non-differentiable functions. Hope this clarifies your lingering question, if you have more related questions, you could post new question.

so RL uses gradient descent and for this the task should be differentiable, however, as You mentioned we have AutoML, so now I am confused - if RL requires differentiability, but also, as you said, there are solutions for non-differentiable questions with reinforcement learning, I am completely lost in the algorithm's operation

Not surprised by your lost since RL is a wide area in AI and complex algos such as AlphaZero contains RL and MCTS and other tricks, but for simplicity sometimes we classify it as RL. But indeed you may have F1 or BLEU loss functions in some rare case, then such RL is not differentiable. So the claim that RL requires differentiability is incorrect strictly speaking. Hope this finally clarifies and useful to you.


text:
How to train a model to make predictions for larger sequences than those in the dataset?
I'm working on a project where I need my model to predict a sequence of n 3x3 matrices given an input sequence of n 3x3 matrices for a physical simulation. (the n's are all perfect squares e.g. I only need to consider sequences of 9, 16, 25, 36, etc. matrices all the way to 324). I need the model to approximate the mathematical relationship between the input and output (which can be computationally expensive for large n, hence the need for an ML model in the first place).
However, the problem I have is that my dataset only contains input/output values for sequences of 9, 16, ... 81 matrices (after that it becomes hard to compute).
I am currently a novice in machine learning and just know the basics of Pytorch, but so far I have trained a simple neural network with alternating Linear and ReLU layers for the n=9 case, with error ~ 10^-4. However, now I want to expand to the general case, but I am not sure of the best way to do so.
What I am thinking is to make my neural network big enough for the largest case n=324 and use some kind of RNN to make the predictions accurate for large n even though my dataset does not cover all those values.
I just wanted to ask if this is the right approach or if there is a better way to transfer the information learned from small dataset sizes to make predictions for larger cases, and also what type of RNN I should use.
Thank you!