text:
For a task that searches for an image artifact within a picture, can existing tools can be used or do I need to design the process myself?
I am familiar only with basic AI/NN concepts but never worked with any libraries/tools as tensor flow. Currently, I have a task for which AI might be ideal: detection of a certain image artifact in a picture (lets say I want to detect a black circular spot of a variable size). Because the spot can be very small or very large, I guess the NN would have to somehow process the whole picture and then proceed in smaller regions? Anyway, for such a task, do I need to learn more about machine learning or there are already tools that I could simply train (e.g. providing "clear" and "stained" image samples in their training sets) without worrying about internal details?

For detecting something like a circle, or other geometric shapes, modern machine learning may be a bit overkill.
You should look at Haar Cascades, which you can find implemented in OpenCV. These are the reason cameras have been able to detect faces since the early naughts; They are efficient and easy to train; They are trained by providing positive and negative examples. They also scale better than something like a convolutional neural network; The implementation in OpenCV is made specifically to be able to detect objects of different sizes.
Here is OpenCVs tutorial to get started.

This, of course, assumes that the artifact is of a known geometric shape. If not, you'll probably have to look at convolutional neural networks and anomaly detecting autoencoders.
Object detection with neural networks is generally done with RCNNs; These break up the image into regions using various methods and do object detection on these regions. The most popular algorithm in this family is probably YOLO, which learns how to break up the image. Many other approaches exist.

Thank you! I will certainly look into Haar Cascades and YOLO, as you advised. In general, do I need to study some theory or these can be used directly (configured and fed data)?

@JohnV Both have open implementations; So, you can find tutorials online on how to train Cascade classifiers and YOLO using OpenCV and Yolov5, respectively; Follow the links in my answer. Both contain tutorials.


text:
How to compare memory requirements for tabular Q-learning vs deep neural network?
I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but consider the following case.
A Q-table has the size 14 states *169 actions= 2366 entries and (say) a fully connected DNN whose number of parameters comes out to be like >8000. Space complexity/memory-wise, isn't storing a look-up q-table of 2366 size better than storing 8000 parameters of neural net?  I never implemented a DNN before so no idea how much space neural net parameters take.

Please, do not ask for opinions. Ask for evidence and facts. We have a specific close reason that can be used to close posts that ask for opinions. Moreover, please, reformulate your title in the form of a question. I see only one question in the body of your post and it's slightly different than what you have in the title, which is more general.

Ok thanks. I will be careful in future questions. I have edited the title now. I thought I can ask this question since it is based on some actual numbers that I am dealing with. And not any general theory question. I asked for the opinion of people who have experienced such scenario so that I get clear as I don't have experience with dnn but only with q table..thanks for keeping the question though.. Do you know of any other site where you think this is more appropriate to post (for future reference I mean)?

I think your problem is fine for our site, but the way you had formulated the question wasn't ideal (you can ask people to share with you their experience and empirical findings - that would not really be an opinion). So, every question on theoretical/conceptual/mathematical aspects of AI (including RL) can and probably should be asked here. I think your question can actually be asked here (because you're asking about the theoretical computational requirements - you're not asking e.g. which GPU you need to buy, which would be more a hardware question, and so off-topic), so don't worry!

You don't say, but I suspect from your description, that you have designed the neural network to operate over one-hot-encoding representations of states and actions. Using such a representation offers no benefit whatsoever compared to a simple table. That is because the intended benefit of using neural networks, or any kind of approximation, is generalisation.
It is not possible to generalise results between states and actions, if those states and actions are simply enumerated. If you represent $\mathcal{S}$ as $\{s_0, s_1, s_2 ... s_{13}\}$ and $\mathcal{A}$ as $\{a_0, a_1, a_2 ... a_{168}\}$ to any function, then the best generalisation that can be made are only based on the mean of $Q(s, a)$ for all states or all actions. You cannot use experience gained for $Q(s_0, a_0)$ to say anything about expected return for $Q(s_1, a_1)$. That is true even if $s_0$ and $s_1$ are similar in some way - the representation does not capture that similarity, so the approximator cannot use it.
State and action representation are very important details when adding approximation to reinforcement learning.
To gain generalisation, an approximation scheme needs to have feature data that encapsulates how elements in the space are similar or different to each other. For example, if the 169 actions are arranged on a 13x13 grid, then each action can be represented as a 2-element vector. This will work for approximation if actions that are close to each other in this 13x13 space usually have similar expected returns.

Space complexity/memory-wise, isn't storing a look-up q-table of 2366 size better than storing 8000 parameters of neural net?

Yes, typically neural network parameters are 32-bit floats, which would also a suitable storage type for the q table values.
It is not common to have a neural network use more space to store its parameters than would be required to fully describe the entire function domain that it is approximating. So you are right to find it unwanted/unexpected. The answer to this puzzle is likely to be in your choices for state and action representation.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What if we modify some Q-values while taking the action?
Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical rule? (2) a reduction of efficiency?
Update: By modification I mean the corresponding value after the action is taken. I am not referring to the Bellman update step. I am assuming we have filled our Q-table and now implementing in real scenario. And in this real scenario upon deciding on an action can I update the corresponding Q-value by say adding some constant factor? Not updating through Bellman eqn. Only after certain time steps I will re-train (if that's the correct word) and refresh my Q-table using the Bellman. I am thinking of like training and testing kind of phases here, if that make sense. I say so because my rewards are dependent on a random dynamic parameter and so I cannot just re-update my Q-table right after each time-step (to prevent unnecessary swaying of Q-values) and want to do so after certain n time-steps. In between, I just use the table to look up and take the actions without updating the corresponding Bellman update equation. But since I have & want to do something to reflect some criteria upon taking the action, I am thinking what if I just modify the corresponding Q-values only by adding some constant factor until I retrain. This would help take better actions even until the n time-steps are over and ready for re-train....

Could you clarify what you mean by modify: Changing the values stored in the table, or changing the values used to decide current action? What kinds of change are you considering - e.g. is it based on some mathematical property of the state, or some other collected data or random mutation? How often/by how much will you be changing the values? Without those details it is very hard to answer your questions (1) and (2).

@NeilSlater I have added the required details. Please see it and give any suggestions.

Thanks for the update. I don't understand why you would not want to use the Bellman update equation in your scenario? It is usually OK to use it in a live environment, and if you want to change the Q values, it would be the usual way to change them, whilst adding some constant does not seem to serve any purpose that I can think of. What are your concerns, and what do you think you will gain by adding this constant? You mention a Q-table a few times, but you have also tagged the question DQN, where there is a neural network instead of a Q-table - this may be an important detail.

@NeilSlater oh yes, I deleted the tag. I was arriving at a similar conclusion to what you said but I think I will first process what you said w.r.t my application. Will edit my code & then get back here to comment better on what you asked.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Graph-Level Regression Task
I'm currently working on a system that predicts energy consumption of a set of buildings using graph convolutionals networks (GCN), which is a Graph-Level regression task (1 prediction for every subset of buildings represented as a graph). However, I don't find any existing implementation / tutorial on how to do this type of tasks in Py Geometric, DGL, and StellarGraph libraries.
Some Stackoverflow Questions and Github Issues mentioned to add a linear layer without any activation function at the end of the model's class to achieve this. However, I'm still unsure that it's the right thing to do especially that I don't get clear signs of convergence in the loss.
Here is my current code that I toke from here:
class GNN(torch.nn.Module):
def __init__(self, hidden_channels):
    super(GNN, self).__init__()
    # Multiply hidden_channels to scale up the network size
    self.hidden_channels_gcn = hidden_channels * 2
    self.hidden_channels_gat = self.hidden_channels_gcn * 2
    self.hidden_channels_gin = self.hidden_channels_gat * 2
    # Assuming layers are defined as before
    self.conv1 = None
    self.conv2 = GCNConv(self.hidden_channels_gcn, self.hidden_channels_gcn)
    self.gat_conv1 = GATConv(self.hidden_channels_gcn, self.hidden_channels_gat)
    self.gat_conv2 = GATConv(self.hidden_channels_gat, self.hidden_channels_gat)

    mlp = torch.nn.Sequential(
        torch.nn.Linear(self.hidden_channels_gat, self.hidden_channels_gin),
        torch.nn.ReLU(),
        torch.nn.Linear(self.hidden_channels_gin, self.hidden_channels_gin)
    )
    self.gin_conv1 = GINConv(mlp)
    self.out = torch.nn.Linear(self.hidden_channels_gin, 1)

def forward(self, x, edge_index, batch):
    # Dynamic creation of the first GCNConv layer
    if self.conv1 is None:
        self.conv1 = GCNConv(x.size(1), self.hidden_channels_gcn).to(x.device)

    x = F.relu(self.conv1(x, edge_index))
    x = F.relu(self.conv2(x, edge_index))
    x = F.relu(self.gat_conv1(x, edge_index))
    x = F.relu(self.gat_conv2(x, edge_index))
    x = F.relu(self.gin_conv1(x, edge_index))

    # Aggregating to graph-level feature
    x = pyg_nn.global_mean_pool(x, batch)  # Pooling node features to the graph level

    x = self.out(x)
    return x

Does someone know how to deal with this situation?

Can you please put your specific question in your title? It's not fully clear what you're asking here.


text:
How to estimate conditional density using neural network?
Conditional Variational Autoencoders (CVAE) and Mixture Density Networks (MDN) are supposed to address this issue. However, these models provide the distribution parameters, e.g., mean and standard deviation, for each given sample, while I need a single underlying distribution that the given data is generated from.
To put it simply, I would like to find the parameters of a normal distribution that estimates $P(Y \mid X)$, given $X$ and $Y$. Let's imagine the given data, $X$, have an $(n, m)$ dimension, where $n$ and $m$ indicate the number of samples, and features, respectively.
Using CVAE/MDN I get the parameters with dimension $(n, d)$, where $d$ is the dimension of the parameters. But I am looking for a model to provide parameters with dimensions $(1, d)$.


text:
How to calculate number of connected neurons with filter
let's say I have a conv layer i with 64 feature maps and a filter size of 3x3.
The previous conv layer i-1 has 32 feature map.
Step-size is 2 and padding 1.
My question is now how to know how many neurons of the previous layer are connected to each of the neuron in i?
My first thought was 32 because there are 32 neurons/filters and I thought that number of feature maps = number of filters.
But I am not sure

Firstly, step size and padding have no influence on the number of parameters, they just determine where to apply your parameters, i.e. the convolution operator.

My first thought was 32 because there are 32 neurons/filters [...].

If you have 32 input channels and a kernel size of $3 \times 3$ then each entry of your $3\times3$-filter acts on a 32-dimensional vector. Therefore the number of parameters of your kernel is $3 \cdot 3 \cdot 32$. Or a more visual explanation: One convolutional filter squashes a $k \times k \times C^{in}$ cube of data into a single scalar, where each number in the cube has its own scalar in the kernel.
If you want 64 output channels, you have to have 64 of these kernels, so the convolution you describe should have $3 \cdot 3 \cdot 32 \cdot 64$ parameters.
As a generic formula: $\text{#Parameters}(\mathbf{W}) = k^h \cdot k^w \cdot C^{in} \cdot C^{out}$. Here $k^h$ and $k^w$ denote kernel height and kernel width respectively, and $C^{in}$ and $C^{out}$ denote the number of in- and output channels.

[...] and I thought that number of feature maps = number of filters

That's right, but each filter $w$ is not actually a 2-dimensional window, but a 3-dimensional window, i.e. a rank 3 tensor $\mathbf{W} \in \mathbb{R}^{k \times k \times C^{in}}$. You can see it in some visualizations of CNNs (s. image bloew). The input image has 3 channels and the first filter has 3-channels too and computes a single output pixel with only 1 channel.
For further reading, this blog post seems to summarize convolution well.

Image taken from here

Ah thanks! Can I say 64 output channels = 64 neurons on this layer? Does that mean that each neuron have 3 * 3 * 32 * 64 connections?

Yes, 64 output channels correspond to 64 neurons. However, these neurons are not interconnected so one neuron has $3 \cdot 3 \cdot 32$ connections. Because you have 64 neurons, the number of parameters is $3 \cdot 3 \cdot 32 \cdot 64$.

If you think about it visually: The kernel is a pattern detector that detects a learned pattern of size $k \times k$ across all channels at once ($k \times k \times 32$) and outputs a scalar that describes how well the pattern matches. If you take 64 output channels your model has the capacity to learn 64 different patterns and produces an output scalar for each one - thus you get 64 output channels.

Thanks!! The number of parameters are the learnable parameters, right?

Jepp, you go it!
Ah and just for completeness: A convolutional layer might also use a bias vector (by default in most deep learning libraries), in that case you'd get $3 \cdot 3 \cdot 32 \cdot 64 + 64$ learnable parameters. Because the bias is a 64-dimensional vector that is added to the final output before the activation function is applied.


text:
Regression model is doing exceptionally very well on time series
I have the following task to do: I have time series data. Training by the consecutive 3 days to predict the each 4th day. Each day data represents one CSV file which has dimension 24x25. Every datapoints of each CSV file are pixels.
Now, I need to do that, predict day4(means 4th day) by using training data day1, day2, day3(means three consecutive days), after then calculate mse between predicted day4 data and original day4 data. Let's call it mse1.
Similarly, I need to predict the day5 (means 5th day) by using training data day2, day3, day4, and then calculate the mse2(mse between predicted day5 data and original day5 data)
I need to predict day6(means 6th day)by using training data day3, day4, day5,  and then calculate mse3(mse between predicted day6 data and original day6)
..........
And finally I want to Predict day93 by using training data day90, day91, day92,calculate mse90(mse between predicted day93 data and original day93)
I want to use in this case , Linear regression,and we have 90 mse for this model.
import os
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import MinMaxScaler
import matplotlib.pyplot as plt

# Paths
data_folder = r'C:\Users\alokj\OneDrive\Desktop\jupyter_proj\All_data'
output_folder = r'C:\Users\alokj\OneDrive\Desktop\jupyter_proj\90_days_merged'

# Ensure the output folder exists
os.makedirs(output_folder, exist_ok=True)

# List all CSV files in the folder
csv_files = [f for f in os.listdir(data_folder) if f.endswith('.csv')]

# Sort the files based on the numeric part extracted from the filename
csv_files = sorted(csv_files, key=lambda x: int(x.split('_Day')[1].split('_')[0]))

# Prepare data
data_list = [pd.read_csv(os.path.join(data_folder, file), header=None).values for file in csv_files]
data_array = np.array(data_list)  # Shape: (num_days, 24, 25)

# Flatten the data for easier handling in regression models
num_days, rows, cols = data_array.shape
data_flattened = data_array.reshape(num_days, -1)  # Shape: (num_days, 600)

# Prepare features and target matrix for range (3, num_days)
X = np.array([data_flattened[i-3:i].flatten() for i in range(3, num_days)])  # Shape: (num_days-3, 1800)
y = data_flattened[3:num_days]  # Target is the 4th day in each sequence

# Train-Test Split and Validation (Separate fixed split)
train_size = int(0.8 * len(X))  # 80% for training
X_train = X[:train_size]
y_train = y[:train_size]
X_test = X[train_size:]
y_test = y[train_size:]

# Scaling the data
scaler_X = MinMaxScaler()
scaler_X.fit(X_train)  # Fit on training set
X_train_scaled = scaler_X.transform(X_train)
X_test_scaled = scaler_X.transform(X_test)

scaler_y = MinMaxScaler()
scaler_y.fit(y_train)  # Fit on training set
y_train_scaled = scaler_y.transform(y_train)
y_test_scaled = scaler_y.transform(y_test)

### Linear Regression
lr_model = LinearRegression()
lr_model.fit(X_train_scaled, y_train_scaled)

y_pred_test_scaled_lr = lr_model.predict(X_test_scaled)
y_pred_test_lr = scaler_y.inverse_transform(y_pred_test_scaled_lr)

# Validation for Days 3 to 93
XX = np.array([data_flattened[i-3:i].flatten() for i in range(3, 96)])  # Shape: (90, 1800)
yy = data_flattened[3:93]  # Target for validation

yy_pred_lr = lr_model.predict(scaler_X.transform(XX))
yy_pred_lr = scaler_y.inverse_transform(yy_pred_lr)

# Calculate residuals for Linear Regression
residuals_lr = [np.mean((yy[i] - yy_pred_lr[i])**2) for i in range(len(yy))]

# Plot residuals for all models
days = [f'Day {i+1}' for i in range(90)]  # Start labels from Day 4 to Day 93
plt.figure(figsize=(12, 6))
plt.plot(days, residuals_lr, label='Linear Regression Residuals', marker='o')

# Configure plot
plt.xticks(ticks=range(0, len(days), 2), labels=[f'Day {i+1}' for i in range(0, len(days), 2)], rotation=45, ha='right')
plt.xlabel('Days (Validation Set)')
plt.ylabel('Residuals (MSE)')
plt.title('Residuals for Models (Validation Set)')
plt.legend()
plt.grid(True)

# Save and show plot
plt.savefig(os.path.join(output_folder, 'residuals_plot_models_comparison_with_naive.png'))
plt.show()

My result:

We know that linear regression models often do not do very well with time series data because the assumption of independent and identically distributed data is usually violated.
But in my case from the above plot, regression model is doing exceptionally very well(means mean squared error are very low, closes to zero), would anybody check all the my regression model inside the code(if I made any mistakes or bugs that I might not be aware of.)?
My all 93 days data folder link that I used for code.

@cinch it's tec data, each data points are pixels. So, would you write the code review?

@cinch for any day we have only 15 mins data available, this data is 24×25 csv filé. On which day the data is available, it's 15 mins data.

@cinch but after shuffling lr model is working worst as expected, but lstm model is started working well than lr model after shuffling. what you comment on this scenario? since we know that in low amount of data( in my case 3-day window), the sophisticated model like lstm is often work worst, but after your suggesting shuffle, lr model is working as expected but lstm model not working as expected. what would you comment on this?

@cinch would I share with you complete code with lstm?

This question is about lr model not lstm. lstm starts working better than lr for all 3-day window test samples doesn't necessary mean lstm is not working as expected since it may still learn local patterns that persist even in the absence of strict time-based structure, though lstm generally needs much more data and certainly is not recommended for your 3-day prediction problem. The behavior you're observing after shuffling, if the new MSE is much larger than the original MSE, suggests that shuffling exposes the intrinsic low noise temporally-dependent regular structure of your data.

@cinch thank you for your valuable comments. I have one little confusion, without shuffle ridge regression is working bad than linear but after shuffle I get the reverse, what would you comments on this?

Sure. Depending on the sense of 'bad', it may be statistical flux without significance, generally with 1800 features you need some regularization.

You pandas flatten sounds right to me. And unlike psychological stock price, TEC ionospheric physical data noise are more white and you certainly have more regularity as your advantage to more accurately forecast ionospheric movement than in general. Empirically the trial behavior you're observing after shuffling of your flattened time series data within all 3-day windows, if the new MSE is much larger than the original MSE, suggests that shuffling exposes the intrinsic low noise temporally-dependent structure of your data with regularity.
The autoregressive AR(3) model should excel at your problem due to the exposed strong temporal dependency, while the cross-sectional linear regression model fails badly when shuffling features breaks such temporal order. Therefore when you originally apply linear regression model without shuffling, you get abnormally good result in the sense of very small average MSE as it's essentially equivalent to the AR(3) model. Further since cross-sectional models won't propagate errors like AR models, linear regression model may be even better to predict any days ahead directly for your problem.

Would tell me my code is right? I request you, kindly please review my code. If I get more satisfactory answer, I will award 100+ for existing answer.

@S.M. I’ve checked the critical flattern part, so far so good as already expressed in my answer. If you can share your raw csv data which includes both training and test dataset from your code, I can run and check further.

In question, below link I have provided compressed csv filé links for all the available day.

I need your opinion, please let me know if you find the file or not.

I've run the  code in your OP with your 234 csv data files, also I plot the RMSE which makes more sense when comparing residues with labels than MSE, they are all on the range of 5e-14 but the shape is different from your OP's plot, mine is more volatile with more visible peaks and valleys in a scaled output plot. Again I think for tec data there's much regularity like the near-term weather forecast, LR model could predict with high accuracy easily here with enough data like yours. Hope this helps. You could also ask in data science exchange where this type of code review question is more fit.

Also bear in mind the (R)MSE here is the average for each data point's residue out of the daily 600 total data points.

You mean, my code is in right track?

Yes from my perspective, since here it's essentially a multivariate lr model with shapes as design matrix XX[90, 1800], labels yy[90,60] after your flattening.

As you have written in above comment, yy should be yy[90, 600] not yy[90,60].

sorry, you're correct, just typo. If you know structural equation modeling (SEM) in causal statistical inference, tec data here suggests all data points may follow some simple deterministic linear structural equations with some small white noise.


text:
FrozenLake-v0 not training using REINFORCE
I am implementing a simple REINFORCE (policy gradient) algorithm for openAI's FrozenLake-v0 environment. However, it does not seem to learn anything at all.
I have used the same neural architecture for openAI's CartPole-v0, and trained it using REINFORCE (policy gradient), and it works perfectly. So, what I am doing incorrectly for the FrozenLake-v0 environment? I think this has to do with the nature of the environment, but I am unsure which aspects of training REINFORCE must be altered to accommodate the dynamics of FrozenLake-v0. It seems like a very simple environment to solve, given that it has only 16 states.
My code is as follows:
import gym
from gym.envs.registration import register
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.autograd import Variable
import matplotlib.pyplot as plt

# helper function for conversion of a state into an input to a neural network
def OH(x, n):
    '''
    :param x: state id
    :param n: n_states
    :return:  1-hot encoded numpy array of size [1,n]
    '''
    one_hot = np.zeros((n,))
    one_hot[x] = 1
    return one_hot

def running_mean(x, n):
    N=n
    kernel = np.ones(N)
    conv_len = x.shape[0]-N
    y = np.zeros(conv_len)
    for i in range(conv_len):
        y[i] = kernel @ x[i:i+N]
        y[i] /= N
    return y

# architecture of the Policy Network
class PolicyNetwork(nn.Module):
    def __init__(self, state_dim, n_actions):
        super().__init__()
        self.n_actions = n_actions
        self.model = nn.Sequential(
            nn.Linear(state_dim, 256),
            nn.ReLU(),
            nn.Linear(256, n_actions),
            nn.Softmax(dim=0)
        ).float()

    def forward(self, X):
        return self.model(X)

def train_reinforce_agent(env, episode_length = 100, max_episodes = 50000, gamma = 0.99, visualize_step = 50, learning_rate=0.003):

    # define the parametric model for the Policy: this is an instantiation of the PolicyNetwork class
    model = PolicyNetwork(env.observation_space.shape[0], env.action_space.n)
    # define the optimizer for updating the weights of the Policy Network
    optimizer = optim.Adam(model.parameters(), lr=learning_rate)

    # hyperparameters of the reinforce agent
    EPISODE_LENGTH = episode_length
    MAX_EPISODES = max_episodes
    GAMMA = gamma
    VISUALIZE_STEP = max(1, visualize_step)
    score = []

    for episode in range(MAX_EPISODES):
        # reset the environment
        curr_state = env.reset()
        done = False
        transitions = []

        # rollout an entire episode from the Policy Network
        for t in range(EPISODE_LENGTH):
            act_prob = model(torch.from_numpy(curr_state).float())
            action = np.random.choice(np.array(list(range(env.action_space.n))), p=act_prob.data.numpy())
            prev_state = curr_state
            curr_state, _, done, info = env.step(action)
            transitions.append((prev_state, action, t+1))

            if done:
                break
        score.append(len(transitions))
        reward_batch = torch.Tensor([r for (s, a, r) in transitions]).flip(dims=(0,))

        # compute the return for every state-action pair from the rewards at every time-step
        batch_Gvals = []
        for i in range(len(transitions)):
            new_Gval = 0
            power = 0
            for j in range(i, len(transitions)):
                new_Gval = new_Gval + ((GAMMA ** power) * reward_batch[j]).numpy()
            power += 1
            batch_Gvals.append(new_Gval)

        # normalize the returns for the batch
        expected_returns_batch = torch.FloatTensor(batch_Gvals)
        expected_returns_batch /= expected_returns_batch.max()

        # batch the states, actions, prob after the episode
        state_batch = torch.Tensor([s for (s, a, r) in transitions])
        action_batch = torch.Tensor([a for (s, a, r) in transitions])
        pred_batch = model(state_batch)
        prob_batch = pred_batch.gather(dim=1, index=action_batch.long().view(-1, 1)).squeeze()

        # compute the loss for one episode
        loss = -torch.sum(torch.log(prob_batch) * expected_returns_batch)

        # back-propagate the loss
        optimizer.zero_grad()
        loss.backward()
        # update the parameters of the Policy Network
        optimizer.step()

        # print the status after every VISUALIZE_STEP episodes
        if episode % VISUALIZE_STEP == 0 and episode > 0:
            print('Episode {}\tAverage Score: {:.2f}'.format(episode, np.mean(score[-VISUALIZE_STEP:-1])))

    # Training plot: Episodic reward over Training Episodes
    score = np.array(score)
    avg_score = running_mean(score, visualize_step)
    plt.figure(figsize=(15, 7))
    plt.ylabel("Episode Duration", fontsize=12)
    plt.xlabel("Training Episodes", fontsize=12)
    plt.plot(score, color='gray', linewidth=1)
    plt.plot(avg_score, color='blue', linewidth=3)
    plt.scatter(np.arange(score.shape[0]), score, color='green', linewidth=0.3)
    plt.show()

Hello. Welcome to Artificial Intelligence Stack Exchange. It seems to me that this is or could be a programming issue. Let me know if I am wrong or right. Generally, questions about programming problems are off-topic here and should probably be asked on Stack Overflow. Here, we focus on the theoretical, philosophical, social and historical aspects of Artificial Intelligence. Please, take the time to read our on-topic page. So, if you think this is not a programming problem, you should edit your post to clarify that.

Thanks for the pointer! However, I suspect this is more than a 'programming issue'. I think the reason the environment does not converge using REINFORCE, has to do with the nature of the environment. I was looking for suggestions on which specific aspects of the environment are an impediment to policy learning. I have edited my question to clarify this :)


text:
Why is val accuracy 100% within 2 epochs and incorrectly predicting new images? (1,000 images per class when training)
My CNN tensorflow model reports 100% validation accuracy within 2 epochs. But it incorrectly predicts on single new images. (It is multiclass problem. I have 3 classes). How to resolve this? Can you please help me understand these epoch results?
I have 1,000 images per class that are representative of my testing data. How can validation accuracy reach 1.00 in just the first epoch when I have a dataset of 3,000 images in total, equal amount per class? (I would expect this to start at around 33% percent -- 1/ 3 classes.)
I understand overfitting can be a problem. I've added a dropout layer to try to solve this potential problem. From this questionWhat to do if CNN cannot overfit a training set on adding dropout? I learned that a "model is over-fitting if during training your training loss continues to decrease but (in the later epochs) your validation loss begins to increase. That means the model can not generalize well to images it has not previously encountered." I don't believe my model is overfitting based on this description. (My model reports both high training and high validation accuracy. If my model was overfitting I'd expect high training accuracy and low validation accuracy.)
My model:
def model():
  model_input = tf.keras.layers.Input(shape=(h, w, 3))
  x = tf.keras.layers.Rescaling(rescale_factor)(model_input)
  x = tf.keras.layers.Conv2D(16, 3, activation='relu',padding='same')(x)
  x = tf.keras.layers.Dropout(.5)(x)
  x = tf.keras.layers.MaxPooling2D()(x)
  x = tf.keras.layers.Flatten()(x)
  x = tf.keras.layers.Dense(128, activation='relu')(x)
  outputs = tf.keras.layers.Dense(num_classes, activation = 'softmax')(x)

Epoch results:
Epoch 1/10
/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:1096: UserWarning: "`sparse_categorical_crossentropy` received `from_logits=True`, but the `output` argument was produced by a sigmoid or softmax activation and thus does not represent logits. Was this intended?"
  return dispatch_target(*args, **kwargs)
27/27 [==============================] - 13s 124ms/step - loss: 1.0004 - accuracy: 0.5953 - val_loss: 0.5053 - val_accuracy: 0.8920
Epoch 2/10
27/27 [==============================] - 1s 46ms/step - loss: 0.1368 - accuracy: 0.9825 - val_loss: 0.0126 - val_accuracy: 1.0000
Epoch 3/10
27/27 [==============================] - 1s 42ms/step - loss: 0.0020 - accuracy: 1.0000 - val_loss: 5.9116e-04 - val_accuracy: 1.0000
Epoch 4/10
27/27 [==============================] - 1s 42ms/step - loss: 3.0633e-04 - accuracy: 1.0000 - val_loss: 3.5376e-04 - val_accuracy: 1.0000
Epoch 5/10
27/27 [==============================] - 1s 42ms/step - loss: 1.7445e-04 - accuracy: 1.0000 - val_loss: 2.2319e-04 - val_accuracy: 1.0000
Epoch 6/10
27/27 [==============================] - 1s 42ms/step - loss: 1.2910e-04 - accuracy: 1.0000 - val_loss: 1.8078e-04 - val_accuracy: 1.0000
Epoch 7/10
27/27 [==============================] - 1s 42ms/step - loss: 1.0425e-04 - accuracy: 1.0000 - val_loss: 1.4247e-04 - val_accuracy: 1.0000
Epoch 8/10
27/27 [==============================] - 1s 42ms/step - loss: 8.6284e-05 - accuracy: 1.0000 - val_loss: 1.2057e-04 - val_accuracy: 1.0000
Epoch 9/10
27/27 [==============================] - 1s 42ms/step - loss: 7.0085e-05 - accuracy: 1.0000 - val_loss: 9.3485e-05 - val_accuracy: 1.0000
Epoch 10/10
27/27 [==============================] - 1s 42ms/step - loss: 5.4979e-05 - accuracy: 1.0000 - val_loss: 8.5952e-05 - val_accuracy: 1.0000

Model.fit and model.compile:
model = model()

model = tf.keras.Model(model_input, outputs)

 model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
              metrics=['accuracy'])

hist = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=10
)

Code to predict new image:
def makePrediction(image):
  from IPython.display import display
  from PIL import Image
  from tensorflow.keras.preprocessing import image_dataset_from_directory
  img = keras.preprocessing.image.load_img(
  image, target_size=(h, q)
  )
  img_array = keras.preprocessing.image.img_to_array(img)
  img_array = tf.expand_dims(img_array, 0) #Create a batch

  predicts = model.predict(img_array)
  p = class_names[np.argmax(predicts)]
  return p

Going to the "data" directory and using the folders to create a dataset. Each folder is a class label:
from keras.preprocessing import image
directory_data = "data"
tf.keras.utils.image_dataset_from_directory(
    directory_testData, labels='inferred', label_mode='int',
    class_names=None, color_mode='rgb', batch_size=32, image_size=(256,
    256), shuffle=True, seed=123, validation_split=0.2, subset="validation",
    interpolation='bilinear', follow_links=False,
    crop_to_aspect_ratio=False
)

tf.keras.utils.image_dataset_from_directory(directory_testData, labels='inferred')

Creating dataset and splitting it:
Train_ds code: (Output: Found 1605 files belonging to 3 classes.
Using 1284 files for training.)
train_ds = tf.keras.preprocessing.image_dataset_from_directory(
  directory_data = "data",
  validation_split=0.2,
  subset="training",
  seed=123,
  image_size=(h, w),
  batch_size=batch_size)

Val_ds code: (Output: Found 1605 files belonging to 3 classes.
Using 321 files for validation.)
val_ds = tf.keras.preprocessing.image_dataset_from_directory(
directory_data = "data",
  validation_split=0.2,
  subset="validation",
  seed=123,
  image_size=(h, w),
  batch_size=batch_size)

it can be data leakage. Check whether train and val images are same or not.

@PrakashDahal Thank you for your help! I tried my best to check this. I don't think they are the same, and I believe I have split correctly. I would appreciate your help to ensure that data is not leaking, as I am new to ML :)

I also checked my batch size to ensure it is reasonable. (If all the data is being passed to the model in 1 or 2 epochs then it may make sense for val accuracy to be high within first 2 epochs) My batch size is 32, so the model will not see all data in the first 2 epochs.

The batch size is used to control number of items used per weight update step, it does not affect the size of the epoch - an epoch is by definition a loop through all the training data. In your case you would expect around 40 weight updates per epoch.

@NeilSlater Thank you for explaining! I would appreciate any additional pointers for things to consider as I try to understand why val acciuracy is 1.00 in the first epoch and why my model then incorrectly classifies all new images.

Sorry I cannot see anything wrong. Looks like something basic has gone wrong though, and I suggest you check your assumptions about training and validation data by inspecting some of it mid-process.

@NeilSlater Than you for your advice. Any particular techniques you recommend me trying to inspect mid-process? It has been quite challenging to understand where I'm going wrong here :)

Any debug tool with breakpoints, or simple logging, or you could try tensorboard https://www.tensorflow.org/tensorboard

@NeilSlater Thank you. Can you please check my understanding of overfitting? I don't believe my model is overfitting because my model reports both high training and high validation accuracy. If my model was overfitting I'd expect high training accuracy and low validation accuracy. Does training continue to occur (weights are updated) after validation accuracy is reported as 100%?

I agree that the issue does not look like overfitting, although the result does match what you might see if you were overfitting with a data leak in the validation set, which is why Prakash mentioned it.


text:
LSTM - What kind of data should contain every dimension of input LSTM matrix, where does specific dimension points to?
I am a beginner and I have a hard time understanding inputs and outputs of LSTM.
So from the begining, I am trying to create multivariate input&output LSTM for time series forecasting.
Thankfully I've managed to operate on multivariate inputs&outputs, but, what I don't understand are the indices and what is stored in them.
For example, I have an intial input 2d numpy array of 4 variables:
a  b  c  d
1  2  3  4
5  6  7  8
9 10 11 12

Lets say that the shape of this array is 5000x4 - thus 4 features and 5000 rows,
So for the LSTM input I'd need an array of x_train = (samples, timesteps, 4 features),
And the code for input is as following:
   model = keras.models.Sequential([
    keras.layers.LSTM(128, return_sequences=True, input_shape=[None, 4]),
    ...
    keras.layers.Dense(4)
])

Also since I want 4 features as output I need 4 output neurons.
Now for example my y_train values are in the following array:
y_train = (samples, timesteps, 4 features)

Now I'd like to understand in what dimension (samples, timesteps or 4 features) in x_train should be values from inital array?
To what values timesteps dimension points, same question about samples and features.
Now let's say I've trained my model, from the output and I get predictions array of the same dimensions as x_train and y_train, in what dimension are the exact predictions?
Since I'd like to plot them or something with them?
EDIT: Additional question
What type of data should be stored in samples, timesteps, features.
Can you give me a simple example?

Rather than writing "Need help understanding X and doing Y", can you please just put your specific question in the title?

@nbro done, hope it's ok


text:
Would it be a good idea to mutate half of the offspring of each GA generation 100% of the time and the other half 0% of the time?
I was reading about genetic algorithms, and to my understanding a genetic algorithm (GA) is an algorithm that starts with an initial population of chromosomes, where each chromosome has associated with it a fitness score, and it evolves the population such that the chromosomes in the population have an on average better fitness score than the chromosomes in the initial population. A GA accomplishes this by selecting the chromosomes in the population with the best fitness scores, and then it combines those chromosomes using a crossover function to produce offspring chromosomes. Those offspring may or may not be mutated randomly according to a probability that the offspring will be mutated. This process of selection, crossover, and mutation is iterated (with each successive iteration of the population being called a 'generation') until the fitness score of the population as a whole is deamed satisfactory. Please correct me if any of this is wrong.
My idea is with regards to mutating the offspring produced during the crossover phase. In the relatively simple implementations of GAs that I've looked at on the internet, most seem to randomly mutate the offspring according to a mutation rate. For example, the offspring may have a ten percent chance of mutating. When the GA is ran, the fitness function generally improves a lot over the first several generations, but it often stagnates for a while when one chromosome takes over essentially the entire population and little mutation occurs. This problem can be partially solved by increasing the mutation rate, but that also increases the probability of having offspring with a lower fitness score than the parents if the mutation poorly affects the fitness score. My idea is to make it so that half of the offspring produced with each generation have a one hundred percent mutation rate, and the other half has a zero or relatively low mutation rate. This could potentially lower the number of generations necessary to reach a satisfactory fitness score.
Is this a good idea? Would it work? Would it be better than other methods of mutating the offspring of each crossover?


text:
Do learning rate schedulers conflict with or prevent convergence of the Adam optimiser?
An article on https://spell.ml says

Because Adam manages learning rates internally, it's incompatible with most learning rate schedulers. Anything more complicated than simple learning warmup and/or decay will put the Adam optimizer to "complete" with the learning rate scheduler when managing its internal LR, causing model convergence to worsen.

I have found the same convergence issues in my own work when using both Adam and a StepLR scheduler.
I understand that Adam adjusts the learning rate on a per-parameter basis, which perhaps negates the need for a learning rate scheduler, but why does this lead to convergence issues?
Is there any mathematical reason/proof why using both the Adam optimiser and a learning rate scheduler causes convergence issues?
Is it true that they really "compete" with each other?


text:
Low accuracy and high loss in multi-class classification
I'm trying to classify images in 17 flowers dataset which consist of 1360 images of 17 classes (80 images per class); I have to use DNNs only therefore I made my model with the following settings:
images_net = models.Sequential([layers.Dense(300,input_dim = 3072, activation='relu'),
                                    layers.Dense(30,activation='relu'),
                                    layers.Dense(17,activation='softmax')]);

images_net.compile(optimizer='SGD', loss='categorical_crossentropy',metrics=['accuracy']);

my_net = images_net.fit(x_train,y_train,validation_data=(x_test,y_test),batch_size=32,epochs=10)

finally, when I evaluate my model I get the following result:

Can anyone explain the result and what the problem is with these settings?

Hard to help you without more information. What images are you trying to classify, what are the stats of the dataset your using (e.g. size), do you have an expectation about the final performance of the model, i.e. are you trying to replicate some results or just trying your luck? Also loss and metrics are easier to understand when plotted in separate graphs. For now the only striking thing is that you're using a dense neural net rather than a CNN, any reason for that?

@EdoardoGuerriero I've just edited my question; In addition, I don't have any specific expectations from the performance of the model, I actually looking for a reasonable result.

Your model is completely underfit
Reasons:

You have a very small neural network that can't generalize your problem.
you mentioned 17 classes but in the last layers, you are specifying 16 class layers. Dense(16,activation='softmax') try to fix this.
If there is a class imbalance you have to remove this which may make your results worst.
Add more epochs after adding more layers or use pre-trained models.

Try this code and models:
base_model = ResNet50V2(weights='imagenet', include_top=False, input_shape=(512,512,3))

for layer in base_model.layers:
    layer.trainable = False
x = GlobalAveragePooling2D()(base_model.output)

prediction = Dense(17, activation='softmax')(x)
model = Model(inputs=base_model.input, outputs=prediction)
model.compile(optimizer = 'adam' , loss='categorical_crossentropy', metrics = ['accuracy'])
model.summary()
history = model.fit(x_train,y_train,validation_data=(x_test,y_test),batch_size=32,epochs=10)

Evaulate Model:
print(history.history.keys())
#  "Accuracy"
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
# "Loss"
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()

Hard to say without seeing the data, but the model has already 0.9 million trainable parameters so isn't super simple either. But it has very few layers and no convolutions at all, which doesn't seem ideal. But I agree with all of your other points.

share your code I will let them with better accuracy


text:
What kind of NN to use to find misprints in test
I have a bunch of unique full names of users. I made pseudo-physical model to emulate misprints of desktop and mobile users (hence, fatfingering, jumpy fingers, accidentals touches of touch bar etc.)
So, I have pairs like John Snow - joh Snown
I tried first Recurrent networks, LSTM, like some kind of vocabulary to "translate" from bad words to good ones, but it return only known predicted result, and when I try to put unknown last names, it returns wrong results.
I wish to find some patterns in misspelled words, and to predict correct spelling.
Can you please advice some kind of NN to cope with the task, or maybe some contributions in that domain?
P.S. Yes, I know that there exist other AI methods to get things done
P.P.S. This vocabulary is not in English, just in case
UPDATE
LSTM nn works nice with known names and last names endings for new last names. Right now I use 2 different nn, first to correct mistypes, second to determine first, last and middle name.
UPDATE2
Sequence to sequence solution also can normalize name (put them in order), find sex of person, find probability of error, etc.

Are you sure there is a pattern in misspelled words? Also, even if there is a pattern, how can a model distinguish between a misspelled word and a correctly spelled word when it confronts an unknown last name?
For example like in John Snow - Joh Snown, how do we know for sure that the user's name is not actually Joh Snown?

@DKDK there are some patterns that can not exist in real language. As for russian, for example, it is very rare to find some combinations of consonants, sequences of letters, sequences of letters in first word if second has some specific words ending, when second is feminine and first have masculine ending etc. because of a grammar of the language. There will happen some faults, but it is a recommendation system. I know some systems that distinguish mistypes with big rule sets, and they work (not perfectly, but even if I reach +5% more correct names that will save time and money)

I think it would be a hard task for identifying spell errors in first and last names but you can fine-tune a BERT and use Damerau–Levenshtein distance in your final prediction. for further information you can read the paper below.
Misspelling Correction with Pre-trained Contextual
Language Model


text:
How to handle list features in clustering?
I have a dataset where one of the features is a list.
Example:
[
 {
  ... other features ...
  X: ['joey', 'monica', 'ross']
 },
 {
  ... other features ...
  X: ['chandler', 'rachel', 'joey', 'phoebe']
 }
  ... more records ...
]

Feature X in the example can be of different lengths(let's say [0,50]).
The domain for the values in X is of 200 different values.
How should I handle X?
Thanks.
EDIT:
More information about what X is.
I'm trying to cluster failed software tests.
Each test might fail in a different component.
X is a list of failed components.
What I'm trying to understand is how to represent this list of failed components in the vector of each data point.

What do you mean by "How should I handle X?"? What's your problem? Are you asking how to compare the feature vectors X? Maybe you should describe what X is supposed to represent. It seems to be a list of names, but do they have any meaning. You should probably describe your dataset more in detail and what you're using now to cluster it. Do you want to cluster it based on what?

Hi, I've added more information about what X is.

You essentially want to have a numerical value to represent the similarity of the lists of two distinct objects. There are a number of metrics to deal with that, eg the Jaccard Index or Dice's Coefficient.
Which one is the best for you depends on the exact properties and your use case.


text:
How to identify overfitting in LSTM-RNN using metrics?
How can I identify overfitting on a RNN-LSTM with the following metrics: RMSE, MSE, RAE, R-squared ? I have searched papers and google results. I don't see something clear to my mind. Also I rarely see some thresholds in RMSE, for example, that above x value it means that I have overfitting...

What do you want to train with your LSTM-RNN?

Time-series forecasting on temperature...

I see. Do you use other data other than temperature for the training such as location, date, time, and etc?

No, only temperature...

I think you are confused, here are some facts about overfitting:

Overfitting is only evaluated on the loss used to train the model.
Overfitting depends on the difference between training loss and validation/test loss, not on a specific threshold, you should evaluate a plot of train/val loss across epochs during training.
Overfitting is not detectable on metrics. Only on the loss.

So you cannot use any of the metrics you mentioned to detect overfitting, unless they are actually the loss used to train the model. Also a threshold for RMSE does not make sense to detect overfitting, it simply does not work that way.

I use loss function, and the both validation and training loss have almost the same values: Epoch 1/100 1640/1640 - 24s - loss: 0.0150 - val_loss: 7.6152e-04 - 24s/epoch - 2ms/step Epoch 2/100 1640/1640 - 22s - loss: 2.6332e-04 - val_loss: 6.1577e-04 - 22s/epoch - 2ms/step So, because it confused me I am trying different methods of identifying the overfitting...

@just_learning These results do not show overfitting, so nothing to worry.


text:
Extremely good results in RNN-LSTM python code!! How can this happen?
I am using this code here: https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/ and more specifically the python code under the (1st) paragraph "LSTM Network for Regression". I am testing it with a simple .csv file where I have the 1st column (X values) and the 2nd column (Y values). I get extremely good results. My concern is: Do you something strange on the code of the above link?

I mean how can I face so good accuracy? Which are the python commands (from the above code)  more responsible for that?

It is basically because of the simplicity of the data points and nothing strange about the code. If you look at the plot in the blog, you can basically replicate those observations with a simple mathematical function. Additionally, the data points are just in two dimensions which makes the problem even simpler. This data basically has a trend that is linear (of the form mx + c) and a sinusoidal variation with some local low-frequency noise.
It is generally understood and known that for any simple dataset where the patterns are very obvious to the human eye, any machine learning algorithm will easily learn the pattern. If it can also be expressed mathematically fairly in a simple form, then any ML model will learn to generalize it. You can repeat that exercise by replacing the same data with a simple sinusoid and will see similar results.


text:
Are the output dimensions of the first and second convolutional layer in YOLO paper correct?
I was reading the last version of the YOLO paper available in Arxiv, and I don't fully understand the output dimensions (I understand width and height, but not depth) of the first and second convolutional layers.

Shouldn't the output of the first layer be 112x112x64? Shouldn't the output of the second layer be 56x56x192? I thought (and this is the case after the 3rd layer) that the depth of the ouput of a conv layer is equal to the number of filters of this layer. This is why after the first conv layer (that contains 64 filters) I would expect an output depth of 64. And the same for the second conv layer that contains 192 filters, I would expect the output to have a depth of 192.

In the diagram, they write 7x7x64-s-2. What is the meaning of the notation "64-s-2"? That's probably going to answer your question. Feel free to answer your own question below, if you find an answer to it.

I don't think that's the solution. s-2 means stride of 2, and this stide results in the input dimensions being divided by 2 in the output. This is why after the first two s-2 layers (one conv and one maxpool), the 448x448 input is transformed into a 112x112 output. The second s-2 maxpool layer, transforms its 112x112 input into a 56x56 output. An so on. However, this is not related in principle to the third dimension (depth). Thanks in any case!

Yes, I was thinking that s was related to stride. The other idea is that 64*3 = 192. Are they using a 3d convolution?

In any case anyone is struggling with the same problem. It seems that they were simply typos in the original paper. I have downloaded the author's framework Darknet, as well as the configuration and weight files for YOLOv1.
Then, the architecture can be tested with one sample image using this command:
./darknet yolo test cfg/yolov1/yolo.cfg yolov1.weights data/person.jpg

The output of this command also prints the full architecure of the net. There, I see the expected depth values after the first (depth of 64) and second max layers (depth of 192):
   layer   filters  size/strd(dil)      input                output
       0 Create CUDA-stream - 0
     Create cudnn-handle 0
    conv     64       7 x 7/ 2    448 x 448 x   3 ->  224 x 224 x  64 0.944 BF
       1 max                2x 2/ 2    224 x 224 x  64 ->  112 x 112 x  64 0.003 BF
       2 conv    192       3 x 3/ 1    112 x 112 x  64 ->  112 x 112 x 192 2.775 BF
       3 max                2x 2/ 2    112 x 112 x 192 ->   56 x  56 x 192 0.002 BF

So it seems clear to me that the 192 and 256 depth values after the first and second max layer on the figure in the question are just typos.


text:
When should discretization of observations be considered?
I found some literature regarding the design of action-spaces and that e.g. a discretization of continuous actions in video-game environments can be crucial for successful learning (Action Space Shaping in Deep Reinforcement Learning, 2020).
However, I didn’t find anything discussing this topic concerning observations. I’m not talking about normalization of observations, as NN are mainly used in DRL as function approximators and they perform better with normalized inputs.
E.g. continuous observations, which might be discretized into ordinal or categorical values. Imagine having a domain where the agent can close or open a gate, and it should learn whether the next object should pass or not. The agent has also to learn how to close the gate (with actions being "reduce or increase gate width", so the agent has to pick one action multiple times in a row to reach some certain width). Now one can design the environment to provide a continuous value for the gate width. Or discretize this and provide flags that indicate the gate is open enough or not. Are there references that discuss this topic?
The reason I’m thinking about this is because considering a policy as a mapping of actions to states, discretizing the observations will eventually lead to less mapping that has to be learned. Surely this can come at the cost of a lower peek performance, but I could also imagine higher training speed.
Is my thinking correctly concerning that? Or could discretizing the observations harm learning in other ways?

I believe that discretizing the action/state space when using function approximators like NN is only acceptable when losing information is acceptable. Why would you discretize an observation, for example, when the precise value of a continuous feature is important for making a decision?  Imagine, for example, control scenarios, one of the fields that fit the decision-making characteristics of MDPs... if there is a system that produces similar results for decisions based on a range of values such as a sensor that measures temperature it could be plausible to discretize it to reduce the observation space. Contrary, if the system needs the accuracy of some values to make decisions, it might be not a good idea. I saw works tackling this mostly backing in the past for tabular RL, where this was a necessity due to the curse of dimensionality. Nowadays, I mainly see approaches to deal with this more efficiently, such as actor-critic methods (continuous action scenarios). In short, the discretization depends on your specific domain, but the idea is to give enough information for the agent to take the best action. You may discretize your observation space which will then require a lighter architecture to learn at the cost of a less efficient policy. On the other hand, you may be adding irrelevant information to the agent with non-discretized features that do not highlight the important caracteristics of the environment. Representation is a tough task in RL!

I thought about this too and was wondering if there are more specific approaches for environments that are very poor on training data. I thought that reducing continuous observations into more discrete ones could ease learning, but I was also concerned that the agent gets less specific feedback from the environment with discretized observations. Consider the agent has to learn a sequence of actions to reach a state change. If only one action isn't performed, the state change is not happening and that might be seen as an equally wrong behaviour as a random action sequence by the policy.


text:
Are multi agent or self-play environments always automatically POMDPs?
As part of my thesis, I'm working on a zero sum game with RL to train an agent.
The game is a real-time game, a derivation of pong, one could imagine playing pong with both sides being foosball rods.
As I see it, this is an MDP with perfect information, as I use the sensor data provided by the environment to know the exact linear and angular position of the rods and the position, direction and velocity of the ball.
These information will be used as a feature vector, which will be passed into the agents network.
I considered using self-play to improve training speed regarding wall clock time, and was now not so sure if the game is after all a perfect information MDP, as there are two players involved (the same network on both sides) and the strategies of the second player are not represented in the observations fed to the network.
So the game might be a perfect information MDP, but with two "players" involved, is this still the case? Or does the fact that multiple learners are involved only make the environment more non-stationary, rather than partially observable? I also found a somewhat related paper: https://www.researchgate.net/publication/220301660_The_world_of_Independent_learners_is_not_Markovian

Generally, "perfect information" is not a formal trait of MDPs. There is a concept of the Markov property, but it only loosely coincides with "perfect information". For instance it is OK for there to be unknown/hidden state, provided it behaves effectively randomly (when revealed, it is drawn from a consistent distribution). An example might be a solo card game - or one where the opponent has a fixed policy such as most implementations of blackjack - which is fine as an MDP, and would not be a POMDP.
A "perfect information" game is chess. The current state of the board gives all the information required to play, and there is no additional state. When played against an opponent with a fixed policy, then this also has the Markov property. When played in general against many different opponent, and you cannot rely on the opponents' policies being the same as the training, then effectively it is non-Markov, or a POMDP. The plays for maximising reward depend on the opponent.
In many games though, there is the concept of "perfect play", of forcing the best possible result from the current game state. Planning/search routines using minimax approaches can theoretically solve these games, and reinforcement learning agents can learn them by assuming the opponent is also trying to play in this game-theoretic optimal approach. This is not quite the same as playing optimally against any given opponent - by comparison it is very safe and risk-averse compared to trying to trigger a mistake in the opponent due to something that the opponent may do. However, it often leads to stable solutions and strong players in RL.
Self play RL will tend to converge on the same game theoretic perfect play, provided it is stable in the given environment. An agent that uses the same learning engine for both players will progress through one player or the other making a mistake, and its opponent model will take advantage of that until the losing player finds a move that corrects its mistake.
To avoid this mistake/fix scenario becoming a stagnant loop of learning and unlearning, it is common to train the agent against multiple versions of itself (and sometimes other agents) to help it progress. Again this works provided the environment supports a stable game theoretic policy where it is possible to avoid mistakes against all adversarial choices. Most (possibly all? I am not sure of the theory here) two-player games are like this. When you extend to three or more players I believe this becomes a lot harder to figure out.

Let's say that you model a multi-agent system as a single-agent system. In that case, what does it mean to play against multiple opponents? It means that $p$ changes over time. If you assume that the strategy of the opponent is non-stationary (which can also be the case when we're playing against different strategies/opponents), i.e. then you have a non-stationary MDP. So, I am not sure it's "correct" to say that, if the opponent's strategy changes, then it's non-Markov, i.e. $p(s_{t+1} \mid s_t, a_t) = p(s_{t+1} \mid s_t, a_t, s_{t-1:1}, a_{t-1:1})$ does not hold...

...because, basically, you're really changing $p$. Also, POMDPs, as the name suggests, still assume the Markov property of the states, even if you don't know in which state you are. So, being non-Markov is not equivalent to being a POMDP. Or am I missing something?

@nbro: It's fiddly. You can model adversity as POMDP (not knowing the adversary's internal state and assuming it has one), but training with multiple learning models is closer to non-stationary as you suggest. So the best theoretical model may depend on whether you are considering a specific training environment or production environment. I will have a think about it, see if I can write something clearer.


text:
Why do off-policy algorithms suffer from worse computational or time efficiency compared to on-policy algorithms?
When I run Soft-Actor-Critic (off-policy) in my Environment, the calculation of gradient updates takes almost twice the time compared to using PPO (on-policy).
I also saw that ACER has a higher time complexity compared to PPO in this paper PPO with Prioritized Trajectory Replay
In their comparison, ACER took almost 5 times longer (wall-clock time) compared to PPO for 1000 steps in Altantis-v0 benchmark.
That's why it came to my mind that this maybe has to do with their different algorithmic approaches (off- vs. on-policy).
This question is not about the difference between sample- and computational- or time efficiency, as this was quite well explained here.
The question is: Is my assumption correct that off-policy algorithms are more computationally or time complex than on-policy ones? If so, why? Is there a certain component like the replay buffer which increases complexity?
Or is my assumption incorrect and the computational or time complexity differences between the algorithms was only implementation related, meaning that off-policy algorithms doesn't necessarily have to be more computational complex or take more time than on-policy ones?

I think your question in the body of the post is partially a duplicate of this. The questions are not exactly the same, but my answer there should give you some insight on why more sample efficient algorithms might require more computational resources. By the way, you're asking here 2 distinct questions (the one in the title and the other one) - I recommend that you clarify what your main question is.

@nbro Thanks for your reply. I indeed read that question and it addresses the topic sample vs computational efficiency in that it gives examples of algorithms that fall more or less in some category compared to others. However, my question is why is this the case: Like why is it that e.g. SAC has a higher computational complexity compared to PPO? Is it because the sampling of a replay buffer? Or was this just some coincidence and is not linked to algo categories off-/on-policy? I will adapt my question to make my question more clear in this regard.

I don't see where that paper you link talks about computational complexity. Wall time is not the same as computational complexity.

PPO is off policy.

@Taw True, wall time is not the same as computational complexity. I will change my question to ask about time complexity. But PPO is an on-policy algorithm.

@Taw PPO is on-policy, as far as I am aware. It uses trajectories generated by the current policy to make updates to the policy parameters.

@kitaird off-policy algorithms will generally have higher computational complexity than on-policy algorithms simply because you are doing more gradient updates (in general) than on-policy algorithms. Consider the SAC algorithm: at each step of an episode we sample a batch from our replay buffer and make a gradient update. If the batch size is large, then this computational overhead will be large. Compare this to PPO where we only make updates every $T$, it makes sense that the overhead would be less.

It is technically off-policy because the current policy may be slightly different than the policy used to generate the trajectories. But maybe it was too pedantic to bring that up.

@kitaird Its difficult to compare wall times of algorithms because they may use different network architectures. It's not clear to me that the paper you linked (which does not appear to be published) accounts for that.

@Taw I see what you're getting at; I think you could argue for it being a hybrid of both on and off policy, but I would say that because the trust region constraint the updated policy to not stray too far from the original policy that generated the rollouts then it is still considered on-policy.


text:
GANs: Why does iterative gradient descent sometimes optimise $\min_G \max_D V(D,G)$ and sometimes $\max_D \min_G V(D,G)$?
For the following minimax equation for generative adversarial networks (GANs),
$$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})] + \mathbb{E}_{\boldsymbol{z}\sim p_{\boldsymbol{z}}(\boldsymbol{z})}[\log(1 - D(G(\boldsymbol{z}))]  $$
Goodfellow mentions in his paper (https://arxiv.org/pdf/1406.2661.pdf, page 3, 1st paragraph under equation 1) that in practice this minimax game is implemented iteratively (algorithm 1 in the paper).
In one of his tutorials (1:31:52 - 1:33:16) he mentions that $\min_G \max_D V(D,G)$ is desirable, whereas $\max_D \min_G  V(D,G)$ leads to mode collapse. Iterative gradient descent can "sometimes act" like the former, and sometimes like the latter.
I am confused about how the iterative method can sometimes act like min-max or max-min. What about gradient descent causes the change in behavior?


text:
Why is there a Hessian diagonal approximation? And when can we use it?
This topic has been introduced in "Pattern Recognition and Machine Learning, Bishop, 2006", section 5.4.1. I am a bit confused about this method and I have two questions.

Why this method has attracted attention or has been developed?
First, we want to compute the Hessian fast, so we try to approximate it in O(W) time, where W is the number of parameters.
And then, we see that this matrix most of the time is heavily non-diagonal.

My second question is how can we know whether we can approximate a Hessian or not? Is there a hint/clue in problems?

Thanks in advance!

Hello. Please, provide more context about this "Hessian Diagonal Approximation". Provide a link to or name of the paper/book/etc that describes this. Moreover, please, to improve the clarity of your post, put your specific question in the title.

@nbro Thanks for the guidance. I did my best. I hope it is clear enough now.

Ok, thanks. I think your question is a duplicate of this. Let me know if I am correct.


text:
Are foundation models something fundamentally new? Is there a proper definition?
Currently, one can hear more and more about "foundation models" but details of this are not  always clear. Also, I even have the impression, that sometimes people don't talk about exactly the same thing. Sometimes all of this feels a bit nebulous in the spirit of "let's throw as much data as possible at as many GPUs as possible, and something useful and predictive will come out". This clearly can be fun but in the worst case just produce a tremendous amount of CO2...
Therefore I am wondering if there exists a clear definition of what a foundation model is (or is not) and if you know of good examples that demonstrate the usefulness and novelties? Do you believe that this is something truly new or just a scaled-up version of what we are doing anyway so far?
Thanks in advance for sharing your insights!

A foundation model is any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks

from "On the Opportunities and Risks of Foundation Models" seems like a clear and reasonable definition.
"Unsupervised word sense disambiguation rivaling supervised methods" from 1995 is considered self-supervised. Most researchers would not call that part new to foundation models.
The "can be adapted to a wide range of downstream tasks" also certainly pre-dates  modern "foundation models" e.g. "Self-taught learning: transfer learning from unlabeled data" from 2007.
So "is it something fundamentally / truly new"? Not really, but still very impressive.


text:
Is my dataset a time series dataset? and should I use an LSTM?
I have a dataset where I am recording temperature after every 4milliseconds till 500 and another feature "conductivity value". The length of the dataset is around a 1000 rows. I need to find the conductivity value based on the temperature pattern.

t1
t2
t3
....
t5
conductivity

90
91
93
....
96
0.34

92
91
93
....
95
0.36

I am bit confused on how to use the dataset in a time series model such as LSTM because I have all the time sequence in columns and I don't know the conductivity values in between as in t2,t3,t4.
I think the dataset becomes a classification problem with the current format.
Can you guys help me out?

I think that what You could do is to treat your 't' data as input vector and conductivity value as y and, as You said, train LSTM-based network on this. If You want to make it work in real time, You can collect last 5 't' values into a list and then put it into network for prediction. After that erase the oldest value from the list and add new one on the other side. If You'll loop that You'll have real-time conductivity predictions.

I don't think time series model necessarily makes sense if you have one conductivity value to predict for each time series.
A regression like setup makes more sense here: you could model this by letting the vector of time points represent the input. So you'd end up with a $n \mbox{ x } t$ matrix as input to predict the conductivity value.


text:
Do neural networks, trained with backpropagation algorithm, exploit the concept of synaptic plasticity?
Is there some of Hebb's rule behind the concept of backpropagation learning rule of a simple supervised neural network, that for example is trained for classification task ?
I was reading about the concept of synaptic plasticity that is explained in simple words here https://qbi.uq.edu.au/:

Plasticity is the ability of the brain to change and adapt to new information. Synaptic plasticity is change that occurs at synapses, the junctions between neurons that allow them to communicate.
The idea that synapses could change, and that this change depended on how active or inactive they were, was first proposed in the 1949 by Canadian psychologist Donald Hebb.

And more in particular I found in Wikipedia that:

Hebbian theory is a neuroscientific theory claiming that an increase in synaptic efficacy arises from a presynaptic cell's repeated and persistent stimulation of a postsynaptic cell. It is an attempt to explain synaptic plasticity, the adaptation of brain neurons during the learning process.

In an artificial neural network, synaptic efficacy (that is the strength of communication between neurons, from https://www.frontiersin.org) depends on weights that are associated to connections between neurons (from what I understood reading the beginning of paragraph 44.3 The basic unit — the neuron in Handbook of Chemometrics and Qualimetrics: Part B https://www.sciencedirect.com).
Thus, when we train our supervised classifier so that it updates weights thanks to backpropagation algorithm (whose philosophy is the changing of weights to minimize an error function that characterizes the comparison between the network output and the ground truth), are we applying a sort of Hebb's Rule and, more properly, the concept of synaptic plasticity since weights change during the learning process?
Or Hebb's rule means only "Fire together, wire together" as I understand by reading the answer to this SE post How do you explain Hebbian Learning in an intuitive way?.
I read also this other SE answer to the post Is there a Hebb neural network? in which it is clearly explained that neural networks (or models) that can learn in a Hebbian fashion are different from those based on backpropagation algorithm.
But if synaptic plasticity means a changing in the connections between neurons (synapses), that in artifical neural networks means a changing in the weights, during a learning process, can I say that also neural networks based on back-propagation algorithms exploit the concept of synaptic plasticity (and so the Hebb's rule) ?

Even when it is always said that "neural network cells" are "inspired" in biological neurons, there are critical differences that made this similitude only an "inspiration". Thus, comparison of a CNN or any other kind of NN with brain is always a fuzzy comparison.
The biggest difference is probably the learning capacity: the human brain learns by itself, while the neural network needs an external system (the back-propagation algorithm) that feeds the NN with the network parameters (the knowledge).
In other words: a neural network, understood as the set of nodes that execute a basic function (lets say weighted addition and activation function) is absolutely unable to learn anything by itself. The knowledge is written in the neural network as a set of weights and other parameters. This is not an intelligent process, as it is not load a program in a conventional computer. The system who learns is the system who runs the back-propagation algorithm and this system is not a neural net.
In comparison, the human mind, understood as a network of cells,is able to learn by itself from the experience.

Thus, when we train our supervised classifier so that it updates
weights thanks to backpropagation algorithm (whose philosophy is the
changing of weights to minimize an error function that characterizes
the comparison between the network output and the ground truth), are
we applying a sort of Hebb's Rule and, more properly, the concept of
synaptic plasticity since weights change during the learning process?

Absolutely no.
A learning process based in Hebb's rule will:

not need an external system to run the back-propagation algorithm
increase the weight between two cells when an artificial neuron and the next one triggers more or less at same time.

This is not the method of the back-propagation algorithm:

the back-propagation algorithm doesn't execute in the neural net
the back-propagation algorithm compares expected results and network ones and, using math calculus, modifies the network weights.

Nowadays, I do not know of any system based on biological neuron behavior and able to learn. Just a few models as been able to learn the eigenvector of a basic process. Even worst, no body as been able to replicate basic behavior of well know biological brains as the one of c-elegans worm, a brain that is completely mapped in number of neurons and synapses between them.


text:
Sequence Embedding using embedding layer: how does the network architecture influence it?
I want to obtain a dense vector representation of protein sequences so that I can meaningfully represent them in an embedding space. We can consider them as sequences of letters, in particular there are 21 unique symbols which are the amino acids (for example: MNTQILVFIACVLIEAKGDKICL).
My approach is to use a sequence embedding that can be learned as a part of a deep learning model (built with Python using Keras libraries), that is a classifier (supervised) neural network which I train to classify sequences according to the host species they belong to. The steps I follow are the following:

Tokenization. The only way I can tokenize these sequences of amino acids symbols is to consider single characters as tokens. I also found this example in Kaggle Deep Protein Sequence family Classification in Python that classifies different proteins and uses single amino acids as tokens.
Embedding. Stealing words from the answer to the question How the embedding layer is trained in Keras Embedding layer:

An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (for example 26 in the case of lower case alphabet) and on the other axis the dimensionality of your embedding space. The role of the embedding layer is to map a category into a dense space in a way that is useful for the task at hand, at least in a supervised task. This usually means there is some semantic value in the embedding vectors and categories that are close in this space will be close in meaning for the task.

Moreover, useful words from the blog titled as Neural Network Embeddings Explained:

The main issue with one-hot encoding is that the transformation does not rely on any supervision. We can greatly improve embeddings by learning them using a neural network on a supervised task. The embeddings form the parameters — weights — of the network which are adjusted to minimize loss on the task. The resulting embedded vectors are representations of categories where similar categories — relative to the task — are closer to one another.

Thus putting all these pieces together in my case: I choose as tokens the single amino acids, so I have 21 unique symbols (i.e. number of amino acids) and I choose 10 as dimension of the embedding space, thus my embedding layer dimension is 21 x 10. This means that, once the neural network is trained, I can extract the weights of the embedding layer that are 21 vectors (one for each amino acid) and each vector is a 10 dimensional vector.
As the second extract explains, each element of these vectors are values that are adjusted to minimize the loss in the task. I could see each amino acid as if it was a letter in a word and I wanted to classify these words according to something (like positive or negative comment); or as if it was a word in a sentence and I chose as tokens words and I wanted to classify these sentences according to something (like positive or negative comment).
Since I want sequence representation, I have to find a way to put together the embeddings of single amino acids and the only way that I found feasible is to average all the 10 dimensional vectors so to obtain for each sequence a 10 dimensional vector that is the average of the embeddings of all the amino acids. This would for sure highlight if there are over-represented symbols in one sequence with respect to other. Furthermore, since each symbol is associated to vectors whose values are adjusted to minimize loss on the task, averaging should preserve each single amino acid meaningful embedding (meaningful embedding according to the task) and give "a global meaningful embedding" of the whole sequence that minimizes the loss on the task. This, in fact, seems to work for simple sentences embeddings. See the answer to a my question post: How to obtain vector representation of phrases using the embedding layer and do PCA with it in which each word of sentences were considered as tokens and I was looking for vectors embedding the whole sentences. Similarly, here I can see each single amino acid symbol as word and the whole sequence as sentence.
Hence this method should carry/embed two information: frequencies of letters of sequences and classes to which each sequence belong to (in this case host species).
But I would like it to consider also the global structure of letters positions (not only the relative frequencies), because with this method sequences like: MNTQILVFIACVLIEAKGDKICL (sequence 1) and AKGDKICLMNTQILVFIACVLIE (sequence 2) are represented by the same 10 dimensional vector.
Thus... from here the third point:

Choice of neural network architecture. Does the choice of neural network architecture influence the embedding of each single amino acid symbol and, consequently, the embedding of the entire sequence ? For example, if I use a LSTM neural network that should "memorize" the global structure of sequences, would it improve the dense vector representation (both in general and in this particular case) ? I would expect yes since, as reported also in the extracts, this embedding layer (so its weights) is trained as all the others so the backpropagation algorithm updates also these weights. In other words, if LSTM network has the power to recognize the importance of the position of each character according to the task (for example, it is able to learn that: if a letter is in position 1, it means it belongs to human, while instead if it is in position 2, it means it belongs to dog) then weights should be updated also according to this. Differently from a simple deep learning model that is not able to consider this information and to deal with sequences.

I understand that if I average the embeddings of the single amino acids, I would have anyway the problem that sequences like MNTQILVFIACVLIEAKGDKICL sequence 1 and AKGDKICLMNTQILVFIACVLIE sequence 2 would have the same dense vector representation. But does, also in this case, a better choice of the network architecture give a better result in some way ?
I apologize for the long question and I thank you in advance.

Premises: mine is not gonna be an exhaustive answer, also I'm more familiar with classic natural language processing than with embedding vectors applied to protein sequences. Said so, I think I can give a couple of suggestions.
3 Yes, the choice of architecture will produce vectors with different properties. This applies also to vectors generated with classic matrix factorization techniques, not only deep learning models. For example, transformers tend to perform better in regards with the position of a word in a sentence thanks to self attention and positional embeddings, LSTM suffer from long range dependencies issues (so the correlation between the first and last amino acids will be poorly captured).
2 Embedding is not just a trainable layer. In fact, many nlp applications leverage pretrained embeddings. So you could train your own embeddings prior to training your classifier using host species as target labels. There are a variety of approaches to do so, the classic one CBOW, Skip-Gram and GloVe. Of course to train good embeddings you need lot of documents (sequences in your case). This is why pretrianed embeddings are usually the way to go, cause they were trained on billions of documents (an operation that can take not hours but months). Still, even for small corpora, training embeddings with one of these approaches and then refine them on a classification task (so using them as a not randomly initialized but trainable layer) leads to better results.
2 (pt2) The order problem is not that hard to tackle as you might think. Averaging the vectors is indeed a good choice when it comes to combine embeddings to get a whole sentence representation. And indeed averaging doesn't provide any information about the order. I already mention that for example transformer are designed to account for order, but there are also simpler ways. The first that comes to my mind, specific for your case, is to simply increase the amount of embeddings to learn, and move from a 1-gram model (single characters) to 2/3-grams models, so training embeddings for A, B, AB, BA and so on. Computationally it won't be a problem since you're starting from 21, which is a ridiculously small amount of vectors compared to the millions of embeddings trained for classic NLP (and I would argue that also 10 as chosen final dimension is kinda small). Another idea could be to train "dummy" positional embeddings yourself, (dummy cause no transformers of self attention is involved). This would require to stick to a fixed maximum length though. Of course 21 becomes big if you think at all possible permutations of let's say 40 characters ($40^{21}$ is a monster number), but if you train separate embeddings for each combination of character+position, so A1, A2, B1, B2, and so on, then the total amount of vectors to train would be just 40*21=840, which again is nothing compared to normal embeddings dictionaries.

Thank you very much for your helpful answer ! I have just a last thing that I do not understand and that is the point I care about the most:  you said that different network architectures will produce vectors with different properties... for example transformers and LSTM, in some way, capture correlation between symbols relative to their positions. So this means that the vector representing an amino acid has values that take into account also this ?

While instead if I do not use them, but I use a simple deep learning model (like with just an hidden layer Dense(64)),  these vectors would have values that are optimized only according to the task without capturing the importance of their positions?

In few words: assuming that I want to use the method of averaging vectors (with single characters as amount of embedding to learn) to obtain representation of protein sequences and without initializing with pretrained embedding (even though I thank you for the suggestion), using LSTM would improve their representation or not ? Or maybe it is correct to say that is necessary to use networks like LSTM because we are dealing with these kind of data that are sequences?


text:
Is my intuition about RNN wrong?
Until today, my intuition about RNN (LSTM/GRU) was that this is some kind of NN that can remember previous inputs.
Consider a task where you need to predict 0 if the previous input was 1. For example: x=[0,1,0,0], y=[1,1,0,1]. We train the model online, i.e. the number of timesteps the RNN sees at a time is equal to one (in Keras you can make an LSTM/GRU cell stateful so that it passes state (memory about previous inputs?) between batches). And the problem is that, in my tests, RNN cells can not cope with this simple task.
I suspect that my intuition about RNN is built wrong. I would appreciate someone could help me with that.
Test script:
import numpy as np
import tensorflow as tf
from random import randrange

batch_size = 1
timesteps = 1
size = 1
units = 16
shape = (timesteps, size)

model = tf.keras.models.Sequential([
  tf.keras.layers.Input(shape = shape, batch_size = batch_size),
  tf.keras.layers.GRU(
    units,
    stateful = True,
    return_sequences = True
  )
])

model.compile(optimizer='adam',
              loss='mean_squared_error')

print(model.summary())

SAME = False

i = 0
prevIsBlack = False
x_batch, y_batch = [], []
x_steps, y_steps = [], []

while True:
  i += 1

  isBlack = randrange(4) == 2

  if SAME: prevIsBlack = isBlack

  x_train = np.ones(size)   if isBlack else np.zeros(size)
  y_train = np.zeros(units) if prevIsBlack else np.ones(units)

  if not SAME: prevIsBlack = isBlack

  x_steps.append(x_train)
  y_steps.append(y_train)

  if len(x_steps) < timesteps:
    continue

  x_batch.append(np.stack(x_steps))
  y_batch.append(np.stack(y_steps))

  x_steps, y_steps = [], []

  if len(x_batch) < batch_size:
    continue

  res = model.fit(np.stack(x_batch), np.stack(y_batch),
    epochs = 1,
    shuffle = False,
    batch_size = batch_size)

  print(res.history["loss"])

  x_batch = []
  y_batch = []

Can you please be more specific about "RNN cells can not cope with this simple task."? What do you mean by "cannot cope"? What's the problem you're having? How much data do you have? How exactly are you trying the RNNs? There are so many details missing from this question.

I got this batch-based GRU to work:
import numpy as np
import tensorflow as tf

units = 2
seq_length = 32

X = (np.random.rand(int(1e4), seq_length, 1) > 0.5).astype(float)
Y = 1 - X

model = tf.keras.models.Sequential([
  tf.keras.layers.Input(shape=X.shape[1:]),
  tf.keras.layers.GRU(units, return_sequences=True),
  tf.keras.layers.Conv1D(1, 1, activation='linear')
])

print(model.summary())

model.compile(optimizer='adam', loss='mean_squared_error')
history = model.fit(X, Y, batch_size=512, epochs=10, verbose=0, validation_split=0.1)

for k in sorted(history.history.keys()):
    print('%15s: %s' % (k, str(np.array(history.history[k][:10]).round(3))))
print()

history = model.fit(X, Y, batch_size=512, epochs=40, verbose=0, validation_split=0.1)

for k in sorted(history.history.keys()):
    print('%15s: %s' % (k, str(np.array(history.history[k][-10:]).round(3))))

It trains for 10 epochs, prints logs, trains for 40 more epochs and prints logs of the last 10 epochs.
Here is an example run:
Model: "sequential_104"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
gru_39 (GRU)                 (None, 32, 2)             30
_________________________________________________________________
conv1d_68 (Conv1D)           (None, 32, 1)             3
=================================================================
Total params: 33
Trainable params: 33
Non-trainable params: 0
_________________________________________________________________

           loss: [1.137 0.97  0.832 0.718 0.625 0.546 0.479 0.422 0.372 0.329]
       val_loss: [1.041 0.891 0.767 0.665 0.58  0.508 0.447 0.394 0.348 0.307]

           loss: [0.026 0.023 0.021 0.019 0.017 0.016 0.014 0.013 0.011 0.01 ]
       val_loss: [0.024 0.022 0.02  0.018 0.016 0.015 0.013 0.012 0.011 0.009]

Using 8 units helps it to converge faster, after 10 epochs the loss is about 0.1.
I noticed that your network outputs the GRU's state directly as the predicted value.
Maybe there is a problem since the output dimension doesn't match the target value? There might be some broadcasting going on, which hides the issue. <- edit: my earlier assumption, it isn't correct.
My network's Conv1D converts the GRU's units dimensional output to a N x seq_length x 1 prediction. Adding it to your online version seems to fix it. Although the learning is quite unreliable, maybe due to the batch size of one.
Edit: Aah after reading your code more carefully I noticed that the target predicted value isn't the previous value of the input (a single digit) but this:
y_train = np.zeros(units) if prevIsBlack else np.ones(units)

I don't know a neural network architecture in which we'd measure the loss directly on the RNN's output, it is only used by network to "remember" data or state from the past. But it is then processed further by subsequent layers. First of all the GRU's activation funtion is tanh, so it cannot even produce values outside of the range from -1 to 1.
I hope this clears things up.