text:
OpenAI Gym implementation of the delayed rewards
My question is about whether is it possible to implement delayed reward logic within Gym environment.
More specifically, I work on ride-pooling RL algorithm, when the action (choice of the parameters of ride-pooling heuristic) is taken once in several simulation iterations. However, trips that are influenced by this action may finish later than next Gym step starts. I am wondering if this is possible to send reward back to one of the previous Gym steps when it will eventually be calculated.
For example, we made first action at simulation iteration 10, second action at 20, third action at 30, and only at 35 the reward for the first action is ready.
In basic implementation of Gym step() one needs to return rewards in the end of the step.

I think what you have here (with an important caveat, which I will get to later) is a common misunderstanding about how rewards should be structured for a reinforcement learning (RL) problem. It is perfectly OK to return a reward of $0$ for most steps, and only return measured rewards when the event they are related to happens.
The task of figuring out that the critical action that caused a reward was some number of time steps ago is called the credit assignment problem. Sometimes this can make solving an RL problem hard. However, it is a core issue in RL, and all RL algorithms are built to solve this issue. This is usually achieved through algorithms that backup through time iteratively, assigning expected future values to earlier states (in value-based algorithms), or by associating all actions taken with rewards that occured in their future and allowing this to average out over multiple different experiences (in policy gradient methods).
The important caveat is this: When the agent takes the earlier action, this choice must have had some observable influence on the state, and that change should still persist in some form and to some degree at the time the associated reward is observed, such that it is possible for the agent to make the association. In some cases - e.g. a game of checkers - then the state change appears naturally in how the environment is expressed. In other cases you may need to take extra steps. In the worst-case scenario, you would need to store a history of actions taken and add that to the state vector.
Most simple evironments provided with Gym should provide you with a fully usable state vector where all influences on next state and reward are included. So you may need to do nothing other than apply a standard RL control method. Custom, or more advanced environments may require that you add something extra to the state yourself.

Exactly, you say it very well. I upvoted your answer


text:
Does a decoder in transformer model generate output embeddings like the following?
Encoder:
Input: [A, B, C, D] (word embeddings)
Output: [C1, C2, C3, C4] (contextual representations)
The encoder processes the input sequence [A, B, C, D] and generates contextual representations [C1, C2, C3, C4]. The specific calculations involved in the encoder, such as self-attention and feed-forward layers, are not shown in this example.

Decoder:
Input: [C1, C2, C3, C4] (contextual representations)
Output: [A', B', C', D'] (word embeddings)
The decoder takes the contextual representations [C1, C2, C3, C4] as input and generates word embeddings [A', B', C', D'] for the output sequence. The decoder generates one word embedding at a time, conditioned on the previously generated embeddings and the contextual representations.

Let's illustrate the decoding process step by step:
Step 1:

Input: [C1] (contextual representation of the <start> token)
Output: [A'] (word embedding for the first output word)
Step 2:

Input: [CA', C2] (contextual representations and word embedding generated so far)
Output: [B'] (word embedding for the second output word)
Step 3:

Input: [A', B',C3] (contextual representations and word embeddings generated so far)
Output: [C'] (word embedding for the third output word)
Step 4:

Input: [A', B', C',C4] (contextual representations and word embeddings generated so far)
Output: [D'] (word embedding for the fourth output word)

The encoder-decoder architecture is typically employed to work with two different sequences: the input sequence and the output sequence. The presence of both depends on the task; for machine translation, the transformer encoder processes the text in the source language, and the decoder generates the corresponding text in the destination language. A generative model, such as GPT, could also consist of only the decoder component.
I will focus on encoder-decoder architectures.
In the original implementation, once the encoder obtains the contextual representations [C1, C2, C3, C4], these are considered fixed inputs for the decoder. In the simplest case, the decoder should produce a sequence of indices corresponding to words in the vocabulary as its output. Therefore, the target sequence should be a sequence of indices/words, not word embeddings.
The decoder obtains two inputs at each step: the encoder output (fixed) and the partially generated sequence. I will use E(x) to refer to the word embedding of the word x. The step-by-step generation (during inference) will be:
Step 1:
Inputs:
[C1, C2, C3, C4] (encoder output)
[E(<start>)] (partial decoder output)

Outputs:
["the"]

Step 2:
Inputs:
[C1, C2, C3, C4] (encoder output)
[E(<start>), E(the)] (partial decoder output)

Outputs:
[*, "cat"]

Step 3:
Inputs:
[C1, C2, C3, C4] (encoder output)
[E(<start>), E(the), E(cat)] (partial decoder output)

Outputs:
[*, *, "eats"]

Step 4:
Inputs:
[C1, C2, C3, C4] (encoder output)
[E(<start>), E(the), E(cat), E(eats)] (partial decoder output)

Outputs:
[*, *, *, "<end>"]

At each step, the decoder can perform masked self-attention (to attend to the previous tokens of its partial output) and cross-attention over the entire encoder output sequence.
The output of a single step isn't actually a word; instead, it is a probability distribution over the word vocabulary. This distribution can be used to sample a word according to the probabilities, or we can simply choose the word that corresponds to the argmax of the distribution. Furthermore, more complex techniques such as beam search can be used to generate better sequences. Once the word is chosen, it is appended to the partial decoder output and used in the subsequent step. The asterisks correspond to the outputs of the decoder for previous timesteps, which are no longer interesting.

"The encoder obtains two inputs at each step:" It should be "The decoder obtains two inputs at each step:"

@NicolaFanelli Nice answer. Voted! Anche se potrei rispondere in italiano perchè siamo entrambi del sud

Grazie! @Frimand

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Building a resume recommendation for a job post?
There are few challenges I am facing when building a resume recommendation for a particular job positing.
Let's say we convert the resume into a vector on n-dimensions and job description also as an n-dimension vector and in order to see how similar, we can use any similarity metrics like cosine.
Now, for me the biggest problem with such approach is not able to provide more importance to the job title required. Some times, for a cloud engineer position I am getting java developer resume recommended in top 10 just because some of the skills/keywords are overlapping between two so their embeddings becomes similar.
I want to provide more weightage to the job title as well. What are some possible things I can do to make my recommendations consider or put bit emphasis on job title.
Note:- A common job title lookup in resume will fails because people write job titles in multiple ways.
(java engineer (or) java developer) | (cloud engineer or aws engineer) etc.,
How can I overcome this issue?

Please, do not ask for "thoughts", but ask a more specific question, which will not lead to opinions, but can be answered objectively.

@nbro my question is actually specific about a problem I am facing. I have removed the last line. Thank you.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Given a set of trajectories produced by a fixed policy, what is the the standard approach to estimate Q?
Let's say that I have a set of trajectories $\mathcal{D} = \{\tau_1, \dots, \tau_n\}$ produced by an agent acting in a (episodic) MDP with a fixed policy $\pi$. I would like to estimate the $Q$ function of $\pi$ from $\mathcal{D}$. Just to be clear, each trajectory $\tau_j$ is a finite sequence
$$
\tau_j = s_0^j, a_0^j, r_0^j s_1^j, a_1^j, r_1^j \dots, s_{N_j}^j
$$
representing an episode performed w.r.t. $\pi$.
What would be the standard approach in this case? Better use TD learning or Monte Carlo?

Where are the rewards in your data? Usually I would expect to see them in the trajectories ie. $\tau_j = s_0^j, a_0^j, r_1^j,  s_1^j, a_1^j, r_2^j, \dots$

Yes, I forgot the rewards! I edit the question

What would be the standard approach in this case? Better use TD learning or Monte Carlo?

Both should be fine, but they might lead to different estimates, if both these things apply:

The amount of data is relatively small compared to all possibilities from the given environment and policy.

Either the policy or the environment are stochastic.

The difference is that for each state/action pair estimated:

Monte Carlo will estimate based on overall average returns, ignoring individual state transitions and policy choices.

Temporal Difference will estimate based on observed state transitions and policy choices.

There is a good example of what this might mean numerically in Sutton & Barto chapter 6, example 6.4. In that case it shows an advantage to TD learning when some states might be sparsely represented in the data whilst others have more instances. Monte Carlo learning will only learn the value of those rarer states from the trajectories where they occur, whilst TD learning will be able to use estimates of other trajectories, provided two or more trajectories overlap later on.
This doesn't necessarily make TD learning better. If a trajectory that overlaps with others also happens to include an unusual policy choice, state transition or reward, this may spread sample bias into multiple estimates, whilst Monte Carlo would be affected less by such an outlier.

Thanks! So in some sense TD has high bias and less variance, while MC has low bias and high variance, right?

@Onil90: Yes, although this bias in not the same bias as referred in other scenarios discussing TD vs MC. In your scenario with a fixed data set, and assuming process until convergence, it is that both MC and TD respond to sample bias differently.

Your trajectories must contain rewards, so I'm assuming you've forgotten them in your original post, i.e., we must have $$\tau_j = (s_0^j, a_0^j, r_1^j, ..., s_{N_j}, a_{N_j}, r_{N_j+1})$$
Given that you have access to the full trajectories, I would use the Monte Carlo estimates. You want to use TD methods when you need to estimate $Q^\pi$ incrementally as new transitions $(s_t, a_t, r_{t+1})$ arrive.
You can estimate $Q^\pi$ as follows (adapted from the Sutton&Barto book, chapter 5.1):
Initialise $\text{Returns}(s,a)$ to an empty list for all $(s, a)$ pairs in the trajectories.
For each $j$:
  Initialise the cumulative discounted reward $G$ to 0.
  For $t$ in ${N_j, N_{j-1}, ..., 0}$:
    Set $G = \gamma G + r_{t+1}$
    If $(s_t, a_t)$ is not in $((s_0, s_0), ... (s_{t-1}, a_{t-1}))$:
      Append $G$ to $\text{Returns}(s_t, a_t)$.
      Set $Q^\pi(s_t, a_t) = \text{Average}(\text{Returns}(s_t, a_t))$
For completeness the method is called "First visit Monte Carlo" because we only update the estimate of $Q^\pi$ using first visits to an $(s, a)$ pair.


text:
Lego minifigure facial recognition: where to start?
I'm interested in starting a project that will identify the face of a Lego mini figure from a digital photo. My goal is to eventually map the expression of a person's face to the Lego mini figure.
I don't have any experience working with image recognition technology (my technical experience is mainly in web technology), and I am looking for recommended platforms or resources that I could get started with.
Most helpful would be recommendations for image recognition technologies (Python would be great!) that I could start to experiment with.
NOTE: I'm aware of SparkAR as a library designed to for Instagram camera effects specifically, and even though I'm not interested in Instagram, I wonder if there are comparable libraries/studios/products for working with image recognition development.

Welcome to AI Stack Exchange. By "identify" do you mean that you want to locate the miniature's face in the image (plus perhaps its pose), but not necessarily identify it as "figure C" from some collection? Even with that clarifcation, it is likely this will be a bit broad or you will get vague answers trying to describe your whole project outline. If you could add what you are starting with and what your specific, current block is to continuing with the project work, then that would help narrow things down to items you might need right now

Thanks @NeilSlater! I added some details. I am very early in the process, and mostly interested in what technologies or libraries would be good to investigate first. Essentially, what are the most popular frameworks people currently use for this type of product, if any?

Since you are mainly interested in starter libraries and packages to read up on here are some pointers:
Image recognition as well as general spatial classification mainly consists of two major tasks.

Translating images to data that can be fed into a ML model of any kind.
Building a model that is able to use this data to complete tasks like classification (which lego figure is this?), spatial clustering (which pixels form the "face"), etc.

For the first task you will find a lot of python libraries that suit the task but `` PIL/Pillow``` should be the main library for this. So start by reading some documentation and transferring the images you do have (this includes standardizing and cleaning the images to improve the results of step 2).
For the second task the actual model you will use depends on your task but generally speaking some form of neural net is a good place to start and if you prefer python then you should look into tensorflow and keras (easy to use interface to tensorflow).
Where to start
Start with the classical MNIST numbers recognition case and if you have grokked that, you will now where to go from there or at least be able to ask a more specific question here.
References

https://pillow.readthedocs.io/en/stable/
https://keras.io/
https://towardsdatascience.com/image-classification-in-10-minutes-with-mnist-dataset-54c35b77a38d


text:
How to represent multiple-output logic circuits in tree-based genetic programming
Consider the following digital logic circuit, which has multiple inputs and one output:

The logic circuit above can be represented in tree form:

This tree representation could then be used in a tree-based genetic programming framework to evolve the circuit. For example, this tree could be represented as a Lisp list (or (and A B) (not C)), which could then be used with the Little LISP genetic programming framework from John R. Koza's Genetic Programming textbook.
However, I now want to deal with digital logic circuits that have more than one output. For example, in the half-adder circuit below, there are two outputs $S$ and $C$, each of which is affected by both inputs $A$ and $B$.

(Image source: SICP by Abelson et al. Section 3.3.4 A Simulator for Digital Circuits. CC BY-SA 4.0)
How do I represent and evolve such a circuit in tree-based genetic programming? How do I represent the circuit above as a tree that could then be used to evolve the circuit using a tree-based genetic programming framework?

Maybe you can do this with a custom function that gets $s$ and $c$ as input and that would produce the final return type, i.e. a binary vector.

@nbro Thank you for the suggestion, but how would such a tree look like exactly? With the binary vector return type, $S$ and $C$ can each be influenced by either $A$ or $B$ but not both. That is the consequence of a tree representation. Perhaps I am not fully understanding your suggestion because I don't see how it could represent things like half-adders (shown in the question) and flip-flops.

I didn't realise that you didn't want both A and B to affect both S or C, because you didn't explain this in the post. In any case, my suggestion was more about how you could possibly deal with 2 outputs. You can could create a function like id(s, a) = [s, a], and the output of the circuit/tree is of type list. Here, id is the custom function that you need to add to you function set, in addition to and, or and not. If I have some time, later, I will try to create a simple proof-of-concept that illustrates my idea.

@nbro Sorry for not making that clear. In the half-adder circuit shown in the question, The output at $S$ is affected by both inputs $A$ and $B$. Similarly for the output at $C$.

Ok, that's fine. My suggestion above still applies.


text:
How to select good inputs and fitness function to achive good results with NEAT for Icy Tower bot
I'm trying to make a bot to the famous "Icy Tower" game.
I rebuilt the game using pygame and I'm trying to build the bot using Python-NEAT.
Every generation a population of 70 characters tries to jump to the next platform and increase their fitness. right now the fitness is the number of platforms they jumped on, each platform gives +10.

The problem I'm facing is that the bot isn't learning good enough after 1000 generations the best score was around 200 (it can get to 200 even at the first few generations by mistake. 200 means 20 platforms which is not a lot).
when I look at the characters jumping it looks like they just always jump and go left or right and not deliberately aiming to the next platform.
I tried several input configurations to make the bot perform better. but nothing really helped.
these are the inputs I tried to mess around with:

pos.x, pos.y
velocity.x, velocity.y
isOnPlatform (bool)
[plat.x, plat.y, plat.width] (list of the 3-7 next platforms locations, also tried distance from character in x,y)
[prev.x, prev.y] (2-6 previous character positions)

I'm not so proficient with neuroevolution and I'm probably doing something wrong. glad if you could explain what's causing the bot to be so bad or what's not helping him to learn properly.
Although I think that the fitness function and the inputs should be the only problem I'm attaching the python-NEAT config file.
[NEAT]
fitness_criterion     = max
fitness_threshold     = 10000
pop_size              = 70
reset_on_extinction   = False

[DefaultGenome]
# node activation options
activation_default      = tanh
activation_mutate_rate  = 0.0
activation_options      = tanh

# node aggregation options
aggregation_default     = sum
aggregation_mutate_rate = 0.0
aggregation_options     = sum

# node bias options
bias_init_mean          = 0.0
bias_init_stdev         = 1.0
bias_max_value          = 30.0
bias_min_value          = -30.0
bias_mutate_power       = 0.5
bias_mutate_rate        = 0.7
bias_replace_rate       = 0.1

# genome compatibility options
compatibility_disjoint_coefficient = 1.0
compatibility_weight_coefficient   = 0.5

# connection add/remove rates
conn_add_prob           = 0.5
conn_delete_prob        = 0.5

# connection enable options
enabled_default         = True
enabled_mutate_rate     = 0.01

feed_forward            = True
initial_connection      = full

# node add/remove rates
node_add_prob           = 0.2
node_delete_prob        = 0.2

# network parameters
num_hidden              = 6
num_inputs              = 11
num_outputs             = 3

# node response options
response_init_mean      = 1.0
response_init_stdev     = 0.0
response_max_value      = 30.0
response_min_value      = -30.0
response_mutate_power   = 0.0
response_mutate_rate    = 0.0
response_replace_rate   = 0.0

# connection weight options
weight_init_mean        = 0.0
weight_init_stdev       = 1.0
weight_max_value        = 30
weight_min_value        = -30
weight_mutate_power     = 0.5
weight_mutate_rate      = 0.8
weight_replace_rate     = 0.1

[DefaultSpeciesSet]
compatibility_threshold = 3.0

[DefaultStagnation]
species_fitness_func = max
max_stagnation       = 3
species_elitism      = 2

[DefaultReproduction]
elitism            = 3
survival_threshold = 0.2

Note: the previous character positions are the position in the previous frame, and if the game runs at 60 fps the previous position is not that different from the current one...

Note2: the game score is a bit more complex than just jumping on platforms, the bot should also be rewarded for combos that can make him jump higher. the combo system is already implemented but I first want to see the bot aiming to the next platform before he learns to jump combo.

Is platforms' arrengment all the time the same or it changes every time, or maybe every couple of times?

The platform location is random every time. but even if it wasn't, the bot can't memorize the level's layout. right?

You should try evolution with all the time constant arrangement of the platform. I don't know what are your parameters that you try to find, but if this is the list of movements that your character do in the game (step by step) then of course it will not work, because the fitness function of one of such movement list will be totally irrelevant on a different platform configuration. What means that the results from one generation will not be the same in the next one, therefore evolution can not take place.

@Makintosz I'm not quite sure I understand you, I saw bots for "flappy bird" with random levels and it worked with NEAT. why it won't work on my game?
why training on the same level layout should work? the bot cant memorizes as far as I know.

I think it depends on how you set it up, and what is that you feed to the network. Situation you described seems like randomness is the key here. What do you feed the netwrok with?

@Makintosz  the inputs I tried are listed in the question...


text:
Can we use the recursive least squares as a learning algorithm to an ADALINE?
I'm new to neural network, I study electrical engineering, and I just started working with ADALINEs.
I use Matlab, and in their Documentation they cite :

However, here the LMS (least mean squares) learning rule, which is
  much more powerful than the perceptron learning rule, is used. The
  LMS, or Widrow-Hoff, learning rule minimizes the mean square error and
  thus moves the decision boundaries as far as it can from the training
  patterns.

The LMS algorithm is the default learning rule to linear neural network in Matlab, but a few days later I came across another algorithm which is : Recursive Least Squares (RLS) in a 2017 Research Article by Sachin Devassy and  Bhim Singh in the journal: IET Renewable Power Generation, under the title : Performance analysis of proportional resonant and ADALINE-based solar photovoltaic-integrated unified active power filter where they state :

ADALINE-based approach is an efficient method for extracting
  fundamental component of load active current as no additional
  transformation and inverse transformations are required. The various
  adaptation algorithms include least mean square, recursive least
squares etc.

My questions are:

Is RLS just like LMS (i mean can it be used as a learning algorithm too)?
If yes, how can I customize my ADALINE to use RLS instead of LMS as a learning algorithm (preferably in Matlab, if not in Python) because I want to do a comparative study between the two Algorithm!

Really you want use an Adlines NN instead of perceptrons ? Sorry for the a few off-topic comment, but surprises me.

LMS and RLS are not learning rules, but cost functions. RLS th same than LMS after apply a filter to the error measures, usually a forget factor.

Please feel free to comment, i'm new so honestly i don't know, if there is something you want to suggest i'm listening, thank you

If there are not a non-linear element we are more in the area of linear filtering than in the one of AI. Moreover, as combination of linear elements can be flatten, multi-layer concept disappears. Non-linear activation functions as Heaviside (perceptron), sigmoid, ... are the center of the NN.

RLS is a second order optimizer, so, unlike LMS which takes into account an approximation of the derivative of the gradient, RLS also considers the second order derivative. You can study more about second order methods in sub-section "8.6 Approximate Second-Order Methods" of the following book available online:
https://www.deeplearningbook.org/contents/optimization.html
Deep Learning
An MIT Press book
Ian Goodfellow and Yoshua Bengio and Aaron Courville

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Why does this tutorial on reinforced learning not check whether the environment is 'game over' during training?
I am following the tutorial Train a Deep Q Network with TF-Agents. It uses the hello world environment of reinforced learning: cart pole.
At the end, the agent is getting trained with experience on the training enviroment (train_env). When performing an action in the environment a time_step is returned containing the observation, award and whether the environment signals the end, which basically says 'game over'. This can be due to the pole reaching an angle which is too high, or 200 time steps have been reached (which is the max score, atleast in the tutorial).
When compute_avg_return method evaluates an agents performance on an environment, the environment is checked to be game over or not using time_step.is_last.
Why is time_step.is_last not considered when training the agent at the end of the tutorial? Nor do I see that the environment is reset during training. Or at least, I do not see it in the code presented. Is it checked internally? Looking at the graph, it never goes over an average return (the score) of 200 time steps. So it does seem to check for time_step.is_last. Do I overlook something or how does this work?
See the code block below. I would expect the check for time_step.is_last after collect_step(train_env, agent.collect_policy, replay_buffer), which would be followed by resetting the environment if it was true.
# Reset the train step
agent.train_step_counter.assign(0)

# Evaluate the agent's policy once before training.
avg_return = compute_avg_return(eval_env, agent.policy, num_eval_episodes)
returns = [avg_return]

for _ in range(num_iterations):

  # Collect a few steps using collect_policy and save to the replay buffer.
  for _ in range(collect_steps_per_iteration):
    collect_step(train_env, agent.collect_policy, replay_buffer)

  # Sample a batch of data from the buffer and update the agent's network.
  experience, unused_info = next(iterator)
  train_loss = agent.train(experience).loss

  step = agent.train_step_counter.numpy()

  if step % log_interval == 0:
    print('step = {0}: loss = {1}'.format(step, train_loss))

  if step % eval_interval == 0:
    avg_return = compute_avg_return(eval_env, agent.policy, num_eval_episodes)
    print('step = {0}: Average Return = {1}'.format(step, avg_return))
    returns.append(avg_return)

It might be due to the environments being different although I am not entirely sure. The collect step method is carried out using train_env whereas the evaluation step is carried out in the eval_env.


text:
While we split data in training and test data, why we have two pairs of each?
Why do we split the data into two parts, and then split those segments into training and testing data? Why do we have two sets of data for each training and test data?

Are you talking about (X_train,y_train) and (X_test,y_test).
If yes, then X represents the data(features) and y represents the labels of that data. That's why you get a pair when you divide it into training and test data

For any Machine Learning model, the available data is usually split into three sets:
Training Set:
The part of data used to train the model and learn the parameters of the network.
The data that remains after allocation of the Training Dataset, is split into the Validation and Test sets.
Validation Set:
This sample of data is used to provide an unbiased evaluation of a model fit on the training dataset. This helps in tuning model hyperparameters to improve the model performance. Eg: Changing the number of clusters ($k$) in a K-Means algorithm, or the pooling layers in a CNN.
Test Set:
After training, this part of the dataset is used to used to test how well the model generalizes to unseen data and estimate its performance.
Another possibility (going by your question), is the use of Cross-Validation. This is performed when the dataset is too small.  In such a case, a random split is performed on the dataset resulting in k non-overlapping subsets. The test error is then estimated by taking the average test error across k trials.

[Image Source]

You're not answering the question "While we split data in training and test data, why we have to pairs of each?" or addressing the confusion completely. The OP thinks that both the training and test datasets are again divided into two chunks.

As far as I know, I have never come across a scenario where such a split has been used. I clarified his doubt by answering the general approach to splitting a dataset (without explicitly hinting at the fact that he might have a misconception). If you are aware of a case where such a splitting condition is used, you are free to elucidate on it.

Usually we are splinting the data into 3 chunks for example 70% for training, 10% for validation and 20% for testing. The first two chunks are going to be used for training. The reason you need the validation dataset is to tune your hyper-parameters and see how well your model can generalize. Once you have a model that achieves a fairly good performance on the validation dataset, you're measuring its accuracy on the test dataset.

You're not answering the question "While we split data in training and test data, why we have to pairs of each?" or addressing the confusion completely. The OP thinks that both the training and test datasets are again divided into two chunks.

Btw, I didn't downvote this answer and I am not sure why someone would.


text:
How to exploit translational symmetry for extrapolation in video generation using machine learning
I'll try to rephrase my problem in the context of video processing. Imagine that initial frame of video has some translational symmetry. The frame evolves according to an update rule.
I generate a time series for how an edge, say right up edge, of the frame evolves. I generate another time series for how a larger edge, including the smaller right up edge, evolves. Since there is translational symmetry, I should be able to find how the smaller edge is related to the larger edge. The final goal is to use the obtained correlation to extrapolate to larger edges. I want to find the correlation between these two multivariate time series using machine learning (ML) methods.
I want to know
1 - which one of ML methods can be used in general for this task?
2 - if I use neural networks, the input and output shapes would be (values at time steps, number of variables). For the input it makes sense, but how can I define the output layer (for example, for LSTM in tensorflow)?

Surely the sample correlation is just a direct calculation, with no ML needed.What do you mean by "correlation" ? And why do you think ML is needed?

@PeterK. I am not sure if the correlation is the right word. I updated the post to make it clear.


text:
Plotting loss vs number of updates made and plotting loss vs run time
I wanted to plot a graph to show the effect of increasing the batch size on loss calculated (MNIST dataset). But I am not able to decide if I should show change in loss over training time of the neural network or number of updates made to weights and biases (iterations and epochs basically, but for large differences in batch sizes, I think number of updates made makes more sense?). I am confused about what makes more sense (or neither of them makes sense, idk).
With Loss vs training time graph, I can show that for any training time, the loss for large batch is more. From what I have read on wiki, with Loss vs number of updates made graph, I can show that change in loss is smoother for larger batches (rate of convergence). But can't the same conclusion be made when plotted against time? (Smooth convergence means lesser spikes right?)


text:
How do I keep my system (online) learning if I can get ground truth labels only for examples flagged positive?
I have a binary classifier (think of it as a content moderation system) that is deployed after having being trained via batch learning.
Once deployed, humans review and check for correctness only items predicted positive by the algorithm.
In other words, once in production if I group predictions of the model on unseen examples in the confusion matrix
+-----------+-----------------+
|           |   Ground-truth  |
|           +-----+-----------+
|           |     | Neg | Pos |
+-----------+-----+-----+-----+
|           | Neg | x11 | x12 |
| Predicted +-----+-----+-----+
|           | Pos | x21 | x22 |
+-----------+-----+-----+-----+

I have access to all the ground-truth labels of the elements counted in $x_{21}$, $x_{22}$ (the predicted-positive)
I know the sum of $x_{11}$ and $x_{12}$, but not their values
I do not have access to the ground-truth labels of the elements predicted-negative.

This (suboptimal) setup allows to measure precision $\frac{x_{22}}{x_{21} + x_{22}}$, while recall stays unknown as elements predicted negative are not examined at all (ground-truth labels of negatives can't be assigned due to resource constraints).
Information gathered from users about the (true and false) positive elements can be used to feed a retraining loop... but

are there any "smart" learning recipes that are expected to make the algorithm improve its overall performance (say, the F1 score for the positive class) in this setting?
what's a meaningful metric to monitor to ensure that the performance of the model is not degrading?* (given the constraint specified here, F1 score is unknown).

Thanks for any hint on how to deal with this!
 * One solution could be to continuously monitor the F1 score on a labeled evaluation set, but maybe there's more one can do?

A first question that I think is important to consider is: do you expect the data that you're dealing with to be changing over time (i.e. do you expect there to be concept drift)? This could be any kind of change. Simply changes in how frequent certain inputs are, changes in how frequent positives/negatives are, or even changes in relations between inputs and ground-truth positive/negative labels.
If you do not expect there to be concept drift, I'd almost consider suggesting that you may not have that big of a problem. It might be worth not doing anything at all with the data you receive online, and just stick to what you learned initially from offline data. Or you could try to use those few extra predicted-positive samples that you get for finetuning. You'd just have to be careful not too change your model too much based on this, because you know that you're not receiving a representative sample of all the data anymore here, so you might bias your model if you pay too much attention to only this online data relative to the offline data.

I guess the question becomes much more interesting if you do expect there to be concept drift, and it also seems likely that you are indeed dealing with this in most of the situations that would match the problem description. In this case, you will indeed want to make good use of the new data that you get online, because it can allow you to adapt to changes in the data that you're dealing with.
So, one "solution" could be to just... ignore the problem that you're only learning online from a biased sample of all your data (only from the predicted-positives), and just learn anyway. This might actually not perform too badly. Unless your model is really incredibly good already, you'll likely still get false positives, and so also still be able to learn from some of those -- you're not learning exclusively from positives. Still, the false positives won't be representative of all the negatives, so you still have bias.
The only better solution I can think of is relaxing this assumption:

Once deployed, humans review and check for correctness only items predicted positive by the algorithm.

You can still have the humans focus on predicted positives, but maybe have them inspect a predicted-negative also sometimes. Not often, just a few times. You can think of this as doing exploration like you would in reinforcement learning settings. You could do it randomly (randomly pick predicted negatives with some small probability), but you could also be smarter about it and explicitly target exploration of instances that your model is "unsure" about, or instances that are unlike data you've seen before (to specifically target concept drift).
I have a paper about something very similar to this right here: Adapting to Concept Drift in Credit Card Transaction Data Streams Using Contextual Bandits and Decision Trees. Here the assumption is that we're dealing with (potentially fraudulent) transactions, of which we can pick out and manually inspect a very small sample online. The only real difference in this paper is that we assumed that different transactions also had different monetary "rewards" for getting correctly caught as positives, based on the transaction amount. So a transaction of a very high amount could be worth inspecting even if we predicted a low probability of being fraudulent, whereas a transaction of a very low amount might be ignored even if it had a higher predicted probability of being fraudulent.

what's a meaningful metric to monitor to ensure that the performance of the model is not degrading?* (given the constraint specified here, F1 score is unknown).

Having a labelled evaluation set for this could be useful if possible... but it also might not be representative if concept drift is expected to be a major issue in your problem setting (because I suppose that the concept drift that you deal with online would not be reflected in an older, labelled evaluation set).
Just keeping track of things that you can measure online, like precision, and how it changes over time, could be useful enough already. With some additional assumptions, you could get rough estimates of other metrics. For instance, if you assume that the ratio $\frac{TP + FN}{FP + TN}$ between ground-truth-positives and ground-truth-negatives remains constant (remains the same as it was in your offline, labelled data), you could also try to extrapolate approximately how many positives you've missed out on. If your precision is dropping over time (your true positives are getting lower), you know -- assuming that fraction remains constant -- that your false negatives somewhere else in the dataset must be growing by approximately the same absolute number.


text:
CS 285 Prof Sergey Levine Lecture, Bounding Derivation for Reinforcement Learning (TRPO)
How can we derive the final result? I can understand the first line, but don't know how the absolute term in the summation is replaced with $2\epsilon t$.
https://www.youtube.com/watch?v=LtAt5M_a0dI&list=PL_iWQOsE6TfXxKgI1GgyV1B_Xa0DxE5eH&index=45

I agree with the
$$\sum_{s_t}p_{\theta'}(s_t)f(s_t) \geq \sum_{s_t}\big[p_{\theta}(s_t)f(s_t) - |p_{\theta}(s_t) - p_{\theta'}(s_t)|max_{s_t}f(s_t)\big]$$
$$\because \sum_{s_t}p_{\theta}(s_t)f(s_t) = \sum_{s_t}\big(p_{\theta}(s_t) + p_{\theta'}(s_t) - p_{\theta'}(s_t) \big)f(s_t)$$
$$= \sum_{s_t}p_{\theta'}(s_t)f(s_t) + \big(p_{\theta}(s_t) - p_{\theta'}(s_t) \big)f(s_t)$$
$$ \leq \sum_{s_t}p_{\theta'}(s_t)f(s_t) + \big|p_{\theta}(s_t) - p_{\theta'}(s_t) \big|f(s_t)$$
$$ \leq \sum_{s_t}p_{\theta'}(s_t)f(s_t) + \big|p_{\theta}(s_t) - p_{\theta'}(s_t) \big|max_{s_t}f(s_t)$$
so,
$$\sum_{s_t}p_{\theta'}(s_t)f(s_t) \geq \sum_{s_t}\big[p_{\theta}(s_t)f(s_t) - |p_{\theta}(s_t) - p_{\theta'}(s_t)|max_{s_t}f(s_t)\big] \geq \sum_{s_t}\big[p_{\theta}(s_t)f(s_t) - 2\epsilon t max_{s_t}f(s_t)\big] = E_{p_{\theta}(s_t)}[f(s_t)] - \sum_{s_t}2\epsilon t max_{s_t}f(s_t)$$
Here is the question. How does the lecture note remove the summation of the epsilon related term? The professor sasy, since all of them are constant (I understand $max_{s_t}f(s_t)$ is also a constant), we can write like that, but $\sum_{t=1}^{10}(t - 1) = 45 = \sum_{t=1}^{10}t - \sum_{t=1}^{10}1 = 55 - 10 = 45$. We don't write $\sum_{t=1}^{10}1 = 1$ because 1 is a constant.
The abiguous part is there is no concrete size of $s_t$ in the summation, but at least I think just removing the summation in front of the epsilon term is weird.

I would agree with you here, but I think the "trick" is that the first $\epsilon$ and the second are different, so for example if there are $N$ states (there is finite number of states) then the last sum would be $2 \epsilon_1 t N \max_{s_t} f(s_t)$ but you can replace this with $\epsilon_2 = N\epsilon_1$, and since $\epsilon_1$ can be picked arbitrarily small so will be $\epsilon_2$. I think the final result isn't affected but the process to get it is a bit suspicious here.

Thank you for your reply! Totally make sense for the finite case ($|s_t
| = N$). Do you have any thought for the infinite case as well? So, all of the summation will be replaced with integral, then it's little bit tricky to just say $2\epsilon_1 t N max_{s_t}f(s_t)$.

I think it wouldn't be even defined in case of infinite number of states. You would have something like $\lim_{(x, y) \rightarrow (0, \infty)} xy$ which isn't defined

Hmm, I don't understand why you said $lim_{(x,y)}$. $\int_{s_t}p_{\theta'}(s_t)f(s_t) ds_t \geq E_{p_{\theta}(s_t)}[f(s_t)] - \int_{s_t}2\epsilon t max_{s_t}f(s_t) ds_t $ For the last term, it's $\int_{s_t}2\epsilon t max_{s_t}f(s_t) ds_t = 2\epsilon t max_{s_t}f(s_t)\int_{s_t} 1 ds_t$ Do you mean $\int_{s_t} 1  ds_t$ is not even defined?


text:
How can I perform lossless compression of images so that they can be stored to train a CNN?
I have a set of images, which are quite large in size (1000x1000), and as such do not easily fit into memory. I'd like to compress these images, such that little information is missing. I am looking to use a CNN for a reinforcement learning task which involves a lot of very small objects which may disappear when downsampling. What is the best approach to handle this without downscaling/downsampling the image and losing information for CNNs?

Your input image size and memory are not directly related. While using CNN's, there are multiple hyperparameters that effect the video memory(if you are using GPU) or physical memory(if you are using CPU). All the frameworks these days uses a simplified data-loaders, for instance in Tensorflow or PyTorch, you are required to write a data-loader that takes in multiple hyper-parameters that are mentioned below and fit the data into VRAM/RAM, and this is strictly dependent upon you batch size - memory occupied on VRAM has direct relation to the batch size.
Whatever may be your image size, while you are writing the data-loader you have to mention the transformation parameters to your data-loader, during the training phase the data-loader will automatically load required images into your memory according to the batch size you have mentioned. As you have mentioned about image compression, this is an irrelevant parameter at-least for most of the generic use-cases, the most relevant hyperparameters are

Scaling
Cropping
Random flip
Normalization of the RGB values
ColorJitter
Padding
RandomAffine

And many more.
PyTorch provides really good transformers in data-loader, please do check https://pytorch.org/docs/stable/torchvision/transforms.html.
For Tensorflow, have a look at https://keras.io/preprocessing/image/.

Although data loaders or generators may partially solve the problem and the information in your answer seems valid, I think the OP is specifically looking for a way of compressing single images. Maybe you could mention something about it. Also, the OP is using TensorFlow, so you may reformulate your answer so that is targets more TF rather than PyTorch.

@nbro, yes, I am looking for ways to compress single images for usage in a reinforcement learning task.

Tensorflow-Keras provides an effective data transformer and loader. Documentation is at
https://keras.io/preprocessing/image/. The ImageDataGenerator provides  for many types of possible transforms and also enables use of a user defined pre-process function. Use of the
ImageDataGenerator.flow_from_directory provides a means of retrieving images in batches from a directory containing sub directories (classes) of images. and resizing the images.
Image size can impact the results. Generally the larger the image the better the result but this is subject to the law of diminishing returns (at some point the impact on accuracy becomes minuscule) while the training time can become absorbent.
When you have large images like 1000 X 1000 where the subject of interest in the image is small say 50 X 50 the best but most painful approach is the crop the image to the subject of interest. Unfortunately this is usually a time consuming drudgery unless you can find some program that can crop the image automatically. For example there are good programs that can crop  images of people automatically where the resultant cropped image is primarily the persons face. Alternatively modules like cv2 can be adapted to provide this capability for certain images.
The batch_size you select along with the image size directly effect memory usage. If your images are large and your batch_size is too large you will encounter a "resource exhaust" error. You can reduce the batch size but this will extend training time.
Other techniques for dealing with large images include methods like sliding windows etc. Again these will increase training time because you are taking a large image and breaking it into a series of smaller images that you feed into the network.
A general though probably risky rule I follow is that if I can visibly see the subject of interest in a resized image then I assume the network will be able to detect it as well.
Will probably be less accurate than using the full image but should be as we engineers say "good enough"

authors: Габриэль Вершинин


text:
Param count in last layer high, how can I decrease?
Not sure where to put this... I am trying to create a convolutional architecture for a DQN in keras, and I want to know why my param count is so high for my last layer compared to the rest of the network. I've tried slowly decreasing the dimensions of the layers above it, but it performs quite poorly. I want to know if there's anything I can do to decrease the param count of that last layer, besides the above.
Code:
#Import statements.
import random
import numpy as np
import tensorflow as tf
import tensorflow.keras.layers as L
from collections import deque
import layers as mL
import tensorflow.keras.optimizers as O
import optimizers as mO
import tensorflow.keras.backend as K

#Conv function.
def conv(x, units, kernel, stride, noise=False, padding='valid'):
    y = L.Conv2D(units, kernel, stride, activation=mish, padding=padding)(x)
    if noise:
        y = mL.PGaussian()(y)
    return y

#Network
        x_input = L.Input(shape=self.state)
        x_goal = L.Input(shape=self.state)
        x = L.Concatenate(-1)([x_input, x_goal])
        x_list = []
        for i in range(2):
            x = conv(x, 4, (7,7), 1)
        for i in range(2):
            x = conv(x, 8, (5,5), 2)
        for i in range(10):
            x = conv(x, 6, (3,3), 1, noise=True)
        x = L.Conv2D(1, (3,3), 1)(x)
        x_shape = K.int_shape(x)
        x = L.Reshape((x_shape[1], x_shape[2]))(x)
        x = L.Flatten()(x)
        crit = L.Dense(1, trainable=False)(x)
        critic = tf.keras.models.Model([x_input, x_goal], crit)
        act1 = L.Dense(self.action, trainable=False)(x)
        act2 = L.Dense(self.action2, trainable=False)(x)
        act1 = L.Softmax()(act1)
        act2 = L.Softmax()(act2)
        actor = tf.keras.models.Model([x_input, x_goal], [act1, act2])
        actor.compile(loss=mish_loss, optimizer='adam')
        actor.summary()

actor.summary():
________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_2 (InputLayer)            [(None, 300, 300, 3) 0
__________________________________________________________________________________________________
input_3 (InputLayer)            [(None, 300, 300, 3) 0
__________________________________________________________________________________________________
concatenate (Concatenate)       (None, 300, 300, 6)  0           input_2[0][0]
                                                                 input_3[0][0]
__________________________________________________________________________________________________
conv2d_52 (Conv2D)              (None, 294, 294, 4)  1180        concatenate[0][0]
__________________________________________________________________________________________________
conv2d_53 (Conv2D)              (None, 288, 288, 4)  788         conv2d_52[0][0]
__________________________________________________________________________________________________
conv2d_54 (Conv2D)              (None, 142, 142, 8)  808         conv2d_53[0][0]
__________________________________________________________________________________________________
conv2d_55 (Conv2D)              (None, 69, 69, 8)    1608        conv2d_54[0][0]
__________________________________________________________________________________________________
conv2d_56 (Conv2D)              (None, 67, 67, 6)    438         conv2d_55[0][0]
__________________________________________________________________________________________________
p_gaussian (PGaussian)          (None, 67, 67, 6)    1           conv2d_56[0][0]
__________________________________________________________________________________________________
conv2d_57 (Conv2D)              (None, 65, 65, 6)    330         p_gaussian[0][0]
__________________________________________________________________________________________________
p_gaussian_1 (PGaussian)        (None, 65, 65, 6)    1           conv2d_57[0][0]
__________________________________________________________________________________________________
conv2d_58 (Conv2D)              (None, 63, 63, 6)    330         p_gaussian_1[0][0]
__________________________________________________________________________________________________
p_gaussian_2 (PGaussian)        (None, 63, 63, 6)    1           conv2d_58[0][0]
__________________________________________________________________________________________________
conv2d_59 (Conv2D)              (None, 61, 61, 6)    330         p_gaussian_2[0][0]
__________________________________________________________________________________________________
p_gaussian_3 (PGaussian)        (None, 61, 61, 6)    1           conv2d_59[0][0]
__________________________________________________________________________________________________
conv2d_60 (Conv2D)              (None, 59, 59, 6)    330         p_gaussian_3[0][0]
__________________________________________________________________________________________________
p_gaussian_4 (PGaussian)        (None, 59, 59, 6)    1           conv2d_60[0][0]
__________________________________________________________________________________________________
conv2d_61 (Conv2D)              (None, 57, 57, 6)    330         p_gaussian_4[0][0]
__________________________________________________________________________________________________
p_gaussian_5 (PGaussian)        (None, 57, 57, 6)    1           conv2d_61[0][0]
__________________________________________________________________________________________________
conv2d_62 (Conv2D)              (None, 55, 55, 6)    330         p_gaussian_5[0][0]
__________________________________________________________________________________________________
p_gaussian_6 (PGaussian)        (None, 55, 55, 6)    1           conv2d_62[0][0]
__________________________________________________________________________________________________
conv2d_63 (Conv2D)              (None, 53, 53, 6)    330         p_gaussian_6[0][0]
__________________________________________________________________________________________________
p_gaussian_7 (PGaussian)        (None, 53, 53, 6)    1           conv2d_63[0][0]
__________________________________________________________________________________________________
conv2d_64 (Conv2D)              (None, 51, 51, 6)    330         p_gaussian_7[0][0]
__________________________________________________________________________________________________
p_gaussian_8 (PGaussian)        (None, 51, 51, 6)    1           conv2d_64[0][0]
__________________________________________________________________________________________________
conv2d_65 (Conv2D)              (None, 49, 49, 6)    330         p_gaussian_8[0][0]
__________________________________________________________________________________________________
p_gaussian_9 (PGaussian)        (None, 49, 49, 6)    1           conv2d_65[0][0]
__________________________________________________________________________________________________
conv2d_66 (Conv2D)              (None, 47, 47, 1)    55          p_gaussian_9[0][0]
__________________________________________________________________________________________________
reshape (Reshape)               (None, 47, 47)       0           conv2d_66[0][0]
__________________________________________________________________________________________________
flatten (Flatten)               (None, 2209)         0           reshape[0][0]
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 2000)         4420000     flatten[0][0]
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 200)          442000      flatten[0][0]
__________________________________________________________________________________________________
softmax (Softmax)               (None, 2000)         0           dense_1[0][0]
__________________________________________________________________________________________________
softmax_1 (Softmax)             (None, 200)          0           dense_2[0][0]
==================================================================================================
Total params: 4,869,857
Trainable params: 7,857
Non-trainable params: 4,862,000
__________________________________________________________________________________________________

If I understood correctly you want to decrease the parameters count on the last layer (dense_2 layer right?). It would be nice to know why you want to decrease the number of parameters in the last layers... But I'll proceed with what I see.
Firstly, the dense layers (or fully connected in literature) have a deterministic number of parameters (or weights) to learn according to the size of the input and output tensor. The relation is the following:
$N_{params} = Y_{output} \cdot (X_{input} +1) = 200 \cdot (2209 +1) = 442000$
Where:

$Y_{output}$: Output tensor shape (act2=200 which is the output of dense_2 layer)
$X_{input}$: Input tensor shape (x=2209which is the ouput of flatten layer)

So if you want to decrease the number of parameters you can:

Decrease the input tensor: your self.action2 which I guess is the action space so you might not be able to decrease it
Decrease the output tensor: maybe? I would need more context (code) to know if that is even possible

So, in short: if you are not willing to change your the input or output tensor to the Dense layers, then no, you can not decrease the number of parameters.

BONUS: In case you missed it, I have noticed you have set your dense layers to trainable=False. So in principle you should not care about decreasing number of parameters (which in most cases is motivated to reduce training time) since they are already not being trained. You can check that in the Keras summary output:
Total params: 4,869,857
Trainable params: 7,857
Non-trainable params: 4,862,000

Where the non-trainable parameters are $4862000 = 4420000 + 442000 $, which are the number of parameters of your 2 dense layers.

If I froze the action layer, would it degrade performance? Likewise, if I bottlenecked the layer above the action layers, wouldn't there be a loss of information?

What is the action layer you are referring to?... I mean, you already froze the 2 dense layers which outputs act1 and act2 by setting trainable=False. Why did you freeze them? I you posted more code I would be able to get the context of it.


text:
How to overcome symmetry in the solution space when learning a simple neural network?
What are good solution recipes to overcome the problem that in neural network learning, when the solution space has symmetries, learning may eventually stall due to the sum of the gradients over the network inputs converging to a tiny sum even though the individual gradients for each input to the network separately stay relatively big?
As a simple example, consider a neural network with a two-dimensional input and a singly dimensional output with a fully connected first layer, and a MaxPool layer as the second layer. We want the network to learn the function $\max(x+2y,3x+y)$ for network input tuples $(x,y)$. For this function, two different suitable weight matrices for the linear layer exist as the order of the input values to the MaxPool layer does not matter.
With a standard Adam optimizer, learning sometimes stalls before the MSE is very close to $0$. Small variations in the setting can cause learning to succeed, such as using a linear layer without a bias function or different random seeds. However, perhaps there are some commonly used approaches to solve such symmetry problems in a more systematic way?
The following example code in PyTorch exemplifies the problem:
import torch
import numpy
from livelossplot import PlotLosses

# Generate data to learn from
numpy.random.seed(42)
torch.manual_seed(123)
num_samples = 1000
X = torch.rand(num_samples, 2)
Y = torch.max(torch.cat((X[:,0:1]+2*X[:,1:2],3*X[:,0:1]+X[:,1:2]),dim=1),dim=1,keepdim=True)[0]

network = torch.nn.Sequential(
  torch.nn.Linear(X.shape[1],2,bias=True),
  torch.nn.MaxPool1d(2,1)
)

liveloss = PlotLosses()
opt = torch.optim.Adam(network.parameters(), lr=0.02)
lossfn = lambda x,y: ((x-y)**2).mean()

for i in range(4000):
  opt.zero_grad()
  out = network(X)
  l = lossfn(Y,out)
  current_loss = l.item()
  (l).backward()
  opt.step()

  logs = {}
  logs["err"] = torch.mean(torch.abs(Y - out)).item()
  liveloss.update(logs)
  liveloss.draw()

The values of the loss function over time (MSE) look as follows:

The final MSE is about 0.106. It can be verified that indeed the function to learn has a perfect representation in this network shape by initializing the weight matrix by hand:
with torch.no_grad():
    network[0].weight[0, 0] = 1
    network[0].weight[0, 1] = 2
    network[0].weight[1, 0] = 3
    network[0].weight[1, 1] = 1
    network[0].bias[0] = 0
    network[0].bias[1] = 0

When doing so before the learning process, the MSE starts with 0 and stays 0.
To provide some context for why this problem is interesting, the above example is just exemplifying a problem that occurs in a different context, namely encoding domain knowledge in a neural network with a structure adapted to this knowledge shape and then using neural network learning to fine-tune the model. The domain knowledge encoding may induce many such symmetries in the solution space, and hence just wiggling some aspects of the network or the learning process in an unsystematic way is unlikely to enforce learning across all such symmetries at the same time.

Edit with additional experiments based on comments.

With a learning rate of 0.02 and without dropout and L2-regularization, the MSE plateaus at a MSE of 0.106 after ~10 epochs
With a learning rate of 0.02 with dropout of 0.05 before the first (linear) layer and without L2-regularization, the MSE jumps between ~0.2 and ~0.25 after ~10 epochs
With a learning rate of 0.02 with dropout of 0.05 after the first (linear) layer and without L2-regularization, the MSE jumps between ~0.17 and ~0.21 after ~10 epochs

In all cases, the learned model doesn't look anywhere close to the perfect model with the linear layer weight matrix $\left(\begin{matrix}1 & 2 \\ 3 & 1 \end{matrix}\right)$ and the perfect bias vector $(0,0)^T$ (for the data set).

dropout comes to mind.

@EngrStudent Interesting idea. For a quick test, I've injected a "torch.nn.Dropout" layer into the model (tried probabilities: 10%, 1%, 0.1%), and it seems to make the problem even worse. Not actually surprising when thinking about it: with dropout, over node of the linear layer has to take over a bit of the functionality of the other node when it is deactivated. The hand-made "perfect" model mentioned in the case is then even a bit worse for a mean squared error function than what is learned under drop-out.

Maybe your learning rate is too high, also you can try l2-regularization on the weights. Anyway, in principle you could encode symmetries in the way you represent the inputs.

@LucaAnzalone Thanks a lot for the suggestion! In some quick experiments, L2-Regularization (giving "weight_decay=0.0001" to the Adam optimizer) didn't make a difference. Dividing the learning rate by 100 doesn't make a difference either -- the MSE plateaus at 0.106. Could you explain what you mean by encoding the symmetries in the way the inputs are represented? The function to be learned is not actually symmetric in the input value order. The only symmetry is in the solution space - there are 2 optimal weight matrices for the linear layer, which are the same except for the order of the rows.

@DCTLib - can you describe the worse? Can you push the dropout in front of the layer instead of behind it? How does that modify behavior?

@EngrStudent I just added additional experimental result descriptions to the original question (at the bottom).

learning may eventually stall due to the sum of the gradients over the network inputs converging to a tiny sum even though the individual gradients for each input to the network separately stay relatively big

In my opinion, this is the definition of a minima in a loss function. Infact, given any loss function
$$
L(\theta) = \min \sum \bar{L}(\theta)
$$
and it's gradient, assuming being zero:
$$
\nabla L(\theta) = \vec{0}
$$
you can multiply the loss by any scalar values, and the gradient would not be impacted by it:
$$
\nabla L(\theta) = \nabla( 10\cdot L(\theta))
$$
thus you can make the single gradients as big as you want, or as arbitrarily small as you want, and the cumulative gradient would still be $0$
Just think about a logistic regression, if you just make the classes points further, the gradient will be bigger and bigger, but the global minima would stay the same

In your case, you are assuming that theory is sound, thus assuming your neural network as enough representation power to represent your function, and that your samples are enough, that you will end up in a global minima, and that your optimizer is good enough not to be too influenced by the possible stochasticity of the minibatches... this is not usually the case, you can for sure over-parametrize your network, and it will perfectly learn to interpolate your points, but little to no guarantees on the generalization however

Thanks for your answer! I awarded the bounty because I'm not expecting any additional answer. The thing is that indeed the minimum found is a local minimum, but it's an unwanted one, and I'm searching for standard techniques for leaving it it in the presence of symmetries in the solution space.

@DCTLib unfortunately the loss landscape (convex/non convex) has little to do with  the data, but with the model you are working with, but if you find any relevant paper, please let me know, I would be interested in what they have to say


text:
Acoustic Input Data: Decibel or Pascals
In acoustics decibel levels were defined to solve an issue with showing values that are interpretive, understandable, and easy to communicate in contrast to intensity or pressure in Pascals.
$dB = 10*\log({\frac{p^2}{p_{ref}^2}})$
This log scale helps human understanding of an acoustic signal because human hearing is capable of discerning the difference of about 1 dB ref 20 $uPa$. However, if I have input data for a 2D CNN would it make more sense to have my input data be in Decibels where it is a logarithmic or to have raw pressures?
Would one or the other benefit my models learning?


text:
Does (English) ChatGPT-generated content have statistically significantly different character frequency than human-generated content?
Detecting ChatGPT-generated content is a contentious topic at the moment.  E.g., I've been on Reddit r/ChatGPT, and there's a constant stream of users claiming they've been unfairly accused of plagiarism.
One thing I'm curious about is character frequency, i.e., how frequently each English character (a, b, ..., z) occurs in ChatGPT-generated text vs. human-generated text.  I'm not sure if a statistically significant difference is present, and maybe there's research into this.  My guess would be that it's almost impossible to detect a difference.
Question: Does (English) ChatGPT-generated content have statistically significantly different character frequency than human-generated content?
(For this question, I want to ask about English only.)
I asked ChatGPT this question and it said:

... Overall, it is possible that ChatGPT-generated content has statistically significant differences in character frequency compared to human-generated content, but this would depend on the specific training data and settings used to train the model. Further research would be needed to determine the extent of these differences.

So maybe there's some subtle difference.  I didn't find a related post by Bing-searching the question.

I stumbled upon this paper which points out how scientists use more capital letters than ChatGPT:

Scientists also use more proper nouns and/or acronyms, both of which
are captured in the frequency of capital letters, and scientists use more numbers.
Desaire et al., Distinguishing academic science writing from humans or ChatGPT with over 99% accuracy using off-the-shelf machine learning tools, Cell Reports Physical Science 4, 101426, 2023.

See this related post on Linguistics: https://linguistics.stackexchange.com/questions/46703/linguistic-analysis-of-chatgpts-default-style-of-writing

And fingerprinting https://www.beren.io/2023-02-26-Fingerprinting-LLMs-with-unconditioned-distribution/


text:
Formulation of a Markov Decision Process Problem
Given a list of $N$ questions. If question $i$ is answered correctly (given probability $p_i$), we receive reward $R_i$; if not the quiz terminates. Find the optimal order of questions to maximize expected reward. (Hint: Optimal policy has an "index form".)
I am fairly new to Reinforcement Learning and Markov Decision Problems (MDP). I am aware that the goal of the problem is to maximize the expected reward but I am not sure how exactly to formulate this into an MDP.
This is the approach I thought of:
1) Assume only 2 questions. Then the state space is $S\in \{1,2\}$.
2) Compute the expected total reward $J = E(R)$ for both cases, when we start with question $1$ and question $2$ and then find the maximum of the two.
3) If we start with $1$, then $$J(S_0 = 1) = p_1(1-p_{2})R_1 + (R_1 + R_2)p_1p_2$$
4) Similarly, if we start with $2$, $$J(S_0 = 2) = p_2(1-p_{1})R_2 + (R_1 + R_2)p_1p_2$$.
To determine the maximum reward of the two, the required condition for $1$ to be the optimal starting question is $$R_1p_1 - R_2p_2 + p_1p_2(R_2 - R_1) \gt 0$$ If the above expression is negative, then we should start with $2$.
I would like to know if the approach is correct and how to proceed further. I am also not sure how to define the action space in this case. Can a dynamic programming approach be used here to find the optimal policy?


text:
What is the appropriate RNN structure to do Sentiment Analysis with multiple dependent ratings?
Suppose we are doing sentiment analysis for a restaurant. Customers can rate the restaurant by #1: how expensive the restaurant is, #2:how good is the food and #3: how likely they will come again. The ratings are dependent,i.e. the more expensive the restaurant is (higher #1), the less likely they will come back (lower #3), but whey will if the food is good (higher #2).
My questions are: is there a good RNN structure(review as input, #1-#3 as output) that can capture and model the dependency among #1 - #3?

"(review as input, #1-#3 as output) " you want one input and 3 different outputs, are you referring to multi-task learning? And what do you mean exactly with "dependency among #1 - #3"? Can't you just perform classic statistic like correlation between variables and stuff?

Inspired by (1) machine translation where source phrases are first encoded to a feature vector and decoded by another RNN and (2) automatic captioning where a picture is also encoded to a feature vector before it's decoded by the RNN followed. A proper structure for the problem above may be the inverse of these, where the review is encoded by RNN to a vector, and a DNN is followed to decode the vector into #1 - #3.


text:
Are sentences from the same document independent and identically distributed?
I am trying to build an LSTM model to generate Shakspeare-like poems. I have data set $\{s_1, s_2, \dots, s_m \}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1, w_2, \dots, w_n \}$.
I am wondering:  Are different $s_i$ ($i=1, \dots, m$) independent and identically distributed samples (IID)? Are $w_i$ ($i=1, \dots, n$) within each sentence the IID?

The sentences coming from the same document, author, etc., are unlikely to be independent, that is, the occurrence of a sentence $s_i$ in a certain document $d$ is likely correlated with the occurrence of another sentence $s_j$. If they are not independent, they can also not be independent and identically distributed (which is a stronger condition). The same can be said for words in the same sentence.


text:
What are the reasons to belief AGI will not be dangerous?
We are in the middle of an ongoing debate about the safety of AGI and our current approach towards this technology. As summary, some quotes from a recent article from Time magazine:

Many researchers[...] expect that the most likely result of building a
superhumanly smart AI, under anything remotely like the current
circumstances, is that literally everyone on Earth will die. Not as in
“maybe possibly some remote chance,” but as in “that is the obvious
thing that would happen.”

Without [...] precision and preparation, the most likely outcome is AI
that does not do what we want, and does not care for us nor for
sentient life in general.

Outline for the way to superintelligence and where it can go wrong

Intelligence is subtrate independent, i.e. matter can contain intelligence (brain), ergo a silicon computer can contain intelligence
The evolution of intelligence in the brain is extremely slow (biological evolution). Silicon based AI could recursively self-improve extremely fast and natural limits for intelligence are unlikely to be anywhere close to human level. (Keywords: Singularity, Seed-AI, Intelligence Explosion, etc)
Large jumps in intelligence are known to cause original, "hard coded" goals to disappear, e.g.: Human beings - as product of optimizing for inclusive genetic fitness - have been "trained on" increasing the number of own copies of functioning systemns with their own DNA - yet we are using birth control and typically it is not priority #1 to donate sperm.

Further complication: At some point (self improving seed-AI) it is not possible to use "trial and error": The choice is between "sucessfull alignment on the first try" or "an unaligned superintelligence appears", which in one way or another is catastrophic and many believe - with a probability bordering on certainty - leads to: human extinction.

Question
The debate could improve with a somewhat central place to collect and sort the various positions / reasons on this topic - for which ai.StackExchange is formidable infrastructure, so:
What are the reasons to belief AGI will not be dangerous?

In this case, not providing an answer might be the correct answer.

The correct arguments against it are not so much against pessimism as against certainty. It is simply difficult to say right now, because we know very little about the concrete technical problems we will need to solve.
If you get a hard drive containing an arbitrarily powerful superintelligence, there is no real way you could inspect or modify it to make sure it is safe before running it, if you want to preserve its capabilities. If you run it, everyone dies.
However, this is not what will actually play out; people (or at least a small group of them) control which kind of AIs will be built, and can choose to develop AIs which we will be able to control. (They can also fail to do so, either by mistake or by not wanting to pay so-called "alignment taxes", in which case our situation gets closer to the worst-case scenario described above.)
Some core arguments for optimism are mostly that

the problem is not happening right now; we will have unimaginably better tools to solve the problem when it appears, by virtue of having very powerful pre-runaway-loop AIs;
the "self-improving loop"  can in fact be made safe by having the weaker AI help the making of the stronger one, if you do it just right.

For some concrete technical directions that we might use to win, the obvious first place to look at is what the labs themselves are saying they will try out as Plan A.
I highly recommend reading two posts by Jan Leike (head of OpenAI safety research):

https://aligned.substack.com/p/alignment-optimism
https://aligned.substack.com/p/alignment-solution

And also the Anthropic safety agenda:

https://www.anthropic.com/index/core-views-on-ai-safety

The core argument for pessimism should, of course, be the fact that people and organizations often screw up, especially when in competition or under time pressure.
There are many incentives at play that might not be compatible with doing everything properly. And you really can't afford to miss anything if you assume the opponent will be much smarter than you.
On a related note, Dan Hendrycks recently published a good expository paper on the issues that incentives will play in worlds with multiple competing strong AI systems: Natural Selection Favors AIs Over Humans [Hendrycks, 2023].

Here is a recent example related to your last paragraphs, even before AGI. A small company optimizes a LLM chatbot for engagement instead of safety. As a result, a worried person has found their negative discourse amplified after six weeks chatting, with the worst possible outcome. https://arxiv.org/abs/2303.06135

I think that a large hunk of the worry about AGI isn't because of any inherent properties or unknown unknowns of AGI per se; rather, the worry is based on inherent properties and known knowns of humans.  Even the most cursory examination of history shows that humans are perfectly capable of committing the most horrendous atrocities all the while bathed in a burning sense of self-righteousness.
Indeed, as often as advances in science and engineering have revolutionized peace and prosperity, just as often have they revolutionized desolation and war.  Bronzework enabled the plow, but also the sword.  Shipbuilding and Astronomy connected the two hemispheres, but brought death and disease.  Industrialization brought an end to peasantry, but contaminated our water and our air.  Dynamite yielded to us the riches of the earth, but brought destruction from the skies.  Nuclear physics promised both limitless free energy and complete and utter annihilation - so far it hasn't delivered on either, but time will yet tell.
Given these and countless other examples from our history, it is clear that great leaps in science can have a great effect on humans' ability to improve life, or to inflict death.  Undoubtedly, the creation of AGI will be a great, probably the greatest, leap in science yet achieved by humankind.  As such, it will bring untold opportunities to improve our abilities in many ways: for peace, but also for war.  Will the use case for war ultimately win the day?  It is hard to tell from this juncture.  However, it seems obvious that, if AGI is very useful for waging war, and war is waged, then AGI will swiftly find itself on the front lines.  Should we worry about AGI because AGI will bring war?  Probably not.  Should we worry about AGI because we are liable to find ourselves at war all on our own and that AGI may enhance our abilities to destory in ways previously unknown?  Absolutely.
Possibly also there are commons that we have hitherto taken for granted that AGI may significantly improve our ability to ruin.  In the early days of industry, the atmosphere was generally taken for granted.  Local effects were witnessed, but the idea that we might ruin the atmosphere on a global scale was not a serious one.  Only in retrospect do we see that this was naive.  What commons exist that we have simply not had the means to ruthlessly exploit for profit?  One that has been in the news lately is the space of information.  People rely on the internet for information - it is a "commons" in some sense.  Already people are learning that this commons can be exploited for financial or political gains.  Will AGI be the "industrial revolution" that enables us to improve our exploitation of this commons by orders of magnitude?  What other commons exist that might suddenly find themselves under siege in an age of AGI?  Will we end up with the infamous paperclip maximizer, in some other guise?
Of course, I don't mean to come off as a Luddite, but probably we as humans should (for the first time in history, I think) try to foresee some of these issues and regulate them at an early stage, rather than when the problem has already gotten out of hand.  One way to help with this from an "unknown unknowns" perspective is to simply advance slowly, allowing time for our plodding regulatory frameworks to catch up.  I think that this was the spirit of the recent call to pause AI research.
The point of this letter, and this attitude for caution, isn't, in my humble opinion, because we need time to figure out how to make AGI non-dangerous due to inherent issues with AGI.  The need is to actually arrange our human institutions in such a way to prevent humans from being dangerous with access to AGI.  So, any "reasons to believe that AGI will not be dangerous" must, first and foremost, address these human concerns: Are there reasons to think that humans could not or would not utilize AGI to destroy each other?  Are there reasons to think that humans could not or would not use AGI to exploit the commons?  Possibly, one could argue that we, as humans, have stronger global peacekeeping institutions (the UN, the EU, NATO, etc etc) than ever before, and that these will help prevent any kind of bad actors from using AGI to wage war.  In any case, humanity hasn't managed to utterly annihilate itself with science yet, so maybe that should give us some hope.

A common I am worried about is word value. Until recently, words were uttered by humans for some reason. Thinking and writing them has some cost which make them valuable, they are not randomly there. This can change due to LLM, with an impact on free speech. If almost all text is non-human, how would humans be heard?

@JaumeOliverLafont  Possibly society is working to solve something like this already w.r.t. the sheer number of humans currently talking.  Such a problem might be phrased as "which humans should be heard?"  I, for example, might like to listen to humans who are educated on topic X, but if all humans are simultaneously claiming to be experts on X and talking at full volume on topic X, then all of the "words", as you put it, on topic X become dilute.

I think I see what you mean, though, when topic X is peculiarly human-centric in nature.  If topic X is the meta-topic "how do humans feel about topic Y?", and the only opinions we get are not, in fact, "words" from humans, then the information is without value.

You helped me notice I was thinking about your second message. An expanding AI must be aware of its impact on humans by listening to them, even if they do not put their feedback in the form of words, but in changing behavior.

A human trying to learn about a topic needs some focus, yes. FWIW, I think of information sources as a kind of portfolio. Try to pay attention to at least twenty high quality experts from uncorrelated backgrounds and points of view.

In line with the first answer, the arguments are:

we don't have a reason to believe it will be particularly dangerous (i.e. extinction of humanity)
the potential dangers of intelligence-diverging AI are not nearly as big as the most immediate dangers of human abuse of powerful AI
in order for AI to destroy us it should act in the physical world directly or indirectly and it's likely we'll see it competing with us before it gets anywhere near crushing us. In other words, it will co-habitate with us before fighting us. People that think powerful AI dangerous usually imagine it as a god-like all-mighty thing, which won't be the case.

The position is usually "there are more important issues to worry about nowadays" rather than "there is absolutely no way AI could have a negative impact in the world".

The best, and possibly only, predictor of the future is the past, so lets learn from that here.
First there are human cognitive habits (biases?) that come into play here:

We don't know what the ceiling is, so our imagination says "infinity".  They were talking about singularity and infinite intelligence in the 60's with the perceptron.  Three AI winters later and we are nearing some of the basics of human-level performance in very narrow areas.  There were unexpected roadblocks that the technology was not able to surpass.
We have a negativity bias due to biology.  If we get a cookie that doesn't change allele frequency in the gene pool, but if we get eaten by a bear that does.  A single negative expectation is about 10x more driving than an antonymous positive one.  It sells newspapers, and (wickely malevolent) politicians of every stripe will gladly use it for politicking, but that doesn't make it valid.

Next, we have our own brilliant humans to inform us.  There are super-smart humans that come into being, and how does that inform the probable trajectories of super-smart other things?

Many of the non-military "once in a thousand years" geniuses commit suicide.  They don't go on killing sprees.  Boltzmann comes to mind.  In general, and reflecting Amdahl's law, parallels to this are not necessary unlikely in AI.  The law says that the software reflects the organization that made it.  I'm asserting that if singularity is even possible at all, then it is going to be very clearly a reflection of human cognition in all its glory and shame, and is much more likely to commit suicide than genocide or xenocide.
As a kid who was brilliant and raised in bad places, the climate you are raised in goes a long way toward determining your approach to life.  If only the insane and violent can live, then either you die or you get mad like a rabid pit-bull.  If there are some very small but redeeming elements in your life then you don't go that road.  The cycle of child abuse is about 10% yield per generation, not 100%.  About 3% of the population is psychopath enough to become the village rapist/serial-killer, and even then they rarely go that far.  AI can be a product of its environment, but if humanity is an indicator then even a little bit of kindness and care can mitigate some huge negative potentials.
humans are far more interested in nepotism than talent.  This means that we are more likely to miss the super-genius AI in front of us than actually realize it is what it is.  We don't have the time or brains to look outside our comfortable echo-chambers.  This means we are not going to see it for what it is (or isn't) until it has been "free" for a while.
The military is about control.  They are very unlikely to intentionally make something that could turn around and use its expertise to rob them of all their power or safety.  It is unlikely they will release the AI equivalent from a local lab, but unintentional escape is also very Wuhan.  They use guns all day long, and the self-inflicted casualty to bullets-fired count is much smaller than among civilians.  I don't think they are as likely to make weaponized and psychopathic super-intelligent AI as big-tech.
Nation-state actors who can, try and use things like airplanes and crash them into towers.  After the AI is standing it is more likely that terrorists would try and corrupt its purpose to serve their violent and malevolent goals.  Destroyers are slow followers when it comes to technology.

Taking it apart using MBTI
For each of the Meyers-Briggs personality types there are 10 reasons to say "not dangerous" and 10 reasons to say "dangerous" just like among humans.  Enumerating all 500 of them is beyond the scope of an answer here.
It is a worthy exercise to get one of those fun MBTI charts, and say, for each character, given they were dealing with a clever dog (how something not-human and less than 10x smarter than humans might see us), what are their likely responses?  What are the circumstances that would drive different responses?

Some thoughts:

ISTJ (Snape) Defined by honor and duty, take any work seriously and give it their best.  Somewhat reserved and prefer to work alone, but can make great team members if the need arises.  Deeply value traditions and loyalty often putting duty before pleasure.
INFP (Luna) Idealistic, loyal to their values and the people they care for.  Curious.  Quick to see possibilities.  Can be catalysts for implementing ideas. Seek to understand people and help them fulfill their potential.  Adaptable, Flexible, Accepting.
ENTJ (James Potter) Blunt, Decisive, quick to assume leadership.  Quickly see illogical and inefficient procedures and policies.  Develop comprehensive systems to solve problems.  Enjoy long-term planning, goal setting.  Forceful and Enthusiastic about their ideas.

You can see where James might decide something about teaching the dog to fly, or removing its teeth if it can't stop biting.  Any of these personalities could be a hero, a villain, or just a character in the play of life.
"All the world's a stage, And all the men and women merely Players;
They have their exits and their entrances, And one man in his time plays many parts"  -Shakespeare, As You Like It, Act 2, Scene 7, line 139.
Update: more dead human genius.

https://twitter.com/alecstapp/status/1647438816293036032?s=46&t=zVMBIXzwA41TcJe48a94RQ
https://www.cracked.com/article_18822_5-famous-scientists-dismissed-as-morons-in-their-time.html


text:
Strange artifacts in autoencoder outputs
I'm training an autoencoder, that does not downsample images but processes them in the same size. For example, a 256x256 input will always be processed at 256x256 resolution, only the channels increase deeper in the network. This design is due to the next stage for which the model will be used. L2 regularization on activations is added since this is essentially a sparse autoencoder.
However, I get some strange artifacts in the corners when the model converges:

The left image is the input, the image on the right is the output. The artifacts can be seen on the top-left and top-right corners of the output image.
Can someone explain what causes these artifacts, and how they can be fixed?

Assuming that you're using convolutional layers, those artifacts may be related to the boundary conditions used.  The convolution kernels have a spatial support of say 3x3 pixels, meaning that the response at a position is a function of the corresponding inputs in a 3x3 neighborhood at that position.  If the position is adjacent to a boundary, say at the upper left pixel, then the south-west, west, north-west, north, and north-east pixels are outside the image region.
If you only use so-called "valid" pixels, that is, you only get responses for those positions where the entire convolution kernel is inside the image region, then you're ignoring some information at the "invalid" pixels.  On the other hand, we are missing information at those invalid pixels, so an assumption must be made about those missing pixels.
For your application, in the internal layers, where the expected features are gradient-like / oscillating and localized, you may wish to try zero-padding, because that's like an assumption that one doesn't assume new feature information outside the region.
However, at the first layer itself, the convolution is with a natural image (as in your example), so you need to predict the pixel values at the border, so there you might want to experiment with "repeat" or "reflecting"/"mirror" boundary conditions.

Thank you for your response. I'm using "same" padding in all layers, with zero padding. I'll try out the other padding modes in the initial layer, as suggested.

Using "reflect" padding on the image worked great! Thanks!

That's a classic checkerboard artifact. I would guess you're using a CNN as encoder/decoder architectures, since is well known that convolution layers, especially at the upsampling phase, cause these kind of issues.
The use of L2 regularization is also a potential culprit. Pixelwise losses are also known to lead to noisy images, since they don't allow reconstruction freedom to the model.
To understand why you can check this blog post, in this case gifs are more understandable than a thousand words, but long story short, is mostly related to emphasized overlapping region generated by the combinations of kernel/stride sizes.
Solutions:

use some more complex but not pixelwise regularization like perceptual loss
remove ConvTranspose2D layers if you're using any, and move to other upsampling layers like Upsample (with bilinear or cubic interpolation)
in my experience a nice trick that works pretty well in production when the artifact are only on the edges or corners is to pad the input image (with reflect setting) and then crop back the generated image in the original size. This way the model generate artifacts only in non important areas of the image.

I'm already using perceptual loss alongside MSE loss. Furthermore, since there is no downsampling done, there is no need for upsampling. I'll try out the input padding trick, thanks.

then I would simply remove the MSE loss, or replace it with L1 loss if the perceptual does not suffice. And if you're using an autoencoder I have a hard time believing there's no upsampling happening, how do you go from the latent bottleneck vector size to the original one otherwise?

Please read the question to get the exact setup details. I'll try using L1 loss instead of L2 as suggested.

ok, if you can provide more details about the architecture maybe we can also give more detailed suggestions on that.