text:
What pre-processing of the image is needed before feeding it into the convolutional neural network?
I can't figure out what preprocessing of the image is needed before feeding it into the convolutional neural network. For example, I want to recognize circles on a 1000 by 1000 px photo. The learning process of a neural network occurs on 100 by 100 px (https://www.kaggle.com/smeschke/four-shapes/data). I'm having a little difficulty wrapping my head around the situation when the circle in the input image is much larger (or smaller) than 100x100 px. How then the convolution neural network determines that circle if it was learned on a dataset of a different picture's size.
For clarity, I want to submit a 454 by 430 px image to the network input:

Example of the dataset for the learning process (100 by 100 px):

Finally, I want to recognize all the circles on the input image:


text:
Why do hypercube latent spaces perform poorer than Gaussian latent spaces in generative neural networks?
I have a quick question regarding the use of different latent spaces to represent a  distribution. Why is it that a Gaussian is usually used to represent the latent space of the generative model rather than say a hypercube? Is it because a Gaussian has most of its distribution centred around the origin rather than a uniform distribution which uniformly places points in a bounded region?
I've tried modelling different distributions using a generative model with both a Gaussian and Uniform distribution in the latent space and the Uniform is always slightly restrictive when compared with a Gaussian. Is there a mathematical reason behind this?
Thanks in advance!

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why a single trajectory can be used to update the policy network $\theta$ in A3C?
The Deep RL bootcamp on policy gradient techniques gives the update equation for the policy network in A3C as
$\theta_{i+1} = \theta_i + \alpha \times 1/m \sum_{k=1}^m\sum_{t=0}^{H-1}\nabla_{\theta}log\pi_{\theta_i}(u_t^{(k)} | s_t^{(k)})(Q(s_t^{(k)},u_t^{(k)}) - V_{\Phi_i}^\pi(s_t^{(k)}))    $
However in the actual A3C paper, the gradient update is based on a single trajectory and there is no averaging of the gradient over $m$ trajectories as defined in the video ? The simple action-value actor-critic algorithm also does not seem to require an averaging over m trajectory.

I guess the gradient of the expectation of the Utility function, $\nabla_{\theta}J(\theta)$ in policy gradient methods where $\nabla_{\theta}J(\theta) = E_{\tau \sim p(\tau ; \theta)}[r(\tau)\nabla_{\theta}log p(\tau;\theta)]$ can be approximated using a single sample trajectory as shown in a deep reinforcement learning lecture by stanford where $J(\theta) \approx \sum_{t > 0}r(\tau)\nabla_{\theta}log\pi_{\theta}(a_t |s_t)$ and an average over sampled trajectories is not needed to compute the gradient for $\theta$ update in the direction of gradient.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Importance sampling eq. 5 in paper "Residual Energy-based Models for Text Generation"
In the paper "Residual Energy-Based Models for Text Generation" (arXiv), on page 5, they write that equation 5 is an instance of importance sampling.
Equation 5 is:
$$ P(x_t \mid x_{<t}) = P_{LM}(x_t \mid x_{<t}) \, \frac{\mathbb{E}_{x'_{>t} \sim P_{LM}(\cdot \mid x_{\leq t})}[\exp(-E_\theta (x_{<t}, \, x_t, \, x'_{>t}))]}{\mathbb{E}_{x'_{\geq t} \sim P_{LM}(\cdot \mid x_{\leq t-1})}[\exp(-E_\theta (x_{<t}, \, x'_t, \, x'_{>t}))]} \ \ .$$
The goal is to approximate sampling from a distribution from which sampling is intractable $P_\theta(Y \mid X) = P_{LM}(Y \mid X) \, \frac{\exp(-E_\theta (X, Y))}{Z_\theta(X)}$, by sampling from $P_{LM}$, from which sampling is cheaper.
I understand that they are marginalizing over $>t$ in eq. 5, and I understand the basic idea of importance sampling to change $\mathbb{E}_{x \sim p}[f(x)]$ into $\mathbb{E}_{x \sim q}[f(x) \frac{p(x)}{q(x)}]$. However, eq. 5 is not a mean or aggregate, it is a probability.
What is happening? I don't see how eq. 5 fits in the importance sampling scheme (or a self-normalizing importance sampling scheme, link). Thanks in advance!


text:
Calculating accuracy for cross validation
I'm struggling with calculating accuracy
when I do cross-validation for a deep learning model.
I have two candidates for doing this.
1. Train a model with 10 different folds and get the best accuracy of them(so I get 10 best accuracies) and average them.
2. Train a model with 10 different folds and get 10 accuracy learning curves. Now, average these learning curves by calculating the mean of 10 accuracies of each epoch. So now we get one averaged accuracy learning curve and find the highest accuracy from this curve.
Among these two candidates which one is correct??

I guess you could train your model with 10 different folds and in each fold calculate the average accuracy. So you would have 10 values - one corresponding to each fold. And then take the mean of all of them to get the average accuracy of your model.
Your first option doesn't seem great because you take the highest accuracy among folds. If for some reason, the variance between accuracies is high for a fold, this would bias your numbers. Taking mean or maybe median of accuracies might be more reasonable.
Does that help?

But then how should I calculate the average accuracy of each fold?

Unless your model has some dynamic component, maybe like RF, I don't think you would need to average for each fold. Actually, you can just evaluate your model once over the model and take that accuracy as the accuracy for the fold.

In most cases we choose to take the mean of the k accuracies of k-fold cross validation; that is each time take the one that corresponds to the fold and when every fold has been used as validation set, find the mean accuracy of them.


text:
Conversion of strided filter gradient to convolutional form
I'm implementing strided 2D convolution. My formula looks like this:
$$y_{i, j} = \sum_{m=0}^{F_h - 1}\sum_{n=0}^{F_w - 1} x_{s\cdot i + m, s\cdot j + n}\,f_{m, n}, \tag{1}$$ where $s$ is the stride
(some sources might refer to this as 'cross-correlation' but 'convolution' is consistent with PyTorch's definition)
I have calculated the gradient with respect to the filter as:
$$\frac{\partial E}{\partial f_{m', n'}} = \sum_{i=0}^{(x_h - F_h) / s}\sum_{j=0}^{(x_w - F_w) / s} x_{s\cdot i + m', s\cdot j + n'} \frac{\partial E}{\partial y_{i, j}} \tag{2}$$
and some simple dummy index relabeling leads to:
$$\frac{\partial E}{\partial f_{i, j}} = \sum_{m=0}^{(x_h - F_h) / s}\sum_{n=0}^{(x_w - F_w) / s} x_{s\cdot m + i, s\cdot n + j} \frac{\partial E}{\partial y_{m, n}} \tag{3}$$
Equation $(3)$ looks similar to the first, but not exactly (the $s$ is on the wrong term!). My objective is to convert the second equation into 'convolutional form' so that I can calculate it using my existing, efficient convolution algorithm.
Could someone please help me work this out, or point out any errors that I have made?

I did in fact find an answer to this and will write an answer soon

Were you able to figure this out?

@Iain check out this file in my repository: https://github.com/nkarve/twistml/blob/main/src/trainable.py


text:
DQN not showing the agent is learning in a snake grid environment game
I've been trying to train a snake for the snake game in DQN. Which the snake can essentially just move up, down, left and right. I'm having a hard time getting the snake to stay alive longer. So my question is, what are some techniques that I can implement to get the snake to stay alive for longer?
Some of the things that I've attempted but doesn't seem to have done much after about 1000 episodes are:

Implementing the L2 regularization
Reduce the exploration decay rate so it give the snake more chance to explore
Randomize the starting point for the snake for each episode to try to reduce "local exploration"
I've tweeked some hyper parameters such as learning rate, policy/target network update rate

The input neurons are fed with the state of the board. For example, if my board size is 12*12 then there are 144 input neurons each representing the space of the environment. I've checked that the loss decreases fairly quickly but no improvements on snake lasting longer in the game.
As a side note my reward function is simply a +1 for every time step that the snake survives.
I'm out of ideas of what I can do to get the snake to learn, maybe 1000 episodes is simply not enough? Or maybe my input is not providing good enough information to train the snake?


text:
Why aren’t heuristics for Connect Four Monte Carlo tree search improving the agent?
I’ve created an agent using MCTS to play Connect Four. It wins against humans pretty well, but I’d like to improve upon it. I decided to add domain knowledge to the MCTS rollout stage. My evaluation function checks how “good” an action is and returns the best/highest value action to the rollout policy as the action to use.
I created a “gym” application for one agent, who’s not using the evaluation function, to play against an agent who is using the evaluation function.
I would have expected the agent using the heuristics to perform better than the agent who isn’t, but the inclusion of the heuristics doesn’t seem to make any difference! Any ideas why this might be the case?

It might be the case that if you perform a large number of random rollouts, the "best action" as chosen by the agent without the domain knowledge, is same as the agent with the domain knowledge. I guess what you can do is try to reduce the number of rollouts and see if the performance changes.


text:
Should we start with a small batch-size and increase during training to improve sample efficiency?
Just made an interesting observation playing around with the stable-baseline's implementation of PPO and the BipedalWalker environment from OpenAI's Gym. But I believe this should be a general property of deep learning.
Using a small batch size of 512 samples the walker achieves a near-optimal behavior after just 0.5 Mio steps. The optimized hyperparameters in the RL Zoo suggest using a batch size of 32k steps. This definitely leads to better performance after 5 Mio steps but takes 2 Mio steps until it reaches a near-optimal behavior.
Therefore the question:
Shouldn't we schedule the batch-size to improve sample efficiency?
I believe it makes sense because after initialization the policy is far away from the optimal one and therefore should update quickly to get better. Even when the gradient estimates using small batches are very noisy, they still seem to bring the policy quickly in a quite good state. Thereafter we can increase the batch-size and make less but more precise gradient steps. Or am I missing an important point here?


text:
What should the dimension of the input be for text summarization?
I am trying to build a model for extractive text summarization using keras sequential layers. I am having a hard time trying to understand how to input my x data. Should it be an array of documents with each document containing an array of sentences? or should I further break it down to each sentence containing an array of words?
The y input is basically a binary classification of each sentence to check whether or not they belong to the summary of the document.
The first layer is an embedding layer and I'm using 100d Glove word embedding.
P.s: I am new to machine learning.

Briefly:
I think what you are looking for is an RNN network (Either LSTM or GRU) with the many-to-many topology.
Explanation:
Clearly your input is the sentences (or to be more precise, the an embedding of your sentences, because you cannot feed the raw text to the network). then for each sentence you want to assign a value, which means for n inputs, you need n outputs. This is the many-to-many architecture.
Moreover, you might want to check the Bi-directional LSTM for your study. Not relevant to your question, but worth mentioning.
For more information, refer to this


text:
Benchmarking SAC on Pybullet
So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?


text:
Super-Resolution with Convolutional Neuronal Networks, why interpolation at the beginning?
I have read several papers about super-resolution with CNNs, where a low-resolution image is reconstructed to a high-resolution image.
What I don't understand is, why it is necessary to interpolate the low-resolution image at the beginning to a size that matches the high-resolution image target.
What is the idea about that? If I have an image to image transformation, what are the benefits in a Neuronal network to have the same input size as the same output size?


text:
Where does reinforcement learning actually show up in Deepmind's game engines?
From the brief research I've done on the topic, it appears that the way Deepmind's Alphazero or Muzero makes decisions is through Monte Carlo tree searches, where in the randomized simulations allows for a more rapid way to make calculations than traditional alpha-beta pruning. As the simulation space increases, this search approaches that of a classical tree search.
Where exactly did Deepmind use neural networks? Was it in the evaluation portion? And if so, how did they make determinations on what makes a "good" or "bad" game state? If they deferred the evaluations of another chess engine like Stockfish, how do we see AlphaZero absolutely demolish Stockfish in head-to-head matches?


text:
How to add a pretrained model to my layers to get embeddings?
I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the next layer. How do I approach this?
The inputs would be an array of documents and each document containing an array of sentences.
The input to the model itself is a list of sentences where it will return a list of embeddings.
This is what I've tried but couldn't solve the errors:
def get_embeddings(input_data):

    input_embed = []
    for doc in input_data:
      doc = tf.unstack(doc)
      doc_arr = asarray(doc)
      doc = [el.decode('UTF-8') for el in doc_arr]
      doc = list(doc)
      assert(type(doc)== list)

      new_doc = []
      for sent in doc:
        sent = tf.unstack(sent)
        new_doc.append(str(sent))
        assert(type(sent)== str)

      embedding= model.encode(new_doc)  # Accepts lists of strings to return BERT sentence embeddings
      input_embed.append(np.array(embedding))

    return tf.convert_to_tensor(input_embed, dtype=float)

sentences = tf.keras.layers.Input(shape=(3,5)) #test shape
sent_embed = tf.keras.layers.Lambda(get_embeddings)

x = sent_embed(sentences)

I think you should use Keras embedding layer. It will be too easier than what you are doing.
Steps

Create Embedding Matrix
add matrix to embedding layer while building model.

You will find detailed article
https://www.cs.uaf.edu/2011/spring/cs641/lecture/04_05_modeling.html


text:
Applications of polar decomposition in Machine Learning
Assume there exists a new and very efficient algorithm for calculating the polar decomposition of a matrix $A=UP$, where $U$ is a unitary matrix and $P$ is a positive-semidefinite Hermitian matrix. Would there be any interesting applications in Machine Learning? Maybe topic modeling? Or page ranking? I am interested in references to articles and books.

Hi and welcome to AI SE! Maybe have a look at geometric deep learning. There are some models there that make use of adjacency matrices and other matrices.

Thank you @nbro. Do you have any references I could check out?

Off the top of my head, I don't have a specific reference on GDL that talks about adjacency matrices (because it's been a while since I had to deal with GDL), but I guess that many papers on graph neural networks will talk about it. Maybe ask another question on the site about it. And I (or someone else) will try to provide a more formal answer.


text:
Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?
I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/.
It seems to me that the number of LSTMs in the encoder would have to be the same as number of words in the text (if word embeddings are being used) or characters in the text (if char embeddings are being used). For char embeddings, each embedding would correspond to 1 LSTM in 1 direction and 1 encoder hidden state.

Is this understanding correct?
E.g. If we have another model that uses encoder-decoder for a different application (say text-to-speech synthesis described here https://ai.googleblog.com/2017/12/tacotron-2-generating-human-like-speech.html) tha uses 256 LSTMs in each direction of the bidirectional-encoder, does it mean the input to this encoder is limited to 256 characters of text?
Can the decoder output has to be same length as the encoder input or can it be different? If different what factor describes what the decoder output length should be?


text:
Transformer encoding for regression
I have a string of characters encoding a molecule. I want to regress some properties of those molecules. I tried using an LSTM that encodes all one hot encdoed characters, and then I take the last hidden state fed into a linear layer to regress the property. This works fine, but I wanted to see if transformers can do better, since they are so good in NLP.
However, I am not quiet sure about two things:

Pytorch transformer encoder layer has two masking parameters: "src_mask" and "src_key_padding_mask". The model needs the whole string to do the regression, so I dont think I need "src_mask", but I do padding with 0 for parallel processing, is that what "src_key_padding_mask" is for?
What output from the transformer do I feed into the linear regression layer? For the LSTM I took the last hidden output. For the transformer, since everything is processed in parallel, I feel like I should rather use the sum of all, but it doesn't work well. Instead using only the last state works better, which seems arbitrary to me. Any ideas on how to properly do this, how do sentiment analysis model do it?


text:
My LSTM text classification model seems not learn anything in early epochs
I am trying to use LSTM to do text classification and monitor the training process with tensorboard. But it seems that this model doesn't learn anything in early epochs. Is it normal for LSTM networks?
Here is the definition of model:
class RNN(nn.Module):
    """
        RNN model for text classification
    """
    def __init__(self, vocab_size, num_class, emb_dim, emb_droprate, rnn_cell_hidden, rnn_cell_type, birnn, num_layers, rnn_droprate, sequence_len):
        super().__init__()
        self.vocab_size = vocab_size                # vocab size
        self.emb_dim = emb_dim                      # embedding dimension
        self.emb_droprate = emb_droprate            # embedding droprate
        self.num_class = num_class                  # classes
        self.rnn_cell_hidden = rnn_cell_hidden      # hidden layer size
        self.rnn_cell_type = rnn_cell_type          # rnn cell type
        self.birnn = birnn                          # wheather use bidirectional rnn
        self.num_layers = num_layers                # number of rnn layers
        self.rnn_droprate = rnn_droprate            # rnn dropout rate before fc
        self.sequence_len = sequence_len            # fix sequence length, so we dont need loop
        pass

    def build(self):
        self.embedding = nn.Embedding(self.vocab_size, self.emb_dim)
        self.emb_dropout = nn.Dropout(self.emb_droprate)
        if self.rnn_cell_type == "LSTM":
            self.rnn = nn.LSTM(input_size=self.emb_dim, hidden_size=self.rnn_cell_hidden, num_layers=self.num_layers, bidirectional=self.birnn, batch_first=True)
        elif self.rnn_cell_type == "GRU":
            self.rnn = nn.GRU(input_size=self.emb_dim, hidden_size=self.rnn_cell_hidden, num_layers=self.num_layers, bidirectional=self.birnn, batch_first=True)
        else:
            self.rnn = None
            print("unsupported rnn cell type, valid is [LSTM, GRU]")
        if self.birnn:
            self.fc = nn.Linear(2 * self.rnn_cell_hidden, self.num_class)
        else:
            self.fc = nn.Linear(self.rnn_cell_hidden, self.num_class)

        self.rnn_dropout = nn.Dropout(self.rnn_droprate)

    def forward(self, input_):
        batch_size = input_.shape[0]

        x = self.embedding(input_)
        x = self.emb_dropout(x)

        if self.rnn_cell_type == "LSTM":
            if self.birnn:
                h_0 = torch.zeros(self.num_layers * 2, batch_size, self.rnn_cell_hidden, requires_grad=True).to(device)
                c_0 = torch.zeros(self.num_layers * 2, batch_size, self.rnn_cell_hidden, requires_grad=True).to(device)
            else:
                h_0 = torch.zeros(self.num_layers, batch_size, self.rnn_cell_hidden, requires_grad=True).to(device)
                c_0 = torch.zeros(self.num_layers, batch_size, self.rnn_cell_hidden, requires_grad=True).to(device)
            output, (h_n, c_n) = self.rnn(x, (h_0, c_0))
        elif self.rnn_cell_type == "GRU":
            if self.birnn:
                h_0 = torch.zeros(self.num_layers * 2, batch_size, self.rnn_cell_hidden, requires_grad=True).to(device)
            else:
                h_0 = torch.zeros(self.num_layers, batch_size, self.rnn_cell_hidden, requires_grad=True).to(device)
            output, h_n = self.rnn(x, h_0)

        if self.birnn:
            x = h_n.view(self.num_layers, 2, batch_size, self.rnn_cell_hidden)
            x = torch.cat((x[-1, 0, : , : ], x[-1, 1, : , : ]), dim = 1)
        else:
            x = h_n.view(self.num_layers, 1, batch_size, self.rnn_cell_hidden)
            x = x[-1, 0, : , : ]

        x = x.view(batch_size, 1, -1)           # shape: [batch_size, 1, 2 or 1 * rnn_cell_hidden]
        x = self.rnn_dropout(x)

        x = self.fc(x)
        x = x.view(-1, self.num_class)          # shape: [batch_size, num_class]

        return x

Parameters of this model:

vocab size: 4805
number of classes: 27
embedding dimension: 300
embedding dropoutrate: 0.5
rnn cell type: LSTM
rnn cell hidden size: 1000
bidirectional rnn: False
number of lstm layers: 1
dropout rate at last lstm layer hidden: 0.5
padded sequence length: 64

The Optim:
criterion = nn.CrossEntropyLoss().to(device)
optimizer = optim.Adam(model.parameters(), lr=args.lr, weight_decay=1e-6)

learning rate here is 0.001, batch size is 32.
The tensorboard graph:

It seems that this model starts learning after epoch 15. Is it normal?


text:
Is there a difference between using 1d conv layers and 2d conv layers with kernel with size of 1 along other than time dimension?
Let's assume I use convolutional networks for time-series prediction. Data I feed to the network have 1 channel depth, height of number of periods and number of features is the width, so the frame size is:
[1, periods, features]. Batch size is not relevant here.
Is there a difference between using 1d convolutions along time (height) dimension and 2d convolutional that will have a kernel size of for example (3, 1) or (5, 1), so that the larger number convolutes along the time dimension, and there is no convolution along features dimension?


text:
Training a model for text document transformation?
I have a bunch of text documents, split into source documents and transformed documents. These text documents have multiple lines and are edited at specific locations, in a specific way.
I make use of the difflib package available in Python to identify the associated transformation, for each source document and the resulting transformed document.
I wish to train and implement a ML technique which will help in identifying and automating this conversion activity.

Here is a sample result of how the transformation result looks like:
(NOTE: This example contains only one line, but my actual use case contains several lines)
import difflib

Initial = 'This is my initial state'
Final = 'This is what I transform into'

diff = difflib.SequenceMatcher(None, Initial, Final)

for tag,i1,i2,j1,j2 in diff.get_opcodes():
    print('{:7} Initial[{:}:{:}] --> Final[{:}:{:}] {:} --> {:}'.format(tag,i1,i2,j1,j2,Initial[i1:i2],Final[j1:j2]))

#Result:
equal   Initial[0:8] --> Final[0:8] This is  --> This is
insert  Initial[8:8] --> Final[8:23]  --> what I transfor
equal   Initial[8:9] --> Final[23:24] m --> m
delete  Initial[9:10] --> Final[24:24] y -->
equal   Initial[10:13] --> Final[24:27]  in -->  in
delete  Initial[13:14] --> Final[27:27] i -->
equal   Initial[14:15] --> Final[27:28] t --> t
replace Initial[15:24] --> Final[28:29] ial state --> o

This helps in outlining the transformation steps to transform Initial to Final. I wish to make use of ML to identify the common pattern in such transformation between a large collection of txt documents and train a model that I can use in future.

What will be the best method to approach this problem? I am not facing a problem in identifying and classifying text data, but in identifying the nature of editing and transformation of strings.


text:
Policy Gradient on Tic-Tac-Toe not working
I wanted to implement the Policy Gradient on Tic-Tac-Toe.
I tried to use the code that worked for any environment like CartPole-v0 to my Tic-Tac-To game. But it is not learning. There are no errors. Just the result is so bad.
RandomPlayer ("Player X") vs PolicyAgent ("Player O")

So one can see that the Policy Agent is not learning after 500 battles. Each battle consists of 100games against the random player. Together 500 * 100 games.
Can someone tell me the problem or the bug in my code. I can not figure it out. Or what I have to improve. It would be so great.
Here is also a project which did the same, which I want to do, but with success.
https://github.com/fcarsten/tic-tac-toe/blob/master/tic_tac_toe/DirectPolicyAgent.py
I did not get what I am making different.
Code:
Packages:
import torch
import torch as T
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

import numpy as np
import gym
from gym import wrappers

Neural Net:
class PolicyNetwork(nn.Module):
    def __init__(self, lr, input_dims, fc1_dims, fc2_dims, n_actions):
        super(PolicyNetwork, self).__init__()
        self.input_dims = input_dims
        self.lr = lr
        self.fc1_dims = fc1_dims
        self.fc2_dims = fc2_dims
        self.n_actions = n_actions

        self.fc1 = nn.Linear(self.input_dims, self.fc1_dims)
        self.fc2 = nn.Linear(self.fc1_dims, self.fc2_dims)
        self.fc3 = nn.Linear(self.fc2_dims, self.n_actions)

        self.optimizer = optim.Adam(self.parameters(), lr=lr)

    def forward(self, observation):
        state = T.Tensor(observation)
        x = F.relu(self.fc1(state))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Policy Agent:
class PolicyAgent:
    def __init__(self, player_name):
        self.name = player_name
        self.value = PLAYER[self.name]

    def board_to_input(self, board):
        input_ = np.array([0] * 27)
        for i, val in enumerate(board):
            if val == self.value:
                input_[i] = 1
            if val == self.value * -1:
                input_[i+9] = 1
            if val == 0:
                input_[i+18] = 1
        return np.reshape(input_, (1,-1))

    def start(self, learning_rate=0.001, gamma=0.1):
        self.lr = learning_rate
        self.gamma = gamma
        self.all_moves = list(range(0,9))
        self.policy = PolicyNetwork(self.lr, 27, 243, 91, 9)
        self.reward_memory = []
        self.action_memory = []

    def turn(self, board, availableMoves):
        state = self.board_to_input(board.copy())
        prob = F.softmax(self.policy.forward(state))
        action_probs = torch.distributions.categorical.Categorical(prob)
        action = action_probs.sample()

        while action.item() not in availableMoves:
            state = self.board_to_input(board.copy())
            prob = F.softmax(self.policy.forward(state))
            action_probs = torch.distributions.categorical.Categorical(prob)
            action = action_probs.sample()

        log_probs = action_probs.log_prob(action)
        self.action_memory.append(log_probs)

        self.reward_memory.append(0)
        return action.item()

    def learn(self, result):
        if result == 0:
            reward = 0.5
        elif result == self.value:
            reward = 1.0
        else:
            reward = 0

        self.reward_memory.append(reward)
        #print(self.reward_memory)

        self.policy.optimizer.zero_grad()
        #G = np.zeros_like(self.action_memory, dtype=np.float64)
        G = np.zeros_like(self.reward_memory, dtype=np.float64)

        #running_add = reward
        #for t in reversed(range(0, len(self.action_memory))):
        #    G[t] = running_add
        #    running_add = running_add * self.gamma

        #'''
        running_add = 0
        for t in reversed(range(0, len(self.reward_memory))):
            if self.reward_memory[t] != 0:
                running_add = 0
            running_add = running_add * self.gamma + self.reward_memory[t]
            G[t] = running_add
        for t in range(len(self.reward_memory)):
            G_sum = 0
            discount = 1
            for k in range(t, len(self.reward_memory)):
                G_sum += self.reward_memory[k] * discount
                discount *= self.gamma
            G[t] = G_sum
        mean = np.mean(G)
        std = np.std(G) if np.std(G) > 0 else 1
        G = (G-mean)/std
        #'''

        G = T.tensor(G, dtype=T.float)

        loss = 0
        for g, logprob in zip(G, self.action_memory):
            loss += -g * logprob

        loss.backward()
        self.policy.optimizer.step()

        self.reward_memory = []
        self.action_memory = []

I am of the conviction that policy gradient (in its naive form at least) does not converge for Tic Tac Toe.  Looking for confirmation from others....

Some suggestions:

You have a loop in which illegal moves by the RL agent are ignored.  In other words, when the agent makes illegal moves, it is not punished, nor is there any +/- rewards for it whatsoever.  In my program I treat illegal moves the same as losing the game.

Try to play a few "pre-moves" to make the game easier.  For example I start with this (now is for Player 'X' to move):

Then you may see convergence sooner.  Always a good idea to start from a dead-easy case.

Notice that the optimal value is not the "perfect score".  If you take win=2, lose=-2, draw=1, then the optimal score for the above game for Player 'X' is 1.5, not 2.0.  (You can check the math).
I wrote a small Python program to calculate the optimal score for an RL agent playing against a random opponent (both players assumed to never make illegal moves).  See this answer.

For me it took much more than 50K games to see convergence.  It's more like 1-2M games.  (Correction:  With more trails, I found that policy gradient seems unable to converge to the optimal value, but it did improve over time to reach a sub-optimal score.)

Your neural network may be a bit too small.

Remark: This is a good question and also very significant to artificial intelligence, as Tic Tac Toe is a game suitable for logic-based agents, and it is important to see how deep learning performs in such "logical" domains.  My own research is focused on combining logic structure with deep learning.
My code is here：
https://github.com/Cybernetic1/policy-gradient

Really great response!


text:
Banding artifacts in CNN
I was working on a CNN for HDR image generation from LDR images. I used an encoder-decoder architecture and merged the input with the decoder output. However I'm getting some banding artifacts in the model prediction as shown.
1)Input LDR image

2) Ground Truth HDR

3) Predicted output

Notice the fine bands in the prediction. What might be causing these bands? Also I trained only for 20 epochs yet. Is the problem due to inadequate training?
Here's my model:
 class TestNet2(nn.Module):
    def __init__(self):
        super(TestNet2, self).__init__()

    def enclayer(nIn, nOut, k, s, p, d=1):
        return nn.Sequential(
            nn.Conv2d(nIn, nOut, k, s, p, d), nn.SELU(inplace=True)
        )
    def declayer(nIn, nOut, k, s, p):
        return nn.Sequential(
            nn.ConvTranspose2d(nIn, nOut, k, s, p), nn.SELU(inplace=True)
        )

    self.encoder = nn.Sequential(
        enclayer(3,64,5,3,1),
        #nn.MaxPool2d(2, stride=2),
        enclayer(64,128,5,3,1),
        #nn.MaxPool2d(2, stride=2),
        enclayer(128,256,5,3,1),
        #nn.MaxPool2d(2, stride=2),
    )
    self.decoder = nn.Sequential(
        declayer(256,128,5,3,1),
        declayer(128,64,5,3,1),
        declayer(64,3,5,3,1),

    )

  def forward(self, x):
      xc=x
      x = self.encoder(x)
      x = self.decoder(x)
      x = F.interpolate(
        x, (512, 512), mode='bilinear', align_corners=False
      )
      x=x+xc
      return x


text:
How to make a multivariate forecasting if one of features becomes known for the future with some confidence level, e.g. weather forecast data
Let's assume that we make forecasting of another metric partially based on forecasts of the weather forecast, e.g. of temperature, pressure, then we can potentially obtain those forecasts from one of public APIs and have some information about future values of these features and do more precise prediction of another parameter taken as a label.
What is the approach to use in this case if one or more of the features in multivariate forecasting have some forecasted values for the predicted horizon? It looks that in this case not only values from historical data can be used but also predicted values, though not clear how to organize the model in this case, e.g. it can be multivariate LSTM.


text:
Low accuracy during training for text summarization
I am trying to implement an extractive text summarization model. I am using keras and tensorflow. I have used bert sentence embeddings and the embeddings are fed into an LSTM layer and then to a Dense layer with sigmoid activation function. I have used adam optimizer and binary crossentropy as the loss function. The input to the model are the sentence embeddings.
The training y labels is a 2d-array i.e [array_of_documents[array_of_biniary_labels_foreach_sentence]]
The problem is that during training, I am getting the training accuracy of around 0.22 and loss 0.6.
How can I improve my accuracy for the model?


text:
How could a NN be trained to output a cyclic (e.g. hue) number?
I was thinking about training a neural network to colourize images. The input would be the luminosity/value for each pixel, and the output would be a hue and/or saturation. Training data would be easily obtained just by selecting the luminosity/value channel from a full colour image.
Suppose all channels are scaled to 0.0-1.0, there is a problem with pixels whose hue is nearly 0.0 or nearly 1.0.

The input data may have sharp discontinuities in hue which are not visible to the human eye. This is an unstable, illusory boundary which seems like it would destablilize the training.
Also if the network outputs a value of 1.001 instead of 0.001 then this should NOT be penalised since.

Possible workarounds might be to preprocess the image to remap e.g. 0.99 to -0.01 if that pixel is near a region dominated by near-0 hues, or similarly to to remap e.g. 0.01 to 1.01 if that pixel is near a region dominated by near-1 hues. This has its own problems. Similarly, outputs could be wrapped to the range 0-1 before being scored.
But is there a better way to encode cyclic values such as hue so that they will naturally be continuous?
One solution I thought of would be to treat (hue,saturation) as a (theta,r) polar coordinate and translate this to Cartesian (x,y) and have that be the training target, but I don't know how this change of colour space will affect things (it might be find, I haven't tried it yet).
Are there alternative colour representations which are better suited to machine learning?

Your solution is pretty much on spot. It corresponds to the YUV scheme used in television and designed to match human perception characteristics. As you already noticed, such an encoding wouldn't suffer from discontinuities.

And its cousin YCbCr. Which apparently in some contexts means the same thing.


text:
Simplifying Log Loss
I am reading through a paper (https://www.mitpressjournals.org/doi/pdf/10.1162/0891201053630273) where they describe logloss as a ranking function and can be simplified to the margin of the training data $X$. I am not sure what the transformation is in each step and could use a bit of help.
A precursor to this ranking loss is standard logloss which may clarify my understanding as well:
In this loss I only get from step 2 to here:
$$-\sum_{i=1}^n log(\frac{{e^{y_iF(x_i,\overline{a})}}}{1 + e^{yF(x,\overline{a})}})$$
$$=-\sum_{i=1}^ny_iF(x_i, \overline{a}) - log(1 + e^{yF(x,\overline{a})})$$

And here is the full ranking loss I am having trouble on:


text:
Training Conditional DCGAN with GAN-CLS loss
I am trying to implement conditional GAN using GAN-CLS loss as described in paper: https://arxiv.org/abs/1605.05396
So, while training discriminator, I should I have three batches of data:

[Real_Image, Embeddings]
[Generated_Image, Embeddings]
[Wrong_Image,Embeddings]

And, while training generator, I should have one batch of data i.e [Generated_Image, Embeddings].
Is this correct way to train the model?


text:
Is there a classification task with multiple attribute regression?
I'm trying to look for a task that predicts a discrete label first (classification), and then predicts the multiple continuous attributes of the predicted class. I found some papers about multi-output regression, but it wasn't what I wanted. Perhaps such a task exists in robot control or in video games, but I haven't found it yet. At the same time I want to train within a supervised learning framework, not reinforcement learning.

Hi and welcome to AI SE! Can you explain a little bit why you first need to predict a class and then predict attributees of that class? Maybe the context can also be helpful.


text:
Understanding the role of the target network in this DQN algorithm
I've found online this interesting algorithm:

From what I understand reading this algorithm, I can't figure out why I should "perform the opposite action" and consequently storing that second experience as then it is never used to update or doing anything else. Is this algorithm incorrect?

It's used, they store both observations in $M$ and then they sample from it, just because they use notation $e_i = (o_i,a_i,r_i,o_{i+1})$ it doesn't mean they sample only observations from $\mathcal E$.

@Brale just another question on this algorithm: when computing $y_i$ they use the argmax operator but w.r.t the main DNN. I've seen other pseudo algorithm which include the target network that compute the action w.r.t the target netowrk. What is the difference?

This is some variation of double Q learning, look up Deep Double Q Learning


text:
Can you find another reason for sample inefficiency of model-free on-policy Deep Reinforcement Learning?
The following mindmap gives an overview of multiple reasons for sample inefficiency. The list is definitely not complete. Can you see another reason not mentioned so far?

Some related links:

interactive mindmap
Reddit post asking the same question
related SE question with only a few reasons all mentioned here


text:
Incorporating domain knowledge into recurrent network
I am currently trying to solve a classification task with a recurrent artificial neural network (RNN).
Situation
There are up to 350 inputs (X) mapped on one categorical output (y)(13 differnt classes).
The sequence to predict is deterministic in the sense that only specific state transitions are allowed based on the past. A simplified Abstraction of my problem:

y - Ground Truth: 01020
y - Model Prediction: 01200
Valid Transitions: 01, 10, 02, 20

The predicted transition 12 is consequently not valid.
Question
What would be the best way to optimize a model to make as less invalid transition predictions as possible (ideally none)?
(temporal shifts of the predictions in comparison to the ground truth are still acceptable)

By the integration of the knowledge about the valid transitions into the artificial neural network. Is this even possible to code hard restrictions into a RNN?
By a custom loss function which penalizes these invalid transitions
Another approach

Current Approach
With a bidirectional recurrent network (one gated recurrent units layer (2000 neurons)) in a many to many fashion an accuracy of 99.5% on the training and 97.5% on the test set could be reached (Implemented with TensorFlow / keras 2.2)


text:
Why is this GAN not converging?
This GAN being trained with CelebA dataset doesn't seem to mode collapse, discriminator is not really over confident, and yet the quality is stuck on these rough Picasso-like generator images. Using Leaky-ReLU, strided conv instead of maxpool, and dampened truths helped a little, but still no better than this. Not sure what else to try. training clip Discriminator feed is in top left corner.


text:
Does anyone know of a model for comparing the eyes of people in two images to see if they match?
There’s a lot of talk of undercover cops intentionally starting violence in otherwise peaceful protests. The evidence, primarily, are images like this.
https://images.app.goo.gl/4n3o2EXwFzMQfsKq6
It looks pretty convincing, but I’d like something more solid. Does anyone know of a model that can detect with a high level of certainty if the “mask” area of two photos represents the same person?

The model you are looking for is probably a Siamese Neural Networks.
It's very used for Biometrics. This kind of model learns to extract relevant and features (For example, facial metrics) that are invariant on several conditions (that won't change on light, angle, or even occlusions) presented on the dataset.
But keep in mind no model is an Scy-Fi AI, like FBI movies enhancing the image 1000x. There is natural limitation, so maybe not even the best AI possible could give you a satisfactory answer.


text:
Actor-Critic implementation not learning
I've implemented a vanilla actor-critic and have run into a wall. My model does not seem to be learning the optimal policy. The red graph below shows its performance in cartpole, where the algorithm occasionally does better than random but for the most part lasts between 10-30 timesteps.

I am reasonably sure that the critic part of the algorithm is working. Below is a graph of the delta value (r + Q_w(s',a') - Q_w(s,a)), which seems to show that for the most part the predicted future reward is quite similar to the approximate future reward.

Thus, I am at a loss for what the problem could be. I have an inkling it lies within the actor, but I am not sure. I have double checked the loss function and that seems to be correct to me as well. I would appreciate any advice. Thanks!


text:
How to exclude sections of bad data from time-series data before training an LSTM network
I am using LSTM network for predicting IOT time-series data receiving from un-reliable devices and networks.
This results in several multiple sections [continuous streak of bad data for several days until the problem is fixed].
I need to exclude this bad data section before feeding it to model training.
Since I am using LSTM-RNN network, it requires to do an un-roll data based on the previous records.
How can I properly exclude this bad data?
I thought of an approach as training model separately using each batch of good data, and use subsequent good-data batch for fine-tuning the model.
Please let me know if this is a good approach? or is there a better method?
example data:
"1-01",266.0
"1-02",145.9
"1-03",183.1
"1-04",0  [bad data]
"1-05",0  [bad data]
"1-06",0  [bad data]
"1-07",0  [bad data]
"1-08",224.5
"1-09",192.8
"1-10",122.9
"1-11",0  [bad data]
"1-12",0  [bad data]
"2-01",194.3
"2-02",149.5

Your idea is a good one.
Another idea is to upsample or aggregate your data.  For example, average by week if you generally have a couple of missing days in every week.
A similar question on Stack Exchange:
https://stats.stackexchange.com/questions/374935/how-to-deal-with-really-sparse-time-series-data-for-a-binary-classification-task


text:
What is the most compressed audio that I can feed an AI?
The problem I currently have is that I want to train an AI to produce music, like music that contains voices etc... However, the problem is that with a WAV file, one second of audio can be up to 48,000 inputs, which is extremely detrimental to the ai's learning process and prevents it from really gaining any knowledge about context.
I've tried to do the fast Fourier transformation, but the amount of data coming in varies depending on what part of the song I'm training it on, which will not work since I can't know ahead of time what every single time unit of every single song will have!
And the max number of inputs is still 24,000
Is there any other way of compressing data down in a way that I can give my ai something within the range of a couple hundred inputs for a second or two?


text:
What are the applications of hierarchical softmax?
Apart from its use in word embeddings (e.g word2vec algorithm), are there any other applications of hierarchical softmax? If yes, can you please give me some reference papers?


text:
How to make a LSTM network to predict sequence only after input sequence is finished?
I am learning to use a LSTM model to predict time series data. Specifically, I hope the network should output a sequence (with multiple time steps) only after the input sequence has finished feeding in, as shown in the left figure.

However, most of the LSTM sequence-to-sequence prediction tutorial I have read seems to be the right figure (i.e. each time step of the output sequence is generated after each time step of the input sequence). What's more, as far as I understand, the LSTM implementation in PyTorch (and probably Keras) can only return output sequence corresponding to each time step of the input sequence. It cannot make predictions after the input sequence is over.
I hope to know is there any way to make a sequence-to-sequence LSTM network which starts output only after the input sequence has finished feeding in? And it would be better if someone can show me an example implementation code.

You should try an architecture with an encoder and a decoder. The encoder will consume all the data you give as in put and decoder will give out the series of output.

These are some links that might be useful with some code, considering an LSTM Encoder-Decoder architecture.

A Simple Introduction to Sequence to Sequence Models

NMT: Encoder and Decoder with Keras

A ten-minute introduction to sequence-to-sequence learning in Keras

How to Develop an Encoder-Decoder Model for Sequence-to-Sequence Prediction in Keras

How to build an encoder decoder translation model using LSTM with Python and Keras


text:
Video recognition (specifically video, not individual frames)
There are libraries for recognizing individual video frames, but I need to recognize an object in motion. I can recognize a person in every single frame, but I need to know if the person is running or waving. I can recognize a tree in every single frame, but I need to find out if the tree is swaying in the wind. I can recognize a wind turbine in every frame, but I need to know if it's spinning right now.
So the question is: do technologies, libraries, concepts, or algorithms exist for recognizing objects over a certain period of time? For example, I have a series of several pieces of frames, each of which exactly has a person, and need to find out if person are walking or waving their hands.


text:
Train a model using a multi-column text-filled excel sheet
I have an excel sheet filled with my own personal appreciations of movies I've watched, and I want to use it to train an AI model so that it can predict if I'll like a specific movie or not, based on the ones I've already seen.
My data is formatted as following (just a sample, the spreadsheet is filled with hundreds of movies):

And I would like to use all the columns to train my model. Because I am going to say if I liked the movie or not, I know it will be Supervised Learning. I already cleaned the data so there's no blank or missing data, but I do not know how to train my model using every column.
If required, I can be more specific on something, just ask and I'll edit the post.

Create an embedding of 'description' field, using word embeddings or perhaps contextualized embeddings from a language model. The 'director' field you could either one-hot encode, or create embeddings, since there might be similarities between directors in a vector space, those you could train from scratch using word2vec. The rating field you could use as it is by converting it to an integer. Then you could concatenate all the vectors and train a model, or train 3 different models and ensemble them :) The 'Movie Name' field does prob not contain any signal. Good luck :)

You will need to convert that to something that a neural network can understand.

Movie Name
Is useless. At least you want to judge a movie by its name.

Description
You will need to perform a Tokenization. Grab the x's more common words and convert them on an array. I recommend you to see these videos from TensorFlow. There you can find the Google Colab Links for the job.

Director
There are two options. A One Hot Array or an integer for every director. One Hot Array may be better if you don't have an order of similarity of the directors. But will increase the size of the inputs.
If you have an order of similarity of the directors an integer for each director will work fine.

Rating
Nothing to do here. Is ready to go.

You can perform this work directly on excel, but Tensorflow has great tools for it. It's hard to run the model after you have trained it on excel. If you are to comfortable on excel or you can install new software on your computer, I've made a backpropagation algorithm that runs on excel and give you a formula to paste in a module.


text:
In a DQN, can Prioritized Experience Replay actually perform worse than a regular Experience Replay?
I've written a Double DQN-based stock trading bot using mainly time series stock data.
I've recently upgraded my Experience Replay(ER) code with a version of Prioritized Experience Replay (PER) similar to the one written by OpenAI. My DQN's reward function is the stock return over 30 days (the length of my test window).
The strange thing is, once the bot has been trained using the same set of time series data and let free to trade on unseen stock data, the version that uses PER actually comes up with worse stock returns than the version using a regular ER.
This is not quite what I'd expected but it's very hard to debug and see what might have gone wrong.
So my question is, will PER always perform better than a regular ER? If not, when/why not?


text:
How to predict Q-values based on stack of frames
I decided to train deep Q-learning agent based on getting raw pixels from environment.I have one particular problem:when I input stack of frames, suppose 4 consecutive frames, if action space is 6,then output is 4 by 6 matrix.So which one is real Q-value?I mean, I input batch of frames and it outputs batch of values and question is which is real Q-value out of those batch values?

You do not output the batch of Q-values. Input frame stacking is needed to gain full observability of the environment. In your case the output would be 6 elements for your current frame. If $F$ is a frame then you would stack 4 frames $[F_{k-3}, F_{k-2}, F_{k-1}, F_k]$ and the output would be 6 Q-values for frame $F_k$.

I still don't get the idea, when I input 4 frames It outputs 4 by 6 values, so what are these values?

It outputs 4 by 6 matrix because you defined the network architecture such that it outputs 4 by 6 matrix. Tensoflow, Pytorch or whatever you used cannot read your mind and know how many outputs do you want. You need to fix your network architecture


text:
How to calibrate model's prediction given past images?
I want to predict how open is the mouth given a face image. It's a regression problem (0= mouth not open, 1=mouth completely open). And something between 0 and 1 is also allowed. ConvNet works fine for one person. But when I train with many people with hope that it will generalize to an unseen person, the model suffers from not knowing the limit of a person's mouth.
For example, if a new person uses the model to predict, the model doesn't have a clue whether this person has completely opened the mouth or not. Because it's hard to know how much a person can open the mouth from one image. People's mouth openness capability is not the same. Some guys cannot open their mouth that much, but some guys can open the mouth like they can swallow an apple. The only way you can know how much a person can open the mouth is to look at multiple images of their mouth movements, especially when they open the mouth completely.
I want to know how to make the model know the limit of a person's mouth by using the info from past images.
Is there a way for me to use a few unlabeled images of a new person in order to help the model calibrate its prediction? How do I do it?
This should help the model know the min/max of the person's mouth and also knows the intermediate values between 0 and 1. If you run the model continuously on a webcam, I expect the prediction to be smooth (not noisy).
My idea is to encode many images into an embedding that can be used as a calibration vector. The vector will be fed into the model along with the person's image. But I am not sure how to do it.
Any suggestions/tutorials would be welcomed.


text:
Do you have to add a dense layer onto the final layer of an LSTM?
If my understanding of an LSTM is correct then the output from each LSTM unit is the hidden state from that layer. For the final layer if I wanted to predict e.g. a scalar real number, would I want to add a dense layer with 1 neuron or is it recommended to have a final LSTM layer where the output has just one hidden unit (i.e. the output dimension of the final hidden state is 1)? If we didn't add the dense layer, then the output from the hidden layer I believe would be between (-1,1), if you use the traditional activations in the LSTM unit.
Apologies if I've used wrong terminology, there seems to be some inconsistency with LSTM's when going between literature and definition in TensorFlow etc.

Have you tried to have a look at some reference implementations? Here the author seems to use a dense layer with 1 unit after the last LSTM layer.

Yes, I have used LSTM's in the past and added a dense layer with 1 unit on. This was from a walkthrough I found -- I was more questioning whether you have to do this, although intuitively I think that you do, especially if it is was in a scenario like I had described.
e: I just clicked the link and that was the walkthrough I had followed haha.

Your question is actually interesting. It's been a while I've used LSTMs, so I cannot answer it now. Of course, if the output of the last LSTM layer is a number in the range $[-1, 1]$ and you need a number in $\mathbb{R}$, then you probably need to fix that (maybe by changing the activation function), if you want to try not to use the dense layers.


text:
Corner detection algorithm gives very high value for slanted edges?
I have tried implementing a basic version of shi-tomasi corner detection algorithm. The algorithm works fine for corners but I came across a strange issue that the algorithm also gives high values for slanted(titled) edges.
Here's what i did

Took gray scale image
computer dx, and dy of the image by convolving it with sobel_x and sobel_y
Took a 3 size window and moved it across the image to compute the sum of the elements in the window.
computed sum of the window elements from the dy image and sum of window elements from the dx image and saved it in sum_xx and sum_yy.
created a new image (call it result) where that pixel for which the window sum was computed was replaced with min(sum_xx, sum_yy) as shi-tomasi algorithm requires.

I expected it to give maximum value for corners where dx and dy both are high, but i found it giving high values even for titled edges.
Here are the some outputs of the image i received:

Result:

so far so good, corners have high values.
Another Image:

Result:

Here's where the problem lies. edges have high values which is not expected by the algorithm. I can't fathom how can edges have high values for both x and y gradients (sobel being close approximation of gradient).
I would like to ask your help, if you can help me fix this issue for edges. I am open to any suggestions and ideas .
Here's my code (if it helps):
def shi_tomasi(image, w_size):
    ans = image.copy()
    dy, dx = sy(image), sx(image)

    ofset = int(w_size/2)
    for y in range(ofset, image.shape[0]-ofset):
        for x in range(ofset, image.shape[1]-ofset):

            s_y = y - ofset
            e_y = y + ofset + 1

            s_x = x - ofset
            e_x = x + ofset + 1

            w_Ixx = dx[s_y: e_y, s_x: e_x]
            w_Iyy = dy[s_y: e_y, s_x: e_x]

            sum_xx = w_Ixx.sum()
            sum_yy = w_Iyy.sum()

            ans[y][x] = min(sum_xx, sum_yy)
    return ans

def sy(img):
    t = cv2.Sobel(img,cv2.CV_8U,0,1,ksize=3)
    return t
def sx(img):
    t = cv2.Sobel(img,cv2.CV_8U,1,0,ksize=3)
    return t

if i had to use a custom algorithm i would have used the opencv version of it. I want to implement it on my own to understand each and everything

I can't go into details of the algorithm but here's some intuition about what's apparently going wrong:
The Sobel transformation identifies mostly-vertical and mostly-horizontal edges. For slanted edges, it also shows a response, just a bit weaker.
By using a window and taking the minimum of vertical and horizontal response, you identify points where you have both a vertical and a horizontal response. However, this is also the case at the slanted edges.
Given the structure of this simplistic corner detection algorithm, you can only expect to get meaningful results for strictly horizontal and vertical lines/edges and their intersections/corners. It is just not suitable for slanted edges.

Thanks for your response. Can you suggest a suitable approach for slanted edges ?

I have too little practical experience, but I would try convolution with kernels for more angles than just vertical and horizontal, and match points where one angle and its 90° rotated counterpart are both present. The kernels must be more specific than simple dx and dy, though.


text:
Action recognition using video stream data
Recently, I am working on an action recognition project where my input data is from the video stream. I read some of the concepts like ConvLstm, Convolutional Lstm, etc. I am looking for someone who already those kinds of staff already and can share his work with me that will be a really good help for me?

If you want to build a stream recognition pipeline than you can use OpenCV, Kafka, and Spark together.how to build pipeline of video stream
and for Action recognition using ConvLstm. This link will help you.action recognition using Convlstm


text:
LSTM - MAPE Loss Function gives Better Results when Data is De-Scaled before Loss Calculation
I am building an LSTM for predicting a price chart. MAPE resulted in the best loss function compared to MSE and MAE. MAPE function looks like this
$$\frac{1}{N}\sum^N_{i=0}\frac{|P_i - A_i|}{A_i}$$
Where $P_i$ is the current predicted value and $A_i$ is the corresponding actual value. In neural network, it is always advised to scale the data between a small range close to zeros such as [0, 1]. In this case scaling range of [0.001, 1] is imperative to remove a possible division by zero.
Due to the MAPE denominator, the close the scaling range is to zero the larger the loss function becomes for a given $|P_i - A_i|$. If on the other hand, the data is de-scaled just before it is inserted in the MAPE function, the same $|P_i - A_i|$ would give a smaller MAPE
Consider a hypothetical example with a batch size of 1, $|P_i - A_i| = 2$ (this is scale indipendent) and $A_i = 200$. Therefore scaled $A_i = 0.04$. The MAPE error loss for the scaled version would be $\frac{2}{0.04} = 50$, and for the unscaled version $\frac{2}{200} = 0.01$
This will mean that the derivative w.r.t each weight of the scaled version will also be larger, therefore making the weights themselves even smaller. Is this correct?
I am concluding that scaling the data when using MAPE will effectively shrink the weights down more than necessary. Is that a good reason why I am seeing significantly better performance with de-scaled MAPE calculation?
Note: I am not keeping the same hyperparameters for scaled and de-scaled MAPE but a Bayesian optimization is performed with both runs, In the later a deeper network was preferred but in the scaled MAPE more regularisation was preferred.
Some expertise on this would be helpful.


text:
How to compare SegNet, U-Net and EfficientNet?
SegNet and U-Net are created for segmentation problem and EfficientNet is created for classification problem. I have a task and it is saying that train these models on the same dataset and compare results. Is it possible?

Segmentation is a pixel-wise classification so you can use any classification network with added deconvolution or upsampling layers to upscale to original input resolution.

I see but it seems little bit not make sense because if I add some deconvolutional layers wouldn't the original Efficient model change? If I add deconvolutional layer it would called like "X Segmentation Model based on EfficientNet", right?

Yes it would be but the encoder is still EfficientNet.

Which dataset are they suggesting to train these models on?

There is no specific dataset. I was asked to compare these 3 models on the same dataset which I will choose. That's the why I am confused and it seems to not make sense for me. Brale is right.I could add deconvolutional layer to EfficientNet but I think it would not be EfficientNet anymore. For example, encoder part of SegNet is VGG-16 without fully connected layer. However, after adding some deconvolutional part and some other approaches it is called SegNet instead of VGG-16.


text:
Crossover method for gene value containing a set of values
I have a chromosome where each gene contain s set of values. Like the following:
chromosome = [[A,B,C],[C,B,A],[C,D,],[],[E,F]]

The order in each gene values matters. (A,B,C is different to A,C,B)
Each value should not appear more than once in a gene. ([A,B,B] is not desirable, B is repeated.)

In my current two-point crossover method. The genes values that are crossover is the whole set of values. (E.g the whole of [A,B,C] is crossed to another chromosome)
Soon, I realize my population lacks variations very quickly because the values within a gene always remain the same. Hence, my algorithm is evolving very slowly, and limited by the variation of gene values at initialization stage.
What crossover can I implement to cross values within the set as well?
I am pretty new to genetic algorithm. Any help will be much appreciated. Thank you.

Why do you cross-over the whole gene? Actually, this is typically the case, but, in your case, that may not be a good idea. Anyway, maybe to solve your problem you can increase the mutation rate (and I assume that during the mutation you mutate individual genes too). Have you already tried something like that?

You probably want to tailor your operators for your problem. Given that each "gene" is a sequence where you don't want repetition, I'd look at operators designed for permutation encodings as a starting place. Something like, "for each gene, apply a separate run of Order Crossover or Cycle Crossover to populate the child's gene" might be a reasonable starting point.


text:
Is it harmful to set the learning rate of training a model to be too high if there is some decay function for the learning rate?
It is known that if $\alpha$ is set to high, then the cost function of the model may not converge.
However, would a decaying of the learning rate provide some "tuning" of the $\alpha$ value during training ? In the sense that if you set a high learning rate but you also have some form of learning rate decay, then eventually $\alpha$ value would fall within the "just right" and "too low" range eventually. Is it better to then set an initial learning rate that is more "flexible" in the higher ranges rather than a learning rate that is too low ?

Setting too high a learning rate will extend the time to get a good result.
In my opinion, it is better to set not too big a learning rate but to use learning with momentum. When the learning starts to be ineffective, increase the learning rate to find a better optimal result. It seems to me that this allows to get very good results in a faster time than setting a large value of the learning rate from the very beginning. At least in cases I've dealt with.

authors: Фёдор Курушин


text:
How to use speaker's information as well as text for fine-tuning BERT?
I want to classify my corporate chat messages into a few categories such as question, answer, and report. I used a fine-tuned BERT model, and the result wasn't bad. Now, I started thinking about ways to improve it, and a rough idea came up, but I don't know what to do it exactly.
Currently, I simply put chat text into the model, but don't use the speaker's information (who said the text, the speaker's ID in our DB). The idea is if I can use the speaker's information, the model might better understand the text and classify it better.
The question is, are there any examples or prior researches similar to what I want to achieve? I googled for a few hours, but couldn't find anything useful. (Maybe the keywords weren't good.)
Any advice would be appreciated.

My answer assumes your fine-tuning architecture simply stacks a single fully-connected layer on top of the BERT [CLS] output, as in Figure 4b of the BERT paper.
Generally, when working with mixed data such as continuous and categorical features, the first step is to simply concatenate all the inputs into one long vector. In your case, you would concatenate a one-hot encoding of speaker ID to the BERT [CLS] output for each example. If you're using tensorflow, you'll need to use the functional API to create a multiple input model, as I outline below.
import tensorflow.keras as keras

# bert_input is the wordpiece tokenized input text
# BERT is the BERT model, such as a tensorflow_hub.KerasLayer with a pretrained model from tfhub.dev
bert_out = BERT(bert_input)

# spkr_input is the one-hot speaker ID encoding.
spkr_input = keras.Input(shape=(num_speakers,), name='spkr_input')
dense_input = keras.layers.concatenate([bert_out, spkr_input])
scores = keras.layers.Dense(num_classes, activation='softmax')(dense_input)
model = keras.Model(inputs=[bert_input , spkr_input], outputs=[scores])

You could also try feeding the one-hot encoded speaker ID vectors through a separate fully-connected network first to obtain a continuous representation (i.e. a speaker embedding) and then concatenate that to the BERT [CLS] output and feed the result into your classification layer. Modifying my example above,
spkr_emb = keras.layers.Dense(spkr_emb_size, activation="relu")(spkr_input)
dense_input = keras.layers.concatenate([bert_out, spkr_emb])

You can find a more detailed guide to mixed input models with Keras at https://www.pyimagesearch.com/2019/02/04/keras-multiple-inputs-and-mixed-data/

Thank you for the detailed explanation. I'll try your answer, and accept it (or ask a followup question).


text:
What would be the good choice of algorithm to use for character action selection in an RPG, implemented in Python?
I have developed an RPG in likeness to the features showcased in the Final Fantasy series; multiple character classes which utilise unique action sets, sequential turn-based combat, front/back row modifiers, item usage and a crisis mechanic which bears similarity to the limit break feature.
The problem is that the greater portion of my project depends on the use of some means of Machine Learning to, in some manner, act as an actor in the game environment, however, I do not know of my options under the bare-bones environment of a command line game; I am more familiar with the use of pixel data and a neural network for action selection on a frame-by-frame basis.
Could I use reinforcement learning to learn a policy for action selection under a custom environment or should I apply a machine learning algorithm to character data, find the example outlined below, determine the best action to use on a particular turn state?
+-------+--------+--------+---------+---------+------------+---------------------+------------+--------------+--------------------+--------------+--------------+
| Level | CurrHP | CurrMP | AtkStat | MagStat | StatusList | TargetsInEnemyParty | IsInCrisis | TurnNumber   | BattleParams       |ActionOutput  | DamageOutput |
+-------+--------+--------+---------+---------+------------+---------------------+------------+--------------+--------------------+--------------+--------------+
|    65 |   6500 |    320 |      47 |      56 |          0 | Behemoth            |0           | 7            | None               |ThiefAttack   |4254          |
|    92 |   8000 |    250 |      65 |      32 |          0 | Behemoth            |1           | 4            | None               |WarriorLimit  |6984          |
+-------+--------+--------+---------+---------+------------+---------------------+------------+--------------+--------------------+--------------+--------------+

I would like to prioritise the ease of implementation of an algorithm over how optimal the potential algorithm could be, I just need a baseline to work towards. Many thanks.

Your third paragraph is confusing to me. Importantly, reinforcement learning (RL) is a type of machine learning, and your description of your alternative to RL "machine learning algorithm ... determine the best action" reads like it is RL too. Do you mean that you wiill personally decide what the best action is for a large number of cases, and use that as training data for a supervised learning algorithm?

@NeilSlater I understand why it's confusing, I didn't properly distinguish between actions permissible by a player in the game and actions for a learning agent as, from my knowledge, they would be indistinguishable.    I am planning to both play as an expert and use collected data from a few testers as training data a supervised learning algorithm; though I do understand that this could concur bias towards particular actions.

@SeunOsiko How good does the AI need to be? You might benefit from training the agent against itself, or rather an agent that takes the role of a potential player, with all their actions. I think in this case though, your best bet is to learn a policy for action selection.

@Recessive It's for an undergrad final thesis; I am more concerned about the process of developing the agent and analyzing the actions chosen on a particular game start, rather than the quality of outcome. How would I go about building an agent to learn a policy, could I use libraries such as TensorFlow or PyTorch considering the lack of visual data?

@SeunOsiko I think you'd be best off using something like openai gym: https://gym.openai.com/ . Then, it's just a matter of how you input the data. You could actually represent this as an image if you wanted - just construct an image where each pixel is related to some action or game state. White if present or black otherwise. Keep in mind that images are ultimately arrays, so technically anything you do input can be an image if you wanted it to be.

@Recessive I had considered using a custom env but porting the game to an env brings up more questions than answers. Whilst I think I understand my action space is variable based on the character's usable actions, how should I store them and how to I manage the human players needing manual input to make choices whilst the enemy has pre-determined logic and the agent choices being learnt. Can I use an observation space by using purely numerical data, should I just store the agents data or the data of all allied characters in the game?


text:
Can imbalance data create overfitting?
I am doing human activity recognition project. I have total of 12 classes. The class distribution look like this:

$\color{red}{If \ you \ watch \ carefully, you \ can \ see \ that \ I \ have \ no \ data \ points \ for \ class \ 11 \ and \ class \ 8.}$ Also, the dataset is highly imbalanced. So, I took minimum data points (in this case 2028) for all of the classes. Now my balanced data look like this:

After doing this it looks like a balance data. $\color{red}{But \ still, \ I \ think \ it \ not, \ because \ I \ have \ zero \ datapoints \ for \ class \ 11 \ and \ class \ 8}$. In my opinion the classes are still imbalance.
I am using CNN model to solve this activity project. My model summary is following:

The main problem is, my model starts overfitting heavily when I train it.

Is it due to my imbalance data( class 8 and 11 has zero data points) or something else?
$\textbf{Hyperperameter:}$
$\textbf{features:}$ X, Y, Z of mobile accelerometer
$\textbf{frame size:}$ 80
$\textbf{optimizer:}$ Adam, $\textbf{Learning rate:}$ 0.001
$\textbf{Loss:}$ Sparse categorical cross-entropy

Your zero-classes just do not exist for the model. There is no information about them in training and test sets. I think the reason for overfitting is that your have very small size of the training set comparing to the number of parameters in your model. If it is easier for a model to remember all training entries - it will just do that. You need to prune your model to force it to make some generalizations about the data.

Those images are from the original dataset. Before feeding it to the model , I reduce the class by 1 ( class 1 becomes class 0 and so on). Yes, I know in case of overfitting I have to reduce the model complexity. But I don't think it is the case in this problem,

You have about 20k entries in the training data and 500k parameters to learn on them. The proportion is reversed, you'd better have 5-10m entries in the training data to properly train such a model.


text:
Can StyleGAN be refined without a full training?
Can I refine StyleGAN or StyleGAN2 without retraining it for many days, such that its pretrained model is trained to generate only faces similar to a (rather small) set of reference images?
I would like to avoid creating a large dataset and training for many days to weeks, but use the existing model and just bias it towards a set of images.

Are you asking if it's possible to apply transfer learning to StyleGAN?

I think transfer learning means solving another problem. My problem at hand is that I would like to use stylegan to generate images of people that are similar to a given input set (that may or may not be good represented in the pre-trained model). I hope it is possible to use the pre-trained model and train it on a new set without losing too much of the pre-trained weights.


text:
Is there a way to show convergence of DQN other than by eye observation?
I made a DQN model and plot its reward curve. You can see intuitively that the curve already converged since its reward value now just oscillates. How can I show confidence that my DQN already reached its optimal other than by just showing the curve? Are there any way to validate that it is already optimized?


text:
How to validate that my DQN hyperparameters are the optimal?
My DQN model outputs the best traffic light state in an intersection. I used different values of batch size and learning rate to find the best model. How would I know if I got the optimal hyperparameter values?

If possible, I would try to calculate what the (theoretical) maximum  throughput through the intersection is for a given time interval. If the control behavior that the DQN produces comes empirically close to the maximally possible throughput score, the model is good. Otherwise, you could measure and compare the throughput of different models and choose the best performing one.

What does "throughput through the intersection is for a given time interval" actually mean in this context? What is a "throughput score"?

Each intersection has some limitations with respect to how many cars can cross it eg per hour. After a certain point, there can just not go 'more' cars through it per time interval, eg due to speed limitations.
Given reasonable simulation assumptions, the maximal number of cars that can go through the intersection per hour can be calculated. This measure can then serve as your gold standard to which you compare the results obtained from your DQN controller.
Unless, you want to evaluate it in a static scene, which seems odd to me to do, since it wouldn't assess the controller's dynamics.

Some simulation packages that could come handy (also for training DQN on Traffic light control, btw):
https://sumo.dlr.de/pdf/dkrajzew_MESM2002_SUMO.pdf
https://www.matsim.org/
https://carla.org/

Well, to be honest, the details of the OP's problem and model aren't clear because the part "My DQN model outputs the best traffic light state in an intersection" isn't very explicit and descriptive enough. Anyway, if I understand you correctly, your suggestion is to get some kind of constraint for each situation.

DQN is a Reinforcement Learning algorithm which is trained on solving a particular task in a given environment. Regardless of how ambiguous the original question is, for testing whether DQN does a good job or not, it is not enough to give it e.g. 1000 scene images and consider what the output is, eg because you could assess the respective output traffic light state, but not how long it would decide to remain in it. Therefore, you won't see whether traffic congestion is gonna kick in after some time of letting the controller run (sign for a bad controller) or not (sign for good controller).

Therefore, yes, I would set up a simulation that comes close to the (real-life) conditions of the intersection for which the controller is designed and measure things like the number of cars passing through the intersection per hour or the average waiting time per car in order to get some objective, quantitative results that you can compare against other models.


text:
How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?
I am writing a couple of different reinforcement learning models based on Rainbow DQN or some PG models. All of them internally use an LSTM network because my project is using time series data.
I wanted to test my models using OpenAI Gym before I add too many domain specific code to the models.
The problem is that, all of the Atari games seem to fall into the CNN area which I don't use.
Is it possible to use OpenAI Gym to test any time series data driven RL models/networks?
If not, is there any good environment that I can use to examine the validity of my models?

If I understand your problem correctly, you can test on just about any environment, and just omit parts of the observations to ensure your RNN is learning. For example, you can test on cartpole, ignoring the velocity and angular velocity states. This way the MDP isn't actually Markovian and you'll need the RNN to learn.

Hi harwiltz, appreciate your quick reply. I am in fact not familiar with the Gym environment and didn't know there are observations like velocity. My understanding on Gym comes from reading a few well cited paper such as the one for Rainbow DQN where they seem to use the game frame images as input and hence using CNN as the internal network. I don't know how I can feed such environment data to a LSTM/RNN instead. Or do I not have to use frame image data as input?

Not all gym environments have pixel state spaces. If you want to train from pixels, you should probably make a CNN feature extractor and pass the extracted featured to your RNN. If you just want to test your network though, I'd recommend training on simpler environments (look up cartpole, lunarlander, they're both in openai gym for example), whose state spaces are basically just position, velocity, etc, and discard some dimensions to force your RNN to learn.

I think I get the picture. Will go and do some study. Thanks. Last question, if I only use data like position, velocity etc as state space for training, will I still be able to get good testing results? After all, getting good Atari game results is how I would tell if my own model is good or not.

With Atari, you can only get pixel or RAM states, both of which you'll probably need to implement a feature extractor to preprocess your RNN inputs. Otherwise, when you have access to states like position and velocity and such, that generally makes the problem much easier (the data will have much lower dimension).


text:
Algorithm to train a neural network against differentiable and non-differentiable databases?
Let's say I have two databases, $(\mathbf{x_i}, \mathbf{\hat{p_i}})$ and $(\mathbf{x_j}, \mathbf{\hat{q_j}})$. A neural network with weights $\theta$ can receive an input $\mathbf{x}$ and produce an output $\mathbf{y}$. Mathematically, $\mathbf{y} = f_{NN}(\mathbf{x},\theta)$. To compare the output of my neural network and the database, I need two wrappers, $\mathbf{p}=g(\mathbf{y})$ and $\mathbf{q}=h(\mathbf{y})$.
The problem is: only $g(\cdot)$ is differentiable while writing $h(\cdot)$ in a differentiable manner would take a huge effort.
Is there any efficient way to train my neural network to minimize the following loss function?
$$
\mathcal{L}(\theta) = \sum_i \left\{g\left[f_{NN}(\mathbf{x_i}, \theta)\right] - \mathbf{\hat{p_i}}\right\}^2 + \sum_j \left\{h\left[f_{NN}(\mathbf{x_j}, \theta)\right] - \mathbf{\hat{q_j}}\right\}^2
$$
My thinking
If I am using gradient descent-type algorithm, I can only optimize the first part of the loss function while ignoring the second part.
If I am using evolutionary-type algorithm, I can optimize both parts, but it will take a long time and I don't make a full use of the differentiable property of $g(\cdot)$.

You can estimate the gradients of h(.) with respect to the outputs of your model. You could use methods like finite difference, if it is viable, SPSA or RDSA otherwise. Then backpropagate the approximated gradients.


text:
NLP Identifying important key words in a corpus
I am intrigued with the idea of Zettelkasten but unsatisfied with the current implementations. It seems to me that a machine learning and NLP approach could be productive by helpfully identifying “important” keywords on which to links could be created, with learning to help narrow the selection of keywords over time.
My problem is that it’s been 30 years since AI classes in grad school and things have moved on. I’m sure I could become an nlp expert with study but I don’t wanna. So I’m looking for guidance: what are the right terms to describe identifying keywords in context, ideally with some semantic content; how would I apply ML with my training to improve the keyword identification.
I’d love references, ideas, and packages references. Python is preferred, but not strongly; I write most common (and many uncommon, SNOBOL and COBOL anyone?) languages so language isn’t all that much of an issue.

I am sure there are complex methods to extract keywords, but the standard one which should serve as a strong baseline is the RAKE graph algorithm https://pypi.org/project/rake-nltk/. It should work reasonably well in most text domains.

See, that's the thing — I'm not looking for complex methods, I'm just so far out of touch that it's hard to even make a useful google search. This not only looks like it might work, it gives me a hook for "things like" searches.

Then RAKE should be good choice. Also, a recent package is YAKE https://github.com/LIAAD/yake, which claims to be a better version of RAKE might also be of help to you. It is also easy to set up and use.


text:
Tic-tac-toe: How would standard SARSA and Q-learning yield different results in the agent's behaviour?
I know this is deceptively simple. Tic tac toe is a well studied game for RL.
Assume your agent is playing aggainst a strong opponent.
I know you deal in after states. I know that in Q learning the optimal policy should be converged on faster as the Q(S,A) is becoming closer to the optimal each step. While in SARSA the Q function will not be updated towards the optimal sometimes as it is exploring. If epislon is fixed SARSA will convereg to the epislon greedy policy.
I came across the question above and I don't know the answer, is it that SARSA may play more conservatively? Opting for more draws rather than getting in board positions where one could is likely to either lose or win rather than draw.


text:
How to measure/estimate the energy consumption of CNN models during testing?
Does someone know a method to estimate / measure the total energy consumption during the test phase of the well-known CNN models? So with a tool or a power meter...
MIT has already a tool to estimate the energy consumption but it only works on AlexNet and GoogleNet, I need something for more Architectures (VGG, MobileNet, ResNet...). And I also need a metric to evaluate the well-known architects in terms of energy consumption. So at first estimate or measure the energy consumption and then evaluate the results with a good metric.
With a measuring device I would measure the power consumption before using the CNN and I will repeat this experiment a few times then I will average the results, and do the same thing while using the CNN and at the end I will compare the results. But I have three problems here:
1. how can I know that nothing else is running on the PC that also consumes energy while using the CNN?
2. how can I increase the accuracy of the measurements?
3. I don't find any power meter that measures the energy consumption in short periodes (1s).

Thats why I prefer a tool to estimate the energy consumption, the accuracy of the measurments will not be that good but I didn't find any another tool..
Does someone have an Idea, papers, sites that can help me?
Many thanks in advance for your reply!


text:
Why is 'scatter' used instead of variance in LDA?
I've been reading about Fisher's Linear Discriminant Analysis lately, and I noticed that the objective function (particularly for two-class classification) to be maximized contains scatter terms instead of variance, in the denominator. Why is that?
To clarify, the scatter of a sample is just the variance multiplied by the number of data points in the sample.
Thank you!

I think this may be a better fit at stats.stackexchange.com - I'm not 100% sure, but I think the answer will be to do with how variance of dataset mean varies depending on variance of individual terms. There are far more experts in this kind of pure statistics over at Cross Validated, and LDA is only weakly related to AI and ML.

I think you may find the answer here:
https://stats.stackexchange.com/questions/123490/what-is-the-correct-formula-for-between-class-scatter-matrix-in-lda


text:
Suppress heatmap non-maxima in segmentation with UNet
I'm using U-Net for image segmentation.
The model was trained with images that could contain up to 4 different classes. The train classes are never overlapping.
The output of the UNet is a heatmap (with float values between 0 and 1) for each of these 4 classes.
Now, I have 2 problems:

for a certain class, how do I segment (draw contours) in the original image only for the points where the heatmap has significant values? (In the image below an example: the values in the centre are significant, while the values on the left aren't. If I draw the segmentation of the entire image without any additional operation, both are considered.)

downstream of the first point, how do I avoid that in the original image the contours of two superimposed classes are drawn? (maybe by drawing only the one that has higher values in the corresponding heatmap)

Regarding problem 1 (if I understand it correctly), have you tried something like non-maximum suppression, or something similar?


text:
Structure-preserving layer in a network with respect to a transformation
I'm reading this paper: https://arxiv.org/pdf/1602.07576.pdf. I'll quote the relevant bits:

Deep neural networks produce a sequence of progressively more abstract representations by mapping the input through a series of parameterized functions.  In the current generation of neural networks, the representation spaces are usually endowed with very minimal internal
structure, such as that of a linear space $\mathbb{R}$^n.
In this paper we construct representations that have the structure of a linear $G$-space, for some chosen group $G$. This means that each vector in the representation space has
a pose associated with it, which can be transformed by the elements of some group of transformations $G$. This additional structure allows us to model data more efficiently: A
filter in a $G$-CNN detects co-occurrences of features that have the preferred relative pose [...]
A representation space can obtain its structure from other representation spaces to which it is connected. For this to work, the network or layer $\phi$ that maps one representation
to another should be structure preserving. For $G$-spaces this means that $\phi$ has to be equivariant: $$\phi(T_gx)=T'_g\phi(x)$$That is, transforming an input $x$ by a transformation $g$ (forming $T_gx$) and then passing it through the learned map $\phi$ should give the same result as first mapping $x$ through $\phi$ and then transforming the representation.
Equivariance can be realized in many ways, and in particular the operators $T$ and $T'$ need not be the same. The only requirement for $T$ and $T'$ is that for any two transformations $g$ and $h$, we have $T(gh) = T (g)T (h)$ (i.e. $T$ is a linear representation of $G$).

I didn't understand the paragraph in bold. A structure preserving map is something that preserves some operation between elements in the underlying set. A simple example: if $f:\mathbb{R}^3\to\mathbb{R}$ such that $(x,y,z)^T\mapsto x+y+z$, then
$$f(r+s)=f((r_1,r_2,r_3)^T+(s_1,s_2,s_3)^T)=f((r_1+s_1,r_2+s_2,r_3+s_3))
\\=r_1+s_1+r_2+s_2+r_3+s_3=f(r)+f(s)$$
where the addition in the far left term is in $\mathbb{R}^3$ and addition in the far right is in $\mathbb{R}$. So the map $f$ preserves the additional structure of addition.
In the quoted paragraph, $\phi$ is the structure preserving map, but what's the structure being preserved exactly? And why is the operator on the right different from one on the left? i.e. $T'$ on RHS instead of $T$


text:
What is the big fuzz about SHA-RNN versus Transformers?
In his paper introducing SHA-RNN (https://arxiv.org/pdf/1911.11423.pdf) Stephen Merity states that neglecting one direction of research (in this case LSTMs) over another (transformers) merily because the SOTA in transformers are due to using more computing power is not the way to go.
I agree that finding neat tricks in AI/ML is equally (if not more) important than just throwing more computing power at the problem. However I am a little bit confused.
The main difference (since they both use attention units) between his SHA-RNN and transformers seem to be the fact that SHA-RNN uses LSTMs to "encode the position of words", where transformers do position encoding by using cosine and sine functions.
My confusion comes from the fact that LSTMs need to be handled sequentially and thus they cannot use this large advantage of GPUs, being able to compute things in parallel, whilst transformers can. Wouldn't this mean that (assuming LSTMs and positional encoding are able to acquire the same results), training using LSTMs would take longer than transformers and thus need more computing power, thus defeating the initial puporse of this paper? Or am I misinterpreting this?
Basically my question comes down to "Why would an SHA-RNN be less computationally expensive than a transformer?"

Transformers can be trained with parallelization. So you might have 100 cores training the transformer for an hour (100 total hours of CPU time), whereas the LSTM has one core training for ten hours (10 total hours of CPU time). Even though the LSTM takes longer, it is less expensive to train.

This is what I also had as a final conclusion. So what this means is that if we would train a transformer for 10 hours of CPU time, or a LSTM for 10 hours of CPU time, results might be the same? And that if we invest more time in researching LSTMs we might even find them to be better (performance wise) & less compute expensive?

Right, although to be honest I'm not familiar with the comparison of LSTMs and Transformers in terms of total training time—not sure if they're exactly the same or if one is better


text:
Solving multi-armed bandit problems with continuous action space
My problem has a single state and an infinite amount of actions on a certain interval (0,1). After quite some time of googling I found a few paper about an algorithm called zooming algorithm which can solve problems with a continous action space. However my implementation is bad at exploiting. Therefore I'm thinking about adding an epsilon-greedy kind of behavior.
Is it reasonable to combine different methods?
Do you know other approaches to my problem?


text:
Can I use augmented data in the validation set?
I am trying to predict nursing activity using mobile accelerometer data. My dataset is a CSV file containing x, y, z component of acceleration. Each frame contains 20-second data. The dataset is highly imbalance, so I perform data augmentation and balance the data. In the data augmentation technique, I only use scaling and my assumption is, if I scale down or up a signal the activity remains the same. Using this assumption I augmented the data and my validation set not only contain the original signals but also the augmented (scaling) signals. Using this process, I am getting quite a good accuracy that I never being expected using only data augmentation. So, I am thinking that I performed a terrible mistake somewhere. I check the code, everything is right. So now I think, since my validation set has augmented data, that's the reason of getting this high accuracy (maybe the augmented data is really easy to classify).

You should not use augmented data in the validation nor in the test set.
Validation and test set are purely used for hyperparameter tuning and estimating the final performance, i.e. estimating the generalization error. These two data sets should be as close as possible to other data, which you could have acquired, but you actually haven not, i.e. your true data distribution.
It's fine to augment training data, since it mimics other samples from the true distribution by applying transformations, noise, etc. and thus helps to increase generalization performance (assuming your augmentation assumptions are right). But evaluation should always be performed on original data.