text:
How do I find a similar RNN as a starting point?
I am new to machine learning and neural networks and I want to create a neural network for a study project.
I would like to create a RNN, that uses one (A) or several time series (with the same length, A, B, C...) as an input and creates another time series (Z) as an output from that. Basically all time series are derived from on time series, so their relationship that can be expressed mathematically. ex. B = dA/dt, ... C = moving mean(B)
For training I want to feed the NN with the input and output time series, which I can create with a simulation with known parameters, so that I know the output is correct.
I am new to working with neural networks and couldn't find a NN that seemed to do a similar job. Does somebody have a NN in mind that could do the job?
I asked chatGPT, which said that simple RNNs with or without LSTM or GRU cells could do the job.
Any ideas?

From the information you have given, then yes any basic RNN architecture could do the job. Some might work better than others in practice, depending on your data. You don't really give enough detail to say much more though, and it is not clear what you are expecting when you ask someone to recommend an NN.

Well my professor just said I should ask in a forum to find a similar network. The problem is I don't know what I am looking for exactly. I found something here but I am not sure if it is what I am looking for and if it is usable for the task I am doing. https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/ Should I go for that?

For training I will have variable input time series let's say 5, which are all derived from one series. And when I use the NN then I only give 4 time series and it predicts the 5th time series. I am just looking for a NN that does something like this to read upon.

Yes your linked example looks fine to me, and from your description so far it seems like it would work. I suggest you look into it to the level where you could follow the description and make the example, and then take that to your professor.


text:
how to train virtex automl to recognise 7 segment display numbers
We're working on a project which needs to read seven segment displays from photos.  We've tried a few AI text recognisers but the 'font' is too tricky.  So we thought given that we're looking for a small number of quite distinct shapes AutoML would be worth a go.
We've not done this before so we had a go and our first model is a bit disappointing, and we've not found anywhere which gives good advice on training data.  We don't have a big collection of photos of seven segment displays we can use (let alone a labelled one)
We made 13000 images of single digits (0-9 as well as C and F).  We used a 7 segment font to get the general shape then rendered it on to a flat colour background in random colour combiations, at random locations in the image and with random amounts of skew (using a perspective transform)
this gives pretty random results when tested against photos, it works ok if we generate more images in the same way as we made training data and test against them

is skewing neccessary/useful? or does AutoML account for skew/rotation etc
is a flat background a basic no-no?  should we be putting random photos in the background?  If so where can one get a collection of thousands of random photos or backgrounds to use?
is colour important - we don't care about colour of the digits, in fact we want any colour combination to be recognised.  We've used random foreground/background colour combinations to 'tell' the model this, is this necessary?
I've seen other training data that is entirely in greyscale - e.g. mikegchambers on youtube and his lego brick recogniser-  will people using greyscale training data be converting the image to greyscale before sending it to the model
where is a good place to get practical advice on making training data

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Can AI-written text detection be made more accurate if you know the prompt?
Usually, genAI detection is of the form:

Input: text.
Output: was it generated by AI?

Thus far, AI-written text detection is terribly inaccurate, and if you're a user of r/ChatGPT, you've probably seen myriad "my teacher accused me of plagiarism" posts as a result.
However, if we wanted to e.g. check if a Stack Overflow answer was AI generated, we might instead have the situation:

Input: text and prompt (i.e., the Stack Overflow question).
Output: was it generated by AI?

This seems like it would be a big advantage in detecting AI text.  I remember when I was trying to determine if an answer is or isn't AI-generated, copy/pasting the question into ChatGPT and comparing ChatGPT's answer with the posted answer was quite enlightening.
Question: Can AI-written text detection be made more accurate if you know the prompt?
I'm wondering if there's a research paper in this direction.  I didn't immediately find the answer to this question by Googling the title.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
When should grayscale processing be applied to image inputs in visual reinforcement learning environments?
I am currently working with visual environments in Reinforcement Learning (RL) and have noticed differing practices regarding preprocessing of image inputs. Specifically, in the Atari environment, a common approach is to first convert RGB images to grayscale, then stack the frames. The channel number of the final environment observation equals the number of stacked frames.
On the other hand, in the DeepMind Control Suite (DMC) environment, the common practice is to directly stack the RGB frames. In this case, the channel number of the final environment observation equals the number of stacked frames times three (corresponding to RGB channels).
Here is a snippet of grayscale processing and frame stacking in the Atari environment:
...
observation = rgb2gray(observation)
observation = stack_frames(observation, stack_size)
...

And here is a snippet of frame stacking in the DMC environment:
...
observation = stack_frames(observation, stack_size)
...

I have two main questions about this:

Why are there different practices for frame stacking in the Atari and DMC environments?
Are there any guiding principles or criteria for deciding when grayscale processing should be applied to the image inputs in different visual environments?

Any insights or recommendations on this matter would be greatly appreciated.


text:
How do i approach creating a masked auto-encoder for feature extraction
I trained Masked Autoencoder-based models in order to use the encoder as a backbone for another task. Pretraining has been done in a Self-Supervised manner on image data. Now that it comes to my downstream task, I was wondering what is the best way of creating feature maps on the whole image for the 2 models:

ConvNextV2 (fully convolutional masked autoencoder) FCMAE with a mask ratio of 60% in pretraining: Since we deal with sparse convolutions, it might make sense to just set the mask ratio to 0, such that we do not mask out anything and create features for the whole image area, right? I tried to compare it with creating 2 deterministic masks (50% each) and do 2 forward passes such that I can simply add up the feature maps, but the results are not equal. Can someone explain the reason for that?
ViT-MAE (MAE with vision transformer backbone): From my knowledge, Transformers are constrained to their pre-trained sequence length and thus do not perform well when changing the number of input patches (e.g. 100%) for inference to create feature maps for the whole image. Any remarks on that approach?


text:
How to Train Multiple Deep Learning Models on Multiple GPUs?
I have access to a GPU server with four gpus. Now I would like to train multiple models or folds of one model, one on each gpu. How can I schedule multiple trainings, so that a new training instance will be started, once a gpu becomes available?
I've seen that there is the Nvidia container toolkit, where I can assign a specific GPU at a time. With docker compose I would be able to start another container with the depends option- However, I envision a more automated solution. Something like a single node cluster, which automatically picks a scheduled training container, once another container that was blocking a gpu is finished.
But so far, I only found (data) distributed training approaches.
Does anyone know about a way to accomplish this?
Thanks in advance!
Best,
Eric


text:
In MAML for RL, are new tasks sampled for every meta update, or is the same set of tasks used throughout?
Consider Model Agnostic Meta-Learning, as described here.
For a RL task $T_i$, represented with a model $f$, with parameters $\theta$ and learning rate $\alpha$, where the RL loss function is $\mathcal{L}$:
$$
 \theta'_{i} = \theta - \alpha\nabla_{\theta}\mathcal{L}_{\mathcal{T}_{i}}\left(f_{\theta}\right)
$$
This is standard gradient descent.
The meta-update step samples multiple tasks $\mathcal{T}_i\sim p(\mathcal{T})$, calculates their respective parameters $\theta_i$ from $\theta$, and minimizes the loss of the original $\theta$ with respect to these intermediate $\theta_i$s:
$$
 \min_{\theta} \sum_{\mathcal{T}_{i} \sim p\left(\mathcal{T}\right)} \mathcal{L}_{\mathcal{T_{i}}}\left(f_{\theta'_{i}}\right) = \sum_{\mathcal{T}_{i} \sim p\left(\mathcal{T}\right)} \mathcal{L}_{\mathcal{T_{i}}}\left(f_{\theta - \alpha\nabla_{\theta}\mathcal{L}_{\mathcal{T}_{i}}\left(f_{\theta}\right)}\right)  \\
 \theta \leftarrow \theta - \beta\nabla_{\theta} \sum_{\mathcal{T}_{i} \sim p\left(\mathcal{T}\right)} \mathcal{L}_{\mathcal{T_{i}}}\left(f_{\theta'_{i}}\right)
$$
Quesstion: is $\mathcal{T}_i\sim p(\mathcal{T})$ sampling done for every meta-update? Or, is a constant set of tasks maintained from the beginning? Can the frequency of task sampling be different from the frequency of meta-updates?

definitely sampled from a bigger pool of tasks, otherwise bye bye generalization


text:
Feature Importance for FT Transformer
I hope someone is able to shed light on this. I was reading through the codes on this link with regards to getting feature importance from attention scores.
https://github.com/aruberts/TabTransformerTF/blob/main/tabtransformertf/models/fttransformer.py
and I came across these codes.
    # Prepare for Transformer
    transformer_inputs = tf.concat(transformer_inputs, axis=1)
    importances = []

    # Pass through Transformer blocks
    for transformer in self.transformers:
        if self.explainable:
            transformer_inputs, att_weights = transformer(transformer_inputs)
            importances.append(tf.reduce_sum(att_weights[:, :, 0, :], axis=1))
        else:
            transformer_inputs = transformer(transformer_inputs)

After cross checking with chatgpt, this is what I found from its replies.
"The third dimension in att_weights[:, :, 0, :] is set to 0 to select the attention weights of the first attention head."
Basically what I want to ask is whether the syntax make sense since its only appending the first attention head, which may not give the full information on the attention scores or weights. Rather we should be taking all the scores from the attention heads and calculate the mean from there. Is this reasoning correct?
Because I would deem this to be rather straightforward but the author of the github didn't take this approach and hence I wonder what could be the reasons.


text:
What machine learning methods can be used to minimise the difference to a target 2D boolean matrix with differing sizes?
I have a problem whereby given an input, the difference that is generated from the model to a target 2D boolean matrix needs to be minimised.
The target boolean matrix can be of varying shape, i.e: 10 x 10, 100 x 100, 500 x 500 etc depending on the input. Naively, I was thinking of having 100 output nodes for the case of 10 x 10 matrix but the number of outputs becomes too huge when it is 100 x 100 matrix and so forth.
Alternatively, I was thinking about a regression model that outputs the number of 1 labels. However, this approach doesn't consider the position of the values in the 2D matrix.
Is there a way to efficiently learn a model for this problem?

please provide an actual example

@AlbertoSinigaglia I would like to provide an actual example but it would be divulging an unpublished work that I'm working on for my research. However, please let me know if anything is not clear with my posting. I'm happy to clarify.

well, you give little to no information about the problem... the best I can say is to lookup gans generator or diffusion models, they usually have to deal with high dimensional outputs... but they use a ton of inductive bias to make it happen, whcih might not hold in your case

Maybe you can factorize/decompose your target matrix with a fixed dimensionality, and then let the network learn such fixed factorization, and finally recompose the full matrix by using the components learned by the network.


text:
Updating custom output layers of an LSTM network
I have a text generation task learning to predict the next word with an LSTM network with multiple output layers.
After the generation of a sentence has finished, I calculate a reward for the whole sentence and try to update the output layers participated in the generation (contributing layers get the calculated reward value, others get 0).
My problem is that even if I update only the selected output layers, it seems that other layer's weights got updated instead.
I have a minimized example with dummy data to present the problem:
import random

import numpy as np
import tensorflow as tf

from keras.layers import Input, LSTM, Dense, Embedding
from keras.utils import pad_sequences
from tensorflow.keras.models import Model

def policy_gradient_loss(y_true, y_pred):
    return tf.reduce_mean(tf.math.log(y_pred) * float(y_true))

# Define the model with 3 output layers (named 'a', 'b' and 'c').
input_layer = Input(shape=(4,))
embedding_layer = Embedding(input_dim=10, output_dim=4)(input_layer)
lstm_layer = LSTM(4)(embedding_layer)
output_layers = [Dense(3, activation='softmax', name=name)(lstm_layer) for name in ['a', 'b', 'c']]
model = Model(inputs=input_layer, outputs=output_layers)
model.compile(loss=[policy_gradient_loss] * 3, optimizer='adam', run_eagerly=True)

# Dummy input data.
input_data = np.array([[2, 3, 4, 5]])

# Create target data to reward only the 'b' output layer.
target_data = [np.array([0]) for _ in range(len(model._output_layers))]
target_data[1] = np.array([10])

# Save initial weights.
initial_weights = model.get_weights()

model.train_on_batch(input_data, y=target_data)

# Save weights after the learning.
updated_weights = model.get_weights()

# Compare the before-after weights.
for layer_idx, (layer_name, initial_w, updated_w) in enumerate(zip([layer.name for layer in model.layers], initial_weights, updated_weights)):
    if not tf.math.reduce_all(tf.equal(initial_w, updated_w)):
        print(f'The weights in layer {layer_idx} ({layer_name}). has changed.')

Result:
The weights in layer 0 (input_1). has changed.
The weights in layer 1 (embedding). has changed.
The weights in layer 2 (lstm). has changed.
The weights in layer 3 (a). has changed.

My expectation would be to get the layer 4. (output layer 'b') updated instead of layer 'a' (or at least beside 'a').
What am I missing? Is my expectation or my implementation wrong? (Or both...?)

Never use arrays as inputs for training in TF, they are used when you are dealing with multiple inputs/outputs...

@AlbertoSinigaglia Thanks for your comment! Could you please explicate this a bit more? Is this a bad practice or do you see some actual error in the current setup? What to use instead? (btw I'm dealing here with multiple output layers)

just make the arrays np.arrays, and you will see that errors start popping up

It's a newbie question, but how to do that? I guess you meant of  target_data, since input_data is an np.array already. If I update target_data as:
target_data = np.array([np.array([0]), np.array([10]), np.array([0])])
or
target_data = np.array([[np.array([0]), np.array([10]), np.array([0])]])
then I get Data cardinality or Incompatible shapes issue.
My original intention was to "tell" the model, that the [2, 3, 4, 5] input - that was generated e.g., with the b) output layer - was correct, hence update the weights of that layer only. Does this make sense?


text:
Improving Fuzzy Logic Control System Results
I am new to using skfuzzy and the use of fuzzy logic for classification, but the results I am getting are both good and bad for some conditions.  I am aware of the overarching impact the  defuzzification method (COA, BOA, MOM, SOM, LOM) has on deriving the defuzzied value, but I would like some advice on how others would approach this problem or what I have overlooked, which has thus caused outputs that don't exactly follow the rules I created.
Fuzzy Model:  I would like to rate a streams vulnerability to development based on distance from road and stream order (Strahler Hierarchy).  As the distance from a road to stream increases there will be less vulnerability (floating values from 0 m to 750 m).  On the other hand, as the Strahler order increases there will be a higher vulnerability because of more stream flow, stream width, habitat, etc. (Integer values from 1 to 10).
Vulnerability ranges from 0 to 1, with 0 being not vulnerable to 1 being very vulnerable.
# Antecedent/Consequent objects and membership functions
strmord = ctrl.Antecedent(np.arange(1, 11, 1), 'strmord')
strmdist = ctrl.Antecedent(np.arange(0, 750, 1), 'strmdist')

# Defuzzification method:
# Fuzzy logic calculations are excellent tools, but to use them the fuzzy result must be
# converted back into a single number. This is known as defuzzification.
# There are several possible methods for defuzzification:
# Centroid: 'centroid'
# Bisector: 'bisector'
# Mean of Maximum: 'mom'
# Min of Maximum: 'som'
# Max of Maximum: 'lom'

vul = ctrl.Consequent(np.arange(0, 1.1, 0.05), 'vul', defuzzify_method='centroid')

# Custom membership functions for stream distance (in meters)
strmdist['low'] = fuzz.trimf(strmdist.universe, [0, 0, 150])
strmdist['medium'] = fuzz.trimf(strmdist.universe, [50, 150, 300])
strmdist['medhigh'] = fuzz.trimf(strmdist.universe, [200, 300, 450])
strmdist['high'] = fuzz.trimf(strmdist.universe, [350, 450, 550])
strmdist['vryhigh'] = fuzz.trimf(strmdist.universe, [450, 600, 750])

# Custom membership functions for Strahler Order
strmord['low'] = fuzz.trimf(strmord.universe,[1, 2, 3])
strmord['medium'] = fuzz.trimf(strmord.universe,[2, 3, 4])
strmord['medhigh'] = fuzz.trimf(strmord.universe, [3, 4, 5])
strmord['high'] = fuzz.trimf(strmord.universe, [4, 5, 6])
strmord['vryhigh'] = fuzz.sigmf(strmord.universe, 5.5, 10)

# Custom membership functions for vulnerability (0 to 1)
vul['low'] = fuzz.trimf(vul.universe, [0, 0.15, 0.30])
vul['medium'] = fuzz.trimf(vul.universe, [0.15, .30, 0.5])
vul['medhigh'] = fuzz.trimf(vul.universe, [0.3, .5, 0.65])
vul['high'] = fuzz.trimf(vul.universe, [0.5, 0.65, 0.8])
vul['vryhigh'] = fuzz.trimf(vul.universe, [0.65, 0.80, 1.0])

Membership Functions:

# The Rules
# IF the distance is low and the strahler order is high, THEN the vulnerability will be high vul.
# IF the distance is low and the order is low THEN the vulnerability will be low.

rule1 = ctrl.Rule(strmdist['low']|strmord['low'], vul['low'])
rule2 = ctrl.Rule(strmdist['medium']|strmord['low'], vul['low'])
rule3 = ctrl.Rule(strmdist['medhigh']|strmord['low'], vul['low'])
rule4 = ctrl.Rule(strmdist['high']|strmord['low'], vul['low'])
rule5 = ctrl.Rule(strmdist['vryhigh']|strmord['low'], vul['low'])

rule6 = ctrl.Rule(strmdist['low']|strmord['medium'], vul['vryhigh'])
rule7 = ctrl.Rule(strmdist['medium']|strmord['medium'], vul['high'])
rule8 = ctrl.Rule(strmdist['medhigh']|strmord['medium'], vul['low'])
rule9 = ctrl.Rule(strmdist['high']|strmord['medium'], vul['low'])
rule10 = ctrl.Rule(strmdist['vryhigh']|strmord['medium'], vul['low'])

rule11 = ctrl.Rule(strmdist['low']|strmord['medhigh'], vul['vryhigh'])
rule12 = ctrl.Rule(strmdist['medium']|strmord['medhigh'], vul['high'])
rule13 = ctrl.Rule(strmdist['medhigh']|strmord['medhigh'], vul['low'])
rule14 = ctrl.Rule(strmdist['high']|strmord['medhigh'], vul['low'])
rule15 = ctrl.Rule(strmdist['vryhigh']|strmord['medhigh'], vul['low'])

rule16 = ctrl.Rule(strmdist['low']|strmord['high'], vul['vryhigh'])
rule17 = ctrl.Rule(strmdist['medium']|strmord['high'], vul['high'])
rule18 = ctrl.Rule(strmdist['medhigh']|strmord['high'], vul['medium'])
rule19 = ctrl.Rule(strmdist['high']|strmord['high'], vul['low'])
rule20 = ctrl.Rule(strmdist['vryhigh']|strmord['high'], vul['low'])

rule21 = ctrl.Rule(strmdist['low']|strmord['vryhigh'], vul['vryhigh'])
rule22 = ctrl.Rule(strmdist['medium']|strmord['vryhigh'], vul['high'])
rule23 = ctrl.Rule(strmdist['medhigh']|strmord['vryhigh'], vul['medium'])
rule24 = ctrl.Rule(strmdist['high']|strmord['vryhigh'], vul['low'])
rule25 = ctrl.Rule(strmdist['vryhigh']|strmord['vryhigh'], vul['low'])

# Creation of control system using the defined rules
vulnerability_ctrl = ctrl.ControlSystem([rule1, rule2, rule3, rule4, rule5, rule6, rule7, rule8, rule9, rule10,
    rule11, rule12, rule13, rule14, rule15, rule16, rule17, rule18, rule19, rule20, rule21, rule22, rule23, rule24, rule25])
    vulnerability = ctrl.ControlSystemSimulation(vulnerability_ctrl)

Test Cases:
# Testing of inputs
vulnerability.input['strmdist'] = some value
vulnerability.input['strmord'] = some value
vulnerability.compute()
print ('Vulerability Value: ', vulnerability.output['vul'])
vul.view(sim=vulnerability)

Some cases where the defuzzified values don't exactly align with the rules.  I used the centroid method for the following two cases:

When strmdist = 10 m and strmord = 1, the defuzzified value is ~0.50.  This should be a lower value.

When strmdist = 10 m and strmord = 5, the defuzzified value is ~0.50.  This should be a higher value.

When I repeat #1 & 2 using the 'max of maximum method (lom)', I get a high value for both.


text:
Linear Discriminant Analysis on a transformed space
Let $S$ be a finite subset of a $\mathbb{R}^k$ partitioned into $N$ subsets $S_1, \ldots, S_N$ and let $n_j = |S_j|$. The between-groups sum of squares of the partition is defined as
$$bSS(S_1,\ldots, S_N) = \sum_{j=1}^{N} n_j ||\mathbb{E}[S_j] - \mathbb{E}[S]||^2 $$
The within-group sum of squares of the each $S_j$ is defined as
$$SS(S_j) = \sum_{u \in S_j} ||u - \mathbb{E}[S_j]||^2$$
The sum of squares of the set $S$ is defined as
$$SS(S) = \sum_{v \in S} ||v - \mathbb{E}[S]||^2$$
It is, I think, well-known that
$$SS(S) = bSS(S_1,\ldots, S_n) +  \sum_{j=1}^{N} SS(S_j) $$
Both Linear Discriminant Analysis (LDA) and Partial Least Squares Discriminant Analysis (PLSDA) share a common initial part that aims at finiding a one-dimensional subspace $U$  of $\mathbb{R}^k$ such that the projection map $\pi_U$ of $\mathbb{R}^k$ onto $U$ maximizes the quantity
$$\frac{bSS(\pi_U(S_1),\ldots, \pi_U(S_n))}{\sum_{j=1}^{N} SS(\pi_U(S_j))}$$
The single vector $u$ that generates $U$ is understood as the direction of best separation for the sets $S_1,\ldots, S_n$ and represents a latent variable resulting from a linear combination of the original features (that are associated to the canonical basis of $\mathbb{R}^k$).
Suppose $S_1,\ldots, S_n$ are not linearly separable in $\mathbb{R}^k$ but there is a real vector space $W$ and a non-linear function $f: \mathbb{R}^k \to W$ (such as that computed by the hidden layers of a deep neural network) such that the images $f(S_1),\ldots, f(S_n)$ are linearly separable in $W$.
Suppose one performs LDA or PLSDA in $W$ and finds a one-dimensional subspace $U \subseteq W$ that represents the most important direction for the separation of $f(S_1),\ldots, f(S_n)$. The anti-image $f^{-1}(U)$ is not in general a subspace of $V$. Are there situations in which $f^{-1}(U)$ can be used to understand directions of best separation inside the original input space $\mathbb{R}^k$ ?


text:
Method of encoding vector graphic data for a GAN
I am working with vector data that I want to train a GAN on. The vector data is a combination of the following forms of vector primitives:

Lines - format: [[x1, y1], [x2, y2]]
Circles - format: [[x0, y0, r]]
Solids - format: [[x1, y1], [x2, y2], ...]
Polylines - format: [[x1, y1], [x2, y2], ...]
Arcs - format: [[x0, y0, r, theta0, thetaf]]
Ellipses - format: [[x0, y0, r1, r2, ratio]]

My current approach to training my GAN is to separate the raw parameters of each element into a separate array, and then concatenate the arrays together. Thus, array[0][0] = [[0, 0], [5, 12]] might be the first line in the vector graphic, array[0][1] might be the first circle, etc. etc. There are numerous problems with this approach:

As vector drawings generally do not contain the same number of primitives of each type, the resulting array is a ragged list that must be padded with zeros in order to be made into rectangular tensors for PyTorch/TensorFlow, which changes the meaning of the data to the GAN
Unlike images, where pixels adjacent to each other are adjacent values in an array, and thus neural networks can find causal relationships between adjacent values, encoding vector graphics in this way means that a circle adjacent to a line in the drawing are placed in completely separate subarrays of the array, reducing the GAN's ability to train
While typical GANs working on images can produce workable results even while not fully trained, as human vision naturally interpolates and fills in low-resolution or noisy parts of images, my vector GAN will produce complete gibberish instead, as vector graphics does not allow for the same tolerance

I am certain there is a better way to encode my data, and I really don't want to take the easiest solution - rasterizing all my vectors to be images - because it would defeat the point of training on vector data. What other solutions are out there?


text:
Clarification regarding the training status of 'domain specific encoder' in stable diffusion
I am currently studying the paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et al. Specifically, I am focused on the section 3.3. Conditioning Mechanisms.
In this section, the authors introduce the cross-attention formulas in which $K, V$ are dependent on $\tau_\theta(y)$, with $y$ representing the conditioning information.
My question revolves around the training status of $\tau_\theta$. Is this 'domain specific encoder' pre-trained, or does it undergo training during the Diffusion Model (DM) process? The reason for my confusion is that $\theta$ seems to suggest a trainable parameter. If $\tau_\theta$ is indeed pre-trained, could you please explain why it is represented with the trainable parameter $\theta$?


text:
Small language model vocabulary correction
I plan to train a small language model (less than 4B parameters) that can run on x86 and handle vocabulary correction such as:
in the early days a king rolled the stake

which should be corrected to:
in the early days a king ruled the state

with an update rate of a few words every 1 sec. As in the example, my objective is to fix autocorrection errors and also "sound-alikes", for example from speech recognition errors. I've looked at huggingface.co and e-mailed several lightweight model paper authors, OpenAI, Stability AI, ThirdAI, etc, attempting to locate an existing model but so far no luck.
Is what I'm attempting do-able given the state-of-the-art ? Would I need to limit the vocabulary size and training text / paragraphs (and likewise limit inference input) to achieve this ?


text:
How to check clustering performance?
Background
I'm implementing the DBScan algorithm. I have trained it to cluster a small dataset of random clusters, and want to be able to get a decimal for its accuracy of clustering the groups.
Motivation
This is for some simple unittesting that checks it can cluster basic, separate classes (i.e. checking in my CI/CD that the accuracy is appropriately high).
The Problem
The problem is that it's outputs are its own cluster-indexes (the classes it's discovered/chosen). These do not align with the original integer labels I generated earlier.
An Example to Clear Things Up
Say, I have some original labels:
y_true = np.array([0, 1, 1, 0, 2, 1, 2])

and I have some cluster-predicted outputs
y_predicted = np.array([1, 0, 0, 1, 2, 0, 2])

You can see it has clustered values correctly, however they don't align with the original y_true array values. Therefore we can't use the normal accuracy function of np.mean(y_true == y_predicted).
The Challenge
Whilst preserving the outlier class of -1, how can I check that the model is significantly accurate, despite the fact that its generated classes do not align with original inputs. I understand that clustering algorithms, in practice are not used in this way but this is for testing that my implementation is proper, in my CI/CD automated testing.

After sorting y_true and y_pred and take the average of absolute difference between y_true[i], y_pred[i], assuming the cluster's have similar sizes i.e., len(y_true) == len(y_pred)?

Thank you, I'll try that now!

However, given that the clusters often have similar sizes, could this be a problem?@shaikmoeed
The numpy code I've used is np.mean(np.sort(y_true) == np.sort(y_prediction)) to give a value from 0 to 1

I have found that this doesn't quite work.
y2 = [2, 1, 0, 1, 2]```

The value ends up as **0.6** rather than **1** as it should be here.

Yeah, that's true. Your requirement seems to be close with adjusted_rand_score.

It's an appropriate metric, however I can't find many mathematical explanations of it. Do you know of any good resources? Also, is there any way to get an accuracy metric from 0 to 1?


text:
Understanding the decreasing influence of text embedding in Text-to-Image diffusion models: A Mathematical perspective
I've been reading the paper titled eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers by Yogesh Balaji et. al. Consider the following excerpt from the abstract of the paper

Large-scale diffusion-based generative models have led to
breakthroughs in text-conditioned high-resolution image synthesis,
demonstrating complex text comprehension and outstanding zero-shot
generalization. Starting from random noise, such text-to-image
diffusion models gradually synthesize images in an iterative fashion
while conditioning on text prompts. We find that their synthesis
behavior qualitatively changes throughout this process: Early in
sampling, generation strongly relies on the text prompt to generate
text-aligned content, while later, the text conditioning is almost
entirely ignored, and the task changes to producing outputs of high
visual fidelity.

Afaik, the text embedding always play a role in the denoising process. So, how does the role or influence of the embedding decrease over time while sampling an image from diffusion model? Can someone provide mathematical explanation for understanding the decrease of the influence of text embedding?

"theory will only take you so far"

I think that it's only backed by intuition... if you strongly guide the model using text, the text will highly impact the generation, limiting the "creativity" of the model... in addition to this, small change to the text, which might me insignificant for humans, will strongly impact the generation... instead "conditioning" your generation only at the beginning, will leave the model enough freedom to steer towards that conditioning to generate good images


text:
Implementing momentum is causing calculation exceptions/errors
I am developing my own neural network in order to learn about how they work. I am implementing via C++ and the Eigen library (for matrix multiplication). I have a working implementation that seems to train on MNIST quite nicely.
The next thing I wanted to implement was momentum. I believe that I have implemented the algorithm here but when I run (with momentum factor = 0.9) after a couple of seconds (something like 100 minibatches on MNIST) Eigen spits out a NaN error on the feedforward.
I have looked at the weight values at the point just before the NaN error spits out and some of the values look very big and very small so I am thinking that it is generating something like an INF value which then causes a NaN? For matrix multiplication, is the only thing which generates NaN an INF result?
Thinking about it more, doesn't the momentum formula lead to effectively summing all of the previous gradients?
Edit: I realise exactly the same question has been asked (but not answered) here.

please post the c++ code... also, please use a very basic model, like linear regression, so that you know it's not some weird exploding gradient problem

Please upload the code. Besides that, if there is a implementation problem, please upload it to StackOverflow.


text:
What is the motivation of VQ-VAE?
I don't understand the motivation because I read that the motivation for VAE was that: 'it could be shown that it is not meaningful to interpolate the latent space of regular auto-encoders' (my own paraphrasing) and apparently VAEs were meant to fix that.
But now with VQ-VAEs it seems once again that it is not meaningful to interpolate the latent space? Please correct me if I'm wrong but exactly what benefits does VQ-VAE have over regular auto-encoders?

empirically speaking, vector quantization it has been shown to be very helpful in deep learning in many scenarios.. it's not as clear why is that, but my opinion is because you are removing unnecessary noise... at the end of the day, your aim is to create a generative model, not a interpolation model that can pass from any image to any other image, so if you are only interested in generation, you are better off optimizing only for that


text:
Sparse linear discriminant analysis for regression problem?
So far, Linear Discriminant Analysis has beed used for classification problems http://proceedings.mlr.press/v38/wu15.pdf . I wonder if there are any ways to adapt it to regression problems?


text:
Training Issue in Solving Multi-Dimensional Multiple Knapsack Problem with Transformer Model and PPO and SAC algorithm
I'm reaching out to the brilliant minds of the AI community to seek help with a challenging issue in my project on solving the multi-dimensional multiple knapsack problem using a transformer model. As part of my master's thesis focusing on resource management in cloud computing, this research aims to contribute to the advancement of cloud computing resource management techniques.
In this GitHub repository, I have implemented two variants of Proximal Policy Optimization (PPO) and one variant of Discrete Soft Actor Critic (SAC) to train a transformer model. The goal is to effectively highlight the object-knapsack connections during various stages of the problem-solving process.
However, I'm encountering a significant obstacle in the training process that doesn't yield the desired performance. Its sims model doesn't train after more than 100000 times.
Further investigation has led us to suspect that the lack of a specific sequence in the decoder input has resulted in a significant amount of variation, thereby weakening the model's output. However, there may be other reasons that we have yet to discover. We are uncertain about how to overcome this challenge and would appreciate collaborating with others to find a solution.
I would greatly appreciate any assistance, ideas, or suggestions that could shed light on this matter. If you have experience or expertise in reinforcement learning algorithms, transformer models, or solving optimization problems, please take a look at my project and help me find a solution. Together, let's tackle this challenge and push the boundaries of cloud computing resource management.
Feel free to explore the GitHub repository to learn more about the project.
https://github.com/ImMohammadHosseini/MKP-RL


text:
Multi-instance learning for time-spatio-dependent data
I am trying to use MIL approach from the paper Attention-based Deep Multiple Instance Learning on the data that is a frames of human pose images acquired on different angle on each timepoint (temporal dependency). Bag (sample) is a tensor with size [batch, frames, channel, height, weight], each instance (frame) of the bag could contain one type of image error (pose error) (11 classes of errors). I want to classify type of the error that bag have and then evaluate which of the frames was assigned as one containing errors.
From the original paper in chapter 2.1, they stated that "In the MIL setting the bag probability theta(x) must be permutation-invariant since we assume neither ordering nor dependency of instances within a bag."
I am wondering, if I assume that instances of a bag could be dependent, and bag is not permutation-invariant. Could I still use MIL approach in this case?


text:
How exactly to create voice audio snippets that blend together into an AI voice?
I just asked the more general question, How to create AI voice generator for fantasy language? Now after asking ChatGPT for some details on how that works, I am concerned about how you would go about creating the "database" of sound snippets recording someones voice (like me recording my voice). How does that work exactly, what are all the pieces required to make a great sounding voice like Siri or Google's Map voice? Maybe it doesn't have to be that perfect, but something better than a simple old-school voice generator that sounds mechanical.
I am wondering for a "fantasy" language. I have 27 letters, which each represent a single phoneme (22 consonants, 5 vowels). I don't have every rule written down on which letters can go next to each other to form consonant clusters (there are no diphthong or "vowel clusters" allowed), but maybe I can put a list together for that, if necessary.
ChatGPT beats around the bush when it comes to explaining how exactly I should create sound recordings, so I wanted to clarify how exactly that works in this question:

Do I need to record each sound individually, and as part of a cluster/syllable? How many variants do I need to create for each sound, so it sounds good in the end? Like is 2 versions for each sound enough? If I have at least 100 consonant clusters allowed, and 2000 possible syllables, what am I looking at roughly speaking in terms of the number of recordings I need?
How do I "splice" up the recordings? Do I do it in some automated way or is this part all manual? I don't know how to efficiently splice audio, so I would need someone else to help me with that, so what should I provide them? And what should the end result of splicing be? (I'm not asking "how to splice audio", that is easy for someone skilled, I am basically asking what I should splice into). And how do I label the splicings? If there are going to be required 1000's of splices, is there a way to automate this in some ways?
Now say I have labelled splices. How do I weave them together, and blend them so they sound natural? Is that where the AI training comes in? What do I need to be aware of and try to implement at this stage?
Given I have a way to blend/weave the sounds together, how do I go from text to audio? (Given my letters each mean one sound). If there are multiple variants of each sound (which seems like a lot of work), then how does it select which variant, etc..?

Just looking for some high-level details on the "audio snippet generation" part. Don't need to know actual code, as I suspect I will have to write my own code for this, but knowing what I need to do in detail is the first step.


text:
can I finetune a model on Azure for information extraction based on "question", "context", and "answer" training data?
I am working on extracting certain fields from a large corpus.
I was looking at finetuning an LLM on Azure for the task. I think finetuning is the right idea (as opposed to vector databases, or RAG), since the data was OCR'd from semi-structured forms, and has been tokenized into a less-than-ideal natural language structure.
So I have a dataset in this form:
{"prompt": "find the date in this text", context: "chickens can't fly, blue red green, asdkjfek, Nov 11 2011", "completion": ""Nov 11 2022"}

I want zero generation or creativity in this task -- merely extraction.
For finetuning in Azure, the recommended structure is this:
{"prompt": "<prompt text>", "completion": "<ideal generated text>"}

I'm not sure finetuning will be effective if I include context with my prompt. I'd basically be doing "in-context finetuning", so to speak. Not sure that's a thing. I've done a lot of [reading], yet still not sure if it makes sense.
Any advice greatly appreciated.


text:
Multi Output Regression but num of objects to predict vary per sample
so recently I came across a problem of predicting the positions of objects from a pulse wave. My biggest concern here is that for each data sample, the number of objects varies. I know that this should be treated as a Multi-output regression problem, but how do I cater to the variation in the number of objects per sample? Also, what would fare better, XGboost or some neural network architecture?

look for objects detection papers in the CV literature, this was one of the early problems they had


text:
Is it possible to combine SGD with an unsupervised learning approach effectively
Before I undertake quite a large project I would like to clarify whether my idea for training a multi-layer neural network will work. I plan to make an AI that can land a rocket from randomly generated scenarios (altitude, velocity, rotation, wind etc). I have both a rocket simulation and a back-propagating neural network which I have coded from scratch in C++. Of course the network works for standard supervised learning scenarios such as regression - however I will need to adapt it for such a project.
My plan is to calculate errors for the output layer via a brute force approach of: increase  the weighted sum by some dx and determine the change to the cost through running the scenario (which in this case looks something like this C(finalRot, finalVx, finalVy, time)  ), from this I will calculate the error in each neuron in the output layer - and rather than do this for the other layers I will back-propagate this error - saving a huge amount of time on calculations - and then optimise parameters conventionally as you would with a supervised model.
Before I implement this, does anyone know if such an approach is viable?

just look at RL, this is literally its application


text:
What if in DPR (dense passage retrieval), the answer belongs to more than one passage?
In the DPR paper
the dataset is expected to be in this format D = {<qi, pi+, pi,1-, ... >}
With only one positive passage, but it is possible that the question requires an answer that spans knowledge from more than one positive passage.
Another issue is that, if we consider other positive answers from the batch as negative, (the Gold strategy in the paper) it is possible that those other positive passages are relevant for the question into consideration.
How should the DPR model be trained in such a case?


text:
Any research in "probe-tuning" of LLMs?
Is there any research in "probe-tuning" of LLMs, i.e., tuning LLM's parameter weights such that a specific probe (classifier) is more reliably detecting certain markers throughout the context, such as grammatical errors, aggression, manipulation, certain political bias, etc.?
This is different from classical fine-tuning and RLHF. As well as classical fine-tuning, probe-tuning is a supervised ML method: it is based on human-annotated texts (contexts). However, probe-tuning should be more effective than classical fine-tuning for detecting many occurrences of a certain marker throughout the context. Probe-tuning doesn't train on LLM's own "original rollouts" at all, only on LLM's activations during the context pass through the LLM.
I imagine than before doing actual probe-tuning, first we should determine which probe in the LLM is most aligned to the training data (annotations) already, so that probe-tuning likely just attenuates some vaguely existing concept within the LLM.

how are you going to do probing on a variable length input (given that you want to catch high level problems)?

@AlbertoSinigaglia probe on activations within some layer of the NN, so the input is fixed-sized.

LLMs manage sequences, pretty sure they don't have a fixed size

@AlbertoSinigaglia input of the probe, e.g. a particular layer of activations.


text:
Has anyone tried to derive linguistic information from GPT internals?
In Stephen Wolfram's write up on the workings of GPT, he suggests that chat GPT may have identified invariant rules of human language that haven't been formalized yet, ie new linguistic findings, and that a much simpler model taught these rules could perform simiarly:

What does it take to produce “meaningful human language”? In the past, we might have assumed it could be nothing short of a human brain. But now we know it can be done quite respectably by the neural net of ChatGPT. Still, maybe that’s as far as we can go, and there’ll be nothing simpler—or more human understandable—that will work. But my strong suspicion is that the success of ChatGPT implicitly reveals an important “scientific” fact: that there’s actually a lot more structure and simplicity to meaningful human language than we ever knew—and that in the end there may be even fairly simple rules that describe how such language can be put together.

It sounds like in the ~96 layers of processing in GPT, some portion of it might be
doing general language processing and some later portion might be involved in knowledge storage/access. I've also heard that sometimes eg in OCR, we can inspect the intermediary layers of the neural net and begin to guess what features each part is looking for.
So, I'm wondering if anyone has tried to inspect the layers of a publicly available chat model eg Llama and see what the intermediary processing stages are doing, or if it is too complex to be understandable in any respect.


text:
What types of econometrics model is necessary for the tittles: determinants of farmers preference of species for smallholder homegarden agroforestry
What types of econometrics model is required for the tittle; determinants of farmers preferences of species in smallholder homegarden agroforestry


text:
Create samples out of documents for Causal Language Modelling
I want to create an input source for Causal Language model using Llama 2 model in hugging face. I have a set of documents which are scraped from a specific website and want to fine-tune on them. Each document its basically a different corner of this domain. Some documents can be very short while others (such as term and condition) large enough. The input size for Llama 2 model is 4096 tokens and only 2% of the documents are above this threshold.
I've seen that the most common strategy for language modelling using this tool is to concatenate context and split by eos token such as :
***samples 1***: sentence 1 <eos> sentence 2 <eos> .... sentence n <eos> <pad as necessary> ***samples 2***: sentence n+1 <eos> sentence n+2 <eos> .... sentence n+j <eos> <pad as necessary>
Where
Document 1: sentence 1. sentence 2...sentence i
Document 2: sentence i+1. sentence i + 2...sentence i+k
Document 3: sentence i+k+1. sentence i +k+ 2...sentence n

and so on
A sample can have sentences from different documents/topics and each sample size is up to the initial 4096 tokens
So, I've got the following questions:

Should I split documents into different samples (and maybe decrease the total input token size down for 4096) or is it just alright to include multiple sentences concatenated with multiple EOS tokens in a single training sequence?
The  token, should i use to split context of different documents or should I use as in the example above marking the end of each distinct sentence?
Can someone provide a heuristic on how to select the chucking parameter used by the model to split the input texts above?


text:
GNN Training loop for node regression
I am currently working with the PYG library on a graph neural network project focused on node regression. In order to train this network, I generate my own collection of graphs using a "for loop." Within this loop, I create specific data objects including node features, edge indices, and target values. These data objects are then compiled into a list, which becomes my training set. To manage the data efficiently, I utilize a DataLoader.
The core architecture of my GNN model is a Graph Convolutional Network (GCN). For the training process, I am struggling with the question of how to structure the "for loop." Specifically, I am uncertain about whether I should use batches and how best to feed the graphs into the model(should I feed the model one graph at a time? And how to code this). This aspect of the training loop is posing a challenge, and I am seeking guidance on the most effective approach.


text:
Distance functions used in clustering analysis
From what I've seen in clustering, distance is taken as a hyper parameter (which is to be selected) when inferring the relationships/clusters between points.  Examples of distances I've come across are Euclidean, taxicab, Mahalanorbis, and Minoowski.
What are some examples of highly-cited papers doing this, and is there any concrete empirical evidence supporting this practice?  And what non-Euclidean distance functions do they use?

supporting this practice what practice? hyper-parameters tuning?

No Turing the distance function as s hyperparameter


text:
story generation with strong restrictions
I try to write a story with only a large subset of a given language, for instance, all French words not containing the letter 'a'.
The tools I tried mostly understood the request, writing a story with MOST words not containing "a", but still leaving a strong proportion of "bad words" (such as the word "bad" :).
From my understanding of story generation algorithms, it does not seem too difficult to modifiy the core algorithm to just assign probability 0 to words not containing the letter 'a', but I don't know these tools well enough to see if one of them gives this possibility to the user.
Would you know such a tool able to follow perfectly such a strong restriction? I emphasize that I am not interested in writing a story without the letter "a", this is just an example, I just want to be able to feed a (very large) set of words, say a few thousands, and the algorithm should write a story with just those words.
Thanks in advance


text:
Interpreting TensorFlow playground simple neural network

The hidden layer neuron output images are meaningful to me. But I don't understand how the model can output any number except zero for the III quadrant when all hidden layer neurons output 0 for any point in the III quadrant. I have read the TensorFlow playground documentation, but I didn't find anything helpful. I came up with a thought about the output being mapped with the sigmoid function, but I am unsure about it.

they are probably hiding the bias


text:
Simple model to predict road traffic?

Given GPS tracking data of many trips belong to different drivers, different type of vehicles
Each GPC point recorded after 10 seconds
I can map matching the GPS data back to the roads sections
Right now I use simple calculation of GPS points density and calculate avg speed for a road section belong to what time of the day, what day of the week

But some road section, same time (say 18:00), same day but maybe very different speed, some road section just have few trips data. And I'm looking for a better way to calculate and predict speed of a (road section, datetime).

Can AI help to calculate the speed of road section that I already have data better?
Can AI help to predict the speed of road section I don't have historical data? Given assumption that speed of related road section can be related with the other.

I'm a total newbie and have zero knowledge of AI/ML, I hope you can give me more details. For example what kind of algorithm fit my data, if any. (And maybe a Google Colab with simple script for starting point).


text:
How to setup correctly a sequence generation task with RL/policy gradient learning?
I've a pretrained model for sequence generation that I'd like to improve with RL but there are several shady points.
So, I have the following model and loss function:
def policy_gradient_loss(y_true, y_pred):
    return -tf.reduce_mean(tf.math.log(y_pred) * y_true)

embedding_layer = Embedding(input_dim=vocab_size, output_dim=128)(input_layer)
lstm_layer = LSTM(128)(embedding_layer)
output_layers = Dense(vocab_size, activation='softmax')(lstm_layer)
model = Model(inputs=input_layer, outputs=output_layers)
model.compile(loss=policy_gradient_loss, optimizer='adam')
model.load_weights(weights)

The states or items of the sequences are dependent on all the previous states (I think it's called higher order Markov chains).
For the RL training, I generate a batch of sentences. After the generation of a sentence has finished, I calculate a reward for it. The reward is a single float between -100 (worst) and 100 (best) and it values the whole sentence. After the generation of a batch of sentences has finished, every sentences are padded (with 0) to the length of the longest sentence. model.fit(...) is then called to update the network:
def _get_reward(sentence):
    # This much more complicate ofc, just to show the interval.
    return random.randint(-100, 100)

def _input_generator(batch_size):
    while True:
        x = [], y = []
        for _ in range(batch_size):
            sentence = generate_sentence()
            x.append(sentence)
            reward = _get_reward(sentence)
            y.append([reward])
        max_len = max(map(len, x))
        yield np.array([pad_sequences([sentence], maxlen=max_len, value=0)[0] for sentence in x]), np.array(y)

model.fit(x=_input_generator(batch_size))

My questions:

Is the used loss function and network setup suitable for this scenario?
Does the interval of the rewards direct the learning into the expected direction (-100 to avoid, +100 to prefer)? I've doubts since the log() of a number in the range (0, 1) is negative, multiplying it with -1 makes it positive, but if the reward is negative, then it will be negative again. And if this negative value is minimised, then the model will learn to generate the wrong results. I've read a similar question here: but the user provided the answer had also some doubts.
Should I use some kind of discounting on my rewards? If yes and IIUC then it means to assign a discounted reward value to every item of a generated sequence. But then, what to assign to the padding values or how to handle padding?
Is there any other issue with my approach that an experienced RL professional can spot immediately?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.


text:
Optimal Neural Network Structure for 'Crystal Rush': Single vs. Modular Networks?

Problem
I'm attempting to design a neural network for the bot programming game Crystal Rush. Given the game's complexity, I anticipate needing a vast neural network to manage bot movements, resource requests, and world perception. By large, I mean an input and output list smaller than the length of 50.
I have already created a simple neural network for a codingame puzzle called Code vs. Zombies with five sensors on the front and three in the back, with four output neurons to control movement. A replay of my neural net is here.
Game Rules (Brief Overview)

Bots start on the board's left edge and can move towards cells to collect ore.
Bots collect minerals and return them to the headquarters (left side) for points. They deposit their haul in the leftmost collum of the board.
Initially, bots can't see the board. They can deploy radars to unveil sections. Radars are placed by mining specific grid cells.
Bots can set mines by mining a cell. If another bot mines this cell, it explodes, destroying that bot. Mines are invisible to radars.
A bot can only hold one ore. After mining an ore, it has to go back to the base to deposit it.
For detailed rules, visit here.

Potential Approaches:

Decompose the Problem:
Should I break down the problem such that each neural network only commands a single bot, replicating this for all bots on the board?
Further Decomposition:
Should I segment the tasks for each bot across multiple neural networks (modular neural networks)? This could allow each network to specialize in a specific task, potentially speeding up training.

Question
Given the above, what's the best approach to designing a neural network for this game? How do I structure the neural network for optimal performance in this game? A structure that would give enough data to the NN for it to make good enough decisions without making the weight and biases list grow too large to get meaningful training in a manageable time frame


text:
Help defining environment with complex action space
I'm working on a personal MARL project with a high-dimensional and continuous action space. The environment is designed to give positive rewards to actions between some moving limits of the action range, and negative rewards to the actions outside of those limits. For example:

Global action range: (0, 1000)
Desired action range for first 100k steps: (0, 10)
Desired action range for 100-200k steps: (30, 40)
...

Therefore, the main challenge of the environment is that actions with positive rewards on certain stage of the environment would return negative rewards on the following stages.
How should I define the actions of the agent? I've tried the following methods without success:

Simply scale actions between 0 and 1000 and hope that agents learn the moving distribution of rewards
Transform actions to percent variations and scale actions over a non-observed moving average (I tried adding the moving average to the observations but the results stayed the same)
Observations do consider a dimension that serve to differentiate when a distributional shift happens
Also, I've tried using SAC and DDPPG to model agents

Feel free to share any comments or suggestions.

Not 100% clear whether your goal is to create adaptive continually learning agents, assessing them for ability to cope with rules changes? Or if your goal is to teach the agents to perform optimally on reruns of the entire environment, having learnt to anticipate the changes


text:
Algorithms for community discovery in multigraphs
In order to group unstructured or sem-structured texts for a timeline construction approach, I consider several types of correlations among such texts. These different correlations induce a weighted multigraph, where the texts act as nodes. All edges have a normalized weight between $0$ and $1$.
I wonder what algorithms exist to find community structures within such multigraphs. The problem of community discovery in static and even dynamic graphs without multiple edges between nodes is well-studied. See e.g. this article and this one for accounts on these respective subjects.
However, it appears that the community discovery problem on multigraphs has been studied considerably less extensively. I've found a few relevant articles on the topic, including this 2012 paper by Martschat, and this article by Tang et al. (2009) considers community discovery in multiple graphs with the same nodes.
Question: are there more recent community discovery algorithms that are described in the scientific literature, and include open-source code that allows one to implement the algorithm on a data set?

https://cs.stackexchange.com/q/161790/755


text:
Which meta-learning algorithms are well-suited for "many-shot learning" scenarios, where the target training set is large?
Much of the meta-learning literature deals with the few-shot learning problem of using data from a diverse set of "source" tasks (the meta-dataset) in order to train a model that can quickly learn how to solve a new, previously unseen "target" task. Here quickly means that the training set for the target task is very small (i.e., one with only a few labeled samples from).
I am facing a "many-shot" meta-learning problem, in which the main limitation is not the amount of training data for the target task, but rather very stringent accuracy requirements. If I'm fine-tuning a "vanilla" pre-trained model on my target task (one which was trained using ERM on the entire meta-dataset of source tasks), I need a target dataset of ~10M samples to reach the required accuracy specifications. My goal is to reach the same, or nearly the same, accuracy, with 1-2 orders of magnitude fewer labeled training samples (so ~100K or 1M target samples). If it helps, I also have a nearly unlimited supply of unlabeled samples from the target task. To achieve this goal, I have of course a (labeled) meta-dataset of many similar tasks.
In other words, while few-shot meta-learning algorithms deal (informally speaking) with the "maxmin" problem of first minimizing the target dataset size (number of shots) then maximizing accuracy, I'm facing the "minmax" problem of first maximizing the accuracy then minimizing the dataset size without hurting the performance too much.
I was not able to find any references that deal with this "many-shot meta-learning" scenario. So my questions:

Which meta-learning approaches would be well-suited to deal with this scenario?
Do you know of references to works that have studied this scenario?


text:
How to change network that works on a subset of the input?
To clarify my question, I'll take an example: let's say you're trying to classify pictures, by determining whether there's a dog or not in the picture. Let's assume the pictures to be 1000x1000 pixels. Let's also assume that the dogs are always located in a specific region of the pictures, in the training/validation/test set and in the real world-data the network is going to encounter. The rest of the picture is irrelevant to the classification task.
I know that an easy thing to do would be to only select this region and train a neural network on the cropped dataset. But if you don't know where this region is or whether that region even exists, you're going to train a network on the full dataset.
So I have a few question: first of all, would a good neural network on the full dataset find by itself the region of interest and consider the other parts of the pictures as noise? And secondly, if there's a network that's capable of having a good accuracy on the cropped dataset, how would you change it to have good accuracy on the full dataset?
I have been working on a similar problem (though in 1 dimension and not 2), but I can't find a network that performs better than random chance on the full dataset, even though I have found a network that performs better than random chance on the cropped dataset. Is noise that much of a problem for a network?

CNN segmenters are asked to find any dogs in a picture so they certainly can determine the region of interest.  They're not my specialty but there is often a network which decides which segments of the picture have something of interest and a second which classifies it (what you describe as the network which operates on the cropped dataset).

A network may or may not have an issue with noise surrounding data of interest.  But one thing is sure, it will likely need many examples of surrounding noise to learn it is to be ignored.


text:
actor critic gradient inconsistent with mathematical proofs
From different books and the courses (including Sutton and Barto, page 332 and cs229 notes, page 222, the gradient with respect to $\theta$ with baseline is proved as:
$$\bigtriangledown_\theta \eta(\theta)_a = E_{\tau \sim P_\theta}[ \bigtriangledown_\theta \sum_{t=0}^{T-1} log\pi_\theta(a_t | s_t).(G(s_{t}, a_{t}) - V^{\pi}_{\theta}(s_{t}))]$$
But many implemnetations, including the one on the tensorflow documentations, define the gradient as:
$$\bigtriangledown_\theta \eta(\theta)_b =  E_{\tau \sim P_\theta} [\bigtriangledown_\theta[\sum^{T-1}_{t=0} \log\pi_{\theta}(a_{t} | s_{t}).(G(s_{t}, a_{t})  - V^{\pi}_{\theta}(s_{t}))]]$$
That means they move the term $G(s_{t}, a_{t})  - V^{\pi}_{\theta}(s_{t})$ inside the gradient.
But clearly, $\bigtriangledown_\theta \eta(\theta)_a \neq \bigtriangledown_\theta \eta(\theta)_b$, because:
$$E_{\tau \sim P_\theta} [\bigtriangledown_\theta[ \sum^{T-1}_{t=0} \log\pi_{\theta}(a_{t} | s_{t}) V^{\pi}_{\theta}(s_{t})]] \stackrel{product rule}{=} E_{\tau \sim P_\theta} [\sum^{T-1}_{t=0} \log\pi_{\theta}(a_{t} | s_{t}).\bigtriangledown_\theta V^{\pi}_{\theta}(s_{t})] + E_{\tau \sim P_\theta} [ \sum^{T-1}_{t=0} \bigtriangledown_\theta\log\pi_{\theta}(a_{t} | s_{t}) .V^{\pi}_{\theta}(s_{t})] = E_{\tau \sim P_\theta} [\sum^{T-1}_{t=0} \log\pi_{\theta}(a_{t} | s_{t})\bigtriangledown_\theta V^{\pi}_{\theta}(s_{t})] + 0 = E_{\tau \sim P_\theta} [\sum^{T-1}_{t=0} \log\pi_{\theta}(a_{t} | s_{t})\bigtriangledown_\theta V^{\pi}_{\theta}(s_{t})] \neq 0 $$
Did I do something wrong? If not, with what justification people use $\triangledown_\theta \eta(\theta)_b$ instead of $\triangledown_\theta \eta(\theta)_a$?


text:
NN architecture for a sparse pixe-wise image2image mapping

Hi,
I have pairs of images like below (inputs and labels), they are both single channel images with real-valued pixels. I want to train a CNN that maps from one to the other. I have tried UNet and variations of UNet, and it somehow works okay but I get very smoothed outputs (as you can see the target has discrete individual pixels). I have tried using MSE loss and when I switch to L1Loss the loss doesn’t decrease. Also I’ve explored other options such as (SSIM, perceptual loss etc.) but none worked.
Maybe I should explore different architecture.
My questions now are:

What kind of architecture would be suitable for this “simple” 1-to-1 pixel-wise mapping problem?
What loss function to use?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.


text:
Backpropagation in a transformer
I have a transformer for timeseries forcasting based on this article https://arxiv.org/abs/2001.08317
Given a source containing $src=(x_{t-5},x_{t-4},x_{t-3},x_{t-2},x_{t-1})$ and a target of $tgt=(x_{t-1},x_{t},x_{t+1})$, the model then tries to output $out=(x_t,x_{t+1},x_{t+2})$.
When training, I currently provide the model with the $src$ and $tgt$ to produce $out$, which I then compare with real data using the LMSE metric. This approach seems to have limitted success.
However, similar to the NLP tasks, I want to first input $src$ and $tgt=(x_{t-1})$ to the model. Then, I could measure the loss and backpropagate before trying training again on $src$ and $tgt=(x_{t-1},x_{t})$ and continue like this until the model predicts the required length of the output.
Alternatively, I could do the same thing, but backpropagate only in the end.
Which of these strategies is better? Is there an alternative that comes to mind?

I'd recommend that you put your specific question in the title.


text:
What is the best type of input for a 3D UNet?
I want to use 3D U-Net (or similar) network to create a 3D reconstruction of my microscopy data.
The original paper for the 3D U-Net (https://arxiv.org/abs/1606.06650) describes the implementation with orthogonal slices (xy, yz, zx planes) of the dataset used for training, which makes a lot of sense as the objective is to segment 3D objects in the data.
However, most implementations that I have come across online (github) use slices from the same plane (xy) for training. I am actually confused about how this works at all because it is not defining an object in its completeness. Can someone explain how this type of input works for 3D segmentation and if it can provide results comparable to the former?
(I understand that a lot of biological image acquisition is done in the form of continuous 2D slices in one plane across a z-stack and thus it might be more convenient to use the 2D slices as they are, but there are methods still to get the orthogonal slices from that too, right?)


text:
Does Embeddings and Vector Databases solve the need of having longer context windows?
I am learning to use the OpenAI API to build LLM-based agents. I recently came across the concept of vector databases, which use embeddings to convert text into vectors and store them in a database for easy retrieval. This technique has been shown to be very useful for long-memory applications.
My question is whether it is necessary to have longer context windows in models that use the embedding + database technique. I know that there has been some research on expanding the context window of transformers, such as the Scaling Transformer to 1M tokens and beyond with RMT paper and Microsoft's LongNet architecture. At what point does increasing the context window lead to better performance than using a vector database approach? Are there any examples or experiments that demonstrate this?


text:
Mechanism of Prediction Readjustment in Supervised Learning and Role of Self-Attention in Sequence Data Relationships
In supervised learning, when the prediction deviates significantly from the expectation, how does it "readjust"?
And... LLMs are a subset of deep learning, just as generative AIs are.
Is the use of neural networks constant in all categories? At any point in the context of artificial intelligence, do we not use neural networks?
And Does the concept of self-attention introduced in the paper "Attention is all you need" give it the power to create relationships within sequence data? Wasn't this already being done? (Why was the paper revolutionary in this aspect?)
I need help


text:
Neural Network (NN) to predict the duration (in seconds) of a fault, how to recognize the elapsed time in prediction of an active fault?
I'm working on developing a neural network (NN) to predict the duration (in seconds) of a fault. However, I've encountered a couple of challenges:
Model Performance: My neural network seems to be predicting near the average fault duration, and the loss is currently high (over 10,000). For instance,
with previous data:
Fault type: xyz
Duration (y in seconds): 180
In real-life scenarios:
Fault type: xzas
Duration (y): Needs prediction
Active Fault Constraint: In the next phase, I want the model to consider an ongoing fault with its elapsed duration. The prediction should be greater than the elapsed time. For example, if a fault has been active for 50 seconds, the model's prediction should be more than 50 seconds for the total duration.
Could you provide guidance on addressing these issues? Specifically, how can I improve the model's accuracy and incorporate the constraint for active faults?


text:
Can we use bounding box cropping to avoid shortcut learning (achieve explainable AI)?
Deep neural networks sometime use shortcut features (pseudo correlation) to predict.
For example, in cat-dog classification, the network may use the background information (e.g. floor, grass) as a shortcut. In lung cancer classification, the network may use features beyond the lung region (see Fig-2).
Such behaviors make the prediction unreliable (and not explainable to human). Hence, in shortcut learning, people study various techniques to avoid shortcut features.
Here, my question is: Why not we just use a detection model (e.g. FastRCNN) to first detect the object (e.g. cat/dog, cancer) from the image, then predict the label based on the cropped image (where the background is masked)?

This naive method must have some problems, otherwise it is not necessary to develop shortcut learning/robust learning papers.
From this post, I find some possible answers:

training two networks (detection, classification) is costly in time.

the detection model may generate mistaken bounding box.

However, I feel these answers are more from a technical aspect. None of them points out the intrinsic problem of the naive method.
Any suggestion, reference paper, etc. are welcomed.

Well, to do so you need to be aware that the background is problematic: there could be some other kind of images for which is not easy to say which "high-level features" could cause a shortcut. Also the bounding-box may be imprecise and contain irrelevant information: to this end a segmentation mask may be better.


text:
Where to find all the documentation regarding StableDiffusion's special prompts (like mixing two persons)
I am currently playing on ClipDrop.co with Stable Diffusion, and I am confused with the tutorials I can find online.
For instance, for mixing two celebrities in one pics, I read all the following syntaxes:

[Person A:Person B:0.5], where 0.5 is the multiplier to determine the number of steps before switching to Person B.
(Person A:Person B)
(Person A|Person B)

I've tried multiple ratio, switching A and B, and played with all these syntaxes, but I have no way of really quantifying the differences I can notice in terms of distance to Person A and Person B of the result.
So I start to wonder if the syntax matters and where to find an official manual.

I went through some source code for one of the servers earlier this year and it almost looked that it was turning it into an array an encoding each word before doing inference. So I think a lot of the syntax is just sugar for turning it into an array.

The specific one I used seemed to want something like (person:1.2) (that is it wanted floats, not ints).


text:
FastGAN implementation has redundant SpectralNorm followed by BatchNorm?
I am implementing a version of FastGAN, and it seems like (see the official repo) they use a Spectral norm directly followed by a Batch norm. Doesn't the spectral norm get fully cancelled by the following Batch Norm?
I believe it does, and have coded up a simple demonstration, could someone please confirm I am not missing something basic and that indeed the authors made a mistake?
The reasoning

Spectral normalization takes a linear operator $O$ and rescales it by it's inverse spectral norm $1/\sigma(O)$
Batch norm (without learnable $\beta,\gamma$) effectively shifts and scales the operator after which it is applied: $$\text{BN}(O(x)) = \frac{O(x) - E[O(x)]}{\text{std}(O(x))}$$

so $\text{BN}(\alpha O(x)) = \text{BN}(O(x))$ for any inputs $x$ and factor $\alpha$

My simple demonstration:
# BatchNormed Convolution
conv = nn.Conv2d(3,1,3,1,0, bias=False)
c_dict = dict(conv.named_parameters())
c_dict["weight"].data = torch.ones_like(c_dict["weight"].data)
bn1 = nn.BatchNorm2d(1, affine=False)

# BatchNormed SpectralNormed
conv2 = nn.Conv2d(3,1,3,1,0, bias=False)
c_dict = dict(conv2.named_parameters())
c_dict["weight"].data = torch.ones_like(c_dict["weight"].data)

# Spectral Normalize the 2nd conv
sn_conv = spectral_norm(conv2)

bn2 = nn.BatchNorm2d(1, affine=False)

# Now feed an image into both conv & sn_conv
im = torch.randn(size=(2,3,4,4))
cim = conv(im)
sncim = sn_conv(im)

bn1.train()
bn2.train()

# Feed the conv & sn_conv outputs through their batchnorm 1000 times
for i in range(1000):
    cimbn = bn1(cim)
    sncimbn = bn2(sncim)

display(cimbn)
display(sncimbn)
>>> Gives the same result

I believe the only effective difference between the two (BN(SN(Conv)) & BN(Conv)) is that they will get initialized to slightly different functions?


text:
Why does each row of data have the same bottleneck features in the Autoencoder after training?
I was training an autoencoder for anomaly detection and I wish to extract the bottleneck features of the encoder for K-NN. The model architecture is as such:
class Autoencoder(torch.nn.Module):
   def __init__(self):
      super().__init__()

      self.encoder = torch.nn.Sequential(
         torch.nn.Linear(curr_col_count, 300),
         torch.nn.ReLU(),
         torch.nn.Linear(300, 150),
         torch.nn.ReLU(),
         torch.nn.Linear(150, 75),
         torch.nn.ReLU(),
         torch.nn.Linear(75, 50),
         # torch.nn.ReLU(),
         # torch.nn.Linear(125, 50)
      )

      self.decoder = torch.nn.Sequential(
         torch.nn.Linear(50, 75),
         torch.nn.ReLU(),
         torch.nn.Linear(75, 150),
         torch.nn.ReLU(),
         torch.nn.Linear(150, 300),
         torch.nn.ReLU(),
         # torch.nn.Linear(500, 1000),
         # torch.nn.ReLU(),
         torch.nn.Linear(300, curr_col_count),
         torch.nn.Tanh()
      )

   def forward(self, x):
      encoded = self.encoder(x)
      decoded = self.decoder(encoded)
      return decoded

and I am using the Pytorch feature extractor as such:
import torchvision as tv
import torchextractor as tx

model = Autoencoder()
model = tx.Extractor(
    model,
    ["encoder.6"]
)
loss_function = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr = 1e-4)

I trained the model with the following code:
epochs = 100
training_losses = []

for epoch in range(epochs):
   for row in concur_tensorloader:
      reconstructed, features = model(row)

      loss = loss_function(reconstructed, row)

      optimizer.zero_grad()

      # Backpropagation
      loss.backward()
      optimizer.step()

   training_losses.append(loss)

And extracted the deep features for each data point after training as such:
deep_features = []
for i in concur_tensorloader:
    rec, bottleneck_features = model(i)
    deep_features.append(bottleneck_features)

Before training the model, I retrieved the bottleneck features for each row of data and every row had different values. However, after training the model for just 1 epoch, I realized that every row had the same values for the bottleneck features. Why is this the case?

Do you see the same thing after training a few more epochs?

Yes, i see the same thing with 30 epochs

Are the two snippets from the same file, so you are training and attempting some analysis of the extracted features in the same run? That should be fine, but you should stil show how you are extracting the features with the lines of code that perform that (currently you only show that you are using torchextractor). Also that would appear to be in two places (before and after training with only one of them failing), so please show both

Well, first one needs to clarify what one means by "bottleneck features". What do you mean by that? I don't see anything in your code that prints or plots any features. Yes, I see tx.Extractor, but nothing is being done after that.  Can you provide more info about your data you're training the model with? Dimensions, etc

Made the necessary edits! Thank you everyone

I still dont understand what you mean by 'bottleneck features'. Please assume that, for somewhat lesser trivial definitions, nobody knows what exactly you mean. 'Bottleneck features' is not a trivial concept, so please define exactly what you mean by it.

By bottleneck features, I mean the feature variables at the last layer of the encoder network


text:
Temporally Non-Aware RNN
I am trying to classify whether or not a specific object is in panoramic photos. The issue is, a panoramic photo can be any width, so the input to my neural network can't be fixed in that dimension.
I've been using RNNs (QRNNs to be specific, as I am trying to reduce the number of parameters as much as possible), but they always learn where the object usually is in the image, and then have a really hard time classifying an image with the object in a different place.
I'm looking for something similar to CNNs, where it doesn't have a spatial dependance (or in this case, a temporal dependance?), but it can't have a fixed input width.
Any ideas?

pretty sure you can just use a bunch Conv layers, then do something like a global average pooling and some dense layer for the classification

As I said, I don't know the input size of my network, so conv layers are out.

You are probably missing something on how convolution works, as they do not at all rely on input size (width and height wise), you can definitely train convolutional layers without any knowledge of the input size of each image

CNNs cannot be trained with variable input sizes unless you use padding, which won't work in my case.

Again, yes, you can, convolution does not depend on the width and heigh of your images, please try and you will see that it's indeed the case, of if you want to be more theoretical, check the definition of convolution, and you will see that they indeed can.

The output side of a convolutional layer is defined as follows: o = ((i + 2p - k - (k - 1)(d - 1)) / s) + 1, where o is the output size, i is the input size, p is the padding, k is the kernel size and d is the dilation. I don't know who told you that convolutions don't depend on your input size, but they are simply incorrect.

Listen, this is not an answer to your question, but it seems that you are missing the whole point of convolution.
Simplified explanation:
Convolution is just a weighted sum of the neighbors of a pixel
You see how this is not dependent on the size of the image?
Take a 3x3 filter, on a NxM image, apply convolution, and you will get a (N-2)x(M-2) image as output
Now take a TxS image, and apply the same filter over it, what you get (T-2)x(S-2)
you see now that you can apply a convolutional layer to any arbitrary sized image?
You still don't believe me? take this code, and you will see that you can input two images of different sizes to this neural network and it won't complain:
network = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, kernel_size=3, activation=tf.nn.leaky_relu, padding="SAME"),
    tf.keras.layers.Conv2D(32, kernel_size=3, activation=tf.nn.leaky_relu, padding="SAME"),
    tf.keras.layers.Conv2D(32, kernel_size=3, activation=tf.nn.leaky_relu, padding="SAME"),
    tf.keras.layers.Conv2D(32, kernel_size=3, activation=tf.nn.leaky_relu, padding="SAME"),
    tf.keras.layers.GlobalMaxPooling2D(),
    tf.keras.layers.Dense(NUM_CLASSES, activation="softmax"),
])

you're welcome :-)

You just proved me right. Applying a 3x3 kernel to a TxS image gives you a (T - 2)x(S - 2) output, which is the MATHEMATICAL DEFINITION of dependance. Regardless, this is not an answer to the question. Please stop.

This actually is an answer to the question (if not, it's not entirely clear what the real question is: indeed the title of the question doesn't seem to match the body of the question too clearly). @Ghull Yes, conv layers depend on the input size, in the sense that their output size depends on the input size. Bigger input size --> bigger output size. But that's great, that means they can handle different input sizes, which is exactly what you need. Just need to make sure to, after conv layers, have something (like pooling) that reduces everything to a single, fixed dim (num. classes).

@Ghull seems like somebody agrees with me, the number of parameters of convolution is not dependent of the input size, so it's said not to be "dependent from input" as a fully connected layer for example. You can perfectly use the code I provided to solve your problem without any problem

@DennisSoemers thank you


text:
Is mobile-phone speech-recognition geographically/dialect weighted?
When I pronounce the English word "sir" as it is pronounced here, which is American English (at least when I visit that page) my mobile phone thinks I'm saying the name "Sarah" when it is in dictation mode in the messaging app.
I am curious if speech-recognition algorithms used in the major mobile phones are geographically/dialect weighted. I know there are some English dialects where the final -r- sound is pretty much left off, and the duration of the word is much briefer than it is in most American English dialects.


text:
What is the detailed experimental setup for class-incremental continual image generation?
Do you condition the generative model (let's say, VAE) on the task identity or the class label or both? If I condition the VAE on both task identity and class label, then I have to provide both the two information to the VAE when testing, right?
Why I'm asking this is that the biggest difference for task-incremental and class-incremental in continual image classification is whether the model is aware of the task identity during test phase, then what is the detailed experimental setup for class-incremental continual image generation?


text:
How to train a machine learning model using reinforcement learning for Battleship to predict next best move?
Question
How can I train a machine learning model to predict best next move using reinforcement learning for Battleship?
(using reinforcement learning is the key of this question. I want it to learn the formulas for "best move from a statistical standpoint")
The best move is the one that minimized the number of turns until all ships are sunk.
I plan to use Tensorflow. From my understanding, this is how the model should look like for this problem:
# Each input neuron is a categorical variable:
# 0.00 < x <= 0.25: missed shot
# 0.25 < x <= 0.50: ship hit by shot
# 0.50 < x <= 0.75: sunk ship
# 0.75 < x <= 1.00: cell that has not been chosen/shot yet

tf.keras.models.Sequential([
    tf.keras.layers.Dense(5 * 5),  # input is a 5 x 5 board
    tf.keras.layers.Dense(4),  # hidden layer, can be any number

    # output is a 5 x 5 board,
    # each with a value 0.00 <= x <= 1.00
    # low values are bad or invalid moves,
    # and high values are good moves
    tf.keras.layers.Dense(5 * 5),
]

and in pseudocode for the whole algorithm:
# Creates a game board of size 5x5 with ships randomly placed in a valid configuration
game = setup_board(x=5, y=5)

until game.is_over():
    game.make_random_valid_move()

    # TODO: where and how should the reinforcement learning take place?

Statistically best move
From a statistical standpoint the best:

first move is anywhere not near edge
move when there is a hit, but unsunk, ship is an adjacent cell

Battleship rules (non-standard, simplified for StackExchange question)

Grid/board is 5 units by 5 units
Ships are randomly placed in horizontal or vertical orientation

Ships do not move across turns

Ships must have at least one unit of padding between ships
There are 1× 2-unit ship and 1× 3-unit ship

Example configurations

· denotes a missed shot
× denotes ship hit by shot
█ denotes sunk ship
  denotes cell that has not been chosen/shot yet

Empty board

Starting board for all games

░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░

First shot

Sub-optimal first move

░░░░░░░░░░░░░░░░░░░░░
░ · ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░

Optimal shot

Since there are two hits, and ships cannot be adjacent, then then two shots must be hitting the same ship

the next shot should either be to the left or right of the existing shots

░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░ × ░ × ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░

Choose left
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░ · ░ × ░ × ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░

Choose right
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░ · ░ · ░ · ░ · ░ · ░
░░░░░░░░░░░░░░░░░░░░░
░ · ░ █ ░ █ ░ █ ░ · ░
░░░░░░░░░░░░░░░░░░░░░
░ · ░ · ░ · ░ · ░ · ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░

the × becomes █ to denote that the ship is sunk
ships cannot be adjacent, so the adjacent cells are ruled out

Game over
░░░░░░░░░░░░░░░░░░░░░
░ █ ░ █ ░ · ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░
░ · ░ · ░ · ░ · ░ · ░
░░░░░░░░░░░░░░░░░░░░░
░ · ░ █ ░ █ ░ █ ░ · ░
░░░░░░░░░░░░░░░░░░░░░
░ · ░ · ░ · ░ · ░ · ░
░░░░░░░░░░░░░░░░░░░░░
░   ░   ░   ░   ░   ░
░░░░░░░░░░░░░░░░░░░░░

All ships have been sunk, so the game is over

Welcome to AI Stack Exchange. This could be the basis of an interesting question, but it looks like you need a full tutorial on how to understand and implement reinforcement learning. That's not really feasible, it would be too much. Plus resources for that already exist. I suggest find a tutorial that does something similar to your game, and attempt to continue your project that way. Ask a more focused question when you hit a specific block that stops you following the example or tutorial

As a simple suggestion, I think you can solve this problem in various ways. Probably the classic alpha-beta search would be enough to solve small grids and should be way simpler to implement and debug than RL. Also, you can try a tabular RL algo like policy iteration to avoid the burden of deep RL. Another option, would be a model-based RL approach like using monte carlo tree search (MCTS).


text:
Changing Gym Environment Mid Training
I am using a custom gym environment for a research project. As my agent solves the environment, I want the task to get progressively harder. Right now, I am doing that like so:
if correct_solved == 20:
    current_knot += 1
    env.starting_braid = knot_list[current_knot % len(knot_list)]
    state = env.reset()
    correct_solved = 0

Where env.starting_braid is what determines the task for the agent to solve.
I fear, however, that the environment is not actually changing, so the task is staying as is.
Am I doing this the right way? Is this even a possible task?
Thank you.

As this is a custom environment, it is not possible to answer here. The correct answer depends on what setting env.starting_braid end then doing env.reset() actually does. To me, it looks logically correct and that the agent should be looping through different starting braids in index order. This also depends on the code incrementing correct_solved as you are expecting it to as well - you don't show that, but even if you did we still don't know the environment so cannot say whether what you are doing will work as expected.


text:
Question about feature matrix and notation in the paper Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
Let me know if this is not the place for this question. I'll take it down happily if that's the case. Also, I emailed the authors, but it seems like I won't be getting a response, so that's why I am here.
In the second paragraph of the preliminaries section almost at the end they have the following: "We refer the output of the penultimate layer of the deep Q-network as the learned feature matrix $\phi$, such that $Q(s,a)=w^T\phi(s,a)$, where $w\in R^d$ and $\phi\in \mathbb{R}^{|S||A|\times d}$.

The matrix notation I am familiar with is such that if $A$ is a $m\times j$ matrix and $B$ is a $j\times n$ matrix, then $AB$ is a $m\times n$ matrix. According to the statement above $w^T\rightarrow 1\times d$ and $\phi\rightarrow|S||A|\times d$, then the dimensions would not match, unless $\phi\rightarrow d\times|S||A|$. What am I misunderstanding?
Assuming fully connected layers, the output of a layer is a vector, so how is the feature matrix $\phi$ obtained from the output of the penultimate layer of the deep Q-network?

I'd appreciate if any can answer my questions and maybe share a code implementation of this paper. I have searched, but didn't find anything.
For reference, the paper can be found here

Instead of saying "Question about X" in the title, can you please just put your specific question in the title?

I will, but the question itself is not about a topic per se, but their paper and their notation. In other words, someone who hasn't looked at the paper could not answer the question.


text:
The impact of allowing the reward to be negative in a contextual bandit problem
I posted this question on stats.stackexchange.com 3 days ago. But there has been no answer. I would like to try my luck here.

It seems the contextual bandits problems, as shown in these two papers Tong Mu, Yash Chandak, Tatsunori Hashimoto, Emma Brunskill: Factored DRO: Factored Distributionally Robust Policies for Contextual Bandits, and Nian Si, Fan Zhang, Zhengyuan Zhou, Jose Blanchet: Distributionally Robust Batch Contextual Bandits, generally assumes the reward $R$ are bounded $0<R<M$. I am wondering what would be impacted if $R$ is allowed to be negative.

unofficially: none, sound RL method are indifferent to reward function shifting... however, I can tell you all the stories you want in the practical side of RL, but i'm not a theory person, so can't help you with the papers

@Alberto: I would tend to agree. But this seems to be a standard assumption in the contextual bandits problem. Why?

Gave a veery quick look at the papers and seems just one of those assumptions that you can make to have an easier life proving stuff without loss of generality (since other cases can be mapped to this bounded positive case)

@Alberto: Could you be more specific after reading more deeply into the papers? Much appreciated.


text:
Proof of Difference in Return Between Two Policies
I am attempting to understand why Lemma 6.1 holds in this paper on reinforcement learning. I have two questions. First, when defining the value function V(s), why is there a leading (1-γ) term? In the Bellman backup, I understand that it's necessary for normalization. However, in the paper's definition below I don't see the need.

My second question pertains to the equivalence below. Specifically, I'm not sure how they pulled a V_pi(s) out of the expectation and how there's an extra γ*V_pi(s_t+1) term inside of the expectation.

Any help is greatly appreciated!

Split this post into 2, one for each question. thanks.


text:
Yolo object detection metrics
I have made some predictions and saved the results to YOLO format. Then I made a program to calculate metrics, every metric looks fine except Precision/Confidence curve. I guess the flatline at the start is because the prediction was made with confidence >= 0.3, but the weird downward spike on the right is ok? or it has to be some bug?


text:
Loss function increases dramatically after transfer learning (or parameters initialization)
I am modeling a CNN network with some customized layers that performs 2D FFT and IFFT, I have a dataset with features representing time domain OFDM symbols, and labels representing frequency domain OFDM constellation samples.
The network is modeled as follows : [some CONV2D layers with activation 'tanh' and BN ---> FFT layer ---> some CONV2D layers with activation 'tanh' and BN ---> IFFT layer ---> some CONV2D layers with activation 'tanh' and BN] this entire block repeated several times.
The network have to produce the same input shape at the output (exactly 1x1024 matrices with 2 channels i.e. (None, 1, 1024,2)).
I trained the model's parts between FFT and IFFT layers separately and I used the obtained parameters for the full model initialization.
After that I got very good loss (only for forward propagation i.e. prediction without training), but after that when I tried to train the model to get lower loss, it directly increased the loss from 7e-6 to 1.9.
I would like to ask for some help to identify the possible problem, is it from the optimizer itself (maybe it did not take this initialization in consideration, some flag should be set or ...)? Or the problem can be from the network it self? Or maybe I need to add some extra trainable layers and try transfer learning from the base model and make it as untrainable?

I am using tensorflow/keras as library.
The loss function = mean squared error.
Optimizer => I used SGD and Adam.
The CONV2D layers are modeled as conv2d depthwise separable layers.

Thank you.


text:
Perspective distortion (fisheye effect) for checking subject in image is 2D or 3D
I've read from research papers (link1, link2) and some posts (link3, link4) that the fisheye effect (perspective distortion) will be observed as we move camera closer to 3D subject (to be concrete 3D subject here is face). Conversely, no significant perspective distortion occurs when moving the camera closer to 2D.
I'm trying to take advantage of this fact for liveness detection / anti-spoofing app. This should hopefully reduce spoofs that are Presentation Attack Detection (PAD) spoofs (printed faces, face images from displays, etc.)
I don't know how to accomplish this? My thoughts are: if there is a way to measure perspective distortion then using some threshold value for it maybe used to determine if subject is 3D or 2D.
How to achive this?

The problem is that if you have a video played in front of the camera, the video will also have that distortion in its recording, so you won't be able to check it... instead, you should take 2 pictures from your camera at different distances, then you might be able to do it with some ML (opinion)

@Alberto thank you for comment, indeed I've observed distortion in video reply attacks. I've managed to utilize distortion effect and it works very good, only issue is as you said video reply attacks. Maybe you have some idea about state-of-the-art methods for detection video replys?


text:
Implement perceptron given decision boundary function
Suppose we have a linear classifier for the classes ω1 and ω2 with characteristics vectors
Xa=[a a]^T and
Xb=[-a -a]^T
correspondingly. Also suppose that the decision  boundary that is defined by Perceptron at the repetition t is given by the relation g(X1,X2; γ_t)= X1+X2+γ_t*a=0
Also suppose P_t=1 for each t, and the correct classification is done for γ_t in the (-2,2) range, where Xa is misclassified for γ_t<-2 and Xb for γ_t>2
How would I prove that if during the (t)th application of the perceptron algorithm the value of γ_t is outside the range of correct classification (-2,2), after the (t+1)th application, the updated form of the decision boundary will be gt(X1,X2;γ_t)=X1+X2+γ_(t+1)*a=0
and that γ_(t+1) will be
(γ_t*a+1)/(a+1), if γ_t<-2 and
(γ_t*a-1)/(a+1), if γ_t>2
I thought about defining original weights Wt=[1,1] and updating the γ_t=γ_t+1 but that didn't get me anywhere

Please, edit your post to format the formulas with MathJax.