text:
Kernels on a trained CNN seem random
I saw this question and am having the same issue: kernels on a trained CNN look random.
I am using Pytorch to train a CNN, and based my code on examples that train the MNIST characters: ~50 convolutional layers of 20x20 kernels, that feed two fully connected layers, with the SGD optimizer, and the MSE or Cross Entropy loss function.
This is just a learning experiment (learning Pytorch for ~3 months now). I am interested in examining the kernels when a sufficiently low loss and high accuracy is reached.  I am training the network on 100 100x100 grayscale images that contain round or square pulses like these

or

The output is a 3 neuron layer that can be 000=round, 001=square, etc.  In all data pairs, 1 is mapped to 0.8 and 0 to 0.1.
As training proceeds, I was hoping to watch the kernels evolve from random to forms containing corners, lines and slopes; in other words, general features of my data.  Here's what I see:
If I choose a learning rate that gives finite, numerical loss values (lr=0.01, etc.) the kernels look like this

and do not change at all over 1000s of epochs, even though the loss can get arbitrarily low (1e-10 if I want), with 90%+ accuracy is classifying the pulses.
If I choose a large learning rate, like (lr=2 or lr=5), I get nan values for the learning rate, but neat looking kernels like this:

but also do not change at all over 1000s of epochs.
Am I wrong to expect the kernels to evolve from random to containing features of the training set as training proceeds?  I see no evolution of the kernels at all.
Thanks for any input. I am hoping features seen in kernels of trained networks might help those in the basic sciences (physics, chem, etc.) with new perspectives on their data.
Edit a day later:
I kept working on it, and normalizing the output (using Torch's normalize) seems to help. Watch this:

Edit on 12-Aug-2024, in response to @NikoNyrh's comment:
Dear @NikoNyrh: Thanks for your input and for sharing your fine results. I've kept at it on my end, and was able to find some similar things as you. In particular:

I agree my input figures are simple (maybe too simple). If I use 30+ kernels (as per a typical MNIST model), the kernels stay random looking, even though the loss lowers and converges. As you indicated, this number of kernels with the relatively simple inputs may be too many parameters, making a loss minimum easy for the network to find, without revealing any structure in the kernels.

I decided to restrict the network some, cutting the kernels down to 5 (20x20 kernels on 500x500 images), to force more a more selective minimum.  For more diversity in the images, I allow the waveforms to appear at any position in the image (even if it gets cut off by the image boundary).

I reworked my training plan. I got access to a GPU and trained on 10,000 images and look for accuracy on 100 images not in in the training set. I get about 17s/epoch on a 4090 consuming 71 Watts.

I too achieve 99%+ accuracy with the 100 test images.

I found the network very finicky to train, but this seems to work: L1Loss, Tanh activation on the fully connected layers, lr=0.1, momentum=0.0, batch_size=1,000 (on the 10,000 input images), and very selective DropOut.

My network is huge (200M parameters) as shown here:

    class neural_net(nn.Module):
        def __init__(self):
            super(neural_net, self).__init__()
            self.conv_layer_count = 5
            self.K = 25

            self.conv1 = nn.Conv2d(in_channels=1, out_channels=self.conv_layer_count, kernel_size=self.K, stride=1, padding=0)
            self.relu = nn.ReLU()
            self.mp1 =  nn.MaxPool2d(2)
            self.conv2 = nn.Conv2d(in_channels=self.conv_layer_count, out_channels=1, kernel_size=self.K)
            self.act = nn.Tanh()
            self.dropout1 = nn.Dropout(0.01)
            self.fc1 = nn.Linear(226*226, 4096)
            self.fc2 = nn.Linear(4096, 1024)
            self.fc3 = nn.Linear(1024, 3)

        def forward(self, x):
            x = self.conv1(x)
            x = self.relu(x)

            x = self.conv2(x)
            x = self.relu(x)
            x = self.mp1(x)

            x = self.dropout1(x)

            x = torch.flatten(x, 1)
            x = self.fc1(x)
            x = self.relu(x)

            x = self.fc2(x)
            x = self.act(x)

            x = self.fc3(x)
            x = self.act(x)

            return x

The Loss profile is interesting, with all kinds of ups and downs on the way to convergence.

Here are some of my eventual kernels that like yours, show the vertical and horizontal lines, including some 45 degree slopes, and wisps of other curves as well.

Thanks again @NikoNyrh!! Very interesting results.

Neat GIF.  You mentioned before the Edit that you saw "no evolution of the kernels at all."  Can you say more about how you normalized the output to achieve the evolution visible in the GIF?

@SapereAude: In my forward() function, I literally added x=F.normalize(x), where x is the vector coming off of the activation of the last and fully connected layer in the network.  x out of the normalization is then returned by forward().

This question turned out to be more interesting than I thought. Basically I think that the problem is too simple for CNNs, and any random-looking feature extraction gets it done.
To get the model to converge to a structure I was expecting, I had to make the network very simple, and use a specific initialization. First of all, here is the Keras code:
model = tf_models.Sequential([
    l.InputLayer(X.shape[1:]),
    l.Conv2D(7, 9, activation='relu', kernel_initializer=keras.initializers.Constant(init)),
    l.BatchNormalization(),
    l.GlobalMaxPooling2D(),
    l.Dense(1, activation='sigmoid', kernel_initializer='zeros', use_bias=False)
])

The generated data looks like this (vertical slopes very tricky, because I used the formula $f(x,y) \approx 0$), each input is a 128 x 128 single-channel image:

The last layer's weights were initialized to zero, here is how they (along with loss and val_loss) changed trough-out the training procedure:

So some Conv2D kernels learned to indicate a gaussian curve, and others indicate the pulse.
Here is the interesting part: Conv2D weights are initialized to "linear" gradients, to seven different directions. They are shown on the first column of this image. Subsequent columns show how the weights changed trough-out the training procedure. Final weights are shown on y-axis labels. Negative weights detect gaussians and positive ones detect pulses.

The pattern becomes a bit more clear if we sort the kernels by their corresponding dense layer's weights:

So gradients on roughly 45 degree angles didn't change much (they are already good detectors for pulse's specific parts), while vertical gradients become "hard" vertical edge detectors.
If instead I use random initialization for Conv2D, I don't get so clear patterns or conclusions. On one instance I got five detectors of vertical lines, and two almost identical detectors for -45 degree slopes:

For what it is worth, I got a model with only seven parameters to learn greater than 99.9% train and validation accuracy.


text:
Does probability flow ODE trajectory (in the context of diffusion models) represents a bijective mapping between any distribution to a gaussian?
I have read several papers about diffusion models in the context of deep learning.
especially this one
As explained in the paper, by learning the score function $(\nabla \log(p_t(x)))$, following probability flow ode trajectory one can uniquely map any data point in data distribution (X(0),let's say an image) to a point in multivariate gaussian distribution (Prior or X(T)) of the same dimension, as shown in Figure 2 in the paper(attached here as well).
I have several questions:

We know that we can draw a random a sample from the gaussian distribution ($P_T(x)$ in the middle of the figure) and the reverse SDE would generate sample from the data distribution $P_0(x)$, in this case a realistic image. Is that also true for the reverse ODE? I.e. any point in the gaussian would be mapped to a realistic image if we follow the probability flow ODE (white in the figure) path?

Can we say ODE trajectory is a bijective mapping from the data distribution to the gaussian distribution?

Can we say in such gaussian distribution every element (dimension) is independent of the others? please note that in the figure, the dimension is shown to be only one for simplicity, however in general it should have the same dimension as our data.

The motivation for my questions is, I want to find a mapping or a new representation of my data, in which every element is independent of the others (some sort of disentangled representation, but interpretability doesn't matter for me). Can the proposed mapping replace the ICA algorithm?

One advantage of the probability flow ODE approach is that it guarantees the generative model is identifiable, thus if the score matching neural network can recover the true score function of the data generating process, after training any given data point will map to a unique point in latent space and semantic interpolation can be meaningfully and interpretably performed in the latent space to generate images with properties that are in between two arbitrary input examples.
Of course in practice score function can only be approximated with errors, but reverse ODE proposed by Song et al could still generate realistic images as confirmed in Karras et al's “Elucidating the Design Space of Diffusion-Based Generative Models”, though they may have worse quality compared to SDE or DDPM models.

Deterministic sampling offers many benefits, e.g., the ability to turn real images into their corresponding latent representations by inverting the ODE. However, it tends to lead to worse output quality than stochastic sampling that injects fresh noise into the image in each step. Given that ODEs and SDEs recover the same distributions in theory, what exactly is the role of stochasticity?

One key insight is that both the forwards and backwards SDEs are a sum of the probability flow ODE and a time-varying Langevin diffusion SDE, through which the cited reference explains why ironically injecting noise in the reverse denoising process is helpful.

The Langevin term can further be seen as a combination of a deterministic score-based denoising term and a stochastic noise injection term, whose net noise level contributions cancel out... This perspective reveals why stochasticity is helpful in practice: The implicit Langevin diffusion drives the sample towards the desired marginal distribution at a given time, actively correcting for any errors made in earlier sampling steps.

In summary the stochastic nature of SDEs and DDPMs allows for continuous correction of errors during the generation process. In probability flow ODEs, once an error is made, it can propagate through subsequent steps potentially leading to lower quality results.
Finally in the context of normalizing flows, an ODE-based trajectory can indeed be seen as a bijective mapping from the data distribution to a (multivariate) Gaussian distribution (or vice versa) due to the invertibility of the continuous flow defined by the ODE and the fact that all intermediate latent variables $\mathbf{x_t}$ share the same dimension as the input $\mathbf{x_0}$. And in the same context, when we transform a complex data distribution into a multivariate Gaussian distribution, we often assume the target Gaussian distribution has independent dimensions with a diagonal covariance matrix for model simplicity and training tractability, and the series of invertible transformations applied by the normalizing flow can capture complex dependencies in the original data distribution. By contrast, ICA typically assumes a linear mixture of non-Gaussian components in the source signal and is designed for linear transformations, thus it's more limited in applicable scope.

Thanks a lot @cinch for your reply.
so you confirm the validity of the assumptions that such a mapping would enable us to find a disentangled representation of any data point sampled from the source distribution. I.e there is no dependence between the elements of the representation in the mapped representation.

You have multiple questions here and please note SE question should focus on one single question. I've answered your question 3 in my last section about the diagonal covariance assumption for $\mathbf{x}(T)$ in DDPM, which is typical for most hierarchical Bayesian models with continuous input features (DDPM could be viewed as a special genre of hierarchical VAEs). However, unlike normalizing flows, due to the stochastic nature of diffusion process and forward sampling for training and score function error, the actual $\mathbf{x}(T)$ represented by samplings won't be exactly disentangled.

The data sample gradually loses its distinguishable features as the step $t$ becomes larger, even in theory only when $T$ approaches infinity $\mathbf{x}(T)$ is equivalent to an isotropic disentangled Gaussian distribution at the end of the forward diffusion process.

Thanks for your reply.
I have a follow up question which I will ask as new question.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Does ideal probability flow-ode models generate good quality data starting from any (high probable) points in the Gaussian distribution?
I have read the question and the given answer Does probability flow ODE trajectory (in the context of diffusion models) represents a bijective mapping between any distribution to a gaussian?
Now I have a follow up question: Let's assume we are talking about the ideal case, e.g. we have access to $T=\infty$
Imagine we have learnt the score function associated with distribution P.
Now, imagine we take an (clean) image from the distribution $P_0$ and then we follow the probability flow ode trajectory (we don't inject noise). At $T=\infty$ , the image will be encoded to a gaussian distribution of the same dimension (let's say d-dimensional).
My question is: Can we freely change any of those d elements (independent of anything), and then follow the reverse trajectory and get an image from distribution $P_0$ ? In the other word, if we start from any point in the gaussian and follow the reverse ode (not SDE/ no noise injection) do we end up in a good looking image with a high probability?

Since in this ideal case well-trained probability flow ODEs by score functions are essentially same as normalizing flow models with well-defined bijective transformations, the quality of generated image should in general as good as normalizing flow models' result as long as the cumulative numerical errors during deterministic transformations computation are bounded and negligible (the referenced answer talked about why injecting stochastic Langevin diffusion noise would actually correct these cumulative errors on average) and your sampling is from the high probability regions of the disentangled Gaussian latent space. Most of the image's qualities are already encoded in the reverse bijective transformations, similar to DDPM's reverse diffusion random sampling process originated from the same final-layer's Gaussian latent space to reconstruct the input data after training phase of the amortized variational noises is done successfully.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Trying to understand backprop with convolution mathematically
I'm trying to understand how backpropagation works mathematically with convolutions.
I have a VERY simple setup.
X input to Y output through a linear filter:
kernel = torch.ones((1, 1, 3, 3)) / 9  # Normalized kernel
the FOM is calculated from Y through some nonlinear function.
The Gradient matrix G contains the derivative of the FOM for each element in Y
Now, what is this gradient multiplied with to get the derivative of the FOM with respect to each element in X?
I know I can get this value easily but I dont know whats happening under the hood.


text:
Architecture of an LSTM with multiple (dependent?) time series
If you have multiple time series data for a given problem (e.g predicting house prices and data is available per city).
Per city there is a list of features and the target feature.
If you want to train one LSTM of all the cities together, how you would approach that?
I was thinking of using a stateless LSTM architecture where I organize my input in such a way that each batch represent a time series of a city. If that approach would work, are there more things I need to account for?
What about making additional features with distance to other cities, thoughts on that?


text:
PPO has hard time finding solutions at the boundary of action space
I am applying PPO to a custom environment and it struggles when the optimal action seems to be at the boundary of the action space. I replicated it in the following simple environment. There are two state variables: time and a variable x, and at each step the reward is -(x+a)^2 + 1, where a is a parameter. At each step the agent should choose x for the next period. x must be in a certain interval. Totally run 10 periods.
The theoretical optimal policy should be just to choose -a at each step. But what I have found is that when the optimal solution is close to the boundary, for example when period reward is -(x+0.9)^2 + 1 and the constraint for x is [-1,1], PPO at best can give me a policy that picks -1 at each period. PPO can solve the problem though, when the solution is "interior" enough (e.g., with reward -x^2 + 1), and it converges to a very deterministic policy.
I looked into what is going on during the training, and it seems that when solution is near the boundary, the policy never goes down to a very deterministic one (std of the Gaussian policy remains at a relatively high level, ~0.3-0.4), and hence it just cannot settle down. In fact, mean_reward does not fully converge (as it should have, in such a simple case).
I can kind of see why this happens, but I do not fully understand what feature of PPO lead to this, and how to fix it. I am also surprised that the optimal solution is not that near the boundary yet my PPO agent cannot find it. Is this something that can be fixed via hyperparameter tuning (if so, how?), or is there any other algorithm better for this kind of tasks (I wanted to stick to PPO because it works for the more complicated task that I am actually trying to solve)?
I am using stable-baseline 3. Here is a minimum script that can re-produce the problem
import numpy as np
import gymnasium as gym
from gymnasium import spaces
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.common.callbacks import EvalCallback
from stable_baselines3.common.utils import set_random_seed
from stable_baselines3.common.env_util import make_vec_env
import os
import torch

# Setting up the environment
home_dir = os.path.expanduser("~")
dir = home_dir+''
GLOBAL_SEED = 0
set_random_seed(GLOBAL_SEED) # This is to ensure reproducibility
#%%
class SimpleOptEnv(gym.Env):
    def __init__(self, max_steps=10):
        super(SimpleOptEnv, self).__init__()
        self.action_space = spaces.Box(low=-1, high=1, shape=(1,), dtype=np.float32)
        self.observation_space = spaces.Box(low=-1, high=1, shape=(2,), dtype=np.float32)
        self.max_steps = max_steps
        self.current_step = 0
        self.state = None

    def reset(self, seed=GLOBAL_SEED, options=None):
        super().reset(seed=seed)
        self.current_step = 0
        self.state = -0.9
        return np.array([self.state, self.current_step / self.max_steps], dtype=np.float32), {}

    def step(self, action):
        self.current_step += 1
        reward = -1 * (self.state+0.9)**2 + 1
        self.state = np.clip(action, -1, 1)[0]
        done = self.current_step >= self.max_steps
        return np.array([self.state, self.current_step / self.max_steps], dtype=np.float32), reward, done, False, {}

# Set up the environment
env = make_vec_env(SimpleOptEnv, n_envs=4)

# Set up logging
log_dir = os.path.join(dir, 'simple_opt_tensorboard')
os.makedirs(log_dir, exist_ok=True)

# Define the learning rate schedule
def linear_schedule(initial_value: float):
    def func(progress_remaining: float) -> float:
        return progress_remaining * initial_value
    return func

# Create the PPO agent with improved hyperparameters
model = PPO("MlpPolicy", env,
            learning_rate=1e-3,
            n_steps=512,
            batch_size=64,
            n_epochs=20,
            gamma=0.99,
            gae_lambda=0.95,
            clip_range=0.1,
            ent_coef=0,
            vf_coef=0.5,
            max_grad_norm=0.5,
            verbose=0,
            tensorboard_log=log_dir)

# Set up evaluation callback
eval_env = make_vec_env(SimpleOptEnv, n_envs=1)
eval_callback = EvalCallback(eval_env, eval_freq=1000, verbose=0,
                             deterministic=True, render=False, n_eval_episodes=1)

# Train the agent
print(f'TensorBoard command: tensorboard serve --logdir="{log_dir}"')
model.learn(total_timesteps=100000, callback=[eval_callback], tb_log_name='PPO')
print("Training completed. You can now run TensorBoard to view the results.")

# Evaluate the trained agent
mean_reward, std_reward = evaluate_policy(model, eval_env, n_eval_episodes=10)
print(f"Mean reward: {mean_reward:.2f} +/- {std_reward:.2f}")

Hi @Hank Chen and welcome to AI Stack Exchange! What is the value of the entropy  coefficient hyperparameter in your PPO implementation? Sometimes this coefficient is represented as $\beta$.

Hi @DeepQZero! Thanks for your response. I updated my question to include a script that can replicate the issue. I currently use ent_coef=0, since the optimal policy is non-random. I tried values like 0.01 and it does not move the needle much.

My initial thought was that the entropy coefficient was too high - obviously it isn't at 0. I might take a look at this question in my spare time, but I hope someone else has an answer for you soon.

Thanks! What is really puzzling is that the same hyper-parameter can solve cases in which optimal policy is not near the boundary, and std drops to near zero. But it just cannot really converge near the boundary. I wander if some features of PPO, or maybe it's the use of clipping to deal with unbounded actions causes this...

I am guessing that it's the randomness of your first reward that the value function has a hard time predicting. To check try to either computer the reward after setting the new self.state which would make the reward completely deterministic, alternatively I would decrease gamma to maybe 0.95 to disregard long-term reward structure and focus more on immediate rewards.

Hi @Chillston! Thanks for pointing this out. I realized the same issue this morning so I changed the code a bit to just set first x to be the theoretical optimal value. It does make the learning process a lot less noisy, but the the same problem persist: if the goal is to optimize -(x+0.1)^2 + 1 over [-1,1], the agent can find the optimal policy (x=-0.1) each period. But if reward is -(x+0.9)^2 + 1, it will only find x=-1. It seems that when solution is too near the boundary, std cannot effectively go down, because too many sampled values are clipped, but I am not sure how to fix this..

Ah that's not what I meant, sorry - I mean to swap the two lines of code that computes the reward and that sets the new state otherwise your reward would be lagging behind one action, no?

Also more things to consider: 1) If your assumption is true, you would see it in the entropy of the action choices. Does that stay large over training? 2) Your observations contain time but the action doesn't depend on time so this might make it much harder to learn the correct action 3) I would still decrease gamma (and gae_lambda) as your action also doesn't depend on history 4) You have a large number of epochs, this can also lead to premature convergence of value/policy function. 5) Check your value function loss too, if large increase vf_coef.
It's a lot but hope it helps :)


text:
How to combine language models and theorem provers?
Suppose that a mathematician wants to find a proof of a conjecture or understand details of some proof. Can one combine language models like ChatGpt or Copilot and theorem proving software like Coq to and iterate outputs until a proof is found?


text:
How to quantify the tone of a textual paragraph? If there is historical communication available, how to check for consistency in tonality for new i/p?
Certain aspects of NLP such as the basic Polarity, Subjectivity, and Positivity, can be obtained with ease, but keyword consistent usage and the "Style" or the "Tone" of writing cannot be well quantified to maintain the consistency which is helpful for brands in their marketing campaigns.
Tried to obtain different quantifiers, and get the aggregate values for the historical communications data, and ranging from this aggregated value would be an accept window, wherein if the new sentence/para lies within the range, it can be deemed consistent, but this is not working out well and lacks accuracy.

Have a look at the relevant research in stylistics and other fields of linguistics.


text:
SSIM in validation higher then SSIM in training for image denoising
I'm working to denoise microscopy images using a 2D U-Net. I'm training my network on images taken at different z-levels, and these images have ground truth, which is the mean of the images in z. Therefore, some images share the same ground truth. I've separated the images into z-groups to place them in either train, validation, or test sets. I'm computing the SSIM on each batch and averaging the results at each epoch to plot them afterward.
The problem I'm facing is that the SSIM in validation is always higher than in training.

I'm not using dropout or batch normalization in my network, which can cause the validation to have higher results. I made sure that images were not duplicated in train
Why is this happening and how to address it?
Thanks


text:
Exact backpropagation formula for gradients vs online source
I visited this website that showed a simple backpropagation technique being applied on a built-from-scratch ANN. The code is in python and looks like this:
    # once you understand the chain rule it becomes super easy
    # to implement with a 'for' loop -- simply loop over the
    # layers in reverse order (ignoring the last two since we
    # already have taken them into account)
    for layer in np.arange(len(A) - 2, 0, -1):
        # the delta for the current layer is equal to the delta
        # of the *previous layer* dotted with the weight matrix
        # of the current layer, followed by multiplying the delta
        # by the derivative of the nonlinear activation function
        # for the activations of the current layer
        delta = D[-1].dot(self.W[layer].T)
        delta = delta * self.sigmoid_deriv(A[layer])
        D.append(delta)

and the second part:
    # since we looped over our layers in reverse order we need to
    # reverse the deltas
    D = D[::-1]
    # WEIGHT UPDATE PHASE
    # loop over the layers
    for layer in np.arange(0, len(self.W)):
        # update our weights by taking the dot product of the layer
        # activations with their respective deltas, then multiplying
        # this value by some small learning rate and adding to our
        # weight matrix -- this is where the actual "learning" takes
        # place
        self.W[layer] += -self.alpha * A[layer].T.dot(D[layer])

Assuming  D[-1] in the first code example is equal to $2(predicted - actual)$ and there is only one output neuron.
This code seems to indicate that the gradient for  the second layer of weights would be:
$2(predicted - actual)\cdot(W^{(L-1)})^T \circ (a^{(L-1)}\circ(1-a^{(L-1)}))\cdot a^{(L-2)}$
since in the final dot product, both tensors have the same dimensions(assumption) then the result would be a single scalar gradient for ALL the weights. I would have thought that since in the original case for the first layer of weights, $ 2(predicted - actual)\cdot a^{(L-1)}$, gives a vector gradient for all individual gradients for each weight, the next layer down should do the same. This just does not make sense to me. Can someone explain why this is not the case? What happens if $a^{(L-1)}$ and $a^{(L-2)}$ have different dimensions(how do we  calculate the gradient then)?
link to the  tutorial:
https://pyimagesearch.com/2021/05/06/backpropagation-from-scratch-with-python/


text:
Cross validation and MICE imputation
I'm working on a binary classification problem where I have some missing data. My initial idea was to use MiceForest. I'm also using stratified k fold technique (Data is imbalanced). I also want to minimize data leakage.

When should I impute missing values using MiceForest? For each fold? Or to the whole dataset in the beginning?
Should I use SMOTE to address class imbalance in each fold as I'm getting a lot of false positives (when using only stratified k fold without oversampling).
I get really good performance when I do mice imputation to the whole dataset, address class imbalance with smote and then do cross validations. I feel like this is overfitting? Is this because data leakage?

Can you please put your main question in the title? If you have multiple distinct questions, you should ask them in separate posts.

Is the imbalance really a problem in need of remedy? Statistics mostly says otherwise.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.


text:
Normalizing the embedding space of an encoder language model with respect to categorical data
Suppose we have a tree/hierarchy of categories (e.g. categories of products in an e-commerce website), each node being assigned a title. Assume that the title of each node is semantically accurate, meaning it's consistent with the category (the titles of the children nodes) it represents. Now, take an encoder language model (like BERT or Word2Vec) and produce embeddings for each node/category. My goal is to ensure that these embeddings are representative of the categorical hierarchy. I wonder:

Should the embeddings (mostly) be consistent with the categorical hierarchy, even if the categorical hierarchy is very niche and/or uneven (meaning that categorical granularity is not necessarily evenly distributed, if that makes sense)? By consistent, I mean that the embeddings of sibling nodes under one parent should be closer to each other than to children nodes of other parents, and closer to their parent's embeddings than to other parents' embeddings. Is there a nice metric to measure/validate this?
If not, what would be good ways to re-map a latent space to make it "nicer" with respect to the categorical hierarchy? I'm thinking either 1) we directly transform the latent space or 2) fine-tune the language encoder model (i.e. BERT). For example: 1) let's say that the shape formed by a category/parent's sub-categories/children is a thin oval, would it be possible and make sense to want to map the space such that this thin oval becomes a circle? Also for 2) would fine-tuning BERT along the task of classifying leaf embeddings' under the correct parent embeddings do the same thing?

I think that this requires the categories to not be so niche and to have some correspondence to regular language usage (and even more after providing context), which is my case.
This is just my intuition, so let me know your thoughts. I would be interested in exploring anything from the most simple heuristic approaches to SOTA NLP techniques that could pertain to this problem.


text:
What's the simplest deep learning approach in signal processing? A 1D CNN?
I have a three datasets each containing one signal of a specific type (normal, periodically jammed, constantly jammed). I would like to experiment with this dataset to predict wether a signal is normal or if being jammed, the type. Fot this experimentation I thought about CNN but I read that they work better with 2D data and images. What would you recommend?
EDit: It is a time series.

What is the data type, vector or time-series/sequence? If vector, use MLP; If sequence, go with RNN or 1d CNN.

Can you please elaborate more?

Sure,  but I at least need to know the data type (i.e., the shape, dimensionality) :) I mean what is this "signal" you referring to? It is a sequence of features of shape (N, T, D)? Where N is the number of data samples, T the sequence length, and D the dimensionality of num. features-

Thank you for the reply, https://github.com/AMHD/Jamming-Detection-in-IoT-Wireless-Networks-An-Edge-AI-Based-Approach ...in this repo, in the dataset portion you can see the signal values (they have been recorded at a constant interval of 30 minutes) are in a 1D array form. What's the simplest deep learning appraoch that will fit? I'm trying to experiment with jammer signal & normal signal identiication through DL approaches but I want to start with simple approach. Thank you for your time.

Since the datasets are just a list of numbers, I think the simplest approach to be classification. Basically, you concatenate the three datasets into one while associating a label to each number: you should obtain something like (x, y=1=normal) or (x,y=0=jammed) for each entry/sample. Then you feed the x to the neural net classifier to predict the y. Let me know if this works. Also, remember to shuffle the data while training the neural net

Hi Luca, I was wondering would concatenating and shuffling be a good idea? because to my understanding, the list of numbers in one file is one complete signal so if I were to shuffle the numbers, the sequence would be interrupted as the values vary in some pattern. The rest I understand, it can be a multi-class classification problem, y=0 for normal, y=1 for a randomly jammed signal and y=2 for a constantly jammed signal / splitting into testing and training bit and so on. What do you think? I'll try the classification method and definitely let you know how it went.

Ok if, so you can also group the single numbers into consecutive chunks of size T, classifying a sequence of T of them each time. Doing so results in sequences of shape (B,T) or (B, T, 1). Then you can still apply MLP but also RNNs. But I suggest you to try the simplest thing first, then try with something more complex.

I see, that makes sense, I'll do the grouping approach definitely. I'll let you know how it goes or if I face another challenge


text:
Is there any scientific work that has measured the perplexity achievable using just word2vec?
I know word2vec is not enough to achieve high quality text prediction just by itself. But has any scientific work tried to do it anyway, just to know what the baseline is that you have to beat with more sophisticated ideas? It doesn't have to be word2vec specifically, any similarly simple model would be fine as well. I'm only interested in the achieved perplexity of next-token-prediction, accuracy and other performance metrics don't interest me. If you're wondering what the point is: text compression. The original word2vec paper unfortunately doesn't concern itself with perplexity. Yes, I know I can do it myself, and I eventually will, but I still want to read what others have written about this topic.

My own answer for the enwik8 dataset ( https://huggingface.co/datasets/LTCB/enwik8 ) and a vocabulary size of 1024 is about 58.6 (corresponding to a cross entropy of about 4.07 nats). This assumes that the dimension of the vector space used for word2vec is essentially unlimited. I haven't tried it with other datasets or vocabulary sizes yet.


text:
What is the definition of a perceptron?
What is the definition of a perceptron?
OK, you are right, this question looks like lacking effort; it doesn't.
On the net, I have found many definitions that restrict the input of a perceptron to $n$ binary inputs. If that was really true, there would be many possible ways to simplify a perceptron, obviously to replace it with a truth-table, with good gain on performance.
https://www.w3schools.com/ai/ai_perceptrons.asp tells me inputs are 0 or 1.
I have seen other sites stating the input would be binary; then, I was in good faith-mode and didn't collect the links.

For a more modern treatment, see Elements of Statistical Learning by Hastie et al. The perceptron discussed in this book allows for any real valued vector input. In modern literature, the perceptron is used as building block for the feedforward neural network, AKA multilayer perceptron models.

Indeed originally Rosenblatt's perceptron's input was conceived to be binary only from biological consideration, but in modern sense perceptron can handle real-valued inputs as well.

In the modern sense, the perceptron is an algorithm for learning a binary classifier called a threshold function: a function that maps its input ${\displaystyle \mathbf {x} }$ (a real-valued vector) to an output value ${\displaystyle f(\mathbf {x} )}$ (a single binary value)

Therefore it can handle both binary and real-valued inputs, making it versatile for various classification tasks.

Quoting Wikipedia is not really the best idea. Wikipedia can be written by anyone, including people that have no idea what they are talking about. It's better to quote reliable books or research papers, if you really want to quote.

@nbro looks quite like the articles at w3schools  ... . They at least should have noted that there is something odd comparing their definition with the method of training they present. May I and ask for a hint to a solid introduction? I'm tired of that buzzword bingo that keeps showing up when I look for AI tutorials on the net.


text:
Dealing with likely class imbalance for unlabeled test data
I have a classifier model that has train/val/(initial)test data that is balanced between known (i.e., labeled) 500 negative and 1200 positive samples. The classifier model itself performs well on this labeled data. My concerns arise due to the negative data likely not comprehensively representing the theoretically negative high-dimensional data space. My positive data likely does comprehensively represent the diversity of the theoretically positive space.
For my unlabeled test data of 4 million datapoints, where likely some ~1% are positive (this condition is rare), I am unsure how to deal with this, as my current classifier is basically classifying this unlabeled data as 50%/50% positive/negative, which seems inaccurate.
I suppose my problem is basically that I have a Positive Unlabeled learning problem, but I want to make use of the known negatives that I do have. What are strategies for this? Thanks.

if it is trained (well) on a balanced dataset and the test data is not out-of-distribution, then what is the problem? you'll have to be careful with your evaluation, of course, but I think that performance should not be affected by imbalance in the test data.

This sounds not a class imbalance problem which is usually worried about during training stage with few available data points, it's a label shift calibration issue with OOD operational data since you already have a well trained probabilistic classifier. Is this what you really asking?

@cinch Thanks. My training data has negative data (500) that occupies what I suspect to be a small part of some high-dimensional space that encapsulates negatives. The positive (1200) training data is fairly comprehensive of its space I believe. For my goal, I have 4,000,000 unlabeled diverse datapoints for which I want to get a probability of being positive. My current classifier is basically classifying these 50/50 pos/neg, which seems inaccurate. Do you have any recommendations for this problem, recognizing the negative training data was likely not comprehensive.

As an alternative approach, could some combination of classification + anomaly detection (or positive unlabeled) model work? Thanks again

i chose the title because the classifier model itself performs well on the labeled data. my concerns arise due to the negative data likely not comprehensively representing the theoretically negative space. Thus, for my unlabeled test data of 4 million datapoints, where likely some ~1% are positive (this condition is rare), I am unsure how to deal with this. My problem is basically that I have a Positive Unlabeled learning problem, but want to make use of the negatives that I do have. Thus, i first made a binary classifier. If you'd prefer I make a new thread better phrasing this problem, i can

Ok, I thought your "My current classifier is basically classifying these 50/50 pos/neg, which seems inaccurate" meant your trained ML model is not good for train/val data set. Now I understand they're good to represent your train/val dataset, just 'seems' inaccurate for test data. And you don't have the 4 million test data available yet or never will have, so you're just worried but actually didn't have a test result 50/50 on these 4 million test data which if true it certainly would be bad. Is it correct?

@cinch yes that is correct. I've modified my initial question to better clarify

This sounds not a class imbalance problem which is usually worried about during training stage with very few available data points, it's a label (prior) shift calibration issue with out-of-distribution (OOD) operational data. Firstly imbalanced training data as a ratio is not a problem in itself, as long as you have enough training data points of the imbalanced class with representative features and not to train a deep learning model with enormous number of parameters.

Poor models... are often a result of fitting deterministic classifiers, using re-sampling or re-weighting methods to balance class frequencies in the training data and evaluating the model with a score such as accuracy... No re-sampling technique will magically generate more information out of the few cases with the rare class... It is well known that these problems typically have multiple incomparable Pareto optimal solutions... A more explicit way than oversampling or downsampling could be... assign explicit costs to misclassified samples and then minimize the total costs via cost-sensitive machine learning.

Now based on your further description of your problem, you already have a good probabilistic binary classifier from balanced training data with a selection bias towards the imbalanced class. Assuming the only thing changed in the target distribution is the label and no new labels at test time as you specified, you can safely apply a MLE based approach called MLLS.

Under label shift, the label distribution p(y) might change but the class-conditional distributions $p(x|y)$ do not... a maximum likelihood estimation approach, which we call MLLS, dominates empirically... This paper provides a unified framework relating techniques that use off-the-shelf predictors for label shift estimation. We argue that these methods all employ calibration, either explicitly or implicitly, differing only in the choice of calibration method and their optimization objective... we believe that this work, which addresses a foundational theoretical problem, does not present a significant societal concern.

It's not a easy solution but a very principled rational approach, where the basic idea is to estimate the OOD label distribution $q(y)$ then adjust your trained probabilistic classifier $p(y|x)$ via $q(y)$ as follows:
$$q(y|x)=\frac{q(x|y)q(y)}{q(x)} \approx \frac{p(x|y)q(y)}{q(x)}=\frac{p(y|x)q(y)}{p(y)} \frac{p(x)}{q(x)}$$
The term $\frac{p(x)}{q(x)} \approx 1$ with above assumption, so we can plugin our estimate of $q(y)$ to get calibrated result. Of course you may also train your model from scratch using 'imbalanced' data following the same target distribution with enough data points (perhaps with the same number of positive samples) using my above referenced cost-sensitive ML with greater cost weight on recall for your specific purpose.

Thanks for this!! I'll look into MLLS. To your last point, I already tried using SMOTE to oversample the negative data to match the target distribution (~1:1000 pos:neg), though I still wasn't finding the resulting predictions particularly convincing, likely because the resampled negative data still wasn't diverse.

SMOTE only plays with sample distribution without firm decision-theoretic basis, that's why you'd better employ 'cost sensitive' ML to actually address your concerned precision/recall cost structure aspect in your objective.

For my data, it is basically molecules. Positive denotes a certain functional property. The labeled training data I have is a dataset with positive examples and negatives examples that have slight modifications from positives so as to remove functionality. My unlabeled test dataset is a very diverse molecule library. You mentioned MLLS for label shift. I don't think my data would quality as label shift because it is not anti-causal; I think the molecular features X do cause the functional property Y; thus, would label shift be appropriate here? Or would a covariate shift method be better?

Your described case is mainly label shift (ignoring the slight covariate shift from the selection bias that you didn't include enough negative high-dimensional sample in your balanced training). Usually in practice several shifts would occur, even concept, but for practical purpose the major apparent imbalance issue here is label shift. Note if you have significant label shift, of course the overall covariate would shift, but covariate shift only means shift within each class label! Anti-causal here is not mechanistic but purely Bayesian view. Hope this completely clarifies the main issue here

Thanks! So in terms of practical implementation of MLLS as you have described, I basically multiply the predicted probability by q(y)/p(y)? So, if my labeled training data is 70.6% positive and my unlabeled test data is estimated to be 1% positive, I multiply the predicted probabilities by 1/70.6?

Thanks! One quick thing: I'm using softmax ouputs as probabilities - do you have recommendations for alternative ways to get probabilities or best practices for calibrating softmax outputs to treat as probabilities? Really appreciate your insight. Thanks

You only have two fixed classes here without open world it's probably not a big issue, but that's an entirely different question unrelated to this one, you can post another one here if you like. Please also upvote the answer you already accepted.


text:
Combination of components to maximize a multi-criteria objective function
I have been given a list of components, with various “contributions” (or weights) which put together in a weighted combination have a combined aggregate effect. I then have the task of suggesting replacements to some of the components with different weights,  which will maximise the overall aggregate effect.
In Bayesian optimization, I would just be trying to find the right weights of each component that will maximize the objective function, but I am not aware of a model that can generate replacement suggestions in order to maximize the final aggregate objective function.
For example, let’s say that you have parts that make up a car engine, and now you have to change a few parts to bring the overall cost down as you have a lower budget, but you want to keep the engine’s quality as close to the original one as possible. Then you have a few questions to answer like:

What parts should be removed and replaced with what parts?
How many new parts (weights) do we need?
Is the weighted combination of parts able to satisfy our criteria and maximize it too?

What ML model should I be looking into for such a task?

Are you looking for ML model to help you define your objective function for the combined aggregate effect? Is there any additional detail for the combined aggregate effect?

@cinch No, I am looking for (possibly) a generative model that suggests replacements of components in such a way that it maximizes a fixed objective function.

You need to first have reason to adjust weights in a (generative) way possibly in a continual learning ml model, is it because the operational data distribution will change over time? Generative models are not usually used to suggest adjustment based on some objective but generate data similar to its trained input distribution.

This is not a continual learning setting where the data distribution changes over time. As an example, let’s say that you have parts that make up a car engine, and now for some reason you have to change a few parts to bring the overall cost down as you have a lower budget, but you want to keep the engine’s quality as close to the original one as possible. Then, what parts do you change/replace and with say what quantity of each part?

This sounds like a Pareto optimization problem with a user defined cost structure, you need to add these details in OP with any other more details possible, no relation to Bayesian so far. Sounds like you would like a Bayesian update process to adjust those weights which is inconsistent with your cost-structured objective at least directly

Are you suggesting a combination of a Pareto optimization problem for dealing with multiple objective functions, while using Bayesian Optimization to adjust the weights of the components to arrive at a Pareto optimal solution sooner?

So, if I were to pose my problem as a multi-objective Pareto opt problem, where say I have tradeoffs like lowering the cost or adhering to some budget while maximizing some other utility, then how do I find the optimal “replacements” of parts and combine them in the most optimal manner to solve the problem? I would imagine my search space would explode if I had to manually change the parts and then see if they were optimal. Is there no way of exploring this space optimally? Also, would it help if I restricted the number of parts you could replace?

Of course to simplify things you need to restrict the # of parts, with a clear cost structure for each part, you may first use the straightforward Linear Programming with total budget constraints as a baseline which may be good enough for your practical case. I don't see how Bayes and generative models help in your case since it's just one time adjustment not continual learning as you described.


text:
RAG and LangChain for the Generative AI Chatbot to look at only the provided data
Requirements:

It would behaves like ChatGPT, however if user is asking something that is not in the provided data, it will say "Sorry, I do not know such information" <-- Please kindly refer to the example(s) below

Within organization only and not taking other sources other than the provided data below.

Only take up to 10 text file(s) for the data source, each of the file contains the information of 1 website.

This is what I think it should happen:
Example(s):

Whenever user is asking something like this to Generative AI Chatbot:

Who is the CEO of Google
Who is the CEO of Microsoft
Who is the CEO of Apple

It will say something like "Sorry, I do not know such information".

Whenever user is asking something like this to Generative AI Chatbot:

Who is the CEO

It will search through the provided data only and if not found will say "Sorry, I do not know such information".

Is there any free or paid subscriptions that I can use to achieve something like above? I am trying with ChatGPT, and it has global information and I am wondering is there anything that can be private use while using open-source LLMs, models, etc?
What I know is using RAG with LangChain, however how do I achieve this?


text:
Why doesn’t this optimization method work?
My idea is simple, for a loss function $L$, learning rate $\alpha$ and weights $W_n$ define the function $u_n(\alpha)$ as:
$$u_n(\alpha) = L(W_n - \alpha \nabla_n L)$$
If we find the minimum of $u_n$, we’ve found the best learning rate for $\nabla_n L$. Using Newton-Raphson method to approximate the point in which $u_n’$ is zero we can find the minimum of $u_n$:
$$\alpha_{n+1} = \alpha_n - \frac{u_n’}{u_n’’}$$
Finally update the weights:
$$W_{n+1} = W_n - \alpha_{n+1} \nabla L$$
I used this method to train a neural network on MNIST(calculated $u_n’$ and $u_n’’$ using the finite difference method), and while it was better than regular SGD, it wasn’t impressive compared to Adam. Why doesn’t this method work well? Is there any way to improve it?

Could you post some learning curves comparing your method vs SGD vs Adam? I think it could help to figure out an answer

Your attempt to find an optimal adaptive learning rate at each update step makes sense, however, even the finite difference Newton-Ralphson application for your one-dimensional problem is likely sensitive and unstable numerically since the loss landscape in deep learning is nonconvex (far from quadratic as the classic Newton-Ralphson assumes) even locally. This can lead to learning rate updates unstable. You might try out more stable quasi-Newton methods such as BFGS, but they share the same sensitivity issue for second-order derivatives.
Also if you combine your method with mini-batch like most other deep learning stochastic optimization methods, they're incompatible since the deterministic result of Newton-Ralphson in one mini-batch may not generalize well to the next batch, leading to inconsistent update.


text:
How should we choose a latent space learner for reinforcement learning?
The DreamerV3 (world models-based) architecture uses VQ-VAE to process visual data from the agent and teaches world dynamics through the latent space. I guess the primary is motivation behind this to reduce unnecessary data, compress the input size, and simplify training.
In the paper, I noticed that the images sampled from the VQ-VAE are blurry and not very sequentially consistent. While I understand that the blurriness might be due to the variational nature of the model but I don't know whether this architecture is of sufficient quality for model-based RL and whether I should go for another model.
For example, when I look for such a latent space learner, I come across latent diffusion models or beta-vae for distangle. However, since they are expensive to train, I can't decide whether they are suitable for RL.
I noticed that in DreamerV3, sequential data is provided as an additional input to the encoder. Maybe I should choose a latent space learner that can handle sequence data. I'm very confused, and I’m not sure whether I should continue with VQ-VAE or look for a better architecture.
In short how can we make the best decisions when selecting a latent space learner for our reinforcement learning algorithm to achieve optimal results?

If you're primarily concerned with computational efficiency and simplifying training while still achieving decent performance, VQ-VAE is a good candidate for model-based RL. The common blurriness in the VAE's reconstructions may not be a major issue if your task doesn't require high-quality visual feedback but rather focuses on the world dynamics and sequential decision-making. To fully leverage the additional sequential data input in VQ-VAE, you can concatenate all past actions or previous states or both with the current visual data input to pass through the VQ-VAE encoder. By providing the additional sequential context, the encoder can learn latent representations that are influenced by the temporal history to be capable for modeling sequential dependencies in the environment.
If your task requires disentangled latent variables such as separating the agent's position, velocity, and object state, models like Beta-VAE might be helpful.


text:
How to implement differentiable sinusoidal basis functions for a convolutional FFT module in torch
I currently have made a convolutional network module in torch that makes the basis functions in the usual way fourier_basis = np.fft.fft(np.eye(frame_size)) and I use this in the forward "FFT" to encode an audio signal into the frequency domain. This is a component in an autoencoder, so there is a corresponding inverse transform using inverse 1d convs.
However, I suspect that the set of periods in the normal FFT basis functions are not ideal for my domain, and I'd like to test that by creating an autoencoder that has learnable basis functions (perfect sinusoids) that differ in frequency.
It seems fairly simple to create a "free" filterbank that has just room for a bunch of parameters and the network figures out the values, like in the asteroid filterbanks API.
But in my case I want the basis functions to be sine waves, not totally freeform waveforms, and let the amplitude and frequencies of the sines be differentiable.
Strangely, I haven't found any implementations online, which leads me to believe either this is a) a bad idea, or b) I am not looking in the right places or describing this concept correctly.
the question: how can I implement a differentiable convolutional FFT module where the sinusoidal basis functions have learnable frequency, phase, and amplitude? (I'm looking for some insight on how to implement, not a full code example necessarily.)
thanks!

What is your question? Please, edit the post to ask a specific question and also put it in the title. Thanks

ok I added the question and edited the title. thanks!


text:
Treating binary classification as 3 class problem with two cutoffs -- what metric to use?
So I have a binary classification problem, however I don't want to predict on places where the model isn't very confident. So let's say for a probability output between 0.4 & 0.7, I return no output. Above 0.7 I return class 1, below 0.4 I return class 0. I obviously want high performance while minimizing this uncertain class.
My question is how do I pick the cutoffs & the model? I am training using a weighted cross entropy function due to class imbalance which only complicates things. There's two questions here, how to pick the model and then how to pick the cutoffs:
For picking the model:
is this test loss itself the best way to determine the best model?
Should I train the model from the onset to be a 3 class classifier with a lowered penalty for guessing this "uncertain" class?
For picking cutoffs:
Should I design my own scoring rule that somehow quantifies error and penalizes middle probabilities when picking cutoffs?

Consider use proper scoring metrics that account for prediction confidence as you specified such as the Brier score which is based on squared error and thus is less sensitive to extremely rare or extremely common classes.

The Brier Score is a strictly proper score function or strictly proper scoring rule that measures the accuracy of probabilistic predictions. For unidimensional predictions, it is strictly equivalent to the mean squared error as applied to predicted probabilities. The Brier score is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive discrete outcomes or classes. The set of possible outcomes can be either binary or categorical in nature, and the probabilities assigned to this set of outcomes must sum to one (where each individual probability is in the range of 0 to 1).

So you can train your classification model using your existing weighted proper scoring cross-entropy accounting for class imbalance or using a well calibrated metric such as Brier score as your loss function. Of course there's nothing wrong to try your own customary metric and I believe you should increase penalty for guessing the "uncertain" class, but bear in mind that it may not be proper in the sense of calibration.
Then you can use a validation set to empirically find the cutoffs that minimize the uncertain class while maintaining high accuracy for class 0 and class 1. Simple calibration technique such as temperature scaling is discussed in this recent answer.

Thank you for the answer and confirming my intuition! Special thanks for your comment on temperature scaling -- I hadn't considered it before, but in the context of my problem is actually extremely valuable.


text:
Proof for Using Q-Function in Policy Gradient Formula
Currently I am reading the OpenAI spinning up document about policy gradient and actor-critic method. In this webpage,replace Return with Action value, I think they are trying to prove that the following two formulas are mathematically equivalent.,
I could not understand the derivation in that webpage. Is there any other resources for a clearer derivation for replacing the return $R(\tau)$ with the corresponding action value $Q^{\pi_{\theta}}(s_t,a_t)$? Thank you very much!
This article presents a different derivation.

the expectation of Q wrt the policy is V, and V is the expected return R

Read the Sutton and Barto derivation

I find another explanation which i think is very good.  link


text:
Why do diffusions learns accumulated noise and not intermediate noise?
The core step is to learn scoring function: $L=\int E_{X_t}\|S_\theta(X_t,t)-\nabla\ln p_{t}(X_t)\|dt$, where $X_t$ is a forward-noise process and $p_{t}(x)$ is a density of X_t.
The trick to actually minimize it is to use simple equality $E_{X_t}\left[\langle S_\theta(X_t,t),\ \nabla\ln p_{t}(X_t)\rangle\right]=E_{X_s,X_t}\left[\langle S_\theta(X_t,t),\ \nabla\ln p_{t}(X_t|X_s)\rangle\right],\quad 0\leq s<t\leq T$
This allows us to rewrite L in the form $L=\int E_{X_t}\|S_\theta(X_t,t)-\nabla\ln p_{t}(X_t|X_s)\|dt$. which enables an easy-to-compute Monte Carlo approximation: $L\approx \sum\limits_{t=0}^T\|S_\theta(X_t,t)- \nabla\ln p_{t}(X_t|X_{s})\|$. However, it's important to note that errors in approximating $\nabla\ln p_{t}(X_t|X_{0})$ for large t can be quite significant.
My question is: why do we need to take s=0? My intuition suggests that it might be more logical to train the loss function in the form $L\approx \sum\limits_{t=0}^T\|S_\theta(X_t,t)- \nabla\ln p_{t}(X_t|X_{t-1})\|$ as $\nabla\ln p_{t}(X_t|X_{t-1})$ is much easier to approximate. While this approach would slightly alter the original loss function, it probably should still effectively learn $\nabla\ln p_{t}(X_t)$. Perhaps the problem will be in the guidance-free approach?


text:
Optimization on predicted confidences
I currently have a classification task, except I use an object detection model. The model outputs detections for all objects it locates in the image, and assigns a confidence to each. However, I am only interested in classifying the entire image as one class, so I will need to perform some postprocessing.
I first have a vector of K threshold values, one for each class, and I threshold all confidences such that if a confidence is less than the corresponding class threshold, I clamp it to zero (essentially deleting the detection).
I also have a vector of K class weights, one for each class, and I multiply all confidences by the corresponding value. This is used to “re-order” the confidences so to improve the accuracy. For example, a certain class could be detected with low confidences by the model, but is a high priority for the user (such that when it appears, the user would always want the image to be classified as it).
I currently have the mathematical problem where I want to find the optimal values for both the thresholds and the weights, such that with these new parameters, I can alter the confidences such that the highest accuracy is achieved (the highest confidence detection, after thresholding/weighting, in each image coincides with the ground truth).
I know that this scheme runs a lot of risks with regards to overfitting, or that accuracy is not the best metric when the dataset is not well-balanced, etc. but I am purely interested in this as a mathematical question.
Currently, for the weight optimization, I am running a gradient descent algorithm using a binary cross-entropy loss. I use binary cross entropy because I cannot handle the “None” (no detections) class using regular cross-entropy, as far as I know (please correct me if I’m wrong). This works with decent results.
But I am not sure how to optimize the thresholds, as it’s non-differentiable, short of using a brute-force or heuristic algorithm like genetic algorithm or grid search. I have tried using an iterative local search but it is slow and is slow to improve.
Any suggestions on improvements would be appreciated.


text:
Two tower recommender system
I am exploring two tower architecture for user and content recommendation. The data that I have, solely consists of positive user and content interactions. (i.e. I do not have any data for scenarios when an user ignored the content that was presented).
I am struggling with the following implementation details.

In-batch negative sampling seems a popular method for training the candidate generation phase. In a batch an user Ui and content Cj is treated as a negative sample if 'i' is not equal to 'j'. But if a batch has multiple interaction records for a specific user, in-batch negative sampling could lead to conflicting training data for the model. How do we handle the issue? Do we need to ensure that each batch has only a single interaction record for a given user?

If I want to measure the model performance in the validation set (or want to use early stopping), how do I generate negative samples for the validation set. I can use the same methodology as in-batch sampling. But I will run into the same problem of multiple interactions for a given user.

Could you please recommend a good way to split the data into training and validation set? Should I ensure that I have a set of users and all their interactions only in the validation set. i.e. for those users there are no interactions in the training set

Any inputs and suggestion would be greatly appreciated.
Thank you!

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.


text:
Training a deep learning classifier on a mix of features from noise and the source of interest
I'm working on classification problem where features are sparsely observed e.g. only 60% of features are observed (out of ~500,000 features). Some of the observed features are from a source of noise (say 70% of observed features is noise), the rest is of interest for classification.
The positions that are observed from the noise source or the source of interest are completely random. I have some example samples of the background noise.
My current approach is to train a fully connected feed forward network on simulated samples that follow these rules. This seems to work reasonably well. Would there be a better solution e.g. filtering the signal of interest in some way to train a better classifier?


text:
How should I see this problem as? Normal or Time Series?
I have multiple independent simulated time series data (for physic experiment). Each time series has t = 0 to 10000, and 7 features with 1 target. I want to build a model that I can train on all the simulated data and make prediction on dataset Test which has only 7 features and the Time column (t=0 to 10000) without the Target. (Completely independent case of the training set)
I tried to think this as a time-series problem since target is time dependent but my goal here seems not? Because if I train a time series model, my Test set needs a historical part, but in my case,my data does not.

welcome to the community.
Let me imagine this problem to be a classification task depending upon the smartwatch accelerometer and gyroscope data collected over time (time series);
Say we have a target variable where we need to predict what activity the human is doing;
You can proceed to input these numeric data normally in your algorithm (either a DNN or some ML algorithm) and let the algorithm figure out what the target variable would be. Don't forget to standardize your features!
To proceed with these kinds of problems, I always suggest doing some EDA, i.e. checking how these 7 features vary depending on the target class. For example, on the smartphone data (accelerometer) for an idle person (sitting or lying)  is drastically different from a dynamic person (walking, running).
If you see these patterns already, you can expect your algorithm to perform reasonably well. Good luck :)


text:
Is the feed forward layer in the last encoder block of a transformer unnecessary?
I am trying to wrap my head around the transformer model using this great resource:
https://jalammar.github.io/illustrated-transformer/
I may need some clarification.
My understanding is that we take the key and value vectors from the last encoder block's self-attention layer and use them as input for the cross-attention layer of each decoder block.
But if that is the case, then everything after the last encoder block's self-attention layer (i.e. the feed forward and normalization layers) is not necessary anymore, right?
I assume they are in the illustrations and also implementations to keep things consistent, but for the sake of inference and training time, I think we could technically skip these layers, like this:

I am asking because the illustration makes it seem like we take the output of the encoder as input of the decoder, but that wouldn't work because the dimensions don't fit (key and value vectors have different length than the embedding vector). Or does the dashed line indicate general flow of information from the encoder as a black box?
Can anyone confirm that these layers are technically not needed? And as a bonus question: Is this usually taken into account in frameworks like tensorflow to conserve training and inference time?
Update: As per @Alberto's answer, my initial premise was wrong. We do not take the key and value vectors from the last encoder's self-attention layer. Instead we take the last encoder's output and compute new key and value vectors in the decoder's cross-attention layer.

I would suggest that for in-depth details on how stuff is done, stick with the researcher materials, so either the code or the paper itself.
In particular, no details are given for this manner in the main paper, but the authors published the code
In particular here, you can see that the encoder output is passed as both key and value.
So, no, you cannot remove all that final part of the transformer

Thanks for providing further material. Looking at the code really cleared that up for me. My mistake was that I thought we capture the k and v vectors from the encoder, but instead we calculate fresh k and v vectors from the encoder output using the decoder's Wk and Wv weights.

@Cerno no worries, i knew the code had the answer because i used to have the same doubt as your, and i found the answer there, good luck with your journey


text:
Is it useful for ML or DL algorithms to create differential variables from cumulative variables in a dataset?
I'm working on a dataset that contains several cumulative variables, which are values that always increase and depend on their previous values (such as an odometer reading in a vehicle). My aim is to train ML or DL models to perform regression and classification tasks in supervised learning from the variables in my dataset.
I wonder whether it makes sense to create differential variables from these cumulative variables to train ML or DL models. For example, calculating the difference between consecutive records to obtain a rate of change between them (and even dividing by the time interval between samples to create the time derivatives). Could this approach provide any advantages for machine learning models? Is it a standard practice?

In most cases, classification at the final decision point is attempting to linearly separate classes.  Complex data is rarely linearly separable so it is common to perform transformations on the data until it is.
The training algorithm must find any important transformations.  If some expert knowledge exists that a transformation is important, it’s often good to do it in pre-processing so it doesn’t need to be recreated by the model.
However, a transformation might also hide information or make the data harder to linearly separate.  So the best answer is pre-processing is sometimes trial and error.
There is a clear pro to your specific example:
Unless a datapoint contains both current and previous odometers reading, speed could never be derived.  Any prediction dependent on speed could not be made.  The same could be said of any point to point derivation.
Similar arguments can be made in regression where the goal is to find a simple polynomial for best generalization.  It may take some transformations to get to a representation where the polynomial is simple.


text:
How to categorize and identify the most requested features from 1000+ GitHub issues using an LLM agent?
I have scraped over 1000+ GitHub issues from a repository using Crawlee. My goal is to segregate these issues based on their descriptions, title, number of comments and statuses. I want to identify the most frequently discussed features or issues.
There are a few challenges:

Some features or issues may be duplicates or related to one another.
Some issues might have user comments (e.g., "+1") indicating
support, and I’m unsure if I need to count these.
The current approach doesn’t return the desired results when querying the LLM just by giving entire context.

Here’s the code I’ve built using Langchain and the Google Gemini API:
text_splitter = RecursiveCharacterTextSplitter(chunk_size=10000, chunk_overlap=1000)
chunks = text_splitter.split_text(text)

embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001")
vector_store = FAISS.from_texts(chunks, embedding=embeddings)

prompt_template = """
Answer the question as detailed as possible from the provided context, make sure to provide all the details, if the answer is not in
provided context just say, "answer is not available in the context", don't provide the wrong answer\n\n
Context:\n {context}\n
Question: \n{question}\n

Answer:
"""

llm = ChatGoogleGenerativeAI(model="gemini-1.5-flash", temperature=0.3)
prompt = PromptTemplate(template=prompt_template, input_variables=["context", "question"])
chain = load_qa_chain(llm, chain_type="stuff", prompt=prompt)

question="What is demanding issue?"
docs = vector_store.similarity_search(question)

response = chain.invoke(
    {"input_documents": docs, "question": question},
    return_only_outputs=True
)

Problem:

The LLM doesn’t return a reliable answer when simply querying for the "most requested issue."

I probably need to identify duplicates and categorize the issues (e.g., "Bug," "Feature Request") before proceeding.
I’m confused about how to approach counting comments like "+1" to assess user demand.

Question:
What logic or approach should I use to automatically categorize these GitHub issues (e.g., classify them as bugs, feature requests, etc.), find duplicate issues, and determine the most requested features or issues based on the total number of useful comments?


text:
How to 'induce' or 'teach' pretrained model to a continous Transformer token-pruning Algorithm
I am currently looking for ways to improve Transformers performance in image processing, especially in image segmentation. I found this paper by Kong, Z., et al called "SPViT: Enabling Faster Vision Transformers via Latency-aware Soft Token Pruning" where in their method, TLDR, they run multiple rounds of Transformer training and pruning until they reached a consensus between accuracy and training weight. So basically the what the paper do is trying to preserve accuracy with lighter training by taking unimportant tokens of which each Transformer heads are making.
From what I understand, they first run transformers on 'maximum performance cost' first because the first tokens are unpruned, then the cost will lower because of pruned tokens and reached the expected lighter yet still accurate training. So from the images, I expect maximum training cost which I'm actually trying to avoid and the cost will be lower as the tokens are more pruned in Phase 3

My question is, is there a way I can 'skip' the 'maximum performance Transformer training' by inducing a model from somewhere else like from segmenter-github-by-Strudel,et al.. But isn't that called inference? How to transfer the 'context dictionary' from one model to another?
So I can transfer these 'contexts' from image I just inference with pretrained model to the Transformer in SP-ViT algorithm

What do I actually look for?


text:
Using hill climbing to solve the polygonal obstacle problem
I will first explain the problem. We start at an initial node S in the 2D plane and our goal is to reach a goal node G. In order to get to the end node we have to get past polygonal obstacles. Here is an image:
A solution path is a list of vertices starting at S and ending at G that form a path from S to G.
We can start by giving the hill-climbing algorithm a user-generated path through the polygonal obstacles. What I am struggling to understand is how we generate neighbours? Because a neighbour also has to be a path that starts at S and ends at G. We have to perturb the path in some way to generate a neighbour.
Any ideas? I have googled this but nothing useful comes up.
An answer has been given, but I would like to understand how to implement such a method. In my head it is going to be really difficult to change the path in such a way that it remains valid.

How can you argue your hill climbing local search heuristic algo is the right approach here since it's not clear this is a convex problem? Why not try A*/Dijkstra/RL?

I have already solved the problem using A* and Dijkstra. IT is part of an exercise to solve it using hill climbing. What is RL?

Hill climbing is a local search algo which applies in convex case only without local optima, however, based on the many obstacles of your case, it may very well likely stuck in a local optimum even your evaluation function successfully filter out all intersected neighbor paths, curved or straight, after rasterization, say.

To solve the polygonal obstacle problem using hill climbing, you could creatively explore potential paths by adjusting waypoints slightly, rotating, or translating the entire path to avoid obstacles. By shifting sections of the path or tweaking its direction, you can uncover new solutions that might bring you closer to an optimal route. With every move, aim to keep the path valid and gradually improve its performance, whether through reducing path length or avoiding new obstacles. This incremental approach can help guide you toward a better solution.

Could you help me understand your answer a bit more? For instance, suppose I enter the initial path to be [A,B,F,T,G,Y,X]. If I was to alter that path, in any way, it will most likely invalidate it.

Could you tell me how you would adjust the path to generate valid neighbours?


text:
Learning Curve of Learnig Machine is not good, how to improve?

I am having a problem with a machine learning model used to monitor alerts for binary classification, the image of the learning curve looks like this, obviously there is something wrong because the training curve is almost a horizontal straight line, but what is the problem?
I have tuned the parameters of the model many times and it still looks like this. The model I'm using is XGboost.

Boosted trees iteratively train on misclassified examples.  This makes it focus on hard to classify examples.  This would typically cause overfitting but the ensemble helps.  But from a training perspective it will look like overfitting (high training, lower validation).
In your case as you increase training examples the training accuracy is always overfit but as more examples are included validation accuracy rises.

Thanks for your answer.But I tried another two times, I got a decrese and a increse trainng curve, y-label is accuracy, which is right?

How are you choosing training samples?  Deterministic or random?

Yes， I use  GridSearchCV and Cross-Validation to search best params.


text:
Low validation loss from the first epoch?
The initial validation loss is low from the first epoch and then decreases slightly. What does this actually mean? Does it indicate that the model can effectively and quickly identify patterns for this task?
I can see that the model works in practice, but the results (some image restoration) aren’t ideal yet, so I want to improve its performance even further.

Given this low loss from the first epoch, should I focus on training with more data or on adjusting the architecture and layers to be even more complex, etc.?

Given the small differences between the first epoch and the last, is it more likely that the model was barely able to improve performance during these epochs, or could the difference in loss still be meaningful?

The dataset count was of 10,000 images - 0.9 for training, 0.1 for validation.
First epoch loss:
Epoch [1/50], Training Loss: 0.026428, Validation Loss: 0.023727
Last epoch and plateau:
Epoch [34/50], Training Loss: 0.020682, Validation Loss: 0.020651

What loss are you using? How many samples are in your training, validation and test datasets? How are you calculating the losses? Have you randomly sampled the the different datasets?

The intilal low validation loss from your first epoch suggests a few key points about your model:
Initial Low Loss:
The model is likely learning useful patterns quickly from the dataset, so you see a low loss from the first epoch. This could mean that the task is not overly complex, or the model architecture is well-suited for initial learning. However, it might also indicate that the model is not learning more fine-grained details, especially if the task involves image restoration, which can be challenging.
Plateau in Loss:
The loss plateaus around epoch 34 with little further improvement suggests that the model might have reached a point of diminishing returns. This means that either the model has learned everything it can from the given data and architecture, or it is struggling to capture more intricate details or patterns.
to slove this issue you can try following things

Training with more data

=> If your dataset is not large or diverse enough, the model might have learned the patterns quickly but lacks enough examples to generalize well. More data, especially if it covers a wider range of scenarios (like different image qualities or variations), can help the model improve its performance. you should the augmentation of your dataset since you have mentioned about the 10,000 images but if they are mostly same it would be not a good dataset here in dataset the quality of the data that matters instead of the quantity of the dataset

Adjusting the Architecture:

=> If the model learns quickly but struggles to refine its output or learn more detailed patterns, the architecture might not be complex enough for your task. This is often the case for tasks like image restoration, where deep networks with residual connections, multi-scale architectures, or specialized layers (like attention mechanisms) may help capture finer details. if you want to try unet with attention you can follow this article


text:
Mathematica code To extract tabulated data after conversion from pdf to text
I've written the following code in Mathematica to extract the data I need from invoices which are in pdf format but can vary in terms of layout and terminology:
(*Prepare Training Data*)
trainingDataItemDesc = {
  "Description" -> "SimilarString", "DESCRIPTION" -> "SimilarString",
  "Product Name" -> "SimilarString",
  "Product Details" -> "SimilarString", "Products" -> "SimilarString",
   "Product" -> "SimilarString", "Details" -> "SimilarString",
  "Desc" -> "SimilarString", "ITEM" -> "SimilarString",
  "Product Description" -> "SimilarString", "" -> "SimilarString",
  "Item Description" -> "SimilarString", "ITEMS" -> "SimilarString",
  "Description of Goods Sold" -> "SimilarString",
  "Part Number/Description" -> "SimilarString",
  "PRODUCT NAME" -> "SimilarString",
  "PRODUCT DETAILS" -> "SimilarString", "PRODUCTS" -> "SimilarString",
   "PRODUCT" -> "SimilarString", "DETAILS" -> "SimilarString",
  "DESC" -> "SimilarString", "PRODUCT DESCRIPTION" -> "SimilarString",
   "ITEM DESCRIPTION" -> "SimilarString",
  "DESCRIPTION OF GOODS SOLD" -> "SimilarString",
  "PART NUMBER/DESCRIPTION" -> "SimilarString",
  "description" -> "SimilarString", "product name" -> "SimilarString",
   "product details" -> "SimilarString",
  "products" -> "SimilarString", "product" -> "SimilarString",
  "details" -> "SimilarString", "desc" -> "SimilarString",
  "item" -> "SimilarString", "product description" -> "SimilarString",
   "item description" -> "SimilarString", "items" -> "SimilarString",
  "description of goods sold" -> "SimilarString",
  "part number/description" -> "SimilarString",
  "Description:" -> "SimilarString",
  "DESCRIPTION:" -> "SimilarString",
  "Product Name:" -> "SimilarString",
  "Product Details:" -> "SimilarString",
  "Products:" -> "SimilarString", "Product:" -> "SimilarString",
  "Details:" -> "SimilarString", "Desc:" -> "SimilarString",
  "ITEM:" -> "SimilarString",
  "Product Description:" -> "SimilarString",
  "Item Description:" -> "SimilarString", "ITEMS:" -> "SimilarString",
   "Description of Goods Sold:" -> "SimilarString",
  "Part Number/Description:" -> "SimilarString",
  "Total" -> "NotSimilar", "TOTAL REMITTANCE:" -> "NotSimilar",
  "[GBP] TOTAL" -> "NotSimilar", "Total:" -> "NotSimilar",
  "Grand Total:" -> "NotSimilar", "Invoice total" -> "NotSimilar",
  "Total VAT" -> "NotSimilar", "Total Paid" -> "NotSimilar",
  "Amount Due:" -> "NotSimilar", "AMOUNT (USD)" -> "NotSimilar",
  "Order Total" -> "NotSimilar", "" -> "NotSimilar",
  "Invoice Total:" -> "NotSimilar", "Gross Total" -> "NotSimilar",
  "Net Price (per unit)" -> "NotSimilar", "TOTAL" -> "NotSimilar",
  "Total paid" -> "NotSimilar", "Invoice Total" -> "NotSimilar",
  "TOTAL INC. VAT:" -> "NotSimilar",
  "Total Amount Due" -> "NotSimilar", "TOTAL:" -> "NotSimilar",
  "GRAND TOTAL:" -> "NotSimilar", "INVOICE TOTAL" -> "NotSimilar",
  "TOTAL VAT" -> "NotSimilar", "TOTAL PAID" -> "NotSimilar",
  "AMOUNT DUE:" -> "NotSimilar", "ORDER TOTAL" -> "NotSimilar",
  "INVOICE TOTAL:" -> "NotSimilar", "GROSS TOTAL" -> "NotSimilar",
  "NET PRICE (PER UNIT)" -> "NotSimilar",
  "TOTAL AMOUNT DUE" -> "NotSimilar", "total" -> "NotSimilar",
  "total remittance:" -> "NotSimilar", "[gbp] total" -> "NotSimilar",
  "total:" -> "NotSimilar", "grand total:" -> "NotSimilar",
  "invoice total" -> "NotSimilar", "total vat" -> "NotSimilar",
  "total paid" -> "NotSimilar", "amount due:" -> "NotSimilar",
  "amount (usd)" -> "NotSimilar", "order total" -> "NotSimilar",
  "invoice total:" -> "NotSimilar", "gross total" -> "NotSimilar",
  "net price (per unit)" -> "NotSimilar",
  "total inc. vat:" -> "NotSimilar",
  "total amount due" -> "NotSimilar", "[GBP] TOTAL:" -> "NotSimilar",
  "Invoice total:" -> "NotSimilar", "Total VAT:" -> "NotSimilar",
  "Total Paid:" -> "NotSimilar", "AMOUNT (USD):" -> "NotSimilar",
  "Order Total:" -> "NotSimilar", "Gross Total:" -> "NotSimilar",
  "Net Price (per unit):" -> "NotSimilar",
  "Total paid:" -> "NotSimilar",
  "Total Amount Due:" -> "NotSimilar"}; trainingDataSubtot = {
  "Subtotal" -> "SimilarString", "Nett Total:" -> "SimilarString",
  "COMPONENT TOTAL" -> "SimilarString",
  "Sub-Total:" -> "SimilarString", "Sub Total" -> "SimilarString",
  "Subtotal:" -> "SimilarString", "" -> "SimilarString",
  "Total goods ex VAT" -> "SimilarString",
  "Sub-Total" -> "SimilarString",
  "Net value of goods" -> "SimilarString",
  "Total Net Amount" -> "SimilarString",
  "Net Total:" -> "SimilarString", "Sub-total" -> "SimilarString",
  "Goods Value" -> "SimilarString", "Products" -> "SimilarString",
  "SUB-TOTAL:" -> "SimilarString", "Net Amount" -> "SimilarString",
  "Nett Total" -> "SimilarString",
  "Total Nett Goods" -> "SimilarString",
  "Invoice Total" -> "SimilarString", "SUBTOTAL" -> "SimilarString",
  "Net total" -> "SimilarString", "NETT TOTAL:" -> "SimilarString",
  "SUB TOTAL" -> "SimilarString", "SUBTOTAL:" -> "SimilarString",
  "TOTAL GOODS EX VAT" -> "SimilarString",
  "SUB-TOTAL" -> "SimilarString",
  "NET VALUE OF GOODS" -> "SimilarString",
  "TOTAL NET AMOUNT" -> "SimilarString",
  "NET TOTAL:" -> "SimilarString", "GOODS VALUE" -> "SimilarString",
  "PRODUCTS" -> "SimilarString", "NET AMOUNT" -> "SimilarString",
  "NETT TOTAL" -> "SimilarString",
  "TOTAL NETT GOODS" -> "SimilarString",
  "INVOICE TOTAL" -> "SimilarString", "NET TOTAL" -> "SimilarString",
  "subtotal" -> "SimilarString", "nett total:" -> "SimilarString",
  "component total" -> "SimilarString",
  "sub-total:" -> "SimilarString", "sub total" -> "SimilarString",
  "subtotal:" -> "SimilarString",
  "total goods ex vat" -> "SimilarString",
  "sub-total" -> "SimilarString",
  "net value of goods" -> "SimilarString",
  "total net amount" -> "SimilarString",
  "net total:" -> "SimilarString", "goods value" -> "SimilarString",
  "products" -> "SimilarString", "net amount" -> "SimilarString",
  "nett total" -> "SimilarString",
  "total nett goods" -> "SimilarString",
  "invoice total" -> "SimilarString", "net total" -> "SimilarString",
  "COMPONENT TOTAL:" -> "SimilarString",
  "Sub Total:" -> "SimilarString",
  "Total goods ex VAT:" -> "SimilarString",
  "Net value of goods:" -> "SimilarString",
  "Total Net Amount:" -> "SimilarString",
  "Sub-total:" -> "SimilarString", "Goods Value:" -> "SimilarString",
  "Products:" -> "SimilarString", "Net Amount:" -> "SimilarString",
  "Total Nett Goods:" -> "SimilarString",
  "Invoice Total:" -> "SimilarString", "Net total:" -> "SimilarString",
  "Total" -> "NotSimilar", "TOTAL REMITTANCE:" -> "NotSimilar",
  "[GBP] TOTAL" -> "NotSimilar", "Total:" -> "NotSimilar",
  "Grand Total:" -> "NotSimilar", "Invoice total" -> "NotSimilar",
  "Total VAT" -> "NotSimilar", "Total Paid" -> "NotSimilar",
  "Amount Due:" -> "NotSimilar", "AMOUNT (USD)" -> "NotSimilar",
  "Order Total" -> "NotSimilar", "" -> "NotSimilar",
  "Invoice Total:" -> "NotSimilar", "Gross Total" -> "NotSimilar",
  "Net Price (per unit)" -> "NotSimilar", "TOTAL" -> "NotSimilar",
  "Total paid" -> "NotSimilar", "Invoice Total" -> "NotSimilar",
  "TOTAL INC. VAT:" -> "NotSimilar",
  "Total Amount Due" -> "NotSimilar", "TOTAL:" -> "NotSimilar",
  "GRAND TOTAL:" -> "NotSimilar", "INVOICE TOTAL" -> "NotSimilar",
  "TOTAL VAT" -> "NotSimilar", "TOTAL PAID" -> "NotSimilar",
  "AMOUNT DUE:" -> "NotSimilar", "ORDER TOTAL" -> "NotSimilar",
  "INVOICE TOTAL:" -> "NotSimilar", "GROSS TOTAL" -> "NotSimilar",
  "NET PRICE (PER UNIT)" -> "NotSimilar",
  "TOTAL AMOUNT DUE" -> "NotSimilar", "total" -> "NotSimilar",
  "total remittance:" -> "NotSimilar", "[gbp] total" -> "NotSimilar",
  "total:" -> "NotSimilar", "grand total:" -> "NotSimilar",
  "invoice total" -> "NotSimilar", "total vat" -> "NotSimilar",
  "total paid" -> "NotSimilar", "amount due:" -> "NotSimilar",
  "amount (usd)" -> "NotSimilar", "order total" -> "NotSimilar",
  "invoice total:" -> "NotSimilar", "gross total" -> "NotSimilar",
  "net price (per unit)" -> "NotSimilar",
  "total inc. vat:" -> "NotSimilar",
  "total amount due" -> "NotSimilar", "[GBP] TOTAL:" -> "NotSimilar",
  "Invoice total:" -> "NotSimilar", "Total VAT:" -> "NotSimilar",
  "Total Paid:" -> "NotSimilar", "AMOUNT (USD):" -> "NotSimilar",
  "Order Total:" -> "NotSimilar", "Gross Total:" -> "NotSimilar",
  "Net Price (per unit):" -> "NotSimilar",
  "Total paid:" -> "NotSimilar", "Total Amount Due:" -> "NotSimilar"};

(*Train a Machine Learning Classifier*)
modelItemDesc = Classify[trainingDataItemDesc];
modelSubtot = Classify[trainingDataSubtot];

(*Set Up the Word COM Interface*)
extractTextFromPDFUsingPDFtoText[pdfPath_String] :=
 Module[{outputPath, result},
  outputPath =
   FileNameJoin[{DirectoryName[pdfPath],
     FileBaseName[pdfPath] <> ".txt"}];
  RunProcess[{"C:\\Poppler\\Library\\bin\\pdftotext.exe", "-layout",
    pdfPath, outputPath}];
  result = Import[outputPath, "Text"];
  If[result === "", extractTextWithOCR[pdfPath], result]]

(*Determine If OCR Is Required*)
requiresOCR[text_String] :=
 StringLength[text] < 50 ||
  StringMatchQ[text, RegularExpression["[^a-zA-Z0-9]+"],
   IgnoreCase -> True]

(*OCR Extraction Function*)
extractTextWithOCR[filename_String] := Module[{pages, ocrText},
  (*Try to import the PDF as individual images*)
  pages = Import[filename, {"PDF", "Pages"}];
  (*If import fails,return a failure message*)
  If[pages === $Failed, Return["$Failed"]];
  (*Apply OCR to each page*)ocrText = TextRecognize /@ pages;
  (*Combine the recognized text from all pages*)StringJoin[ocrText]]

(*Combined Text Extraction Function*)
extractTextFromPDF[filename_String] :=
 Module[{text},(*First,try to extract text using Microsoft Word*)
  Print[filename];
  text = Quiet[Check[extractTextFromPDFUsingPDFtoText[filename], ""]];

  (*If the extracted text is empty or looks like gibberish,apply OCR*)

  If[requiresOCR[text], Print["Requires OCR"];
   text = extractTextWithOCR[filename]];
  text]

(*Search through all subfolders in a folder and extract all.pdf \
filenames*)
getAllPDFs[folder_String] := FileNames["*.pdf", folder, Infinity];

(*Specify the training and main folders where the PDFs are located*)
mainFolder =
  "Z:\\Shared\\03 - Operations\\06 - Supply Chain\\Invoice Data \
Extraction\\Example Invoices";

Print["Getting a list of all PDF files in the folder and its \
subfolders..."];
(*Get a list of all PDF files in the folder and its subfolders*)
pdfFiles = getAllPDFs[mainFolder];

(*Initialize a list to store the first entries in \
extractedNumbersSubtot for each file*)
firstEntriesItemDesc = {};
firstEntriesSubtot = {};

(*Process each file in the folder*)
Do[

  (*Load the file content*)
  invoiceText = extractTextFromPDF[pdfFiles[[i]]];

  (*Find positionsSubtot of phrases in the file content that match \
similar strings*)
  allPositionsItemDesc = {};
  allPositionsSubtot = {};

  Do[
   (*Find positionsItemDesc of the current phrase in the file content*)

   positionsItemDesc = StringPosition[invoiceText, phrase];

   (*Classify the phrase if it's found in the text*)
   If[Length[positionsItemDesc] > 0 &&
     modelItemDesc[phrase] === "SimilarString",
    (*Add all positionsItemDesc of this phrase to the list*)
    allPositionsItemDesc =
     Join[allPositionsItemDesc, positionsItemDesc]
    ],
   {phrase, Keys[trainingDataItemDesc]}
   ];

  Do[
   (*Find positionsSubtot of the current phrase in the file content*)

   positionsSubtot = StringPosition[invoiceText, phrase];

   (*Classify the phrase if it's found in the text*)
   If[Length[positionsSubtot] > 0 &&
     modelSubtot[phrase] === "SimilarString",
    (*Add all positionsSubtot of this phrase to the list*)
    allPositionsSubtot = Join[allPositionsSubtot, positionsSubtot]
    ],
   {phrase, Keys[trainingDataSubtot]}
   ];

  (*Extract Numbers Next to Similar Strings*)
  allItemDescriptions = {};
  itemDescriptions = {};
  extractedNumbersSubtot = {};
  numberPattern = RegularExpression["\\d{1,3}(,\\d{3})*(\\.\\d+)?"];

  Do[
   (*Print["positionsItemDesc dims = ",Dimensions[
   positionsItemDesc]];*)

   If[
    Dimensions[positionsItemDesc] =!= {0},
    pos =
     Flatten[positionsItemDesc][[
      2]];(*Get the end positionsItemDesc of the match*)

    (*Extract lines of text directly below the item description \
header*)
    itemLines = StringSplit[StringDrop[invoiceText, pos], "\n"];
    Print["itemLines Dims = ", Dimensions[itemLines]];
    Print["Length[itemLines] = ", Length[itemLines]];
    Print["Min[Length[itemLines]] = ", Min[Length[itemLines]]];
    (*Ensure we handle empty lines*)
    itemIndex = 1;
    qtyIndex = 1;

    (*Match the lines that are under the headers and collect them*)
    Do[
     If[StringLength[itemLines[[i]]] > 0,
      Print["i = ", i];
      itemLine = StringTrim[itemLines[[i]]];
      Print["itemLine = ", itemLine];
      AppendTo[itemDescriptions, itemLine];
      ],
     {i, 1, Min[Length[itemLines]]}]
    ],
   {positionsItemDesc, allPositionsItemDesc}
   ];

  Do[
   (*Print["positionSubtot dims = ",Dimensions[positionSubtot]];*)

   If[
    Dimensions[positionSubtot] =!= {0},
    pos =
     Flatten[positionSubtot][[
      2]];(*Get the end positionSubtot of the match*)

    (*Extract the text after the similar string's positionSubtot*)
    afterText = StringDrop[invoiceText, pos];

    (*Find the first number following the similar string*)
    number = StringCases[afterText, numberPattern, 1];

    (*If a number is found,add it to the list of extracted numbers*)
    If[number =!= {},
     AppendTo[extractedNumbersSubtot, number[[1]]]
     (*Print["No number found after positionSubtot: ",pos]*)
     ];
    ],
   {positionSubtot, allPositionsSubtot}
   ];

  (*Flatten the list to handle any nested structure and get the first \
number*)
  firstExtractedNumberSubtot =
   If[Length[extractedNumbersSubtot] > 0,
    Flatten[extractedNumbersSubtot][[1]], "No numbers found"];

  (*Add the result to the firstEntriesSubtot list*)
  AppendTo[allItemDescriptions, Flatten[itemDescriptions]];
  AppendTo[firstEntriesSubtot, firstExtractedNumberSubtot],
  {i, Length[pdfFiles]}
  ];

(*Final output:List of first extracted numbers for each file*)
Print["All item descriptions for each file: ", allItemDescriptions];
Print["First entries in extractedNumbersSubtot for each file: ",
  firstEntriesSubtot];

In the image below which I converted to a pdf, I would like the code to extract the Item Descriptions: "Endurance Watch SKU: 24-MG01", "Fusion Backpack SKU: 24-MB02", etc and the corresponding Subtotals: "$49.00", $50.00", etc.

How do I update the code to be able to do this?


text:
SpeechT5 Huggingface voice conversion: how to process whole input
I have followed the voice conversion example in the Huggingface blog post, and can replicate that example in a Colab session. As mentioned in the blog, the voice conversion stops consistently in the middle of the example input sentence, due to it interpreting the pause as the end of the sentence.
My end goal would be to use the SpeechT5 model to replicate the product ElevenLabs is offering. There you can upload a sound file up to 50 MB, and expect a voice conversion that time-wise matches the whole input.
I have started writing a Python script that will analyze the input audio clip, detect silence, cut audio only chunks, and send these non-silent chunks into the converter one-by-one. Finally the converted chunks will need to be "glued" together with the all-zero silence chunks, and written into a new WAV file.
Is this the only viable way to make the SpeechT5 model process a whole input file of let's say 10 minutes of 16-bit/ 16 kHz WAV mono input?