text:
Optimal decision with continuous, stochastic signals and rewards
I am performing a task, where I have to decide which projects to pursue at a given point in time, where the projects have different horizons of 30 minutes.
At a given point in time, forecasts are made on the profit of project that terminates 1 halfhour out in time, 2 halfhour out in time and so on up to 6 periods out in time. This give a "signal" on the outcome of the various decision, where time can be allocated to the different projects with different horizons and the profit is determined by how much time is allocated i.e. price/timeunit * timeunit, where the forecast is specified in price/timeunit. As the profit is only "expectations" they are not known for sure but they will either be at least as good as the forecast, or 0, depending on how things evolve within the next 30 minutes. The profit depends on the forecast in reality because the forecast influence the realized price/timeunit, but there is also data on the "optimal price/timeunit" i.e. the largest values that would have been possible in a perfect foresight scenario.
After the next minutes occur, a forecast is then again made from this new state with the same horizons as before. The thing is that the choice of horizon in the previous state influence whether it is possible to take on additional projects in this state because only a certain amount of projects is allowed (specified in time consumption and is NOT an integer number of projects but fractional). It might be the case that projects with longer horizons have a higher profit when being in a current state, but they will also constrain the flexibility of taking on additional (and maybe more profitable) projects as time pass by because time consumption is too high already.
I want to come up with either a decision policy or a way to weight the expected profits at a given point in time, to see if it is ever profitable to take on projects with longer horizon, and how much larger the profit has to be when doing that.
I have data on the forecast of profit and also data on the realized profit when taking the various decisions based on the forecast. And also the perfect foresight profits. All of these can be any non-negative real value as it is not possible to lose money but only make bad decisions resulting in lost opportunity. As mentioned only a pre-specified total "time consumption" but I do not have data on that but I can make various assumptions that I can allocate percentage of the total available time consumptions on various project e.g. 60% on a given range and 40% on another with longer horizon.
The only way that I can come up with on performing this analysis is a Markov Decision Problem with different states and stochastic rewards and stochastic states following a given decision, but I think it is hard to specify the state space and decisions as they depend the forecast and accumulated time consumption, and I dont know whether it is possible to make such analysis without simplifying the signals/rewards by dividing them into groups and not letting the reward and signals be continuous values. Also the process never terminates in reality so I do not know how to cope with that. The overall target is to come up with a way to determine how the available time consumption at a given point in time should be allocated based on the forecast because the projects with a longer horizon sometimes will have larger profit but restrict the possibilities in later states. Do you have any suggestions on how to do that?
Comment
So to elaborate: I get forecasts on the profit of various projects but they are uncertain (if the forecast is to high it will lead to 0 profit, but if it is lower than the upper bound of market participation it will lead to a profit >= forecast), but the goal is to know how to use these forecasts for decision making i.e. which projects to engage in either fully or fractional split between the different projects (1 project pr. horizon). I have timeseries data on the forecasts and the realized profit for taking various decision at given points in time, and the decision at a given state will influence up to the 6 following states as I work with a forecast horizon of 1-6 halfhours and the states is determined by the forecasts happening every 30 minute.


text:
Do GANs have constant running time?
After the model is trained, you just need to input random noise and the generator will output an image, does this mean GANs have constant running time ? I'm asking about both naïve GAN and variants of GAN

constant running time with respect to what?

What do you mean? I just asked about the running time complexity of GAN, does the model have a constant running time or what?

The question is a bit ill defined... usually when we want some bound on the running time, we have to say with respect to what
For example:

sorting is O(nlogn) wrt the size of the input
Transformer is O(n^2 * d) wrt to the number of input words (or number of patches)
Diffusion models have O(1) complexity if we just consider the forward/backward diffusion

Now, as you can see, the problem is "hat are we considering for the measure"... sure if you encode GAN will Fully connected layers, you just have the n^2 for matrix multiplication... however, diffusion models have N step of diffusion to account for, but with the big-O notation taht N is canceled as it is just a constant

Is inference time different from running time, I read from the internet that GAN's inference time is faster than diffusion model ?

@David yes, but for the definition of computational complexity, you cannot see such difference, as they differ by a constant... GANs only need 1 forward pass, diffusion models usually in the order of tens of forward passes

So both run at constant time ? Because you can define the number of steps in diffusion model

@David exactly, from a computational point of view, since it's constant, it's omitted from the analysis (though, in real life, you have definitely to consider it)

I guess diffusion model is better than GAN in every aspect ?

@David depends, diffusion works if the assumption hold, like the gaussian distribution of the targets, GANs don't make any assumption on the distribution

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Could an analysis of GPT4's WAIS score be published?
Similar to this: https://www.scientificamerican.com/article/i-gave-chatgpt-an-iq-test-heres-what-i-discovered/
But more detailed and in depth (subtest breakdown, including image analysis, etc.), WAIS-IV not WAIS-III, better methodology, etc.
Would it be a more interesting paper if it compared overall test scores and subtest scores to other LLMs, like 3.5, LLaMA, Mistral, etc.?
I know it's out of the box so I'm wondering if this is publishable in some journal. It seems that the WAIS interesting as there is both good human benchmark data, and it's never been on the internet, so there is no leakage.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Prefix tuning in LLM uses learnable vectors to fine tune the model
I would like to implement a new architecture for Transformer.
Below description is my thought.
Prefix tuning in LLM uses learnable vectors to fine tune the model.
Is there a way to use the output generated by the Neural network as prefix?
Thanks

Hi @jackson and welcome to AI Stack Exchange! If possible, please consider adding more detail to this question so that people may be able to answer it more quickly. Thanks, and we look forward to seeing you more on the site in the near future!


text:
How to decide on how to define the output function for a neural network?
What I want my simple ML model to do is for a certain case file, to take the type of crime and the amount of damage(converted into dollars), and to make a judgment (in dollars incorporating price for bailing) based on the two input params given. What I kind of got stuck on is how to define the output function. I understand how the ML system works and all, but I was really confused about what relations/connections I needed to make between the input nodes to create the output function. Could anyone help?

Please edit the Question and explain your doubt properly.


text:
Why is chatGPT 4 unable to correct a program given its output?
Recently I used chatGPT 4 to write a small project in Perl which involved reading of an exam file (multiple choice), comparing it to a master exam file, and scoring the student exam including fraud detection and statistics.
chatGPT did a great job of providing low-level code such as regex for pattern matching.
However, chatGPT failed miserably when asked for the program logic of a larger task such as reading the whole exam file and categorizing it into questions, answers, and hints.
What I found stunning was that it got the program logic about 80% correct.
However, even when giving it the whole output it seemed unable to adjust the code and sometimes even made it worse. It was like chatGPT hit a roadblock.
Why is chatGPT unable to refine its code even when prompted with the output and specific instructions on how the output should look vs. what it looks like now?


text:
The more a MLP neural network is trained, the value of correct class should increasingly closer to 1, and the other classes to 0? Is this expected?
I am using a multilayer perceptron network with the Iris dataset. This network has 3 output neurons (to represent the 3 classes of Iris, that is, Iris Versicolor, Iris Virginia and Iris Setosa). Therefore, the output of this network is a vector of 3 positions (for each class). The criterion I used to determine which class is predicted is
simply the index of this vector that has the highest value.
I used this neural network presented in this article:
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/
And the Iris dataset: https://github.com/selva86/datasets/blob/master/Iris.csv
I ran several tests experimenting with different amounts of epochs, learning rate, etc. With the tests I was able to notice the following:

If I train this network with just a few epochs, for example, 1 or 4    epochs, the result of all predictions in both the training and
test    data is always 0 or 1, that is, the network always makes
mistakes    because it always predicts the same value, which is
totally expected    as she trained very little.

But, increasing the number of epochs, to 20 epochs or more, I noticed that    the resulting vector output from this neural network
for the test    data is something like [0.999, 0.996, 0.993] (where
the highest value was    that of the first index of the vector, which
represent the first class). In my    test I did, for this sample the
network was correct saying that the    class was 1, and in fact this
was the desired class in the test    dataset.

However,at last, when training for more epochs, for example, 1000 epochs, it was also able to generalize to the test data, but with much
better output vector, with the value of the predicted class more
differentiable from the other values, for example [0.999, 0.345,
0.230]..

In the last test I did, notice that with this large number of epochs, the output vector contained much better values, which for the correct class were increasingly closer to 1, and for the other classes, closer to 0.
Note also that the values were much more numerically differentiable.
I had this doubt: the more a multilayer perceptron neural network (MLP) trains, the more the value for the correct class should become increasingly closer to 1, and the value of the other classes closer to 0? Is this expected behavior during training?

Since the Iris dataset as a ML benchmark quite represents its whole population, the sample error of your trained hypothesis accuracy testing could more or less be an unbiased estimator for its corresponding true error parameter along with likely quite small or practically acceptable variance. You may not need large number of epochs here to tradeoff bias-variance of your MLP model with an appropriate complexity level and weights distribution, do you know the usual ways to balance such bias-variance tradeoff?

@mohottnad Thanks for answering. I don't know. What are usual ways to balance such bias-variance tradeoff? And how does it work?


text:
Requirements for AI to be able to read a book and accurately recall the facts?
As a fun random project I've been working on is learning languages and making cheatsheets for them, dictionaries, etc.. I would love to encode the grammar rules which have been captured in books written in natural language (with some tables and diagrams in the books), across many different languages and writing systems. It is very tedious and time consuming to go into a book written by some grammar master, learn and grok each grammar rule well enough to turn it into structured JSON-like data, and then capture each one like that.
The Astadhyayi of Panini is over 1500+ pages (in translation), which is a Sanskrit grammar. I met someone who said they've spent the past few years converting that Sanskrit grammar into code (maybe 3-5 years), and they are only a fraction of the way through. Multiply this by 100-1000+ languages and you are going to be in for a thousand+ human years to capture the grammar data into structured form.
What is required for AI to be able to solve this problem: read a book (PDF or otherwise), learn the material just in that one book, well enough to abstract out each casually defined grammar rule into structured JSON? Are LLMs even close to being able to do this? Predicting next token seems a far cry from being able to represent and learn explicit knowledge like this. But I don't have enough intuition for determining how far AI is from doing this. Where are we at, and what are the unsolved problems or missing pieces we must address before getting to this level?
As a sidenote, my experience with ChatGPT trying to have it teach me about languages, is it gets 90% of the specific facts wrong, either entirely wrong or slightly wrong. Maybe it's just me. But maybe we need a different approach to reading a book and recalling the facts. I would just like to know at a high level what that might look like, if we have any sense yet.
Side-side note: I write this after seeing so many startups subsumed by OpenAI's latest release, and I'm wondering if I should even spend any time on my task or just wait it out haha, wait for AI to be able to solve it. I am not making a startup, just trying to learn several languages, but still.


text:
Neural Networks Model Predictions on Same Sample
I have built a tensorflow model to reconstruct the given input data sample by using an Autoencoder to predict the Missingness in the sample. However I do not have enough data sample where i don't have missingness in the samples. Hence I tried to check the model predictions with the mean imputed values where the NAN values are present.
However this analysis is not helping me well.
Hence, I thought to predict the same sample multiple times to understand how the model is predicting so that I can understand how the model is behaving in predicting the value where the NAN is present. I would then like to plot a distribution to understand the range of values that my model is trying to predict. However, I am getting the same exact predictions when I tried to predict the sample. May I know why it is happening like that?


text:
How to create meaningful features that allow unsupervised image classification?
I would train features that can later be correlated with categories, once I have some examples for the categories. Let's say one has a set of training images sorted by artist, and wants to create some kind of signature that will later allow unseen images to be grouped by artist, without having seen the images of the artist before.
I think if one trains a supervised CNN, one will get sharp features that match the known artists well, but may be completely random on images of artists that were never seen before.
Would it be a good idea to use a CNN and classify in the last layer with a sigmoid, but try to get it to create meaningful features in the previous layers that could be used to create a kind of signature for the artist, so that the hard cut in "Artist A/B/C/unknown" is only done in the sigmoid layer? How would you force the previous layers to learn diverse features instead of just fitting to the dataset of images with known artists?


text:
Symbolic AI in the medical field, specifically in diagnosing a patients condition. Have i understood the concept behind how it works?
This scenario should demonstrate just how the system along with the rule works in a medical diagnosis where the symptoms of the patient are evaluated by the system to determine what condition he/she could have.
Just say a patient is suspected to be experiencing anaphylaxis, although the onset of anaphylaxis usually happens within seconds to minutes and an EpiPen is usually on standby, it wouldn't have the need to be diagnosed by a Symbolic AI in the first place, for the sake of argument lets say that it did (hypothetical situation), The rule related to the system would look like this.
If the patient has anaphylaxis AND the patient has a weak and fast-paced pulse AND struggles with abdominal pain the conclusion could be that the patient may be going into anaphylaxic shock.
The process the AI goes through to reach this conclusion is by connecting the dots per se through deductible reasoning and making the logical assumption that since the patient has all the symptoms it's most likely anaphylactic shock. New information could come in the form of a new symptom. This new information would lead the AI to deduce it through a reasoning engine like that of a Bayesian network (it leans toward reasoning with the information that's been presented to it rather than just directly relaying information to what's being asked as a search engine would do). The reasoning process involves the need for human intervention when a phenomenon arises (the new symptom could be ear pain). Since the symptom doesn't correlate to anaphylaxic shock it could potentially need a new system in place since it's most likely a symptom of an earache. Regardless of this, the new information by which the phenomena relate or don't relate in this case and potentially in need of a new system must be hard-coded directly into the system's program, updating the rule. However, If the symptom had been struggling to breathe the current system rules would have been updated to include it, since it is a symptom of anaphylaxic shock. The rule coded in is a preconception, in other words, it's a hard assumption. Ultimately the system should be a flexible one (assumption wise) and the input given results in an accurate decision.

What is the question here? Seems there is none.

I'll see if I can clear it up. The question is this the process a Symbolic AI goes through when diagnosing a patient's condition? Have I got all the details correct such as when a new symptom (new information) arises in a patient, say the symptom doesn't correlate to the assumed diagnosis, in this case, anaphylactic shock, does the current system get updated or does it need an entirely new system with a new rule? This is what I'm trying to say. Hope this helps.


text:
How can you add additional features/attributes while doing instance segmentation?
I want to do an instance segmentation of objects in images.
Usually I would stick to something like an Mask R CNN and let it run. However additionally to the image itself and the pre-labeled images, I have additional features that might be interesting for the segmentation.
Example: I want to segment certain products in images from a factory and I have additional information about the products than run at a specific time (like product family, color, brand, etc.).
How do I add these additional features in an instance segmentation?


text:
in vgg16 ssd model, the feature maps from Conv5 and Conv7 are not directly used for prediction. Is my understanding correct?
the following ssd architecture comes from the original paper with 2 arrows i added.

the layer pointed out by the red arrow labelled "Conv4_3", followed by Conv6, and then Conv7. Conv4, Conv7 and the layers after Conv7 are fed to the layer pointed out by the blue arrow, which means Conv5 and Conv7 are not directly used for prediction. Is my understanding correct?
what is the layer pointed out by the blue arrow called? the detector layer?


text:
Are the outputs of layers in Attention Is All You Need interpretable by mapping to tokens?
In the basic transformer model from 2017, I'm a bit confused what the outputs of each layer are supposed to be. Are they embeddings? If so, does that mean you could examine a given output from a given encoder and decoder layer and covert it to tokens basically? In other words, for a given layer output, suppose you used a similarity search over the token embeddings for the entire vocabulary, and then matched the most similar words to the embeddings from the chosen layer. Would this give you some sense of what words it was basically considering at that point in the architecture?


text:
When using Reinforcement Learning with Human Feedback to train a transformer, how do I propagate the feedback through the transformer?
I'm basically trying to replicate the processed used to create Chat GPT:

Am I supposed to backpropagate? How can I do that when these aren't really errors, but rather ranking several response? Can I use a 1-10 rating system where 10 is a perfect response instead so I have something closer to an error signal?
Also I have to rank/rate each output as a whole, but the final layer of the model are token neurons. Do I somehow connect my ranking data to the layer right before the final layer? Is the reward model supposed to be completely separate from the transformer with no direct edges/weight connections? Do I update the transformer model using the Bellman equations?
I am shocked I can't find a tutorial or really any information about this crucial component given that this is the strategy used to make Chat GPT according to OpenAI. Is there something I'm missing that someone could direct me towards? All I can find is information about fine-tuning transformers and RLHF separately, but not combined.


text:
In the conditional GAN (cGAN) architecture, why does the discriminator need conditional variable?
I'm reading about conditional GAN (cGAN) architecture, what I know is that the generator creates images combining both noise vector and conditional variable, the noise vector brings in random elements like colors or shapes while conditional variable is used for maintaining the same object.
As for the discriminator, the input is an image that is either fake (generated by the generator) or real (from the dataset) combines with the conditional variable. What I don't understand is that why do we also include the conditional variable in the discriminator, I get that the generator needs them for guidance, but why does the discriminator, which is just classifying fake or real, require this additional information?

Because otherwise there is no conditioning... consider the case where you condition the generator but not the discriminator: given an image and a label, the generator proposes an image, which will be passed to the discriminator... however, it has no idea what that image should represent, so at most it can at most guess if it comes from the joint distribution or not (aka dataset, as a naive GAN)
So, by giving the information also to the discriminator (take the digit MNIST case), the discriminator learns how a specific digit looks like, forcing the generator to generate realistic images of that specific label

Why does the discriminator need to know what do the images represent ? I still don't understand, I think giving the label to the discriminator is meaningless since the generator already produces images that correlated with the label

@abcd no, the generator is no forced at all to understand your label as conditioning if not forced by the discriminator, it can just ignore such information and fall back to the normal GAN, try and you'll see

Are you saying that the generator will produce something different from the conditional label I provide? I read from the tutorials that they combine the one-hot encoding vector of the label with the noise vector, then the generator will always output the image correlated with the label

@abcd given you also give the concatenated vector to the discriminator, if you don't condition your discriminator, the generator will be free to ignore the label given as input

Thanks for your help

Do you think the discriminator is also ignore the label too ?

@abcd not quite sure what you are referring to, but the discriminator most likely won't ignore it

I thought that in cGAN, the discriminator's role was only to classify images as fake or real, not to check if the images match the conditional labels. What's your opinion on this ?

@David discriminator in cGANs acts as an energy model, giving you an unnormalized score of the probability of the input being from the dataset distribution, thus in addition on checking if the generated image come from p(x), it will check if the pair label/image come from p(x,y)


text:
Using Variable Text Features in Machine Learning Model
I am building a classification model using deidentified patient data with ICD-10 codes as inputs. Each code is a string and represents a diagnosis, and these follow the pattern of 1 letter, followed by 2 digits, followed by numbers and/or letters. However, patients can have multiple associated ICD-10 codes, so I am not sure how to handle this. There are some logical groupings, though, and the initial letter of the code represents a general category of diagnoses (e.g. cardiovascular).
A patient may have multiple ICD-10 codes. For instance, if a patient has hypertension and diabetes, this may be I1310 and E119.
My naive approach would be to make, say, 10 columns for each patient [diagnosis 1, diagnosis 2, etc.] and use that as training data. However, this has the issue that, for instance, even if two patients both have diabetes, patient A may have this as diagnosis 2 but patient B may have this as diagnosis 4. I am assuming this would hinder the model from training properly.
Any thoughts on how to approach this problem?


text:
In policy gradient methods why do we compute the gradient of the objective function through a one-trajectory estimate?
Taking as an example the Advantage Actor Critic, the objective function is:
\begin{equation}
 \nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})=\mathbb{E}_{\tau \sim \pi_{\boldsymbol{\theta}}}\left[\sum_{t=0}^T \nabla_{\boldsymbol{\theta}} \log \pi_{\boldsymbol{\theta}}\left(a_t \mid s_t\right)A^{\pi, \gamma}(s_t, a_t)\right],
\end{equation}
where $A^{\pi, \gamma}(s_t, a_t) = Q^{\pi, \gamma}(s_t, a_t) - V^{\pi, \gamma}(s_t)$ is the advantage function.
In principle in order to compute the gradient we should collect many trajectories and take the average of the desired quantity (which is what REINFORCE does). I guess that computing many trajectory is very costly, and indeed in Actor-Critic-like algorithms, such gradient is estimated by sampling just one trajectory (the policy is updated after every episode).
I understand that we exploit baselines in order to reduce variance for better estimates but I would like to understand why the one-trajectory sample is a good estimator of $\nabla_{\theta}J(\theta)$. Are there any rigorous references about this?

The one-trajectory sample is the Monte-Carlo way to estimate the gradient, which, turns out, to be an unbiased estimator although with high-variance due to relying only on one sample: I think this is proven in the Sutton's RL book. Unbiased means that it converges (in the limit of infinite experience) to the true gradient, which is something good but the high-variance makes everything more difficult, slow and ustable.
To reduce the variance people usually use a baseline, but you can also have parallel environments, which should be uncorrelated (i.e., initialized all differently), allowing you to collect a batch of trajectories that you can average on to reduce a bit the noise due to the stochastic nature of the agent-environment interaction.
There are also combinations of ideas from both MC and TD learning that allow you to trade the bias for the variance, for example by truncating the trajectory early (to reduce variance) and then bootstrapping (which introduces bias.)

Hi, yes I know the N-step bootstrapping to set up a trade off between bias and variance but as I've seen it often refers to the computation of the returns (and then the advantages), as it is explained here https://arxiv.org/pdf/1506.02438.pdf.
Indeed, there it turns out that the empirical returns are good estimators of the Q function and one can obtain them with a bootstrapping procedure. However, in the same paper the gradients is still approximated through an average over many trajectories (eq. 9). I cannot find a similar analysis for the whole gradient in the Sutton's book.

@LorenzoMancini yes, but you need to consider that, in practice, for deep RL the gradient is always approximated because you can't compute a full expectation over all possible trajectories: it's just intractable. What you can do is to sum over a finite num of trajectories, and MC assumes one trajectory is enough.

I know that it's intractable to compute a full expectation, that was not my question. Btw I see the point, ty.

Sutton & Barto's book of Reinforcement Learning mentions below:

When the state-value function is used to assess actions in this way it is called a critic, and the overall policy-gradient method is termed an actor–critic method. Note that the bias in the gradient estimate is not due to bootstrapping as such; the actor would be biased even if the critic was learned by a Monte Carlo method... The natural state-value-function learning method to pair with this is semi-gradient TD(0).

Thus this inherent bias in actor-critic algo using one-trajectory sample during every episode to update both its actor's and critic's parameters at every step (see page 332 for its algo detail) faces the same convergence issue as any TD boostrapping learning algo whose convergence is a theoretical result from stochastic approximation methods as discussed here and the relevant source in the same reference is summarized below. So long as the advantage function as approximated by the difference of the one-step return and the current state value as the baseline (both values are from the same critic) converges to their true values just like TD state values learning, the policy actor and critic parameters would optimize the policy's performance metric per the policy gradient theorem.

converges with probability 1 to an optimal policy and action-value function, under the usual conditions on the step sizes (2.7), as long as all state–action pairs are visited an infinite number of times and the policy converges in the limit to the greedy policy... but not for the case of constant step-size parameter, $α_n(a)=α$. In the latter case, the second condition is not met, indicating that the estimates never completely converge but continue to vary in response to the most recently received rewards. As we mentioned above, this is actually desirable in a nonstationary environment, and problems that are effectively nonstationary are the most common in reinforcement learning.

Therefore actor-critic algo usually needs tons of data for training in order to approximately satisfy above convergence conditions, yet it's often superior mainly due to substantially reduced variance.


text:
kerascv retinanet for gender identification
I am using KerasCV Retinanet to detect people and their genders in images.
I  would like to detect "man", "woman", "boy", "girl" and "baby" in images. I treat them as separate classes in my annotation. Should I detect "person" and then use a separate classification model to figure out the gender or is gender identification by using just retinanet is the right approach?
I have been struggling with the implementation; hence, was not sure if the problem is my approach or the data quality.
Thanks,
Doug


text:
Why my deep learning model (FCNN/ 1DCNN) fails to learn when training on medical dataset?
I am working on a project to predict the severity of the disease, Hemophilia using a deep learning model(FCNN or 1DCNN). I am working based on the information provided in this article: https://www.sciencedirect.com/science/article/pii/S0888754320308193?via%3Dihub#s0070
I have used the raw data files for Haemophilia ‘A’ Factor VIII gene mutations provided under the "Appendix A. Supplementary data" section. The dataset has 7784 rows and 17 columns.  After preprocessing, the final dataset has 5257 rows and 58 columns. Here is the link to the preprocessed data: https://drive.google.com/file/d/1S3t5Ka_mhetB60sndZ9i7OsXaj0jxUVN/view?usp=sharing
I used 1DCNN to train the data, the result was not promising either.  I also uploaded some of the learning curves to demonstrate the loss and accuracy of the models.

How can I fix this issue? Which part am I doing wrong?
The full version of my code is available below:
# Read the CSV file into a DataFrame
data = 'filtered_rows.csv'
df = pd.read_csv(data)
print("The dataset has {} rows and {} columns".format(df.shape[0], df.shape[1]))

# Separate features and labels
X = filtered_rows.drop(columns=['Severity']).values
y = filtered_rows['Severity'].values

print("The dataset has {} rows and {} columns".format(filtered_rows.shape[0], filtered_rows.shape[1]))

##################Class_Weight && Applying_OverSampling##################

# Calculate class weights manually
class_weights = {}
for cls in np.unique(y):
    class_weights[cls] = len(y) / (len(np.where(y == cls)[0]) * len(np.unique(y)))

# Apply oversampling to the 'Moderate' class
smote = SMOTE(sampling_strategy='auto', random_state=42)
X_oversampled, y_oversampled = smote.fit_resample(X, y)

from collections import Counter

# Assuming you have variables y and y_oversampled containing the original and oversampled labels
class_counts_original = Counter(y)
class_counts_oversampled = Counter(y_oversampled)

# Count samples in each class in the original dataset
total_samples_original = len(y)
samples_mild_original = class_counts_original[0]
samples_moderate_original = class_counts_original[1]
samples_severe_original = class_counts_original[2]

# Count samples in each class in the oversampled dataset
total_samples_oversampled = len(y_oversampled)
samples_mild_oversampled = class_counts_oversampled[0]
samples_moderate_oversampled = class_counts_oversampled[1]
samples_severe_oversampled = class_counts_oversampled[2]

print("Original Dataset:")
print(f"Total samples: {total_samples_original}")
print(f"Mild samples: {samples_mild_original}")
print(f"Moderate samples: {samples_moderate_original}")
print(f"Severe samples: {samples_severe_original}")

print("\nOversampled Dataset:")
print(f"Total samples: {total_samples_oversampled}")
print(f"Mild samples: {samples_mild_oversampled}")
print(f"Moderate samples: {samples_moderate_oversampled}")
print(f"Severe samples: {samples_severe_oversampled}")

########################################################################

# Create a StandardScaler instance
scaler = StandardScaler()

# Fit the scaler to your data and transform it
X_normalized = scaler.fit_transform(X_oversampled)

# Display the updated DataFrame
df_normalized = pd.DataFrame(data=X_normalized, columns=filtered_rows.columns[:-1])  # Exclude the 'Severity' column
display(df_normalized.head())

# Create a MinMaxScaler instance
scaler = MinMaxScaler()

# Fit the scaler to your data and transform it
X_scaled = scaler.fit_transform(X_normalized)

# Display the updated DataFrame
df_scaled = pd.DataFrame(data=X_scaled, columns=filtered_rows.columns[:-1])  # Exclude the 'Severity' column
display(df_scaled.head())

###########################FCNN_Model###########################
# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y_oversampled, test_size=0.2, random_state=42, shuffle=True)

# Create a fully connected neural network model
model = Sequential()
model.add(Dense(128, input_dim=X_train.shape[1], activation='relu'))
#model.add(BatchNormalization())
#model.add(Dropout(0.3))
model.add(Dense(64, activation='relu'))
#model.add(Dense(64, activation='relu', kernel_regularizer=l1(0.001)))
model.add(Dense(3, activation='softmax'))

# Compile the model with a custom learning rate
model.compile(optimizer=Adam(learning_rate=0.001), loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Train the model
history = model.fit(X_train, y_train, epochs=200, batch_size=32, validation_split=0.2, class_weight=class_weights)

# Evaluate the model on the test set
loss, accuracy = model.evaluate(X_test, y_test)
print(f"Test Loss: {loss:.4f}")
print(f"Test Accuracy: {accuracy * 100:.2f}%")

###################################

# Model Prediciton
y_pred = model.predict(X_test)
y_pred = np.argmax(y_pred, axis=1)  # Convert softmax output to class labels

# Create the confusion matrix
confusion = confusion_matrix(y_test, y_pred)

# Create a heatmap for the confusion matrix
sns.heatmap(confusion, annot=True, fmt="d", cmap="Blues")

# Add labels and titles
plt.xlabel('Predicted')
plt.ylabel('True')
plt.title('Confusion Matrix')

# Display the plot
plt.show()

###############Classification_Report##################

class_names = ["Mild", "Moderate", "Severe"]

# Calculate precision, recall, F1-score, and support
report = classification_report(y_test, y_pred, target_names=class_names, output_dict=True)

# Convert the report to a DataFrame for easy visualization
df_report = pd.DataFrame(report).transpose()

# List of columns you want to format
columns_to_format = ['precision', 'recall', 'f1-score', 'support']

# Apply formatting to each specified column
for column in columns_to_format:
    df_report[column] = df_report[column].map('{:.2f}'.format)

# Then, display the formatted DataFrame
print(df_report)

###########################1DCNN_Model###########################
# Define the 1D CNN model
#model = keras.Sequential()
#model.add(layers.Input(shape=(X_train.shape[1],)))  # Input layer
#model.add(layers.Reshape((X_train.shape[1], 1)))  # Reshape the input for 1D convolution

# Convolutional layers
#model.add(layers.Conv1D(64, kernel_size=3, activation='relu'))
#model.add(layers.MaxPooling1D(pool_size=2))
#model.add(layers.Conv1D(128, kernel_size=3, activation='relu'))
#model.add(layers.MaxPooling1D(pool_size=2))

#model.add(layers.Flatten())  # Flatten layer

# Fully connected layers
#model.add(layers.Dense(128, activation='relu'))
#model.add(layers.Dropout(0.3))  # Dropout layer to prevent overfitting
#model.add(layers.Dense(64, activation='relu'))
#model.add(layers.Dropout(0.3))
#model.add(layers.Dense(3, activation='softmax'))  # Output layer with 3 classes

# Compile the model
#model.compile(optimizer=Adam(learning_rate=0.001), loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Train the model
#history_1DCNN = model.fit(X_train, y_train, epochs=100, batch_size=32, validation_split=0.2)

# Evaluate the model on the test set
#loss, accuracy = model.evaluate(X_test, y_test)
#print(f"Test Loss: {loss:.4f}")
#print(f"Test Accuracy: {accuracy * 100:.2f}%")
##############################################################


text:
Output Not Changing - Feeding Previous Outputs Back Into a Model
Full disclosure, I also posted this on Stack Overflow I have put a more theory based bent towards the question itself here
I have a simple model in pytorch based on the quickstart except instead of a pre-made data-set I am trying to find parameters for a mathematical function to fit a curve, I am then trying to feed back into the network the "goodness of fit" through the loss for the curve. My output is (6 parameters) for the function.
I gave as input:

The independent input at N inputs (lets call it x), this does not change
The dependent output at N outputs (lets call it y(x)), this does not change
The output of the mathematical function f for each point of x, and parameters p (lets call it f(x, p) where p is the same for all x)
The last set of guessed parameters p

Now I have a simple cross entropy loss function, and stochastic gradient descent for back-propagation in a simple feed forward network, for which the output is a new set of 6 parameters for f, p', I take p' and x and put it into f and back-propagate based on y(x) and f(x, p'), not p, loss is indirectly based on the output.
Behavior I am observing is that the output does not change very much, really just the lower digits (also should it be outside the bounds [-1, 1]), neither has the loss. I have made attempts to divide it out by the absolute value of the absolute maximum value of the output when feeding it back into the network but no luck
Questions: A: Am I doing anything fundamentally wrong? Can I feed the previous results back into a network and expect a change? Can I sort of feed it a "score" to tell it how well its doing? What kind of model may be able to do this, if not feed forward? B: Could this be a normalization problem? C: How might I diagnose this problem?
I'm trying to do a sort of "live" learning/generation or "active" learning perhaps.
Thanks!

Looking at genetic algorithms, or genetic + NN, also possibly baysian optimization, and NEAT


text:
When to know if I am "on the right track" for a CNN architecture
Context
Very new to CNNs and ML in general. I am building a simple binary image segmentation network for generating black and white image masks (white pixels = desired object; black pixels = all else). To this end, I am taking a downsampling/upsampling approach utilizing convolutions and max pooling to downsample, and then transposed convolution to upsample.
My network does extremely well on the training data, mastering it in under 50 epochs, but struggles with the test set. I know that the best ways to mitigate overfitting are reducing complexity, regularization, data augmentation, etc. but I'm really struggling to find that sweet spot between not learning at all and essentially memorizing the test set. I'm aware that a lot of building a robust architecture is mostly empirical.
My question(s)
How can I know if a high-level framework such as the depth or number of filters/kernels in a given convolution, as opposed to smaller tweaks such as hyperparameter tuning or dropout, is "close" to capturing the essence of the task? Is it a "good sign" that my model, despite aggressively overfitting, is able to capture patterns that I'm interested in? Or is it possible that my approach is inherently inadequate for the task?
TL;DR
Is overfitting, as opposed to not learning, a sign that I'm headed in the right direction, architecturally speaking?

It is always good to first attempt to overfit your architecture on a very simple dataset to ensure that your architecture can actually learn something to the extend that you want it. I know of many data scientists that they can often take 10 samples to overfit their models, just to see whether they can. It might not necessarily mean your code is bug free, but it assures you that you are on the right track. There is nothing wrong with this approach. I've even fitted generative time-series models on the standard base sine curve. You might try a single digit from MNIST for example.


text:
How to get exact learning time without loading library time?
I have some questions about get learning time.
Most people think getting learning time like that...
import time
import tensorflow as tf
from tensorflow import keras
... #passing detailed code
start_time = time.time() #check start time
model.fit(x_train, y_train, epochs=10) #do learning
end_time = time.time() #check end time

training_time = end_time - start_time #get learning time

but, I had an inexplicable doubt about that code.
When we do model.fit(), then It load libraries such as CUDA before learning. so, I think training_time includes loading libraries time, so that time is not pure learning time.
So, If this doubt is correct, then how can we get pure learning time exclude loading library time?
Thank you.


text:
How was the word2vec model trained?
Let's take the CBOW (continuous bag of words) model as the example.
Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input vector is $v \cdot c$, where $v$ is the total number of words to be learned.
Suppose the first hidden layer has $d$ neurons, which is the size of the embedding dimension.
So all weights (or: embedding) parameters that are to be learned can be arranged in a $v \times d$ matrix.
My question is concerning how many weight parameters are updated during each training epoch for the word2vec model. Are all the $v \cdot d$ embedding, parameters updated during each training epoch? Or, are only $c \cdot d$ of them updated during each training epoch? Here the $c \cdot d$ embedding parameters are for the input context words.
If all $v \cdot d$ weights got updated during each epoch, it would be much more expensive, considering that the typical value of $v$ ranges from tens of thousands to hundreds of thousands.
I suppose, maybe a detailed backprop formulation would help me understand this. Where can I find such a detailed formulation? I'm not looking for a general theoretical backprop formulation, but seeking the one for the word2vec (e.g., the CBOW model).

Have you looked into this code, https://github.com/tmikolov/word2vec/blob/master/word2vec.c? The programmer, Thomas, is one of writers of Efficient Estimation of Word Representations in
Vector Space (https://arxiv.org/pdf/1301.3781.pdf)

@CloudCho Thank you so much for the code and paper links. They are very helpful. I will look into them. I don't have enough point in this community, otherwise I'd have voted up your comment.

Please, consider putting your study in the Answer section. Putting own answer isn't a bad practice unless you mark as the "Accepted Answer" :).


text:
Base rate effects on object detection neural networks?
Suppose that we are training an neural network for object detection.  It is natural to train the neural network on images that contain the object in question.  However, this might have the effect of biasing the "base rate" of the network: The network "expects" there to be at least one object worth detecting in any image.
Are there any scholarly articles that investigate this phenomenon, and the effect of strategies to alleviate it (such as adding more true negatives to the training data)?


text:
What Kind of Models and Loss Functions for User Churn Prevention by Promo Codes?
The Company Business Model
Bike rental with an app, where riders pay for the time they rented the bikes for.
The Business Case
User (rider) attrition prediction, and ideally, prevention. Basically, once a user hibernated (has not rented) for 30 days, we send a promo code with some discount on their next rental, if hibernated for 45 days, then another promo code etc. This business logic uses something like an RFM analysis. But delivering each code has its own cost, and then, not all users are likely to use the code either.
Data We Have
Users' ride histories, along with some demographic data such as home address, work address, gender, age, income level etc. along with some external data like weather, public transport data etc.
Modelling Objective
Basically, replace the business logic I said earlier with a model, based on historic data, that will select $N$ users everyday (let's say $N=5,000$) to send a promo code, the basic criteria (in plain English) being

Users who are least like likely to engage with the platform absent the promo code
Users most likely to use the promo code

Obviously, we do not want to waste the promo by sending to a user who is likely to ignore it, but also not to anyone who is likely to engage anyway. In other words, how to select users who are most likely to be swayed by the promo. Of course, we will have some postprocessing logics as well to not badger a user with a promo every day, but that's an implementation detail. Also, we are not tuning the promo amount/conditions with the model, the sole purpose of the model (or some blackbox algorithm) would be to select $N$ users to send a promo code.
The Business KPI
Average weekly ridership after the model is deployed. The management does not care about the model details, loss function etc. (obviously). If the ridership after the model deployment shows a lift, then I get a raise, if it drops, then I get fired.
Question
Although I have the data, and know all about developing specific classification, forecasting models etc., coming from a big data engineering background, I am struggling with translating this business problem into a well defined machine learning problem. I can featurise the history and demographic, but what should be the type of label, and the loss function/metric to look for (assuming I am using some variations of feed forward DNN, or other tree based methods like XGB etc.)?
Or, is this question (selecting the best the $N$ users) addressed better by not a model (in ML sense), but some kind of statistical number crunching? I realise the question is probably more business intelligence than data science, but any help or reference would be appreciated, on how would you approach it.

Do you have any data, from previous promo codes, on whether any of the user accounts have used promo codes in the past? It changes the nature of this problem a lot if you have no current ground truth for your target.

@NeilSlater yes, I have historical promo code usage data


text:
Could we use another attention module on the outputs of the attention heads in the transformer architecture?
Before concatenating the heads in MHAtt, could we add another attention module with the heads as input to combine them?
Thus, for each head we would get a value matrice enriched with the outputs of the other heads, possibly allowing for an even more coherent view on the input as each head's output is additionally improved by insights from the other heads.

attention means "how much this thing X is useful for this other thing Y"...  the problem with your approach is that i see how X=head, but i can't see what us Y=?

Ah, I see. Thank you.


text:
Could converting features into PCs, where #PCs = #features, improve model performance?
Let's say I have a dataset with some number of features (e.g., 10) and a target variable.
I create 10 PCs from the dataset excluding the target variable. Then, I run a few classification algorithms (XGBoost, SVC, RF, etc.) on these new PC features to predict the target variable.
Is there a reason to expect that this technique will outperform simply using the original features, in the case where we use all available PCs?
I have some intuition that this may help some models learn by making multicollinearity less of an issue, but I'm not sure this is right.

I'd recommend that you don't use abbreviations in the title. PC is not really a standard abbreviation, so people will not immediately understand the idea of your problem by reading the title.


text:
Should the encoder be trained for more steps in VAEs?
This is based on my interpretation of how the ELBO loss works. The log likelihood of the data is equal to the ELBO + KL Divergence term.
$$\operatorname{log} p_{\theta}(x) = \underbrace{\mathbb {E}_{q_{\phi} (z)}\left [\operatorname{log}\frac{p_{\theta}(x|z)p(z)}{q_{\phi} (z)} \right ]}_{\text{ELBO}} + D_{KL}(q_{\phi} (z) ~||~p_{\theta}(z|x) )$$
Now the LHS is constant with respect to the parameters of the encoder $\phi$ but not with respect to the parameters of the decoder $\theta$. Most explanations I have come across say that because the LHS is constant, maximizing the ELBO would reduce the KL Divergence term and this would tighten the bound. But in practice we optimize both the encoder and decoder parameters jointly. My intuition says that it would make more sense to train the encoder for more steps before training the decoder and repeating the process. So I would really appreciate any clarification on the subject.

Expectations are hard to approximate, you cannot know how many sample you'll need... however, seems that 1 is enough, that's why you dont train it for multiple steps


text:
How to select an equilibrium among the landscape of Nash equilibria in poker?
Reading some articles on counterfactual regret minimization solutions of HULHE and HUNLHE poker published in the last decades -like 1, 2- i was wondering what can be said about the different Nash equilibria, in heads-up poker to begin. First of all am i right to think that if both players play equilibrium strategies, possibly different, they both have 0 expectation, when averaging over both starting positions -dealer and small blind ? The reasoning is that neither can improve his expected utility by deviating so his expected utility is maximal given the other's strategy, but the game being symmetric no player can expect a utility greater than the other at an equilibrium, and the sum of expected utilities being 0 each player has 0 expected utility.
A priori HULHE poker has many equilibria, yet despite the a priori very different strategies, say $S$ and $S'$, and different payoffs for player 1 choosing strategies $S$ and $S'$ for any particular card distribution, in the long run all the different outcomes would even out to give 0 expected payoffs for both $S$ and $S'$. This observation leads me to wonder: how different can equilibrium strategies be ? For simple $n$-player games, like stag hunt or battle of the sexes, the different equilibria are well understood and there is a theory of equilibrium selection -introducing for instance the notions of payoff-dominance and risk-dominance. When passing to a complex game like poker there must be very intricate coincidences between the different kinds of strategy, very interesting tradeoffs, which must be computationally difficult to investigate and often surprising.
Question: Has there been research on the landscape of equilibrium strategies in poker, if only to propose a theory for selecting them, like in terms of risk of ruin or exploitativity ? Where "exploitativity" is a measure -to be defined, with different possible definitions- of how much a strategy exploits nonequilibrium strategies.
Of course this question can be considered in contexts intermediate between the simple $n$-player games mentioned above and the complicated one of poker. One can also consider yet more complex games. But it seems that HULHE poker is a good case to study. For the classical cases the selection of equilibria has been studied by Harsanyi and Selten, but their methods look far from poker.
Thank you.


text:
What algorithm should I use to train an antichess agent?
I will implement an antichess agent and am not sure about which algorithm to use. My current candidates are minimax with alpha-beta pruning, MCTS and proximal policy optimization.
Should I consider antichess just as regular chess in the selection of algorithms? Which algorithm could be expected to give better results?
Antichess, also known as suicide chess, is a chess variant where the objective is to lose all your pieces or be stalemated rather than checkmating your opponent.
For more info: https://lichess.org/variant/antichess

Maybe link what antichess is? I'm assuming some version of chess where the goal is to force your opponent to put you into check? If there are any variants, say which one.


text:
What else could I try to avoid catastrophic forgetting in my implementation of Semi-Gradient SARSA?
I was trying to implement the semi-gradient SARSA algorithm (see p.244 Reinforcement Learning: An Introduction) using PyTorch:
class SemiGradientSARSA(nn.Module):
    def __init__(self, env, hidden_dim, lr, epsilon) -> None:
        super().__init__()
        self.state_space = env.observation_space.shape[0]
        self.action_space = env.action_space.n
        self.goal_position = 0.5 # used for customizing the reward function in MountainCar
        self.starting_position = -0.5 # used for customizing the reward function in MountainCar
        self.epsilon = epsilon
        self.q_net = nn.Sequential(
            nn.Linear(self.state_space, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 2*hidden_dim),
            nn.ReLU(),
            nn.Linear(2*hidden_dim, self.action_space)
        )
        self.optimizer = torch.optim.Adam(self.q_net.parameters(), lr=lr)

    def forward(self, x):
        x = torch.from_numpy(x)
        return self.q_net(x)

    def act(self, state):
        # epsilon greedy policy
        p = random.random()
        if p < self.epsilon:
            action = np.random.randint(0, self.action_space)
        else:
            q_values = self(state).detach().numpy()
            action = np.argmax(q_values)
        return action

    def train(self, n_episodes):
        for i in range(n_episodes):
            print(f'episode {i}')
            done = False
            trunc = False
            state, info = env.reset()
            action = env.action_space.sample()
            while not done and not trunc:
                next_state, reward, done, trunc, info = env.step(action)
                position, vel = state
                # reward for being further away from the starting position
                if not done:
                    reward = -0.1 * np.abs(position - self.starting_position)
                else:
                    reward = 10
                if done:
                    pred = self(state)[action]
                    loss = 0.5 * (reward - pred)**2
                else:
                    next_action = self.act(next_state)
                    loss = 0.5 * ((reward + 0.99 * self(next_state)[next_action]).detach() - self(state)[action])**2
                loss.backward()
                torch.nn.utils.clip_grad_norm_(self.q_net.parameters(), max_norm=1.0, norm_type=2)
                self.optimizer.step()
                self.optimizer.zero_grad()
                action = next_action
                state = next_state

agent = SemiGradientSARSA(env, 128, 0.005, 0.2)
agent.train(n_episodes=300)

I tried different rewards, Dropout Layers, Gradient clipping, different learning rates, different size of layers, literally everything. But the agent does not learn to solve the mountain car environment, especially due to catastrophic forgetting. I also tried other environments like Cartpole, but catastrophic forgetting is an issue there aswell. Any ideas what else I could try or maybe something is wrong in the code?


text:
What is the loss function used when pre-training BERT on MLM & NSP tasks?
I'm new to NLP and was reading through the 2019 BERT paper and am confused about the loss function used during pre-training.
As I understand it, the model is trained on the MLM and NSP tasks. The MLM task is trained by passing the final hidden vectors corresponding to the mask tokens into a softmax function and then minimizing the cross entropy loss function between the softmax output and truth.
For the NSP task, I understand that the goal is to use the final hidden vector corresponding to the [CLS] token, and using that to  determine whether the two "sentences" follow each other or not. Now the paper doesn't exactly say what loss function they use for this is, so I'm assuming they're doing something similar to the MLM case.
Now, in the appendix the authors mention that "The training loss is the sum of the mean MLM likelihood and the mean NSP likelihood". Given the relationship between log likelihood and cross entropy, this makes it seem like both MLK and NSP are used simultaneously during pre-training through a combined loss function.
I'm probably overthinking things, but I was wondering if any of you had a different interpretation of this. Specifically:
What is the loss function used when pre-training BERT on MLM & NSP? Also, what's your source?
I've read the paper start to end, as well as the transformers paper, searched forums, videos, and blogs, and my next stop is to dig into the code. I feel most explanations are either hand-wavy and it's unclear whether they're just guessing (like me) or if they actually know the answer.


text:
Which input embeddings are learned during pre-training in BERT? What about during fine-tuning?
I was reading the 2019 BERT paper and they mention how they use wordpieces that are then represented as the sum of token embeddings, segment embeddings, and positional embeddings. What is unclear to me is whether the token, segment and positional embeddings are all learned during pre-training AND whether they are modified during fine-tuning.
Pre-Training:

Sentence Embeddings: The paper says that the sentence embeddings are learned during pre-training ("we add a learned embedding to every token indicating whether it belongs to sentence A or sentence B"). am pretty confident about this.
Positional Embeddings: The paper does not say whether the positional embeddings are learned during pre-training. However, BERT relies on the transformer architecture from the "Attention Is All You Need" pa[er and I believe the positional embeddings in THAT paper are learned during what would pre-training for the BERT model. - Not so confident about this.
Token Embeddings: I'm guessing the token embeddings aren't learned during pre-training but before pre-training. Specifically, I hypothesize that the authors simply use the word piece embeddings approach to break up sentences into word "pieces" AND map each of these pieces to numbers, but this mapping is not modified during pre-training. - I'm very uncertain about this.

Are positional embeddings and token embeddings learned during pre-training?
Fine-Tuning:
Essentially the same question but for fine-tuning: are these embeddings typically modified during fine-tuning?


text:
How to find distance between class representation's decision boundaries for a neural network?
I have a 5 layer DNN with data containing 10 classes. To study how the model works, one thing among many I am looking at is the class wise representations.
I can extract the representations of each class from the 3rd and/or penultimate layer of the DNN, but how do I find the distance between these representations in the representation space ?
I want to look at decision boundaries and find the minimal distance between each class's boundary to another, but am stuck on how to do this.
I looked at t-SNE plots, where I can feed each class representation as a vector. But this does not include decision boundaries.
Why am I fixed on including decision boundary ? If given 2 models, I want to find how well each of the models separate the representation of one classes from the others.

How exactly are you computing the "representations of each class"? Can you provide more details of what they also look like?


text:
Can the methods of FunSearch that have been applied to the cap set problem be also applied to the enumeration of maximal antichains of sets?
Today on a seminar we discussed with colleagues recent impressive breakthrough in the cap set problem using Google DeepMind's FunSearch. The following specific question has been raised. Among several very similar problems in extremal combinatorics, let us consider the one of enumerating maximal antichains in the powersets of finite sets (including finding good upper or lower bounds for their number).
The question: is it feasible to readily apply the methods of FunSearch to this problem?
About the problem in some more detail. The sequence $a(n)$ in question is A326358, it starts with
1, 2, 3, 7, 29, 376, 31746, 123805914
One possible definition is this: for each $n=0,1,2,3,...$ consider the following graph $G_n$. It has $2^n$ vertices, identified with all possible subsets of $\{1,...,n\}$. Two such subsets $S$ and $T$ are joined with an edge if neither $S\subseteq T$ nor $T\subseteq S$ holds. Then $a(n)$ is the number of maximal cliques (complete subgraphs) in the graph $G_n$.
For example (see the above link A326358), $a(3)=7$ since there are exactly $7$ maximal cliques in $G_3$. Namely, these maximal cliques are:
$$
\begin{aligned}
&\{\varnothing\}\\
&\{\{1,2,3\}\}\\
&\{\{1\},\{2,3\}\}\\
&\{\{2\},\{1,3\}\}\\
&\{\{3\},\{1,2\}\}\\
&\{\{1\},\{2\},\{3\}\}\\
&\{\{1,2\},\{1,3\},\{2,3\}\}
\end{aligned}
$$
The above OEIS page contains $a(n)$ for $n<8$. I don't know whether anybody has computed $a(8)$. I have supplied $a(6)$ and $a(7)$ using the GAP package GRAPE which works by taking into account any symmetries present in the graph. Note that the graphs $G_n$ are indeed highly symmetric.
So, can the LLM methods of FunSearch give some further progress here?


text:
Can RL solve scheduling problems with unknown function
I have the following scheduling problem.

There are $n$ tasks and $m>n$ machines.
Each task $i$ has a requirement $t_i$ that should be guaranteed.
Any task can be scheduled on at least one machine but a machine cannot schedule more that one task.
When a task $i$ is scheduled on machines $S_i\subset\{1,2,\ldots,m\}$, the machines in $S_i$ should be consecutive meaning that the set $S_i$ is of the form $S_i=\{j,j+1,\ldots,j+|S_i|-1\}$, for some $j\in\{1,2,\ldots,m\}$.
Given the scheduling $S_1,\ldots,S_n$, we can compute a satisfaction value for each task $i$ scheduled on machine $j\in S_i$ as $f_{ij}(S_1,\ldots,S_n)>0$, which is unknown. I only have access to a black-box to compute $f_{ij}(\cdot)$. So, given $S_1,\ldots,S_n$, I can find the satisfaction value for each task $i$ and each machine $j\in S_i$.
We have to guarantee that each task $i$ has the sum of its satisfaction values of its machines is at least $t_i$, i.e., $\sum_{j\in S_i}f_{ij}(S_1,\ldots,S_n)\geq t_i$.
The objective is to maximize the sum of all satisfaction values, i.e., $\sum_{i=1}^n\sum_{j\in S_i}f_{ij}(S_1,\ldots,S_n)$.

Is RL a good approach to solve this problem. What RL algorithm do you suggest and why?
I was trying to model it using MAB but when I was defining the arms I figured out that the number of arms can be the possible partitions of $m$ into $n$ tasks (with permutations). So for $m=70$ and $n=6$, I have 11238513 arms. So, I don't know if it is possible to apply MAB. Even with RL, that action space will be very large and should the RL assume terminal reward game or step reward game?

from your formulation, you have more machines than tasks

Yes exactly. A task is heavy and needs more than one machine to be executed.


text:
which the effective ranks of embedding matrices in Large LMs?
The hidden size of current LLMs is well in the 8K and above, and I was wondering, has the rank of the embedding matrix, and the rank of the transpose of the output matrix, been increasing with the size, or is it restricted to an smaller subspace, or growing more slowly?
If it were growing more slowly that could mean that the "semantic vector" uses the rest of the hidden size and it could be useful to separate both parts.
Is there some measure or lore knowledge about this?


text:
Do LLMs based on a diffusion model (as opposed to an autoregressive model) exist?
Is there a such thing (described in title), at least in research papers (not actual models)?
So far all LLMs that I know are autoregressive models.

The question you want to ask is "Do LLMs based on a diffusion model (as opposed to an autoregressive model) exist?"

This is a false dichotomy. Most diffusion models, including Dalle 2 and 3, already are transformers.
However, assuming you meant to ask if any language models use diffusion as opposed to a GPT, the answer is yes. For example, see this recent paper:
https://arxiv.org/abs/2305.09515


text:
If you throw additional compute at DQN will you get the same results as if you threw compute at transformers?
Recent performance in transformer based AI models have done really well by just throwing additional compute and data into the training. Can you expect similar levels of results by throwing more simulations at a Deep Q learning agent?

one is an algorithm, the other one an architecture


text:
What is the approximate minimum coding rate for NLP datasets?
I just realized that it is actually practical to use information theory to compute the maximum viable compression for datasets & that it is easiest to compute for discrete datasets. This makes me wonder what the minimum coding rate (i.e. minimum required bit representation) for well known NLP datasets is? Also approximately how close to this minimum coding rate do modern transformers achieve in the embedding space?


text:
Using reinforcement learning to optimize a state-based potential function
I am using reinforcement learning for a project, and the objective function I want to optimize is a potential-based function. In other words, it is preferable for the agent to move to positions with higher potential.
My questions are: 1. Can I directly use the potential function as the reward function? 2. Since I don't know the optimal location of the potential in advance, the length of the sequence cannot be predetermined. How should I choose the length of an episode in such a situation?
I'll appreciate any suggestion and comment.

Are you required to use RL for this? How is the value/potential value being sampled?

@NeilSlater At the initial state, the agent is randomly assigned to a position. Our goal is for the agent to output an action to move itself, thereby increasing the potential of its current position. It is similar to mountain climbing. I'm uncertain if my description is clear.

OK, I understand that. This is an unusual way to find a solution to multivariable nonlinear optimisation. You can make it work, but RL is an odd choice here, and I wondered why? One reason might be because you really want to see what happens with an RL agent in this situation. However, there are better choices, especially if calculating values of your potential is easy.


text:
Does the NEAT alogrithm disable genes randomly?
It is not mentioned in the original paper, if genes can be disabled randomly. They only mention that the connection gene is disabled, when a node is inserted in it's place. Is it the case, that genes can be disabled randomly as well?


text:
Which epoch is the best for me to choose?
I have trained my deep learning model. I also saved the validation loss to a file and plotted on a graph

I have $2$ questions for this:

Does the validation loss look normal? Is there any issue with the validation loss ?
How can I determine which epoch is the best epoch so I can choose ?

If you need any additional information, I can provide. Thanks

The first question is not well defined. What does normal mean? It is clearly decreasing, so that is good. If you are asking if the fluctuations are normal, yes that is not uncommon. This is not an unusual looking validation curve, but if you want a more complex answer, you need to look at how your loss is defined, what sort of values are reasonable, how it compares to the training loss, and how it compares to the test loss (assuming a three fold train test validate split).

The simplest answer is pick the one with the lowest loss as a good heuristic. The more complex answer, is to get a less biased estimate by doing either a three fold or cross validation. For more info on those two and their tradoffs, see:

https://stats.stackexchange.com/questions/410118/cross-validation-vs-train-validation-test

Let's say we train for 700 epochs, and the lowest validation loss occurs at epoch 530, should we choose the the last epoch 700 or the lowest validation loss epoch 530 ?

in that case best to pick 530. Also please mark answer as correct if it was helpful

Thanks for your help


text:
Is Training Separate NER Models for Specific Labels a Viable Strategy for Improved Accuracy?
In my project, I'm utilizing Named Entity Recognition (NER) to identify crucial variables in prompts categorized into specific domains. However, incorporating numerous labels in the training data has resulted in poor accuracy. Considering this, would creating separate NER models for each category be a  strategy to enhance accuracy and improve variable identification?

Please, add information of your neural network model structure, input data example, expected output, accuracy change, training parameters. These will help us to give you good suggestion.


text:
Is there a database of math conversations with Chat GPT?
I would like to make GPT (or other AI Chat, if someone knows better) to assist me in my mathematical education. But hard to say what GPT is capable to do. Some testing tasks are performed well by him, others worse. Maybe there is a database of such conversations when I could learn what is possible with Chat GPT, even if they are leading evidently to bad solutions? Please let me know.  If not how to test his capabilities to know what is possible for him, and when for sure he will be making errors?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

@bot I'm asking about repository for different math conversations ( to learn from them) about mathematicial characteristics of Chat) Additionally I would like to know methods (fast methods) how to test Chat GPT to define what is possible for him in mathematics  and what is not  at the current moment.


text:
Dimensions and file sizes of AI types (img/txt/sound) that work commercially and projections of future AI based on dimensionality and file simplicity?
Is it accurate that current AI breakthroughs are proportional to low dimensional complexity of datasets and small data throughput?
Image and text file renders are 100's of kilobytes, and they are dimensionally very simple strings and 2D graphs.
(Phrases are ~2D permutations of words from a dictionary of 170k words, and images are 2D graphs that can easily be processed by NN vectors.)
Audio files are 2-3 megabytes and there is a Time-Frequency wave uncertainty so maths cannot analyze audio TF accurately. (voice can be simplified as graphs of vowel formants and sibilants which are 2D so it works).
Mathematics file sizes are often less than 100kb and can be represented by code, so a maths and physics GPT should be definitely a future breakthrough?
Video, 3D, mechanics, chemistry, biology, these all involve data volumes which can cause exponential strain on processing?
Can we make a linear prediction of what future Ais will happen and when simply based on their potential to factor into code and 2D formats (If so, what breakthroughs can we predict? ), with real-world audio/video/mechanics/chemistry being very, very far away?

This question isn't quite clear to me. Is your thinking "text and images are simple / low dimensional and that is why AI has worked in those domains" (?) Where simple = each observation can be represented by a small file (?).  Is "NN vectors" = nearest neighbor vectors? Also, what do you mean by "mathematics files"?

Just from experience of coding arrays for all these media, using 1d/2d/3d/4d arrays, and the data quantities involved with the processing. For 3D, scientists were heavily researching ISOsurfaces, humans rarely work with ISOsurface formulas bigger than 5kb. mathlab files, wiki maths theorems, science journal maths reasearch, are 99% file sizes less than 50kb.


text:
NaN gradients while Training (but loss isn't NaN neither the computational graph is disconnected)
The reason I am asking this here is that I haven't found a bug in my code and maybe there isn't a bug at all (or maybe there is). I just want to validate the idea that I am trying to implement. Here is my idea.
I am trying to estimate a function g using some basis function(s) $f_i$'s where $i = 1,\dots, n$. Further, for generalizability, I am introducing $4$ transformations to the set of basis functions that help in the construction of $g$ from $f_i$, $i=1, \dots, n$. The transformations are $T_1: x \to a_1.x$; $T_2: x \to x^{a_2}$; $T_3: f_i(x) \to  b_1.f_i(x)$; $T_4: f_i(x) \to f_i^{b_2}(x)$. With these transformations introduced, I want to estimate $g$ as the sum of $n$ $f_i$'s transformed using all four transformations. (Note that an identity transformation corresponds to $a_1 = a_2 = b_1 = b_2 = 1$.
As a starter, I tried to learn a set of transformed $f_i$'s using the basis functions which by default have the identity transformation. With the short prelude to my objective, I now present my codebase which can be found here. Anyone can download the code.zip file and run nn4.py which should get the code running.
My problem though is that, while training a neural network, for predicting the parameters of transformations i.e., $a_1$, $a_2$, $b_1$, $b_2$, I get the gradients to be NaN despite trying very small learning rates and several initializations. My training pipeline is as follows:

Suppose the function to be estimated is g. Step 1: Give samples from $g$ to the neural network $NN$ and obtain the set of parameters $a_1$, $a_2$, $b_1$, $b_2$ for all basis functions. (Effective dimension of output is thus $n \times 4$ is $n$ is the number of basis functions) Step 2: Reconstruct the function $g$ by substituting the parameters into each basis function and then adding up all the basis functions. Step 3: Calculate the mean squared error between the samples of the reconstructed function and the samples from the original function. Step 4: Backpropagate this loss

I have ensured that $g$ (the function we are trying to estimate) is scaled between $[-1, 1]$ and the all $f_i$'s are also scaled to $[-1, 1]$ as well. However, scaling the parameters would not make sense in this context so I didn't scale them.
I have no idea, as to why the gradients are coming NaN. I have verified that the loss isn't NaN. Also one important observation that I made is that the parameters given by the $NN$ is NaN from the 2nd step itself (1st step is initialization). I couldn't find a bug in my code so I thought there might be a conceptual bug in the idea.
Kindly help me to resolve this issue. Any help/comments are greatly appreciated.


text:
How to let the AI trained on MNIST recognize my own hand-written number?
I have already trained a simple model without using any frameworks to recognize the hand-written numbers from the MNIST dataset. Actually it's just a copy of the example on the book Grokking Algorithms. Here:
import sys, numpy as np
from keras.datasets import mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()

images, labels = (x_train[0:1000].reshape(1000, 28*28)/255,y_train[0:1000])
one_hot_labels = np.zeros((len(labels), 10))
for i, l in enumerate(labels):
    one_hot_labels[i][l] = 1
labels = one_hot_labels

np.random.seed(1)
relu = lambda x:(x>=0)*x
relu2deriv = lambda x: x>= 0
alpha, iterations, hidden_size, pixels_per_image, num_labels = (0.005, 350, 40, 784, 10)
weights_0_1 = 0.2*np.random.random((pixels_per_image, hidden_size))-0.1
weights_1_2 = 0.2*np.random.random((hidden_size, num_labels))-0.1

for j in range(iterations):
    error, correct_cnt = (0.0, 0)
    for i in range(len(images)):
        layer_0 = images[i:i+1]
        layer_1 = relu(np.dot(layer_0, weights_0_1))
        layer_2 = np.dot(layer_1, weights_1_2)
        error+=np.sum((labels[i:i+1]-layer_2)**2)
        correct_cnt += int(np.argmax(layer_2)==np.argmax(labels[i:i+1]))
        layer_2_delta = labels[i:i+1]-layer_2
        layer_1_delta = layer_2_delta.dot(weights_1_2.T)*relu2deriv(layer_1)
        weights_1_2 += alpha*layer_1.T.dot(layer_2_delta)
        weights_0_1 += alpha*layer_0.T.dot(layer_1_delta)
    if j%10==9:
        print(j, str(error/float(len(images))), str(correct_cnt/float(len(images))))

After training(without any regularization), the accuracy on the test set is about 83%. It sounds fairly high to me, so I want to test this model on some of my own hand-written numbers. I know the   network takes in a (784, 1) shaped matrix as input, with all elements ranging from 0 to 1. Then I found the website Pixilart to actually draw several numbers and downloaded them as images with PNG format. Here's one of them:

According to the blog post, I used the following code to load my own pictures and transformed it to a matrix:
from PIL import Image
img = Image.open("C:/users/22432/downloads/it0.png")
n = np.array(img.getdata())

Then I noticed it is a 784x4 matrix, with the fourth column having actual numbers, the others '0'. So I did this:
n1 = np.array([i[3] for i in n])/255
layer_0 = np.array([n1])
layer_1 = relu(np.dot(layer_0, weights_0_1))
layer_2 = np.dot(layer_1, weights_1_2)

The layer_2 variable should store the results of the recognition. But after several tries, I found that the model behaves unexpectedly poor. There's no problem on the weights, and it's accuracy on the MNIST test set is indeed very high. So there's something wrong with the example I provided. But what's wrong? Is there any more requirements for the sample to be successfully recognized?
This question is indeed very similar to this one. But I didn't get any satisfying answers. Could anyone help me?


text:
Masking during Instruction Tuning for LLM finetuning
I'm currently trying to learn more about LLMs particularly generative decoder only models such as the GPT family of models. I do have one question about masking though.
For me the way masking is performed left-to-right makes sense during pre-training but it is weird to me during instruction fine-tuning when dealing with the input prompt. I therefore have two questions:

Is the input prompt masked in the same way as the text during pre-training?
And if so why is it done in this way and have there been studies on it


text:
Example VQ-VAE code for audio in paper "Neural Discrete Representation Learning"
I want to replicate the paper "Neural Discrete Representation Learning" by van der Oord et al (2018). DeepMind provides an example for CIFAR images on GitHub.
It seems that the model for images is quite different from audio. Here is the description of images in the paper:

We model images by learning a powerful prior (PixelCNN) over z. This allows to not only greatly speed up training and sampling, but also to use the PixelCNNs capacity to capture the global structure instead of the low-level statistics of images.

and here is the description for the experiment with audio:

We train a VQ-VAE where the encoder has 6 strided convolutions with stride 2 and window-size 4. This yields a latent space 64x smaller than the original waveform. The latents consist of one feature map and the discrete space is 512-dimensional.

I found this repository that seems to implement the VQ-VAE on video. But I couldn't find anything on audio; and this issue, open for 3 years, suggests that the authors won't provide it either.
Does anyone know of an existing example of the VQ-VAE from the paper applied to audio data, such as the VCTK (Voice Cloning Tool Kit)?


text:
Intentionally corrupting LLM weights (lobotomy)
It is largely unknown how LLMs work inside. Has anyone scientifically tried to corrupt (open source) model's weights in an organized manner to maybe detect which parts of the model are doing what or are more or less important?
I mean something like setting the weights to zeros/random numbers in random (% of all weights) or organized (whole areas/layers etc.) manner and benchmarking the LLM to see how the results change.
Related reddit post.

isn't that page already answering your question?


text:
Can you illustrate how the weights in transformer model generated from a training sentence can be generalized to an unseen test sentence?
Can you show how the weights in transformer model are generalizable?

I think the easiest way to understand generalisation of models is as follows. Imagine a cloud of 2d points that forms something that has roughly the shape of parabola. Now, you could create a function that goes exactly through all these points. However, this function is more likely to be wrong for new points than a function that just has a parabola shape. This is harder to explain than to show with a picture. So, I'd recommend you look online for a picture. In any case, I doubt you can do this easily for the transformer.

Could also please provide more info about your background? It seems you have little knowledge of ML. It's hard to satisfactorily answer any question in this way.


text:
Same prediction result with little probabilities change
I built a job prediction system leveraging data scrapped from LinkedIn with Random Forest and compared it to XGBoost. XGBoost was used due to its high accuracy after training.
When I made a prediction, I always obtained nearly similar results every time. There is only a slight difference in the probability percentage.
This same goes for the Random Forest.
Does this problem occur due to my bad dataset?
I took a list of skills and years of experience for the user input. My data has about 20k points after done data cleaning.


text:
Bellman equation for MRP
Bellman equation for MRP is:

And it can be written as

or the inverse matrix method, when the transition matrix is known. My question is: how the left $v$ equals to the right $v$ in the second formula, when $v(s)$ and $v(s^{\prime})$ refer to rewards sum begin from $t$ and $t+1$ respectively in the first formula?

Please give a reference where you found these formulae. They look equivalent to me, assuming some things about what the variables $v$, $r$, $P$ represent.


text:
Neural network dynamic input shape vs fixed input shape in input layer for handling NULL values
I have binary classification problem, we know that the output layer will be scalar or dense a.k.a. 1 unit neuron with sigmoid as function activation. 1 means is the subject will die, while 0 means keep alive.
My data type is tabular, where rows represent time (hours) and cols represent features such as gender, age, white blood count, heart rate, and so on.
The time is bounded with first 24 hours when collecting data while the feature is limited to the 19 features
Therefore the model was trained with fixed input shape matrix (24, 19) where 24 represent hours and 19 represent features.
While model in inference mode (forward propagation) after trained, can I change input shape dynamically such as (24,10) in case I didn't get some features for all 24 hours such as heart rate. Or, (10,19) in case didn't get all captured data for 24 hours. Or, (1,1) just a hour and a feature such as only heart rate at that time while another are NULL.
Or, shall I fill the NULL value with an interpolation or an average value to achieve fixed shape?
Theoritically, the weights are saved in hidden layers after training, therefore reloading weights after modify input shape is possible. But I have no idea which one shall I chose


text:
What are the differences between seq2seq and encoder-decoder architectures?
I've read many tutorials online that use both words interchangeably. When I search and find that they are the same, why not just use one word since they have the same definition?

They are not the same, but they can overlap.
An encoder-decoder architecture is composed of an encoder (which compresses the input) and a decoder (which decompresses the compressed input).
A sequence-to-sequence (or sequence transduction) model is a model that converts sequences to other sequences. The most obvious example are models for machine translation, where sequences are sentences in 2 different languages (e.g. English and French). See e.g. NMT or the transformer. These models also use an encoder-decoder architecture.
The variational autoencoder (VAE) uses an encoder-decoder architecture, but it not usually used to convert sequences to other sequences.
So, in conclusion, encoder-decoder architectures are not just used for sequence transduction tasks, and sequence-to-sequence models may not use encoder-decoder architectures, although famous models like the original transformer do.

What about GANs ? Are they seq2seq or encoder-decoder ?

@user I don't think GANs have an encoder-decoder architecture, but I honestly don't remember the details. They have a generator and a discriminator (which are trained in some game theory fashion), but I don't remember if they use this ED architecture. Afaik, GANs are mostly used for image generation tasks, so not for sequence transduction tasks, but I don't exclude they may also be used for these tasks. I'd need to check that (i.e. read the paper again). Maybe later.

Are there any seq2seq problems that don't use encoder-decoder architectures ?

I can't think of a specific model right now, but I am sure there are (maybe check hidden Markov models or even just simple RNNs). But one reason why the encoder-decoder is useful in the case of sequence modeling is because the input and output sequences may have different sizes (e.g. number of words or characters), so you cannot just map one token to another token. With the ED, you just let the model learn a latent representation (which is what the encoder does) and then the decoder tries output sequence conditioned on the latent vector. Or something like that.

Yes, you may have read tutorials or texts using interchangeably because of close relationships, but actually, there is a subtle distinction.
Encoder-Decoder: It contains two main components Encoder and Decoder which take information to create context. The decoder takes context and generates the output. The key point is it can be used in various applications like NLP generation, Machine translation, Image Captioning etc... It Supports CNN and RNN or similar (GRU, LSTM, etc...).
Seq2Seq Model: Sequence to Sequence is a type of model that is specifically designed to handle sequences. It is mainly used in tasks where input and output are both in sequences. So, we can say that the seq2seq model also uses the encoder and decoder-based network specific to the nature of data and task. It only supports RNN, LSTM, or GRU.
So, we can say that seq2seq is an encoder-decoder-based architecture while the encoder and decoder model represents a wider range of applications and data.


text:
ViT fails to detect a white pixel in a black image
I wanted to report you to some experiments in the context of Deep Learning for Computer Vision, in particular for visual reasoning.
The main question I am trying to answer is the difference between the Vision Transformer (ViT: https://arxiv.org/abs/2010.11929v2) and CNNs in visual reasoning benchmarks that require object localization.
TASK
Given an NxN black image with a single white pixel, predict the coordinates of the white pixel.
The following are some properties of the dataset:
the cardinality of the dataset is |NxN|, i.e. the number of all possible black images of size NxN with one white pixel
in the experiments carried out, the training set and the validation set have a proportion of 80%/20%, split with a uniform distribution
I attach the image of a sample entitled the ground truth, i.e. the coordinates of the white pixel.
The shared (between the two tested models) hyperparameters used are:

Optimizer: SGD
Momentum: 0.9
LR: 7e-4
Batch Size: 8 for ViT, 256 for CNNs

ViT hyperparameters used are:

num_layers: 4
num_head: 2
dropout: 0
patch_size: 1
embedding_dim: 8

(both learned and fixed positional encoding are tested)
The CNN is a ResNet (without stem layer and max_pooling) with 8 layers (stride = 2 for layers 1, 3, 5, 7 and channels doubled), the channel sequence for each layer is: 1 (input)-3-3-8-8-16-16-32-32.
The output of the two models is obviously the same, the pair of coordinates of the white pixel, for this reason, both L1 and MSE loss were tested (the results do not change).
Finally, CoordConv encoding was also tested (https://arxiv.org/abs/1807.03247). It consists of creating two channels to be concatenated to the input image which represents the coordinates of the pixels.
QUESTION
The goal of these experiments is the evaluation of the Vision Transformer and CNNs on a visual reasoning task that requires localization. In particular, I expect the choice of patch size for the Vision Transformer to be critical to convergence. In particular, I expected that by choosing a patch size = 1 the ViT would be able to overfit the training set (thanks to positional encoding).
RESULTS
the results showed that:

The ViT transformer works at chance level for any patch size, it is
not able to overfit the training set. In particular, it returns the
same output for any input, behaving shift-invariant (permutation
invariant in the pacification view).
ResNet is able to overfit the training set with a good level of
generalization, but by increasing N (i.e. the size of the input
images and therefore the cardinality of the representation space) the
results worsen
CoordResnet on the other hand is able to overfit and generalize well
for any N

How do you explain the behavior of the Vision Transformer?


text:
Are there guidelines or rules of thumb on how to stack hidden layers in a RNN?
I’m currently working on the prediction of chaotic data and I have decided to see how well would an RNN, namely an LSTM, would do. I am fairly new to the topic of Neural Networks, but I have found a spate of helpful resources along the way, including this forum.
One aspect that I am still having trouble with is how to structure the layers of an RNN? By “to structure” I mean the number of layers and their ordering. For instance, on this Julia discourse post, we can see the following RNN model
function LSTM_model(N,num_of_classes)
    model = Chain(LSTM(N,200),
                Dropout(0.2),
                LSTM(200,200),
                Dropout(0.1),
                Dense(200,101),
                Dropout(0.1),
                Dense(101,num_of_classes))
    return model
end

The author adopted a structure to the model used for time-series prediction, the same application I’m working on. But what makes it better to have two LSTM layers instead of one? Or why is there a Dropout() after every layer? Why have the LSTM layers before the Dense layers?
On the “Julia for Optimization and Learning” online course maintained by the Czech Technical University, there is a section (that might be a bit outdated) dedicated to training and storing a network. They create a model with Chain and describe each layer’s role:
m = Chain(
    Conv((2,2), 1=>16, relu),
    MaxPool((2,2)),
    Conv((2,2), 16=>8, relu),
    MaxPool((2,2)),
    flatten,
    Dense(288, size(y_train,1)),
    softmax,
)

Two convolutional layers extract low-level features from the images.
Two pooling layers reduce the size of the previous layer.
One flattenlayer converts multi-dimensional arrays into one-dimensional vectors.
One dense layer is usually applied at the end of the chain. One
Softmax layer is usually the last one and results in probabilities.

So far, this has been the only resource I’ve found that gives some reasoning to their choice of layers and their ordering.
In my case, I have been trying different architectures with a different number of hidden layers, a mix of LSTM and Dense (Flux.jl), but results have been lacking. Granted, the data I am working with behave chaotically (as if from a Lévy Process). But I was hoping that with nonlinear activation functions, the model would capture features other “simpler” models, like a Recursive Least Squares, wouldn’t. But so far, the average RMSE for a series of sequential predictions has been nearly the same for an LSTM model and a Recursive Least Squares.
Are there guidelines or rules of thumb on how to compose the hidden layers (which ones, their ordering, how many) in a RNN when the goal is to make time-series predictions?


text:
What is the meaning of "dimensionality of the embeddings and hidden states"?
I was reading the GPT-2 and LSTM documents and noticed that they use the terms "dimension of embedding and hidden state". For GPT-2, the size is $768$, and for LSTM, the size is $256$. What does that mean ? Are there any visual graphs or figures for these concepts ? Very difficult to understand any of this.
Thanks

Could you specify which documents you are referring to? In NLP, we vectorize tokens and must assign them a number of dimensions. This is a hyperparameter that we can change, and it's normally determined through experimentation. 'Hidden States' is a term used in RNNs, where we have an output that changes over a number of iterations to process all the tokens in the sequence. These are the intermediate values of the recurrent operation that are passed to the next state in the sequence. These are not the same and to my undertanding they can be of different sizes.

@CesarRuiz, Thanks for your reply, do word embeddings and hidden states have the same meaning ?

No, they refer to different things. Think of word embeddings as a dictionary where you look up a word (token) and it gives you a definition (the vector). So, whenever the model uses a token, it replaces it with its vectorial representation. The hidden states in an RNN are vectors that hold information from previous tokens in the sequence. This is how RNNs can retain information from previous tokens in a efficient way.

@CesarRuiz, I thought Transformer architectures like GPT-2 also have hidden states ?

We could argue that all neural networks possess hidden states, which are the outputs of the hidden layers, but the term 'hidden states' is usually associated with RNNs. Hidden states are linked to sequential data, meaning that the current step requires the previous step for calculation. This is not the case with Transformers, because Transformers do not process data sequentially. Instead, they parallelize the data and then combine it through concatenation.

@CesarRuiz, In this document about GPT-2, there is a function called GPT2Model that is used to output raw hidden states ?

Yes. In this scenario, raw hidden state' refer to the individual outputs of each attention block. This class have the option of saving all the outputs of the attention blocks in the variable all_hidden_states as a tensor. Here, hidden_states refer to the output of a layer. The term hidden_state has a different meaning in the context of RNNs, not just the output of a layer. That's why the first thing I asked was about your resources, because the meaning of the term can change depending on the context :)

@CesarRuiz, Thanks for clarifying. One more question: The description of the class GPT2Model also mentions without any specific head on top. What does this mean ? Do you know ? Thanks

Based on the documentation, "head on top" refers to the addition of a layer to a model for the purpose of fine-tuning it to a specific task. This is a technique for fast fine-tuning of the model. You can see this if you look at the GPT2LMHeadModel class, which explains what "head on top" is. Basically, they are just using the vanilla GPT2 pretrained model without anything else on top.

Thanks for your help.