text:
Is the problem of Language Modelling a Well-Posed Learning Problem?
Hadamard defines (Well-posed problem (Wikipedia)) a well-posed problem as one for which:

a solution exists,
the solution is unique,
the solution depends continuously on the data (e.g. it is stable)

Now for an autoregressive language model, the pretaining objective consists of predicting the next token given all previous tokens. But as I see it, while the objective helps the model pick nuances of the language and learn "language representation", there is no "correct" answer.
Given the example, say, "I would love to have some ____", "chocolate" is just as "correct" as "coffee".
Thus, the solution is not "unique" as required by the definition.
But again, we could think of the problem as taking as input all the previous tokens and returning a probability distribution over the whole vocabulary. In this way of looking at the problem, the solution, i.e., the probability distribution, is unique.
So, is the problem of language modelling well-posed?
Is either of the two approaches the right reason? If not, what is the correct reason?

I'd rephrase the problem as a whole to 'can multi-modal problems be well-posed' and you have yourself a super interesting philosophical (and much more broadly applicable) question. One category mutli-modal problem is problems for which there are multiple correct answers to a single input. Coloring in a grayscale image for example. (snooker balls in an image can be of many different colors).

One of the most interesting theoretical questions ever asked here and still gots no attention. This shows how generate this site and the world are. Unfortunately, the answer below is not really what I am expecting and not good. This question is not philosophical. This is only philosophical in that it makes us think that maybe we're doing something wrong in AI and that we even need to understand what we're doing. That's why AI doesn't really advance and people just brute-force with the hope and dreams of getting rich or famous because of a breakthrough.

Philosophically speaking,
I believe a perspective shift is necessary. When you say, 'A solution exists', the next statement should be, 'Well, what is the problem?'. While you are correct in pointing out instances where the model is unique and not so much so, their correct problem statement is not being talked about.
I am afraid, that the concept of a "Well-posed problem" is theoretical, and while a problem might look absolutely well-posed in theory, things might work out differently in the real world.
Let's talk about the theory of an LLM (will intentionally leave out the technical bits). Ideally, given an exhaustive dataset of all the world's data and a NN with practically infinite training parameters; a trained language model should be able to be attentive enough to completely understand the context before providing any answers. Such a model will work something like this-
User: "Fill in the blank, 'I would love to have some ____'"
LLM: "What do you love"
User: "Among many things, chocolate, ice-creams, dance, booze, etc."
LLM: "Okay, and how are you feeling today?"
User: "Particularly hot, and alone."
LLM: (to itself: maybe some sugar to lift up the mood, and some cold ice cream to help with the hot day)
LLM: Let's go have some Chocolate Ice-cream.
User: Wow! I would never have thought about it myself! Thanks, Overlord.
Strictly theoretically speaking, the Freudian concept of pan-determinism says, given the complete history of a human being, you can pinpoint the exact decision they will make given a situation. So, in this ideal world, where the model thought this information is enough to figure out the answer would be absolutely right. As there will exist only one fill to the blank.
Now, let's travel to the real world, shall we?
Here, everything is limited. From low memory, and limited silicon chip supply, to limited attention spans (of LLMs of course). Because here, our LLM doesn't have (yet) the capability to reason on itself, unless explicitly advised to do so, it will answer based on defined heuristics like Greedy Search (see, now that you think of it, because someone included greedy search in the architecture, suddenly it feels like it was a well-defined problem after all? Because given the limitations of the language models and the architecture set bu the creature, there is only one solution, isn't there? Unless of course, you decide you want to increase the randomness in selection). But, back to the point, transformers or any other kind of NN are essentially numerical models which theoretically speaking have a well-defined convergence to a point but practically never converge; we always judge the trained parameters to be good enough and stop the training.
Given that we can probably never really achieve the ideal doesn't mean that it doesn't exist. With infinite resources, we can build the ideal LLM that can do things that make it a Well-posed problem.
And in due time we might have LLMs good enough to predict human decisions which makes you feel like they only had one solution which they duly produced, but as is with illusions, they are never real.
UPDATE: One more thing that I just recalled is that while it is a well-posed problem, engineers and developers go to great lengths to disturb its "pose" just because again, practicality supersedes idealism. What I mean by that is illustrated best using an example, that of the Bi-directional Encoder Representations from Transformers or the BERT architecture.
The trainers went to great lengths to intentionally introduce errors in training to avoid overfitting and achieve higher efficiency, which on the downside had the side-effect of not predicting a specific token with high confidence. What I am referring to is their use of masking [Mask] only 80% of the time during training, and intentionally using [word] instead of a mask, or a [wrong word] instead of the mask equally over the remaining 20% of the times.


text:
How does a Machine Learning model predict this classification problem?
Let’s imagine we want to create a simple Sentiment Analysis model using Machine Learning not Deep Learning algorithms, so we need to have a set of handcrafted features for this classification problem.
Let's say we have 2 features (F1, F2) for each sentence and a target class consisting of 0 and 1 as positive and negative. So we have a bunch of sentences in a dataset like this:

Well, classification algorithms like SVM, LR, DT, and …. can be trained from the training set and learn the sentiment of each sentence from their features. They can also test themselves using the test set. Now the model is ready to predict the sentiment of any sentence out of the dataset. Till now, there is no problem and everything is clear to me.
My problem is when we want to give a new sentence to the model form outside the dataset. Obviously, when we give a sentence to the model, we don’t give any feature to the model. So here is my question, how the model can predict the sentiment of the new sentence when it doesn’t know how to calculate each of the features?
Should we define what is F1, and F2 in the model first? Should we define a function and for any new input sentence, call the function to calculate the proper value for each feature and then teach the model that the function outputs are equal to F1 and F2 in the dataset?
To make a long question shorter, let me give a clear instance. My new sentence is "She was overjoyed when saw my cat". I want to give this sentence to the model and obviously expect to see a predicted positive sentiment. This sentence is not in my dataset so there is no feature for it and the model has no idea about F1 and F2. On the other hand, my model learned to determine the sentiment of each sentence using F1 and F2. So back to my question, when I don't give new sentence features (F1, F2) to my model and don't specify any procedure to calculate the features for new inputs, how the model can predict the sentiment of my new sentences?

Updated:
As an example of a classification problem, I mentioned the article
" A Pattern-Based Approach for Sarcasm Detection on Twitter," Click here! where the authors proposed four sets of features that cover the different types of sarcasm.
They used those to classify tweets as sarcastic and non-sarcastic.
These features are sentiment-related features, Punctuation-related features, Syntactic and semantic
features, and Pattern features. For instance, in sentiment-related features, they checked whether there
is a contrast between the different components. By contrast,
they mean the coexistence of a negative component and a
positive one within the same tweet. Once the features are extracted, they ran the classification using classifiers like
(SVM).
So till now, everything is clear, with those features they trained their model and then test it with the test set.
The question is when they wanna check the classification for a new sentence, say x for instance, obviously, we should
calculate all four sets of features for the new input x.
Assume that G(x) is a function that gets a new sentence x and returns four sets of features for it (F1,F2,F3, F4), where (F1 = Sentiment-relatedfeatures, F2 = Punctuation-related features, F3 = Syntactic and semantic features, F4 = Pattern features)
How should I show the model to use these features (F1,...,F4) to predict the class for this new sentence (x)?
We have such a code: Ypred = model.predict(x, F1,F2,F3,F4); how the model should know to use F1,...F4 features in the
same way as its train or test phase?

when I don't give new sentence features (F1, F2) to my model and don't specify any procedure to calculate the features for new inputs, how the model can predict the sentiment of my new sentences?

It just cannot.
Any supervised ML model can be seen as a function of some set of features $x$, i.e. $\hat{y} = f(x)$. Since you describe the actual input (the sentence) with two features $F_1$ and $F_2$, and you don't provide them to the model you can't predict on new sentences.
Conversely, if you try to learn another model, say $\hat F = g(s)$, that maps a sentence $s$ to the features, you need to figure out how to encode the sentence to a fixed-length representation. But if you can already do it then there is no point at learning the model $g$ at all, since you can directly learn $f$ instead also dropping the two features.
I think your problem setup is a bit ill posed, you should encode the sentences either with an n-gram approach or with a vocabulary lookup to a dense embedding (instead of one-hot encodings of words.), and then train directly the model to predict the sentiment given the class variable, so skipping the two features.
Update: Assuming you can can compute a set of features $F$ for each sentence, like done in the paper you linked, you need to train a multi-class classifier (e.g. a decision tree, random forest, SVM, or even neural-network) on six classes from $1$ (highly non-sarcastic) to $6$ (highly sarcastic): otherwise you can define just two classes, sarcastic vs non-sarcastic. The way you train the classifier is by pairing features $F$ and targets $y$ (i.e. the desired class label.) So you build a training dataset $\mathcal{D}_{train}=\{F_i, y_i\}_i^N$ for each training sentence $s_i$ ($N$ in total), and a validation dataset (that you can obtain from $\mathcal{D}_{train}$ by taking a random $20\%$ for example). Your train the model, like model.fit(F, y), and then evaluate the performance on a test set $\mathcal{D}_{test} = \{F_j\}_j^M$ for each new sentence $s_j$. Note that for the new sentences you don't have a class label (you want to predict that) but you have only to compute their features $F$. To predict the new classes you do something like: y_pred = model.predict(F_test), where F_test are the features of the test sentences, i.e. $\mathcal{D}_{test}$. Finally, for building/training and evaluating a classifier have a look at scikit-learn (DTs, RF, SVM) or Keras (for NNs).
Basically, to both train and test the model you only need to provide the computed features $F$ as input, and not also the sentence: such kind of models cannot learn directly from words, and so from sentences.

thank you for your response. So you believe It’s better to use n-gram or vocabulary lookup, that’s a good idea. But what if I want to use those features only? Let’s say I add a function to my program to calculate F1 and F2 for any new input. How should I tell the model that the output of the function is equal to F1 and F2. ?

The problem is that I read lots of articles on classification problem like sentiment analysis and sarcasm detection and in most of them authors think about different features and try to teach their models using those features. However, I have no idea how I should predict the classification labels for new sentences. I would appreciate it if you can help me.

If you're able to compute F1 and F2 then you just need to setup a classifier where each sentiment is a class, there is no need to train a model that predicts them. Can you update your question with the papers you mentioned? Knowing them may help, others too

I appreciate your time. I've updated my question with relevant information. I would be thankful for any help.

Thanks for the extra information. I have updated my answer, have a look.

Honestly, no one could answer my question in such a smooth and clear way. I get the point to a great extend. Thanks a million!

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Action space for the A2C algorithm
I am working on a problem where I need to generate a list of 0 - 8 different prices as the action space where generating 0 prices represents doing nothing
NUM_PRICES = 8
action_space = [i for i in range(NUM_PRICES + 1)]  # Include 0 as an option

# Sample an action from the action space
action = np.random.choice(action_space)

if action == 0:
    prices = []
else:
    prices = np.random.uniform(-10.0, 200, action)

I have something like this where prices would be the generated action.
Would something like this work with the A2C algorithm?
I'm unfamiliar with A2C and have only worked with DQN and QLearning
So I'm worried maybe the action space is too complex

A2C is a policy gradient method, that learns a policy $\pi(a\mid s)$ directly.
This means that the actions are sampled from a probability distribution. In practice, you can model a discrete action space with a Categorical distribution or, in case you experience gradient issues, with a Gumbel-Softmax (also called Concrete) distribution which is its continuous relaxation.
Lastly, unless NUM_PRICES is very large you shouldn't experience issue about the action space size.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Math behind Diffusion models explanation?
I am recently reading this wonderful article https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#what-are-diffusion-models about the math behind the diffusion models ,
As i dont have a math background , i am stuck or not able to understand the below statement,

I am able to get how the bayes rule is working, I dont understading what is the reason behind the exponential math below it.
Kindly explain why it is used and what is the reason behind it or suggest any resources for getting an intuition.

That exponential comes from the PDF of the assumed distribution that you might have missed in the blogpost:

and

plug those PDF in that bayes formula and you will end up with those equations

Can you explain why we need to use exponential and what is the reason behiind it ? I

@mat it's literally the gaussian PDF, it's not a "reason", it's its definition


text:
MobileNet validation loss not decreasing over time
I am trying to train a MobileNetV2 on a custom dataset, to image Classification task.
Cardinality is 864 images, split in 70%/20%/10%, balanced between the 3 different classes.
Weights are pre-loaded from imagenet, I froze the net and I added to the bottom of the net a GlobalAveragePooling, a Dropout (with 50% drop probability), and a Dense layer with 3 classes and softmax as activation function, since i want the output layer to give me an output like (1,0,0) if the inference image is from the first class, and so on.

image size: 96x96 (I normalized, too)
batch_size: 32
Learning rate: 0.001
trainable params: 3843
optimizer: sgd ('adam' doesn't improve my accuracy)
loss: categorical cross entropy
metrics: accuracy

Training for 20 epochs gives me these results:

After that I decided to try some fine-tuning, by freezing only the first 100 layers of the net.
Trained again for 10 epochs, that's what I get:

My net is overfitting, but I don't know why it's happening and what am I expected to do in order to improve my accuracy.
Edit: I also tried increasing dataset images with some source images or even with some data augmentation, up to more than 3K images, but it didn't work out at any rate.
2nd edit:  I'm currently using a sgd optimizer with a decay rate of 1e-6. I've just added the stoch dept per each conv2D layer with linear decay surviv rate, up to 50%. I reached a 80% val_accuracy before early stopping. I suppose pretraining can't be adopted to my personal use case, so I'll be exploring some other architectures, hoping for better results.

Hello. Welcome to this community! Could you please put your specific question in the title? "MobileNet not improving" is not really a question although it describes a situation.

Since you already tried to gather more data along with data augmentation, and dropout. I can suggest you to use:

Weight decay, but in the form of AdamW (or similar for SGD) in which the "W" stands for decoupled weight decay update. In this way the magnitude of the penalty is both easier to control and interpret. I think you should use a coefficient of 1e-4 or larger.
Label smoothing, with an $\epsilon=0.05$ or $0.01$.
Stochastic depth: is a regularization technique specific for networks with skip connections, basically dropping a residual path with probability $p$ (think of it as a dropout over entire skip connections.)
Advanced image augmentations: like training on random crops of your images (to increase their number; also be sure to not crop much otherwise the content will be changed), cutout (zero random portion of image), mixup (blend to random images and their classes as well), and cutmix (as before with also cutout.)

If the issue still persists, try to pre-train on a different dataset and/or use a different network architecture.

Thanks for answering. I'm currently using a sgd optimizer with a decay rate of 1e-6.
I've just added the stoch dept per each conv2D layer with linear decay surviv rate, up to 50%. I reached a 80% val_accuracy before early stopping.
I suppose pretraining can't be adopted to my personal use case, so I'll be exploring some other architectures, hoping for better results.

That's useful information, you should add that to your question. Well, if you target dataset is not covered by ImageNet than it won't help much.


text:
Do batches need to be sequential in Transformer traning?
When training a transformer model (I'm using nn.TransformerEncoder from pytorch) is it better to use sequential batches (for example, three sequences [[1,2,3,4], [5,6,7,8], [9,10,11,12]] where next sequence continues where the previous ended)?
Or can I use non-sequential or randomly picked sequences from my dataset (for example, here are three randomly picked sequences [[1,2,3,4], [56, 57, 58, 59], [22, 23, 24, 25]])?
I can make a test myself, but I was just wondering if that's already known.

Each sample in each batch is processed separately. Each batch in turn is handled separately. Having consistency between batches is not going to improve the convergence of your model.
If any, it might result in a less generalized model because your data is not shuffled enough.


text:
can GPU crypto mining rigs be converted to train or run AI?
SHORT VERSION:
"I've got a bunch of GPU mining rigs, surely I can do something useful with them like train my own AI or provide hashing power to astronomy or genome folding operations?"
TLDR:
Having acquired a large collection of GPUs that were previously mining ETH and ETC, as well as a nice stack of opterons with server RAM that were mining Monero I now find myself with a pretty big stack of silicon that isn't being used.
The GPUs seem to me to provide excellent support for graphics algorithms and the opterons I believe might be viable for CPU intensive tasks.
simply put: "I had a good run mining crypto", but I didn't sell my hardware...
in this nearly decade long process I learned quite a bit about Linux (since that's what every rig is running on)
My setup was pretty straightforward, I just used Ubuntu and manually installed each mining app, and then manually tuned each GPU. I simply started off with an eclectic mix of mining software, and eventually I pointed everything at NiceHash
once you tap into a serious silicon collection the tendency is to push it as far as you possibly can (it inevitably took over my entire garage)
that served me quite well until 2021. of course 2022 was rough, and in January of 2023 I have shut down the whole operation. (it's just not profitable to mine on GPUs, ETC and ETH aren't viable choices anymore. half of my setup is 4GB cards so those stopped being useful years ago, but I do still have 8 GB and a couple 12 GB cards)
I have a healthy background in full stack development (JavaScript, PHP, SQL). remoting into the machines constantly to babysit them for a decade gave me a healthy crash course in command line
at this point I'm very good at stripping down the hardware and giving "GPU baths" to the gear, rebuilding it, flashing the cards, cloning Linux drives, setting up auto run scripts,  replacing fans, ect.
I sold off my ASICs, (let me know if that was a mistake, but I don't see how antminer s9s are anything more than paper weights at this point)
I suspect there's probably a pretty large segment of the mining community (just like me) that is just sitting on a ton of silicon that could be put to better use...
I don't know a whole lot about algorithms but I suspect GPUs are a good choice for training AI? (correct me if I'm wrong about that)
I'm also open to getting different GPUs, maybe a stack of Teslas? (those were horrible for mining, but I see them flooding into eBay on the cheap)
I'm thinking "gee, maybe I'd like to have my own AI? or perhaps these could be put to good use genome folding?"
at this point I've consumed a ton of information from Google (countless bookmarks closely related to the question I'm asking here)
but I find myself overwhelmed with the information, can anyone help me with the following questions:
first consider this eclectic mix of hardware: Nvidia GPUs, AMD GPUs (with DDR4 RAM) 4GB, 8GB and 12GB cards, couple Ryzen thread rippers, Ryzen 3800s, 3950s, a couple dual CPU Epyc's and a range of old Opteron processors (with ECC server ram), all of which were previously modded to maximize hash rates for cryptocurrency mining.
The CPU rigs were devoted to Monero, GPUs of course were various altcoins, ETH and ETC

1: can I stick with Linux and train an AI?

2: if so, what would be a good AI platform to start with?

I would be interested in training my own generative art AI or simply crunching data. I am not so much interested in language models (I don't want a chatGPT)
I'm not too concerned about the electricity cost, I have an excellent network and the correct electrical infrastructure. (it's not on an industrial scale, but it is a reasonably serious 'hobbyist' home mining operation)
I feel it's also critically important to point out that I've donated hash power to Folding@Home, and I thought that was a very interesting experience.
I'm also open to gearing these rigs to do genome folding (but I'm not sure where to start)
I'm very interested in training an AI for pattern recognition and image recognition. for example, feeding the AI 10,000 images and having it correctly identify patterns or similarities in those images.
if you were to give a checklist of suggestions to get started in training an AI (in my garage on old mining equipment) what would you suggest?
if this is an illogical pipe dream feel free to point out why (I've considered it might be better to just sell the hardware)
if I were to simplify this into an elevator pitch it would simply be: "I've got a colossal amount of mining equipment that's doing nothing, I think I could put it to good use in the AI sector what would you suggest if I wanted to train my own AI puppy?"
essentially I'm hoping for a series of links and a sobering reality check on electricity cost or hashpower (is it even feasible)
turning a profit is only of minor importance to me, I am more interested in the exploration and the experience of training an open source AI from the ground up.

I don't want a botnet and have no interest in any social media AI related shenanigans
I do think it would be nice if I could just point these rigs at an AI platform (maybe something in astronomy or genetics) and have them turn a small profit on electricity costs, or even just the opportunity to donate the hash power and contribute something meaningful with my 'seemingly' worthless silicon stack
I'm moderately interested in training the AI on stock market data (but I think this is a bit of a fantasy, but one I'm honest about)
I'm extremely interested in training the AI on pattern recognition and generative art
video rendering and 3D render farm opportunities are also of moderate interest (especially if AI is involved)
I would be very interested to see if it's possible to train my own AI to defend websites against bot spam, click fraud and RPA attacks (several of my web clients have indeed been attacked by bad actors using AI and RPA in SEM click fraud attacks)

side note: if you want to provide me with your own "TLDR", I want to point out I will be totally happy to read your entire dissertation. (I don't believe in TLDR because TLDR should be based on character limit)

The answer is "it depends, but generally no".
See, AI operations is all about memory. That is useless for GPU mining. GPU mining cards are generally cost/performance optimized and that leads not to the top end with a lot of RAM, which makes them seriously suboptimal for anything other than the smallest models.
On top, TRAINING AI is a LOT more computational intensive than running it - hence companies sing thousands of GPU for the training part.

"sing" -> "using"? I think worth adding an important additional reason for "no" which is that NVidia refuse to license the CUDA driver software for gaming GPU cards in racks (apart from for crypto specifically) . . . the Tesla cards would be OK though.


text:
Can we model the global statistics of the dataset features using LayerNorm?
When applying layer normalization in a neural network, we consider the mean and variance of each input data point to the layer separately without considering the global statistics in the entire data-set. How does the model learn to, for example, discriminate for a classification based on global statistics?
If we assume a classifier for input 1D time signals of sine waves with a fixed frequency. If we label the signals by amplitude for classification (say high and low amplitude in a binary classifier), aren't we losing the amplitude information in the layernorm?
import torch
import holoviews as hv

hv.extension("bokeh")

TWOPI = 2 * torch.pi
N = 1000
fr = 10
ratio = 5

t = torch.linspace(0, 1, N)
x1 = torch.sin(TWOPI * fr * t)
x2 = ratio * x1

hx1 = hv.Curve((t, x1.numpy())).opts(xlabel="Time", ylabel="Amplitude", color="black")
hx2 = hv.Curve((t, x2.numpy())).opts(
    xlabel="Time", ylabel="Amplitude", color="red", line_dash="dashed"
)

lyn = torch.nn.LayerNorm(N)

y1 = lyn(x1)
y2 = lyn(x2)

hy1 = hv.Curve((t, y1.detach().numpy())).opts(
    xlabel="Time", ylabel="Normalized Amplitude", color="black"
)
hy2 = hv.Curve((t, y2.detach().numpy())).opts(
    xlabel="Time", ylabel="Normalized Amplitude", color="red", line_dash="dashed"
)

hx1 * hx2 + hy1 * hy2

Layer normalization normalizes a batch of data $X$ (usually the output of some intermediate hidden layer) with dimensionality $D$ (e.g. $D$ features/columns), independently on each dimension considering the statistics (e.g. mean and variance) computed on the current batch.
Consider that your batch comprises randomly picked data points, so if your batch size is too small the sampled data won't be representative of the whole dataset.
But even in such situation your classifier is not said to fail since it mostly (if not entirely) relies on (higher-order) correlations among the features and class label.

Thanks Luca.
I understand from your explanation (layernorm applies on output of hidden layer) that any mapping happening before the layernorm could take care of the time (raw) features in the input.

Exactly, usually LN is applied after hidden layers. To normalize the input, if necessary, one can use batch-normalization (which keeps updating a moving average of mean and std statistics.)


text:
Can a reward function have various cases?
I'm doing a Q-learning algorithm and I'm designing my reward function. Basically I'm working on optimizing a network while changing some parameters. My metric to measure its optimization is the delay on a flow of data (generated).
I've discretized my delay values in intervals and now I'm designing the rewards. What I was thinking about is, as I want to prioritize lower delays to use a non linear function such as $1/x^2$.
The range of my delays are usually from $10$ to $40$ secs (depending on the perturbations). It works well when the delay is varying a lot, but way less good when the delay isn't varying that much (with low perturbations).
I was then wondering if there are restrictions on reward functions. What I wanted to do is to have different parts depending on the value. Like if my value is in an interval under $15$ secs can I normalize the value? And use another way to calculate reward when it's in an interval above?
I'm new to reinforcement learning so maybe what I said is non sense, but I would gladly hear any advice or idea.
Adrien

as long as your reward is in $\mathbb{R}$ for all $(s, a, s')$ you can define it however you like.

In general the reward function can be anything that allows you to solve the task: if the reward is misspecified, the env cannot be solved; this is the reward hypothesis which RL relies on.
So, consider the reward function to be defined as $r(s,a)$ or even $r(s,a,s')$ from a state $s$, actions $a$, and sometimes including also the next state $s'$. It can be as much complex as you want, having various cases, parts, and so on. But consider that if too complex you may find a hard time debugging it, to see if the reward fn suits your task. Also, different rewards (assuming they are correct enabling to solve the problem) may result in slower or faster convergence to optimality.

What I wanted to do is to have different parts depending on the value. Like if my value is in an interval under 15
secs can I normalize the value? And use another way to calculate reward when it's in an interval above?

In principle you can do that, as I said above. Try to have a reward range that is bounded and possibly limited (e.g. $[-1, 1]$), avoiding large values and occasional spikes.
For example, you mentioned that $1/x^2$ works well when the delays are quite different. In the case the delays are similar, you can try to transform them (e.g. by $\exp$ to increase the difference) and then apply the above function. For example, case 1 is $1/x^2$, case 2 can be $1/\exp(x)$, case 3 something else, etc..

Ok, I think I get it now ! Thanks for your help !


text:
Is there any justification for excluding core statistics in 'AI Death Predictor' Paper?
Over the last few days I have been seeing a lot of buzz and news articles about an AI death predictor that is 'highly accurate' based on a 'life story'.  Due accuracy being such a poor indicator of predictive power, I tend to be highly suspicious of claims that focus on it.  For example, I can write function return True which will be extremely accurate for many questions such as, "is person X on earth?" I realize that most mainstream media articles are not going to get into more meaningful statistics so that doesn't mean it's necessarily bogus so I decided to look into the specifics of the claim more closely.  I was able to access the article here from my home computer (but not through my work VPN.)
I was a little underwhelmed by what I found.  First off, I cannot find a confusion matrix or values that could be used to build one.  Secondly, I cannot find any reference to the "78% correctness" mentioned in some of the articles I have seen.  I also don't see anything like the claims that this will predict "when you'll die" given your life story.  It seems clear that many of the news articles misrepresent the actual study, but the paper itself has some puzzling aspects.
The actual prediction was whether a person would survive for the 4 years after 2016 based on a number of factors.  However, I can't find the values for: true positives (TP), true negatives (TN), false positives (FP), or false negatives (FN).  All I can find is the mean-corrected Matthew's correlation coefficient (C-MCC) which is given as 0.41 which ranges from -1 to 1 where a value of 1 is completely correct, 0 is random chance, and values below 0 are worse than random.
Assuming I haven't just missed something (I searched visually and on key terms,) this seems a little strange.  The MCC is calculated from these values but you cannot compute those inputs if all you have is the MCC value.  The closest it seems to get is a (pretty confusing, IMO) plot which seems to show a lot of incorrect predictions and kind of looks to me like it mostly predicts that older people are more likely to die.
Question: Is it suspect to exclude the values that were used to calculate the MCC?  This seems pretty basic to me.  At the very least I would expect a paper of this type to provide precision and recall.

Too many questions in the same post. It seems that your main concern boils down to the definition of MCC and it relation to precision, recall, true positives, etc. If your questions are just subquestions of a main question, can you please put your main specific question in the title? Thanks.

"main concern boils down to the definition of MCC and it relation to precision, recall, true positives, etc" I understand how those relate.  Anyone can look that up.  I'm trying to understand why the core statistics seem to be obfuscated in the paper.

I don't find it inherently suspect to report results in the form of summary statistics like the MCC, F1 score, or chi-squared statistic or p-value. Space and reader attention are limited in journal articles, it is simply not possible to include all underlying data or run all possible statistical tests. You mention that you'd have been happier seeing precision and recall, but those themselves are just other choices of summary statistics from a wide range of options. Different readers want different analyses or levels of detail. I'd say it's typically good research practice to release as much underlying data as possible when publishing, but it's quite common not to, so failure to include raw confusion matrices doesn't by itself strike me as a red flag.
Now, it's always possible that the authors have purposefully omitted particular representations of their results in order to make their method look more impressive than it is. There is of course no justification for bad-faith selective presentation of the statistics, but all authors must be selective in some way when communicating to their readers.
In this particular case, I agree it might have been interesting and informative to see if there is any bias toward Type 1 or Type 2 errors. The authors found the summary statistics sufficient, however - from their domain perspective, they're not particularly interested in whether they're incorrectly predicting that survivors will die or that the soon-to-die will survive. The paper seems to be written more from the perspective of general feasibility of the topic rather than actual application, which would be fraught with ethical concerns.

"Space and reader attention are limited in journal articles, it is simply not possible to include all underlying data"  They can include huge diagrams but not 4 values because of space?  Really?

"The paper seems to be written more from the perspective of general feasibility of the topic rather than actual application,." That's sort of the problem I'm pointing to.  The paper is basically saying, "this is doable because we did it."  But did they?  They claim their results performed 11% better than an existing benchmark, which is what, exactly? I'm also not completely sure whether they used 2020 data in their evaluation which would definitely throw the general applicability of this into question.

@JimmyJames Sometimes more data detracts from the message - the authors don't seem to care at all about Type 1 vs. Type 2 errors, so they're not going to point the reader's focus in that direction. The paper reports performance statistics from several dozen models, reporting all the underlying confusion matrices would almost surely have been overkill. I don't really see how their choice of benchmark or practical use case relates to your original question, I'm just pointing out that omitting data may (or may not) be done for practical or well-justified reasons.

"several dozen models" I guess I missed that.  How do they only end up with one MCC value, then?

@JimmyJames They get one final model, but do report a lot more performance stats in the model tuning section. You could argue all that data should be reported too, but you have to draw the line somewhere.

I get they can include everything but if they provide the confusion matrix, the MCC can be calculated, as well as F1 and other summary stats.  We cannot work back to the confusion matrix from MCC, however.  It seems a little like buying groceries and getting a receipt that just says 'food' and a total at the bottom.

@JimmyJames True, but summaries stats do have their use as summaries - only if I am very invested in a paper would I calculate further statistics from raw confusion matrices. It ultimately depends on what you're trying to do - if I just want to know if I spent more on food than last month, I don't need the itemized receipt, just the total. For good or bad, the authors present their method as a high-level summary. It hides some information, but streamlines what remains. You can always write the authors and ask for more detail.

" if I just want to know if I spent more on food than last month, I don't need the itemized receipt, just the total" But if you want to know why you spent more, or if the charge was accurate, you are SOL. You can't question things if you aren't told about them.  Of course, this isn't even a repeatable study so maybe I need to stop treating it like hard science.


text:
RNNs - Is the recurrence at the layer-level or at the network-level?
I am confused by where the recurrence happens in RNNs, especially in the context of deep neural networks. I am trying to transform an ordinary neural network into a recurrent one from scratch.
nn = NeuralNetwork()
nn.add(Layer(2, 2)) # fully connected layer
nn.add(ReLU()) # activation layer/function
nn.add(Layer(2, 1))
nn.add(Tanh())

Does the recurrence happen at the last hidden layer of the network or after each fully connected layer?
I have heard of stacking RNNs, but I am not sure why we can't simply add a layer to an existing RNN. Is it because the recurrence is after the activation function/layer, hence the following code won't work?
import numpy as np

class Layer():
  '''
    Fullly connected layer
  '''

  def __init__(self, input_size, output_size, randomize=True):
    # initialize weights and biases
    if randomize:
      self.W = np.random.randn(input_size, output_size)
    else:
      self.W = np.ones((input_size, output_size))
    self.b = np.zeros((1, output_size))

  def forward(self, X):
    return np.dot(X, self.W) + self.b

class NeuralNetwork():
  '''
    Neural network divided into layers
  '''

  def __init__(self):
    self.layers = []

  def add(self, layer):
    '''
      Add layer
    '''
    self.layers.append(layer)

  def forward(self, X):
    '''
      Propagate input forward through each layer
    '''
    for layer in self.layers:
      # output of current layer becomes input of next layer
      X = layer.forward(X)
    return X

class ReLU():
  '''
    ReLU activation layer
  '''

  def forward(self, X):
    self.X = X
    return np.maximum(0, X)

class RecurrentLayer():

  def __init__(self, input_size, output_size, randomize=True):
    self.layer = Layer(input_size, output_size, randomize)
    if randomize:
      self.V = np.random.randn(output_size, output_size)
    else:
      self.V = np.ones((output_size, output_size))

  def forward(self, X_seq):
    hidden_state = []
    for x_t in X_seq:
      h_t = self.layer.forward(x_t)
      try:
        h_t += np.dot(hidden_state[-1], self.V)
      except:
        pass
      hidden_state.append(h_t)
    return hidden_state

X_sequence = [[1, 1], [2, 2], [3, 3]]
nn = NeuralNetwork()
nn.add(RecurrentLayer(2, 2, randomize=False))
nn.add(ReLU())
nn.add(RecurrentLayer(2, 1, randomize=False))

nn.forward(X_sequence)

[array([[4.]]), array([[20.]]), array([[64.]])]

According to this paper - which discusses RNNs, LSTMs, Stacked RNNs, and even Bidirectional RNNs - a one-layer RNN is built as follows (eq 1 and 2):
$$\begin{align}
h_t &= \mathcal H(W_{xh} x_t + W_{hh}h_{t-1}+b_h)\\
y_t &= W_{hy}h_t + b_y
\end{align}$$
These two lines should be iterated for the sequence length $T$, so you get as output hidden states $h=(h_1,...,h_T)$ and sequences $y=(y_1,...,y_T)$. $\mathcal{H}$ is the activation function.
According to the paper:

Deep RNNs can be created by stacking multiple RNN hidden layers on top of each other, with the output sequence of one layer forming the input sequence for the
next.

This means that the recurrence is at the level of computing the hidden state, $h_t$, meaning that if you have $N$ recurrent layers you compute all the $h^n$ sequences iteratively, and then when at the end you compute the output sequence $y$ which is also the network's output. See Eq. 11 and 12.
In general, I've seen that RNNs have different implementations. For example, in Keras the recurrence occurs on the output sequence $y$: so you compute all the output sequences, and optionally also return the states (if so these will inizialize the states of the next layer), which are fed to the next recurrent layer.


text:
Has anyone tried to use llama.cpp with NVLink?
Apparantly its possible to pool the memory of two 3090 using NVLink (although not with 4090). This would make it possible to run large LLM's on consumer hardware.
https://huggingface.co/transformers/v4.9.2/performance.html
Although before I invest into a new GPU, I would like to verify that it actually works, since conventional wisdom used to be that SLI only doubled performance, not memory.
So has anyone tried yet? Whats the token rate?

memory pooling is not really much of a thing these days: the interface is not really that all of a sudden you get a single address space. You still have individual GPUs, you just specify the enablement of peer transfers. This makes sense from a design point of view, because in order to write efficient software the software has to really be aware of which data is on which physical device so that efforts can be made to optimize and reduce (unnecessary) transfers.
Therefore I'd say the premise of your question is flawed. Perhaps it should be edited.
Anyway, I'm running llama.cpp with dual 3090 with NVLink enabled. llama.cpp does have implemented peer transfers and they can significantly speed up inference. For example 10 tok/s -> 17 tok/s for a 70B model.

I get 17.1 tok/s for a 70B model on dual 3090 WITHOUT NVLink, using a llamafile. Don't have NVLink to test

Yes my answer is outdated, I have known for a few months now that it is indeed possible to have very little data volume shuttling back and forth to inference LLMs across two GPUs. now the GPUs aren't going to be each fully loaded, but you do leverage both their memories so you are still winning.
NVLink is useful for various training use cases. And simulations and other HPC type workloads.

Can you report the speed you get with and without NVLink now, on something like 70B-LLama?

There isn't a way to disable nvlink in software if it is physically connected. I could physically disconnect it but my system setup is a bit unconventional and I can't quite do that right now, sorry. But by monitoring the data transferred via some command like watch -n.5 nvidia-smi nvlink -gt d you can get a good sense of how much the nvlink is being utilized; when below a rate your PCIe bus can provide (which could vary by a factor of up to 64 between x1 gen 2 and x16 gen 4!) it won't provide much benefit.


text:
In DDPM, why can't we get $x_0$ from $x_t$ directly by using equation?
In DDPM, when given $x_{t}$ and time step $t$, U-Net predicts $\epsilon$, if I understand correctly, this $\epsilon$ is the exact same $\epsilon$ in equation
$$x_{t}=\sqrt{\bar\alpha_{t}}*x_{0} + \sqrt{1-\bar\alpha_{t}}*\epsilon $$
Since $\bar\alpha_{t}$ is known, and if we have $x_{t}$, we use trained model to predict $\epsilon$, then why can't we get $x_{0}$ directly by using this equation?

Indeed there's no theoretical reason you cannot get the data $\mathbf{x}_0$ directly from the variational noise(s) after training converges successfully based on your mentioned formula due to the Gaussian noise added to the explicitly designed fixed Markov chain of the forward diffusion process according to a certain variance schedule $\beta_{1:T}$. The only reason is empirical and is explicitly mentioned in the DDPM paper's section 3.2 as follows.

To summarize, we can train the reverse process mean function approximator $\mathbf{µ_θ}$ to predict $\mathbf{\tilde{µ}}_t$, or by modifying its parameterization, we can train it to predict $\mathbf{\epsilon}$. (There is also the possibility of predicting $\mathbf{x}_0$, but we found this to lead to worse sample quality early in our experiments.)... Nonetheless, it is just another parameterization of $p_θ(\mathbf{x}_{t−1}|\mathbf{x}_t)$, so we verify its effectiveness in Section 4 in an ablation where we compare predicting $\mathbf{\epsilon}$ against predicting $\mathbf{\tilde{µ}}_t$.

Therefore it's basically an empirical result that the reparameterization via $\mathbf{\epsilon}_{\theta}$ and sampling through denoising reverse process to generate data is the best algo so far by experiments. Intuitively DDPM is essentially a hierarchical latent variable model whose generative reverse process decoder is a Markov chain with learned Gaussian transitions $p_θ(\mathbf{x}_{t−1}|\mathbf{x}_t)$  reparameterized via $\mathbf{\epsilon}_{\theta}$, so it makes sense to follow this denoising hierarchy in the reverse process via new sampling while not to directly train from the same variational ELBO objective to predict $\mathbf{x}_0$.

You actually can!
What you wrote makes perfect sense. It is actually given in original work on page 6 equation 15.
Even more, the process of how the network can reconstruct the $x_0$ is given in denoising diffusion paper in Progressive generation section on page 7. They study exactly the process you are talking about.
This process of image reconstruction is studied even more in work on perceptual weighted diffusion process, it also can be interesting for you to look at!
Hope it helps.

From my understanding of the DDPM paper, the mistake is assuming that $\epsilon$ predicted by the U-net is the same $\epsilon$ in $$x_{t}=\sqrt{\bar\alpha_{t}}*x_{0} + \sqrt{1-\bar\alpha_{t}}*\epsilon$$
In the DDPM paper, $$\epsilon$$ always refers to the noise sampled during the forward process, this is "unbiased" noise which is not following some learned gradient to arrive at a location in latent space, it is simply normal (no pun intended) noise meant to distort the original data.
$\epsilon_{\theta}$ is the parameterised noise which is learned and subtracted from a diffusion step in order to converge towards $X_0$
That is why:
$$\epsilon_{\theta}(\sqrt{\bar{\alpha_t}}X_0+\sqrt{1-\bar{\alpha_t}}\epsilon,t)$$ is defined as such.
The forward sampled noise is required to calculate the noised sample which the parameterised noise is predicted from.

Thanks, I can somehow understand you point. But in original DDPM paper, page 4, algorithm 1, step 4 and 5, aren't they the same ε?
Also in another paper https://arxiv.org/pdf/2208.11970.pdf, the whole derivation
in page 15, I think the ε_0 in equation-130 is the same ε_0 the author use for getting x_t from x_0 in equation-69/115.  Correct me if I made another mistake, I'm just getting more confused.


text:
Is this just a wild fluke?
I'm not very good at recognizing faces, so I knew it would be taxing to watch Bombshell (2019), since all three lead actresses (Charlize Theron, Margot Robbie, and Nicole Kidman) look very similar to me. But I'm such a fan of Margot Robbie I made the effort.
Cutting to the chase, there was a point (46m) where I didn't know who I was looking at, so I paused the video, clipped this from a screenshot...

...and uploaded it to Reverse Image Search on duplichecker.com. Which unsurprisingly didn't find the "original". But it offered "Find similar images in different search engines".
The first few different search engines all gave worthless results ranging from Bing's We couldn't find any results to Lensio showing me 4 different blond[e?] actresses (none were in Bombshell). But yandex.com returned this...

If you don't know, the picture is actress Charlize Theron playing media personality Megyn Kelly - who at the exact point where I paused the movie is agonizing between staying at Fox or moving to NBC. Yandex has apparently seamlessly moved from less than 3% of a single video frame in a movie to the real-life "celebrity news" context being depicted.
If this isn't a wild fluke, is it evidence of huge variation in competency across different free AI-powered image recognition services?

Any ML model's evaluation has to come from an independent test set which is supposed to be simple random samples from the target population, therefore only one instance is not evidence of huge variation in competency across different models, only an experienced fluke. To gather evidence, you need to prepare much more test samples perhaps beyond 10,000 or much more for a face recognition model. And in your case the result difference may not be due to the underlying ML model, but some reference database containing people's information.

perhaps beyond 10,000 test samples? I was wondering whether to (1) clip closeups of the other two actresses from the same movie, or (2) find a couple more "based on real events" movies and do similar close-up clip searches with them. Are you saying it would still not be significant if Yandex correctly identified the relevant real-world events from two similar clips created by either of those two methods?

Face recognition models generally use standardized benchmark dataset like LFW (Labeled Faces in the Wild) or MegaFace. While testing additional clips can be insightful, two or three clips (or even a dozen) are still too few to draw meaningful conclusions. The results from such a small-scale test would not statistically represent the model's or service's general performance which is not specifically for real-world events portrayed in movies. The threshold of 10,000 samples is a general guideline for large-scale ML systems, however, for a targeted use case a smaller dataset might suffice.

I'm very grateful you took the trouble to respond. It took me quite a while to construct this question, after which I was keen to get back to the movie! But I did check a clip of Julia Roberts in Erin Brockovich (2000) afterwards. Bing and lensio positively identified the actress, but only yandex clearly homed in on the exact frame within the movie (again, the clip was less than 5% of the total frame from which it was taken). Some sites seem to work better than others with that kind of image.

...but I shall now look into what api functionality yandex offers. I don't know if I could create it myself, but I could certainly use a VLC plugin to let me pause playback and quickly draw a box around a "person of interest" in a movie. With a quick response identifying the actor and/or the character they're playing in that movie!


text:
How can a transformer encoder attend to future tokens?
What does attending to future tokens mean? From my understanding, the transformer model works by inputting a prompt and predicting the next word in a sequence and this process just keeps repeating while attending to the words from the prompt and the already generated words. However, the explanations for transformers all mention the prevention of attending to future tokens while generating the next word in a sequence. How do these models attend to a word that hasn’t been generated yet?

Most of the recent hype around transformers is about the decoder-only transformer architecture. This architecture is different from the originally introduced traditional transformer which features both an encoder and a decoder. A transformer only attends to future tokens in the encoder. Hence, models such as chatGPT, GPT-4 etc, do not attend to future tokens.
The traditional transformer (featuring an encoder and a decoder) was introduced for the task of language translation. Hence, you would input a prompt of language A into the encoder and the decoder would output that same prompt in language B. In language translation tasks, you need to attend to future tokens. In different languages, pieces of sentences are ordered differently. You cannot simply translate each word at the same place.
Hence, the encoder of the traditional transformer model attends to future tokens. It can do so because the whole input prompt is fed into the encoder at once. You have to think of the transformer model like a standard Multi-Layer Perceptron (MLP). An MLP calculates the output for the whole input at once. The transformer does the same. It uses information from the complete input prompt to calculate what parts of the input prompt to attend to. This can be at the start of the prompt, but also at the end of the prompt. The decoder then uses this information, as well as the previously translated words, to generate the next translated word/token.
Lots of confusion is currently occurring surrounding transformers because it is not clearly explained that 'the transformer' comes in different forms. The most popular transformer right now is the decoder-only transformer (see full explanation here). It is much easier to understand how the traditional transformer works by first understanding how the decoder-only transformer works. So I'd recommend reading the linked post and then thinking again about whether you now understand the answer to this question. I expect you to easily answer your current question after you complete understand the decoder-only transformer architecture.

If this answer has sufficiently answered your question, please consider assigning it the 'green checkmark' such that future visitors of the site can better find what they are looking for :D


text:
Linear SVM Hyperparameter Selection
I'm trying to train a linear SVM on the CIFAR-10 dataset and I obtained the results in the plot below for the hyper-parameter tuning (learning rate and regularization strength). It looks like the validation accuracy is identical for all the hyper-parameter combinations and I'm not sure how to interpret this given that accuracy computed on the training set is more "informative". The question is, how should this be interpreted? Should the information about the accuracy of the model on the train set be used as hint to pick the right model given that the validation accuracy is all the same? (i.e. From all the available combinations, given this data, the best combination is the one that achieves the highest accuracy on the training set since the validation accuracy doesn't seem to vary)


text:
How do I create an AI controller for Pacman?
How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game?
It takes the pixel values (or likewise) to learn, and play the game, just like a human does. Normally, AIs like enemies - are part of the game code, which learn and play by using the internal state of the game, just like the “enemy” AI.
How can I create this abstraction between the game and the agent?
The most obvious example would be : (I don’t know how viable this is) -
The agent takes in pixel values of the game - learns - and sends “keyboard” inputs to that thread just like a human would, to play the game.
Can someone point me in the right direction - as to how to do something like this?

I don't understand this: "where the “agent” which learns is part of the game code itself. Meaning it’s just like any other AI in the game, but it learns dynamically.". What do you mean by 1. "just like any other AI in the game" (i.e. which "any other AI in the game" are you referring to? Are you referring to rule-based bots?), 2. "is part of the game code itself"? Do you mean that there are no abstractions that separate the "game" from the "agent"? If that's the case, how is that not just a programming issue (i.e. some bad programmer that doesn't know what abstractions are)?

From "separated “controller” thread", it seems that you were talking about creating multiple threads (one for the agent and one for the game, whatever that means).  If that's the case, rewrite your question to make that initial paragraph clearer. Maybe also provide the link to the code you were referring to. Which models or algorithms are you specifically referring to? Also, the title doesn't really match the problem or question in the title. One question per post, so the question in the title should be equal to the equation in the body. Edit your post to fix all these issues!

The answer below just seems a general answer to "how can I create a game AI agent". It doesn't even focus on pacman, threads, or whatever you're really asking here. It assumes that you want to learn from pixels, while you write "taking in pixel values or something else.". So, this post and the answer below couldn't be more problematic, and I don't have an idea of why people upvoted this!

Valid answer to a valid question. He just wants to know about deep rl (as he looks for something that dynamically plays games from pixel values (or other datatypes) and chooses actions like a player would (no tricks)), and not classical rl that is deeply integrated with the game (but learns dynamically). What is almost like simple rule based AI you generally find in games.

Your right the question is quite confusing.

I have edited the question, I hope it makes better sense now.

I think it is clearer what you are asking about now. The separate thread thing is a bit of a red herring (because in practice this is not the usual approach), but an interesting part of the question nonetheless. I am voting to reopen.

@R3sist do you want to know about the different types of AI or more like in you have the AI and you want to know how to integrate it with your game? (you might want to look at openAI gym and libraries based upon that) It has environments (like ms pacman, space invaders, etc.) where you can do obs= env.step(action) where obs can be pixels, ram dumps or else. But it doesnt implement the AI for you (the books and resources I had in my answer will help you with that).

@R3sist I updated my answer, does it answer you question or is it missing something? :) Then I can improve my answer.

 About the environments
For the controller part of your question, I would advice looking at openAI gym.
https://www.gymlibrary.ml/content/environment_creation/ #how to make your own gym enviroment

Those gym enviroments work kinda like this
env = gym.make("ALE/SpaceInvaders-v5")
observation = env.reset()

observation, reward, done =env.step(action)

Where your observation can be pixels,ram dumps, etc. The actions can be internally mapped to key presses. (but the internals don't really matter that much) Gym environments are just an easy way to then have your agent take in the observations and map them to actions. So your game can be stepped through so you don't have to worry about a complicated way to integrate the AI and keeping it synced.

There are already a bunch of artari games included (also ms pacman) https://www.gymlibrary.ml/environments/atari/
About the AI
Since you want to learn directly from the pixels or other more complex data types I advice you to read this:
Deep Reinforcement Learning, a textbook (arXiv:2201.02135)
https://arxiv.org/abs/2201.02135
If you just want some simple tutorial just read "Hands-On Reinforcement Learning with Python", then you can try to implement something like DQN , with some cnn architecture. (similar to how they did in arXiv:1312.5602 "Playing Atari with Deep Reinforcement Learning").
Since DQN is model free, off policy and relatively easy to implement.
(cool thing about off policy is that your agent doesnt have to interact with the environment as it doesnt need to sample "experience" using only it's current policy (way of choosing actions given a state) so you can even play the game itself and collect information (a,s,s',r) while playing and train your agent on that (to nudge it in the right direction if it gets stuck))
if you're not a fan of gradient descent based methods, you can also use methods like ES (evolutionary strategies). As that way you can directly optimize for a reward. (it's a genetic algorithm that scales well in both compute and dimensionality)  (arXiv:1703.03864 Evolution Strategies as a
Scalable Alternative to Reinforcement Learning)

(a,s,s',r) stands for action, state, next state, reward.

I "advice" you to ask for clarifications before providing an answer because the question and post are currently unclear.

It's quite clear what he wants, he is just very unfamiliar with rl. He is dissatisfied with the idea of simple AI that follows some simple rules and wants some separate agent that looks at pixels (like the player) and also can only do the same actions as the player (while poorly formulated it's quite clear). So seeing that I point him into the right direction being deep rl textbooks (since he wants the AI to learn pacman from the pixels).

The OP writes "taking in pixel values or something else.", so why are you so sure that the OP wants something that takes (only) pixel values? The OP may be unfamiliar with RL, but that doesn't mean that the question is clear. The title says "how to create an AI controller for pacman", which is different from "create something like a separated “controller” thread". The OP has already looked into implementations of RL for pacman, so I assume he's not so unfamiliar with it, otherwise, they would not have understood anything in those implementations. In my view, the post is just unclear.

I don't give an answer that takes only pixel values. I point him to deep rl with that he has his controller that can interact with openAI gym to play his pacman directly from pixel values or ram states or whatever he wants.

It's clear that the controller thread is a separate thread/system interacting with the game in a complex way. (through observing pixels values or different (but the data type is complex not deeply integrated into the game))

Ok, I would edit your post then not to start it with "Since you want to learn directly from the pixels". To me, it seems that the OP is suggesting to use multi-threading for learning and interaction (that seems to be the crux of the post), but I am not sure because the post is not fully clear.

You're right about the "Since you want to learn directly from the pixels" I changed that to a more fitting sentence.

"The OP may be unfamiliar with RL, but that doesn't mean that the question is clear". Just that someone can't find the correct terms doesn't mean they are not clear.

"The OP has already looked into implementations of RL for pacman, so I assume he's not so unfamiliar with it, otherwise, they would not have understood anything in those implementations". I point him clearly to deep rl (what is different), as classical rl has problems with high dimensional spaces (input spaces) like pixels or other complex datatypes as they tend to be tabular methods. (because of that they are generally like how OP said, like any other AI in games (meaning simple AI) that are deeply integrated with the game at hand to reduce the dimensionality )

Hello there. I’m sorry the question is confusing, I’ll try to make it more clear.
Like @hal9000 said, simple AI is deeply integrated with the game at hand to reduce the dimensionality. I want to create an AI , which can look at the pixel values or “something” else, some other dataype, just like a human does.
I was asking resources around that. How can I create a controller - which runs on a separate thread - learn by taking in pixel values, and controls that game with that. Not by being a part of the “game” itself - like other simple AIs (like the enemies)

@R3sist I think some of the confusion is that although the term "game AI" is often used for enemies in video games, and they may even use some old school AI such as pathfinding via A* search, many experts here probably won't even register that as being AI (despite the name)


text:
Could a good poker-playing AI be made that didn't look at its own cards?
A bit ago, I found out that researchers had taught a machine to play Texas Hold'Em at a level that beat most champions. However, that AI had access to the information of what cards it was dealt.
So I got to wondering. If a poker-playing AI were made, where the AI did not know what its own cards were (that is, it never 'looked' at them), would it be possible for the AI to reach a similar level of success?

Please, ask only one question per post. The fact that you put 2 questions in the title (instead of one) is a good indication that you probably should split this post into multiple ones! Please, edit your post to fix this issue.

Part of why I'm asking is that if the answer to the above question is 'yes', it segues into a question about whether you could teach a human to play Poker at a high skill level without ever looking at their cards. (Though if the other humans noticed this, they might change their behavior.)

If the AI does not know its own cards, it only has less knowledge and is therefore at a disadvantage. The AI could only act on the basis of the other players' play, while they know their own cards and can use the information.
For example, if you look at a game with 2 players before any action was taken, from the AI's point of view the chance of winning would be 50% while the other player can calculate a much more precise probability based on his own cards and build on that.
A super-human level cannot be reached with this.
If the strategy is cleverly adapted, a beginner level can perhaps be reached.


text:
What model can solve vector to vector prediction?
I am totally newbie into serial prediction.
I am think about which model or AI paradigm can be used to do vector to vector prediction?
For instance, [1,0,1] ^ [0,1,0] = [1,1,1]
Another example could be: [1,0,1]^[0,1,0]^[1,1,0]^... = [RESULT]
(I have to say that this example is not proper enough, because obviously we can use a multi-layer model to simply learn the rule of XOR operation with 100% accuracy, but I think it still can indicate what I want. The point is multi-dimensional input and figure out a non-linear rule. Let's say each row vector can be a row of a matrix, I want to find some pattern across each column entries.)
Another better example:
[[1,2,3], [4,5,6], [7,8,9]] => [10,3,8]
I know seq2seq model, to some extend can do this job, so as pointer network. But I am just skeptical on my knowledge in this RNN area. And I do need some help.
For the record, I think what I want is different from stock prediction or word prediction or sentence classification. My argue is that these tasks will need an embedding layer to output a one-hot or whatever vectorization and feed the embedding to the model.
But my original input has already been vectors. I did try to feed those vectors as an embedding to the model. But theoretically, it does not make sense to me. And it did not work under my practice.
Please enlighten me.
Yes, I dare to share my code. Please be easy on me.
    # model structure
    model = keras.Sequential()
    model.add(Input(shape=(node_size, node_size)))  # seq, input_dim
    # The output of GRU will be a 3D tensor of shape (batch_size, timesteps, 256)
    model.add(LSTM(1024, return_sequences=True, activation="relu"))
    # The output of SimpleRNN will be a 2D tensor of shape (batch_size, 128)
    model.add(Dropout(0.5))
    model.add(LSTM(node_size, activation="relu"))
    model.add(Dropout(0.5))
    # model.add(Dense(128, activation="relu"))
    # model.add(Dropout(0.1))
    model.add(Dense(node_size, activation="sigmoid"))

    # build model
    opt = keras.optimizers.Adam(learning_rate=0.001)
    model.compile(optimizer=opt, loss="binary_crossentropy", metrics=[BinaryAccuracy()])
    model.summary()
    # A_input = A_input.reshape(1, node_size, node_size)
    # label = label.reshape(1, node_size)

    # train
    model.fit(A_input, label, epochs=200, batch_size=10, validation_split=0.3)
```

Taking the example you provided, you can use a simple RNN or any Seq-Seq architecture. Your input should be of the shape (BATCH_SIZE, NO_OF_TIMESTEPS, FEATURES_AT_EACH_TIMESTEP). So, you basically need to reformat your input as arr = np.array([[1,4,7], [2,5,8], [3,6,9]]) - This assuming that 1,4,7 are representing different set of features at the same time-step (or same dim of diff vectors). Your training output (Y) would be of the shape (BATCH_SIZE, TIME_STEPS) OR arr_y = np.array([[10, 3, 8]]) Then, you can have the model predict the output for the next 3 time-steps (or 3 dimensions of the vector). Note that, in your case you are not really constraining the model to look at the link [TF_simpleRNN][1]. Note that you need to use return_sequences=True` and use a single unit as the output

I did try to use [1,4,7], [2,5,8], [3,6,9] as feature. Basically, lets assume batch_size=1, then my input is (1, 3, 3). One round training is basically
'''[3,6,9] -> [2,5,8] -> [1,4,7] -> [MODEL] -> [10,3,8]''' (first three arrows means they get popped into model one after another)
I think I did implement in the right way, but it seems like, the model can not efficiently learn a certain rule, even as simple as vector XOR operation

some of the people say, flat(30, 30) into (1,900) will not impact RNN training, but to me, I think it does effect the RNN training, because flatting (30, 30) -> (1, 900) even change the timestamp and the meaning. I believe mathematically these two has no difference when training, but I just don't feel right.


text:
When are traditional image processing methods preferable to machine learning and why?
By traditional image processing I understand, e. g. using filters to improve the image, extracting edges and then classifying objects using template matching.
My current decision criteria are:

large possible changes from picture to picture (e. g. lighting, angle of view, …) -> ML
more complex problems (e. g. weld inspection) -> ML
difficult generation of a labeled image dataset -> traditional approach
traceability of the decisions -> traditional approach

Are there other general decision-making criteria that lead to a specific choice (especially if both variants would work for a given application)?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.


text:
What is the difference between the triplet loss and the contrastive loss?
What is the difference between the triplet loss and the contrastive loss?
They look same to me. I don't understand the nuances between the two. I have the following queries:

When to use what?
What are the use cases and advantages or disadvantages of the two?
Also, how do they fit with the siamese network discussion?

These are not the same loss, but are often confused because many people use the term contrastive to refer to the triplet loss.

Contrastive Loss is defined in the paper "Dimensionality Reduction by Learning an Invariant Mapping" (link) and works with similarity labels to learn a distance mapping.
Triplet Loss is defined in the paper "FaceNet: A Unified Embedding for Face Recognition and Clustering" (link) and this is where the triplet concept appears, with a anchor and negative/positive samples defined with respect to the anchor. This is a form of contrastive learning, but its not the same as the contrastive loss.

Many papers mistakenly confuse the Triplet loss with the Contrastive loss, and they are not the same.
Note that these solve different problems, if you have known similarity relationships, then the Contrastive loss is appropriate, if you have negative/positive relationships (like for face recognition where people's identity is the anchor), then you should use the Triplet loss.
Both of these losses can be used to train a siamese neural network, but mostly they are used for embedding learning.


text:
How special tokens in BERT-Transformers work?
"[SEP] tokens are useful to differentiate the questions from answers through type_ids" Yes, but how is this helping model to understand that "I should look paragraph and generate answers from here"? We don't have if-else inside the model that will say: "if type_id==1, generate questions from here"
The same question appears for this example too:
[CLS] previous question [Q] current question [SEP] text [EOS]
How model says: "I should look at the previous question, understand the meaning from here too with the current question, and answer the question." We need if-else in here too like:
If there is a previous question:
get the meaning from it and use it with the current question
One more example based on this paper:https://arxiv.org/pdf/2005.01107v1.pdf
In this paper, we have a dataset like this: https://huggingface.co/datasets/iarfmoose/question_generator
In the t5 transformer, if we don't have those  and  tokens, the model will not learn anything. And I have no idea how specifying  and  we are helping the model to generate questions based on that answer in the context.
I hope I am clear with my question.

I think you are thinking too much like traditional programming here. A neural network doesn't need if-else. Providing the special tokens hint to the model about the structure of the question, and with enough data and training, the language model learns to utilize that hint.


text:
How and whether to apply Reinforcement Learning in an Environment with a precise and always available Evaluation?
Say we want to train an agent $A$ in an environment $E$ which provides a continuous loss $L$. That is, we want $A$ to choose its actions $a$ so that it minimizes the mistake it does, i.e., it minimizes the loss $L$.
In an Actor-Critic scenario, if I understood it correctly, the idea of the Critic is to learn to "evaluate" the current state of the environment by learning from rewards/losses. The Actor will then be adjusted so that its performed actions satisfy the Critic.
Now, if the optimal loss is always provided already by $E$, what would even be an argument to use a Critic? Is the whole idea of an Actor-Critic infrastructure in this case just wrong? Is an Actor-Critic approach only reasonable if the reward/loss sparse so that the Critic can learn a better one?
ADDED on 22.12.2023:
As an example for further clarification, $A$'s goal could be to adjust settings of various machines in $E$ so that a measurement, e.g., a pressure inside a tank, stays at a desired value all the time. So the loss would just be the deviation from that value. This is what I meant by "continuous, optimal" loss; a signal that constantly says how well $A$ is doing. This is very much unlike the case of an inverted pendulum that only provides a loss of, say, $-1$, if it fell over, and otherwise constantly $0$.

I'm not sure of the answer here but I would conjecture that A2C still has something to give because although the reward/loss is fixed (I assume this was what was meant by continuous) this is not the Q value of a state.  In a grid environment with a -1 reward for each step, this is not the Q of a state.  The Q should approach -[steps to goal] + [reward for reaching goal].

What do you mean exactly by "continuous/optimal loss"? Could you elaborate a bit?

@foreverska The reward/loss is not "fixed". Please see my added things.

@LucaAnzalone Just did that.


text:
How Can We Create Neural Networks with Different Depths and Widths But Same Number of Parameters?
Right now I am doing a research project investigating how the depth of a Neural Network affects its capacity to learn. In order to do this, I wanted to test different Networks with the same number of parameters but with different depths and widths. The solution to this problem gets reduced to a system of equations with integer solutions. Nonetheless, I haven't come up with an idea to solve it, even by using numerical methods. Therefore, I wanted to ask if there's someone that could help with finding a solution.
Context
A sequential Neural Network is defined by a number of $n$ matrices that correspond to the weights of each layer ($\{W_i\}_{i = 1,\dots, n}$). The shape of each consecutive matrix is directly related to the previous one (since they are composed after applying a non-linear function) such that:
$$
W_i \in \mathbb{R}^{a\times b} \Leftrightarrow W_{i+1} \in \mathbb{R}^{b\times c}
$$
where $a, b, c \in \mathbb{N}$.
On the other hand, the number of parameters of a Neural Network is the same as the sum of the total number of entries of each matrix $W_i$. Assuming that the number of neurons in the input layer and the number of neurons in the output layer remain constant (let's call them $a$ and $c\in \mathbb{N}$) and the number of neurons the same for each hidden layer (let's call it $b_i \in \mathbb{N}$ for a network with $i$ hidden layers) we have that we can calculate the number of parameters in the network. For example, for a network with one hidden layer, the number of parameters would be such that:
$$
p_1 = ab_1 + b_1c
$$
Then, if we add another hidden layer, the number of parameters would be such that:
$$
p_2 = ab_2 + b_2b_2 + b_2c \\
= ab_2 + b_2^2 + b_2c
$$
Therefore, for the general case with $m$ hidden layers the number of parameters is defined by the equation:
$$
p_m = ab_m + \sum_{i=1}^{m}{b_m^2} + b_mc \\
= ab_m + mb_m^2 + b_mc \\
= \boxed{b_m (a + mb_m + c)}
$$
Question
In my case, since I want to create $m$ neural networks ($5 \leq m \leq 12$ would be enough for the purposes of my research) I would need to find a solution to find the $b_i$'s $\in \mathbb{N}$ such that all of the $m$ networks have the same number of parameters $P$, where $P$ can be any positive integer (hopefully the smallest one where the system has solutions).
$$
\begin{cases}
      P = ab_1 + b_1^2 + cb_1\\
      P = ab_2 + 2b_2^2 + cb_2 \\
      \dots \\
      P = ab_m + mb_m^2 + cb_m \\
\end{cases}
$$
$\Leftrightarrow$
$$
\begin{cases}
      P = b_1 (a + b_1 + c)\\
      P = b_2 (a + 2b_2 + c) \\
      \dots \\
      P = b_m (a + mb_m + c) \\
\end{cases}
$$
Is there a closed-form solution for this system of equations in the positive integers? That is, a solution such that we could express the set of $b_i$'s in terms of $a, c,$ and $m$ where $b_i \in \mathbb{N}, \forall i \in \{1, \dots, m\}$?
I would really appreciate any help. Thank you all in advance.
Edit: Relaxing the constraints for the number coefficients to be in a band of 1% or 5% of $P$ would serve too as a solution for the problem. Regarding relaxing the number of neurons in the hidden layers to allow them to have a different number for the different layers, I think it would be useful for the project if the difference between the number of neurons in each layer does not differ to a great extent.


text:
Do NNs suffer from lack of efficiency in network structure and suggesting training parameters?
I am working on dynamical systems using Optimal Control theory and trying to find the connection between this field and Machine Learning. Consider a simple 2-layer Neural Network (NN) where the activation function is considered as $y = x+x^2 $ (I intentionally ignored bias term and supposed this activation function only for illustration purpose). So, if the output of the first layer is calculated as $y_1=w_ix_i+w_i^2x_i^2$, then the input to output mathematical relation can be given as:
$$y_o=w_o(w_1(w_ix_i+w_i^2x_i^2)+w_1^2(w_ix_i+w_i^2x_i^2)^2);$$
By expanding the terms, it results in:
$$y_o=w_ow_1^2x_i+w_o(w_1^3+w_1^4)x_i^2+2w_ow_1^5x_i^3+w_ow_1^6x_i^4.$$
But we know that there could be four parameters $\alpha_1,\alpha_2,\alpha_3,\alpha_4$ as
$$y_o=\alpha_1x_i+\alpha_2x_i^2+\alpha_3x_i^3+\alpha_4x_i^4$$
in the input to output relation, but the NN structure suggests only three parameters $w_i,w_1,w_o$ to be trained. So, the degrees of freedom is less than the actual potential of the given structure.
Will this lead to any inefficiency in NN model? Could you please suggest some references which mathematically studies the NN structure?

It is also possible to ask questions about "Optimal Control Theory" on our Operations Research site. --- If posting the same question on two sites be certain to make a note of that in the question, so people know that there might be efforts put forth elsewhere; so they don't duplicate the work. --- Here you'll get a better Machine Learning answer, while over there you'll get better optimization.


text:
Calculation of the CPTs in a Dynamic Bayesian Network
I am trying to figure out how to build my DBN (with pyAgrum), and I am a bit confused. Let us say I have the network on the next figure.
I am interested in the variable $B$. At each time $t$, I am reading the data $A_t$, $B_t$, $C_t$ and $D_t$ from my sensors to find the likelyhood of $B_{t+1}$.  An then, at time $t+1$ , I start again with the data from the sensors. Is it the correct way to proceed?
Moreover, I do not know the CPTs (Conditional Probability Table) for my DBN (it's actually one thing I want to find), can we update as time progresses the values of the CPTs?
It's one thing I'm not sure with DBN, are the CPTs fixed, or do they change over time?
Thanks in advance for your clarification


text:
Is the accuracy the best metrics to evaluate the performance of Deep Learning model?
Consider a model A that achieved an test accuracy of 99% on dataset-A with the size of 200 images and a model B that achieved only 50% on dataset-B with a size of 50,000 images. Also consider both the datasets split into train,validation and test sets in the ratio of 0.8,0.1,0.1.
But on test data of dataset-B,the model A is failed to attain same accuracy in fact it is giving lesser accuracy that of model B.
So, is the accuracy always the best measure to evaluate the performance of the DL model? Or any other better performance metrics available?

What do you mean by "But in real real-time, model B is more robust than model A."? Do you mean that when you use B to predict on the real data B has better accuracy than A? What do you mean by "more robust" here?

No, accuracy is not by far the best or the only measure you should look. It is not universal, accuracy only applies for classification problems, if you do regression then accuracy cannot be used.
There are a lot of metrics that can be used to monitor learning and predictive performance of models, there are too many to mention here. But the very important point, is that you should select metrics according to the problem and the desired behavior for the model, there is no best for this exact reason, since every problem is different.
It is very important to always think on the loss and metrics to be used for your model and problem, and not to take default choices.

Can you please give some use cases when to use loss as metrics over accuracy?

I support what Dr. Snoopy said. Other metrics include AUC (area under the curve), precision, recall, sensitivity, specificity, positive predicted value, negative predicted value,...
It ultimately depends on the context that you are deploying your model, and what the best standards to judge by are in the field.


text:
Are there any toy classification problems that can't be solved with logistic regression, but can be solved with a NN with exactly one hidden node?
Basically, I'm wondering if there are any small and simple problems that are:

complex enough to be unsolvable with a standard neural network without any hidden layer (ie. input -> output)
simple enough to be solvable with a standard neural network consisting on exactly one hidden node in exactly one hidden layer (ie. input -> one hidden node -> output)

Can such problems even exist at all? If not, why?

With exactly one hidden neuron, I believe the answer is no. The hidden neuron has one input and one output, and thus all it does is scale the final output by its activation function. Thus, it does not accomplish anything.
However, with a very slight tweak, the answer is yes. If we instead consider a network with exactly 2 hidden neurons, this becomes a very famous problem.
The XOR problem cannot be solved without a hidden layer, but is solvable with one hidden layer consisting of 2 neurons.
Famously, this limitation of single layer perceptrons led Marvin Minsky to be very critical of research on Neural Networks in the 1970s, and was a contributor to the first AI winter.
Here is an example of an XOR, which could not be computed with no hidden layer:

With one hidden node, there is one single input to the output neuron.
Therefore, the two neurons are equivalent to a single neuron with activation function the composition of the two activation functions.
This collapses the two layers into one, so no problem can be solved with only one single hidden neuron that is not solved with no hidden layer.

You should clarify that the answer to the OP's question is basically "no" due to their constraint.


text:
How can I demonstrate a novel ML classification algorithm has value?
I designed a ML classification algorithm that's simple, efficient, and effective. It's not perfect, but seems to be widely applicable across domains.
I'd like to submit it for publication, but I don't know how to present my case.
What does an ML research paper need to succeed through peer review to be published? Further, what does it need to be noticed by the ML community as a whole?

If you are gearing towards publishing, I would recommend the following.

There are different ML journals and conferences. Pick what suits you. Conference publications are generally quick compared to Journals. You can refer sample papers from different sources and decide what you would like. In the world of ML, researchers opt for conferences because it allows you to get your findings out to public more quickly compared to journals.
Once you decide your publication medium get some sample papers that have been already published there and try to follow a style similar. Beware writing publication material is hard and will take many iterations.
Write up the theoretical background of the algorithm with required mathematical / analytical proofs.
Generate baseline comparison results with benchmark datasets. Eg. if your algorithm is for tabular datasets, use standard MNIST, IRIS etc datasets to have a baseline comparison with other algorithms which have their results published. (Choose those based on what your algo does - classification or regression).
Once you have good benchmark results, compile them, write a detailed discussion on what worked and what did not with explanations.
Remember to put all necessary material publicly (eg. Github) available to reproduce your results.

For any research paper to be accepted for publication, the method need to be novel with concise proofs and explanations. You should be able to clearly explain and establish the novelty and superiority of the method with sufficient experimental proof.
These are general directions, but I would highly recommend talking to someone in person who has already published research.


text:
How to identify important features in data?
I have a couple opportunities to write a paper, or papers over some of the neural networks I have made.
I was wondering if there are anyways to figure out why the neural network classifies the data I have the way it does. As in, what features of the data the neural network is using to classify the data. The neural networks I'm using consists of ltsm layers, CNN layers, and fc layers.
I have thought about plotting the neural network at everything output, but this doesn't really help to much. Just because there are so many weights that go to each node in a layer it makes it very hard to determine what's happening. I could plot the bias's but I don't know how much influence they have over the weights.
Another thing I considered was adjusting the the values of the input data a little bit at a time and seeing where the classification changes. Which this would work to some extent, but wouldn't allow to me to get the full picture of what the neural network is doing.
So any suggestions on how to do this?
Or any papers that would be help over this topic!

Or any papers that would be help over this topic

I ended up using Guided backprop from TorchRay.
https://facebookresearch.github.io/TorchRay/attribution.html#module-torchray.attribution.guided_backprop

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.


text:
A2C unable to solve Cartpole
I have coded my own A2C implementation using PyTorch. However, despite having followed the algorithm pseudo-code from several sources, my implementation is not able to achieve a proper Cartpole control after 2000 episodes.
import torch

import numpy as np

import torch.nn as nn
import torch.optim as optim

from collections import deque

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

class A2C_agent:
    def __init__(self, observation_space, action_space, batch_size=100, gamma=.99, alpha_actor=1e-3, alpha_critic=1e-3):

        self.observation_space = observation_space
        self.action_space = action_space

        self.alpha_actor = alpha_actor
        self.alpha_critic = alpha_critic

        self.gamma = gamma

        self.batch_size = batch_size
        self.experience_batch = deque(maxlen=self.batch_size)

        # Actor network
        self.actor = nn.Sequential(
            nn.Linear(self.observation_space, 64),
            nn.ReLU(),
            nn.Linear(64, self.action_space),
            nn.Softmax(dim=1)
        ).to(device)

        # Actor weight initialization
        for layer in self.actor:
            if isinstance(layer, nn.Linear):
                nn.init.xavier_uniform_(layer.weight)

        # Critic network
        self.critic = nn.Sequential(
            nn.Linear(self.observation_space, 64),
            nn.ReLU(),
            nn.Linear(64, 64),
            nn.ReLU(),
            nn.Linear(64, 1)
        ).to(device)

        # Critic weight initialization
        for layer in self.critic:
            if isinstance(layer, nn.Linear):
                nn.init.xavier_uniform_(layer.weight)

        self.optim_actor = optim.Adam(
            self.actor.parameters(), lr=self.alpha_actor)
        self.optim_critic = optim.Adam(
            self.critic.parameters(), lr=self.alpha_critic)

    def pred_action(self, state):
        state_tensor = torch.Tensor(state).unsqueeze(0).to(device)
        action_probs = self.actor(state_tensor).detach()
        action = torch.multinomial(action_probs, 1).item()
        return action

    def save_experience(self, state, action, reward, state_next, done):
        self.experience_batch.append(
            (state, action, reward, state_next, done))

        # Learn when the experience batch is full or when the episode ends
        if len(self.experience_batch) % self.batch_size == 0 or done:
            self.learn()
            # Reset experience
            self.experience_batch.clear()

    def learn(self):
        # Initialize the loss
        actor_loss = 0.
        critic_loss = 0.

        # Loop through the experience batch
        for state, _, reward, state_next, done in self.experience_batch:
            state_tensor = torch.tensor(state).unsqueeze(0).to(device)
            state_next_tensor = torch.tensor(state_next).to(device)

            advantage = reward + (1 - done) * self.gamma * \
                self.critic(state_next_tensor) - \
                self.critic(state_tensor).detach()

            # Compute action prob distribution and sample action
            action_probs = self.actor(state_tensor).detach()
            action_distrib = torch.distributions.Categorical(
                probs=action_probs)
            action = action_distrib.sample().unsqueeze(0)
            action_tensor = torch.tensor(action, dtype=torch.int64)

            # Compute and accumulate loss
            critic_loss += advantage.pow(2)
            actor_loss += -action_distrib.log_prob(action_tensor) * advantage

        # Average the loss
        batch_len = len(self.experience_batch)
        actor_loss /= batch_len
        critic_loss /= batch_len

        # Update Actor and Critic networks
        self.optim_actor.zero_grad()
        actor_loss.backward(retain_graph=True)
        self.optim_actor.step()

        self.optim_critic.zero_grad()
        critic_loss.backward()
        self.optim_critic.step()

Is my A2C agent correctly implemented? Or did I misunderstood any part of the algorithm?
Thanks!

Hi @Antonio. From what I recall, MountainCar is a reinforcement learning environment with sparse rewards. Has your agent ever reached the victory state yet (top of the mountain)? If not, there is no reason that it should learn an optimal policy.

Hi @DeepQZero, and sorry! I had a typo when I wrote the question. I meant the Cartpole environment, not MountainCar. I've just edited the question. Thanks for the comment :-)


text:
Designing Reinforcement Learning Environments for Continuing Tasks (Non-Episodic)
I want to create a continuing (non-episodic) reinforcement learning environment for chiller plant operation. There are some tutorials focusing on creating environments for the episodic case, however I couldn't find one for the non-episodic cases.
My problem is involving scheduling 3 chiller plant for energy optimisation. The state, action spaces and reward function are something like what is defined below.
S : {temperature, demand}
A : {c1 condensor, c1 evaporator, c1 compressor, c2 condensor, ... , c3 compressor} - where c1, c2 and c3 refer to the 3 chillers
R : f(suppy, load) - some function defined based on supply and load
As my environment is sitting in a Simulink model I need to solve the RL problem in MATLAB (RLToolBox). However any method of creating an environment in OpenAI gym is fine as well since I believe all these frameworks work in a similar fashion.
The main issue I am having is how we define the "done" ("isdone" in MATLAB) variable, as this case is non-episodic. The general procedure for the episodic case is ...
next_state, reward, done, info = env.step(action) (Gym implementation)
I believe we just don't pass anything for "done" variable in the non episodic case. Am I correct?.
Further if we don't pass anything to the "done" variable how would the algorithm eventually terminate. By this what I mean is for practical purposes even for non-episodic cases must eventually terminate (I think, maybe I am incorrect) so that things like performance criteria can be measured.
Another issue is what would you pass to the environment reset function? Would it be None value?
If so how do define a termination criteria in non-episodic cases? or does it happen from the algorithm side (i.e DDPG implementation in RLToolBox/stable baseline)?. When something like the RLAgent finding the optimal policy (q -> q*) happens?
Any knowledge on this much appreciated as I am just starting out RL.
Thanks

Welcome to SE.AI!  It looks like a lot of what you're asking about are straightforward programming-questions -- those questions might work on StackOverflow.  You could probably just focus on asking what the signatures and call-conventions for the relevant Matlab's RLToolBox methods are, without needing to discuss the background.

BTW, I don't immediately recall having played with RLToolBox, but I'd guess that you could probably just do a coupled (non-episodic) environment the same way that you'd do an $n=1$ series of de-coupled (episodic) environments.

Sure thanks, I'll post it in stackoverflow

Side-note: Careful with that setup. Your environment has hidden state (delays, thermal mass, presumably with a different temperature you don't measure), but many RL algorithms assume a MDP, where the optimal action only depends on the visible state. E.g. they will assume past actions are irrelevant, given the state.

Notice that in newer versions of both OpenAI Gym and Gymnasium, the done flag has been replaced by two variables terminated and truncatedwhere:terminatedindicates whether or not the new state is terminal, andtruncated` instead says whether the episode have been artificially ended, by a time limit for example (which may be useful in your case.)

Thanks , does any one also know anything regarding how continuing environments eventually terminate (since we want at some point to use the learnt policy instead training forever)? Is it when the optimal policy (v*/q*) is found?

You can set a performance threshold that when it's surpassed by the agent the training terminates. But the simplest approach is to use a maximum number of training steps (or updates) criterion that terminates the training when exceeded. Otherwise, you can terminate when the policy converges to some value (which may be suboptimal, e.g. due wrong hyper-parameters.)

Can you please put your specific question in the title?