text:
Do I need a large pool of training data to train a bot to play the 'pegging' game in the cribbage card game
The game of cribbage https://en.wikipedia.org/wiki/Cribbage is a two-player card game played over a series of deal with the goal to reach 121 points.
The game's elements are:

the discard. There are three hands, one for each player of 4 cards selected from six. The discards go face down to a hand which belongs to the dealer (the crib). On average, because non-dealer is optimising one hand, his hand scores slightly more points, say 8 points vs 7.8 for dealer, while the crib scores around 4 points, because one player aims to toss trash (while not damaging his own hand), while the other aims to toss good cards (while likewise maximising his own hand)
pegging. Here the dealer expects to peg on average around 4 points (he plays second, so has an advantage in responding to non-dealer's lead), and non-dealer around 2 points
the cut. This is a fifth card which belongs to all three hands. If the cut is a Jack, dealer scores 2 points, for free.
the order of play. This is:

the cut. Averages 2/13 = 0.15 points for dealer (if a Jack is cut)
pegs (played one card at a time, scoring for each). Dealer will peg at least 1 point, and non-dealer will peg 0 or more. Averages 4 and 2 points.
non-dealer shows his hand. Averages 8 points
dealer shows his hand. Averages 8 points.
dealer shows his crib. Average 4 points.

As should be obvious the hands are generally the main source of points, so players will tend to optimise their hands to maximise points there. This problem is OUTSIDE OF THE SCOPE of this post, since there are exhaustive analyses that have solved this problem reasonably well. E.g.,  https://cliambrown.com/cribbage/
There are 6C4 = 15 discards possible for each hand, and often the correct play will be unambiguous. For example, if we had the hand A49QKK with mixed suits (no more than 3 of any 1 suit) and we are non-dealer, then the correct hold is obvious - A4KK (this hand has two ways to make 15 A4K and A4K for 2 points each, and a pair for 2 points, plus it improve swith cuts of A, 4, 5, T, J, Q K)
After we've held the card we must then 'play the pegging game'. The scope of this post/question is therefore limited to how to play a given 4-card hand that we have already selected using an exhaustive combinatorial analysis, hence I'm assuming that the bots inputs are a given 4 card hand, as well as the three dead cards (discard and cut), and the replies to each play from our opponent.
The scoring for pegging is:

each pair scores 2 for the player playing the second card, 6 for 3 of a kind, 12 for 4 of a kind
each time the count reaches 15, the player playing the card scores 2
each time the count reaches 31, the player reaching scores 2 (it is not permitted to exceed 31, and if you have no card that will keep the count at/below 31, you must call 'go', and if both players call 'go', the last player scores 1 point). After this the cards are turned over, and play continues from zero
each time a run of 3 or more cards is visible (between the cards played by both players), then you score 3, 4, 5, 6, 7, or even 8 points according to the length of the run.

Follows discussion about pegging strategy, which I have blockquoted for those who find this tl;dr

The optimum play for a given hand is not necessarily clear-cut. For
example, with the hand A4KK, then we could lead:

K - on the basis that if dealer replies with another K (for 2 points), then we reply with the third K (for 6 points), and then most
likely dealer does not have an Ace, so we score 2 points for the 31 (K
= 10, A = 1), AND dealer must lead the next round of pegging, which is a considerable disadvantage.
not K - because there's a bias to holding 5s, so if dealer was dealt a 5, then he can reply with the 5 scoring 2 points for the 15, to
which we have no scoring reply. In addition, non-dealer generally has
a bias to leading from a pair, so if we lead the K, then in general a
smart dealer says 'he is likely to have a pair of Ks'. So even if
dealer has a K, he may decline to peg it, especially if he doesn't
hold the A himself
4 - this is the most popular lead in cribbage because 15 cannot be scored on it. As indicated above there are 16 10 cards (TJQK), as
opposed to 4 of every other denomination, so it's more likely to
receive a ten back than any other denomination, and we hold an Ace, so
we would score 2 points following a ten card reply
A - this has the same advantage that dealer cannot reach 15, and again if we get a ten card back we can score 15-2.
It might be that the A is better to lead because the 4 is such a popular lead that dealer is more likely to pair it (risking that we
hold the third 4 for 6 points), in which case we have no good reply.
If we lead the A, then the dealer is [probably!] less likely to reply
with an A (which doesn't help us).
We might prefer to lead the 4 if we think that the A is more likely to allow us to play the last card before 31, in which case dealer is
forced to lead first next round, which is a disadvantage.

During pegging we have considerable information:

which cards we hold and discarded. For example, clearly if we hold 3 4s, and the 4th was cut, then there is no chance that we get one
played against us.
which cards have been played and therefore which cards are likely to remain, based on the hand selection process. For example, if a 4 is
led then x% of the time that will be from A4 TJQK, y% it will be from
4456, etc. As more cards are played, then this becomes more obvious.
For example, if we've seen the 3 cards 5TK, then it's VERY likely that
the 4th card is another TJQK card, and somewhat likely that it's a 5.
It could be ANY card because maybe of the six cards the player was
dealt he ended up with an unconnected card. But we can say that the
chance of this is low.

In terms of exactly what cards are held, then if we analysed millions
of games, we could calculate in general knowing one, two or three
cards what the remaining cards could be.
Although there are in theory 270,725 4 card hands, in pegging terms
(ignoring suits), there are only 1820  distinct hands
(https://docs.google.com/spreadsheets/d/1fxkLBkWC2LA6J06zhku21jcG2ATHqE1RNHlPSDcc4fQ/edit#gid=834958733)
For any given hand, if we for example held the cards 358TTT and we
were non-dealer, then we would choose to hold 5TTT and toss 38. We
would then lead the T. In this spot, clearly of the 1820 possible
hands, then combinations such as TT xy are no longer possible.
As another example if we were dealt A778JK, then we'd toss the JK.
Here we'd likely lead the 7.
Before we play the 7 it's relatively simple to calculate the odds that
dealer was dealt two 7s. Since he had six cards, that is
2/461/456C2, which is 1.45%. The chance that he chose to hold those
two 7s is a different number, and we could theoretically calculate it
by figuring out for each of the 45C6 (but simplifiable!) combinations,
which hold he would make. However it won't be too far from this number
of 1.45%.
HOWEVER, once we have played a 7 and dealer replies with a second 7,
then this number is now very far away from accurate. Firstly because
it's now a simple conditional probability where one condition is
already satisfied, so the chance of any of 5 unknown cards being a
specific card (e.g., the 7 of diamonds) is now almost 1/9. Of course
dealer has 3 cards, not 5, so again the chances of this are not that
number, but not too far off, because of those hands where dealer was
dealt 77, and he holds one seven, then he is seldom going to toss the
other one.
In terms of our possible plays, we have at the count of 14:

play the Ace and score 2 points for the count of 15.
play the 7 and bring the count to 21, which is 6 points. However, dealer is fairly likely to have a TJQK (bringing the count to 31,
scoring 2 points, and making us lead the next deal). In addition,
given that dealer has a similar analysis process to us, in hands where
he holds say 78 then he probably replies to the 7 lead with the 8, as
it doesn't allow us to score 6 points. So given that dealer has
replied with 7, this increases the chance that he holds the fourth 7
as well.  It's hard to say what this chance is, but for example there
is an x% chance of 31-2, a y% chance of 28-12, and a z% chance of
something else. Or we could consider the chance of any given card.
probably not play the 8, because at 778 the count is 22, and dealer has many scoring replies: 6 is a run (3 points), 9 is a run AND 31 (5
points), and 8 is a pair (2 points)

In general it can be seen that there are:

up to 1820 unique 4-card hands
up to 455 3-card hands
up to 91 2-card hands
up to 13 1-card hands

Clearly we could weight each hand by the chance it has to be held, and
this would work well BEFORE a card has been played.
However after 1 or more cards has been played this approach is going
to be hopelessly naive. For example, let's say we led the 4. If dealer
is holding a hand like 5JJK then he's definitely not going to reply
with the 5, because we can play a 3 for 3 points (run), or a 6 for 5
points (run and count of 15).
Further, against non beginner-level players, it will be obvious that
we often hold hands like A4 JK or 23 KK, etc. So if we lead the 3, and
dealer replies with a ten card (TJQK), then it's likely he's NOT
holding cards such as 789, which a non-beginner player would likely
prefer to reply with here. It's possible of course that dealer has the
same cards. For example, if we hold 23JK, dealer might be holding
23QQ. In this case, the Q reply might be preferred by dealer, because
if the play goes 3-Q-2, then dealer can pair our 2, and perhaps dealer
doesn't like to pair our 3 lead, for fear we have a third 3, while he
lacks the fourth.
In addition, while for pegging terms suits don't matter at all, suit
information is during important, so long as the player has a flush.
For example if we were dealt 2468h Jd Js, we'd hold the 2468 of
hearts, because that's a flush worth 4 points. So if during we play,
we've seen 468h from our opponent, then the probability distribution
of possible hands is going to contain a good weight for every heart
card. Whereas if we've seen 4h 6d 8h, then we know after two cards
that he does NOT have a flush, so this weights likely hands towards
hand such as 4568, 4678 and so on, and it's unlikely that the
remaining card is, say, a K.

In general the goal of a pegging bot could be seen as:

score most points or
concede fewest points or
maximise net points

Nuance: In early game it's likely max net points is the best approach.
But if we are at a score like 117-117 (121 is the winning mark) as
dealer, then it's (let's say) 80% likely that non-dealer has 4 points
in his hand, which means he wins if we do not peg 4 points. In this
case non-dealer would try to hold cards that score 4 points (as a
hand), and if there are multiple options, try to hold cards that
reduce dealer's chance of pegging. Meanwhile dealer's realistic route
to victory would be to hold the 4 cards that give him the best chance
of pegging 4 points (remembering that dealer will on average score
more pegs, while non-dealer scores his hand first). If the score was
113-113, then dealer would play differently as he has NO chance of
pegging 8 points, but there's perhaps a 40% chance that non-dealer
fails to score 8 points from pegging and his hand. So in this case
dealer would try to stop non-dealer pegging anything.
So it seems that an AI would need to take into account the current score
to decide how to peg.

I have read a couple of papers on cribbage AI, but they have been superficial and tended to concentrate on the discard game, which can be optimised at least for discarding purposes (without considering the relative pegging value), by simply iterating through the possible hands.

http://r6.ca/cs486/ (does not cover pegging)
https://helda.helsinki.fi/bitstream/handle/10138/273545/SeanLang_MS_thesis.pdf (addresses pegging very superficially by maximising the score of the next card, not considering that that might result in very poor total scores)

Now the question is to what extent this is a problem of machine learning, and to what extent this is an exhaustive analysis? If we ignore the discard problem, and we say that we will input four-card hands to our bot using the output of a process similar to the one here https://cliambrown.com/cribbage/methodology.php then for example:

we could calculate the exact probability that our opponent holds any given four card hand, by iterating through the 45C6 hands (subject to appropriate simplifications) that our opponent could have, and then producing a weight for each of the (up to) 1820 possible hands, and for the 715 different hands of 4 unique denominations, the further chance for each to be a flush
I am not quite clear how computationally expensive this is, but it seems to me that should be calculate the weights in a reasonable amount of time

So we have 4 cards, and we have weighted possibilities for each of 1820 hands.
Clearly it's not appropriate for us to simply randomly iterate through the hands. I.e. there are four choices for our first card, likewise four for our opponent, then 3 for the next. Roughly there will be 4!*4! choices in this way (roughly because the issue of the count resetting at 31 means that the card order after the third card is not always the samme). But our opponent's reply is not random.  If we lead the 5, then he will certainly reply with a TJQK if he has one. He won't possibly reply with a 5.
So it seems to me that some kind of learning process is appropriate. But I am not really familiar with AI to say how this should work. Do we need a pool of human training data? Or do we allow the bot to play against itself?
And what role do probability tables have in this process? Is it going to be productive on a hand level to iterate through  possible hands that our opponent might have, or is a Monte Carlo process just as good?
I should note that humans might make sub-optimal play. So for example, if we play PERFECTLY, then if we hold the hand 6699, and the 4 is led, then we SHOULD reply with the 9, rather than risk a 5 on our 6, conceding 5 points. So the play of a 6 on a 4 SHOULD indicate that we hold a hand such as 5566, in which case then we are informed accordingly. But clearly the chance that we hold 6699 and just made a blunder is not zero. So the bot cannot ever be completely certain about our holds.
Likewise it might be we choose to make 'sub-optimal' plays in order to avoid becoming predictable ourselves. For example, the question 'will our opponent pair our lead' is an important one - if we have a pair, we generally want him to. But sometimes we will hold a hand such as A4 JK, in which case we don't want him to pair our lead. Some players will play more aggressively than others, and against an aggressive player, he might pair our lead every time, and a more defensive player might almost never pair our lead.

Welcome to SE:AI!  Although interesting, this post is very long.  Could you possibly synopsize at the beginning and post the actual question in the opening of the body text?  (Part of the problem I see here is what constitutes "large" in terms of a data set in context?  I get what you're asking, but it's a relative term.)  I think this question would be well served by describing the structure of the problem (i.e. the rules of cribbage) more succinctly.  If you could reduce that to Mathjax formulas, that would be optimal.

Also maybe specify the features of the game in more formal language, if possible.  (The game involves randomness, imperfect information, etc.  The card pool is size x, the number of possible hand combinations y.)  I suspect it's very similar to poker from an AI perspective, and likely the same complexity class.  Since poker has been rigorously studied, and statistical AI recently validated as able to play poker strongly, I might research that game.  For instance Measuring the Size of Large
No-Limit Poker Games

Superhuman AI for multiplayer poker (Nature)

hi, thanks for the comments.
I guess my problem is that I know very little about AI.
OTOH very few people know much about cribbage. Even if they play it casually, they typically focus on the hand discard problem and don't optimise the play of cards.
Rules are one thing, but the approaches I have seen to cribbage play seem to treat it at as a problem that will solve itself, artificially. Whereas I am not sure if AI works like that.

I have edited the post to indent the strategy about pegging so that it can be ignored, and try to focus the question on the extent to which this is a problem of probability analysis from our inputs, and to what extent perhaps the bot can learn without doing such an analyses (presumably via a very large pool of training data).

Thanks for the edit.  My sense is that the same methods used for poker would work, since they are problems with similar structures.  The caveat is that training such a network to play strongly would almost certainly be quite expensive in terms of compute time and memory for an individual developer.  (Just a guess.)

maybe so. essentially there are 4!*4! possible pegging decisions (some illegal for certain card sets = 576), 52C6 cards and 46C6 cards for each player, and 6C4 = 15 discard decisions. This comes to 9.5 * 10^25 decisions. There are also 121^2 score states = 14,641, and there will be different correct decisions for each state, although they will be similar for scores like 0-0 and 10-10, but very different for say 120-120 vs. 110-110 vs. 60-60.

Might be worthwhile to dip your toe into computational complexity theory and algorithmic combinatorial game theory.  Essentially, we can mathematically group problems with similar structures into classes, such that, even though they are distinct games, both poker and cribbage are very likely in the same complexity class, and the same algorithmic methods will work for both.

thanks. I have got a bit further down the road to scoping the problem since posting this.

Deep reinforcement learning might be suitable for your requirement. If possible I would recommend you to add DRL hashtag to your thread, so that it can reach to larger number of  DRL researchers and practitioners, to help you out.
Thanks ,
Durga


text:
Neat Python - Population size explodes because of high mutation rate
Someone has already asked a question about this. But I implemented the suggestion made in the comments without success. So I was wondering if anybody had a better idea.
I changed the mutation power for the bias and the weights to 5. And the number of genomes roughly doubles after every generation. Consequently the genetic algorithm is taken ages.
Is there any solution for this?

Can you please put your specific question in the title and explain in the body why your question is different from the one in the other post? Thanks.

@nbro Actually my question is not so different. Should I delete my question? Also I have found an answer myself. Should I answer in the other post?

If the answers to the other very similar question don't answer your question, then you can keep it, I'd say.

One can overcome the problem of exploding population by increasing the compatibility threshold. This way one need not compromise the mutation rate.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How large should the corpus be to optimally retrain the GPT-2 model?
I just started working with the GPT-2 models and want to retrain one on a pretty narrow topic, so I have problems finding training material.
How large should the corpus be to optimally retrain the GPT-2 model? And what is the bare minimum size? Should it simply be as large as possible or can it flip over and make the model worse in some way?
I am also not certain how many steps you should let the retraining run. I have been using 6000 steps when testing, and it seems not much happens after that, loss only moved from 0.2 to 0.18 last 1000 steps.

After doing many runs with GPT-2 355M I come to the conclusion that anything below 20k tokens yield worse and worse results the less tokens you have regardless of how many steps you train for. For my application the sweet spot is around 30-35k tokens.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why does a neuron in a multi-layer network need several input connections?
For example, if I have the following architecture:

Each neuron in the hidden layer has a connection from each one in the
input layer.
3 x 1 Input Matrix and a 4 x 3 weight matrix (for the backpropagation we have of course the transformed version 3 x 4)

But until now, I still don't understand what the point is that a neuron has 3 inputs (in the hidden layer of the example). It would work the same way, if I would only adjust one weight of the 3 connections.
But in the current case the information flows only distributed over several "channels", but what is the point?
With backpropagation, in some cases the weights are simply adjusted proportionally based on the error.
Or is it just done that way, because then you can better mathematically implement everything (with matrix multiplication and so on)?
Either my question is stupid or I have an error in my thinking and assume wrong ideas. Can someone please help me with the interpretation.
In tensorflow playground for example, I cut the connections (by setting the weight to 0), it just compansated it by changing the other still existing connection a bit more:

There's a few reasons I can think of, though I have not read an explicit description of why it is done this way. It's likely that people just started doing it this way because it's most logical, and people who have attempted to try your method of having reduced connections have seen a performance hit and so no change was made.
The first reason is that if you allow all nodes from one layer to connect to all others in the next, the network will optimise unnecessary connections out. Essentially, the weighting of these connections will become 0. This, however, does not mean you can trim these connections, as ignoring them in this local minima might be optimal, but later it might be really important these connections remain. As such, you can never truly know if a connection between one layer and the next is necessary, so it's just better to leave it in case it helps improve network performance.
The second reason is it's just simpler mathematically. Networks are implemented specifically so it's very easy to apply a series of matrix calculations to perform all computations. Trimming connections means either:

A matrix must contain 0 values, wasting computation time
A custom script must be written to calculate this networks structure, which in the real world can take a very long time as it must be implemented using something like CUDA (on a GPU level, making it very complicated)

Overall, it's just a lot simpler to have all nodes connected between layers, rather than on connection per node.

It doesn't.
Whether or not this is useful is another story, but it is totally fine to do that neural net you have with just one input value. Perhaps you choose one pixel of the photo and make your classification based on the intensity in that one pixel (I guess I'm assuming a black-and-white photo), or you have some method to condense an entire photograph into one value that summarizes the photo. Then each neuron in the hidden layer only has one input connection.
Likewise, you are allowed to decide that the top neuron in the hidden layer should have only one input connection; just drop the other two.
Again, this might not give useful results, but they're still neural networks.

so in short, was it made so that it could be better represented mathematically? So as a unified model? Isn't there some document or something that takes up my question and maybe even explains a meaning?

I don't follow how you get from my answer to "better represented mathematically" and "a unified model". Please explain.

ahh, i read your answer a bit wrong at first, but you actually say that it would also work with one input, but i would still like to know why someone came up with this architecture.

Which architecture?

the architecture that each neuron in one layer is connected to each in the next.

It seems like you're the one who came up with it.

Let us continue this discussion in chat.

If you adopt a slightly different point-of-view, then a neural network of this static kind is just a big function with parameters, $y=F(x,P)$, and the task of training the network is a non-linear fit of this function to the data set.
That is, training the network is to reduce all of the residuals $y_k-F(x_k,P)$ simultaneously. This is a balancing act, just tuning one weight to adjust one residual will in general worsen some other residuals. Even if that is taken into account, methods that adjust one variable at a time are usually much slower than methods that adjust all variables simultaneously along some gradient or Newton direction.
The usual back-propagation algorithm sequentializes the gradient descent method for the square sum of the residuals. Better variants improve that to a Newton-like method by some estimate of the Hessean of this square sum or following along the idea of the Gauß-Newton method.

But I donts see, why just tuning one weight worsen some other residuals? If I have a layer with 3 neurons, I just tune 3 weights (not 9) and ending with the same results. Why should it take more time? It would just look like that here: https://i.imgur.com/HvhaBrV.jpg


text:
Why this single layer perceptron for the add operation not learning the correct weights?
This might be unnecessary but to learn the basics of neural networks, I am trying to create a single perceptron neural network to solve the adding operation of 2 inputs (x1 + x2 = ybar)
The code is written in C# since the math should be very simple and it's a language I use usually more than python.
The problem is the network is not capable of learning the right weights for this simple problem so I am trying to understand why is that before checking another working example on the internet
Note before explaining more about the problem, feel free to include any level of math or technical stuff (although might not be needed for this simple question).

Learns is the time the backpropagation happens. So equal to the nb of batches.
Activation function takes x and rounds it to the nearest integer.
The cost function is yBar - y. The total cost for each batch is the sum of all the costs generated by the samples in that batch divided by the batch size (so the average).
Backpropagation is in the form of w[i+1] = w[i] - learningRate * cost.
The hope is that w1 and w2 should end up having the same value after backpropagation.
To test the code I tried first on 1 input:

It learns a combination of weights for that case to work.
But then I tried generating a random training set of 500 rows and changing the batch size to 10 expecting it to generalize the problem and the weights to be in the form of w1 ~= w2. But it is failing to do so.
Most of the predictions are away from the correct value by 30 or 40 with some exceptions that are far by a 100.
I am not sure about how to think of the problem. This is supposed to be a linear problem so the activation function should not be the issue.
I suspected the way the cost is calculated might be the issue, but there doesn't seem to be a more fitting cost function for this adding operation.
Tried changing the learning rate and the initial weights with being pretty sure that these are not the problem, and it also didn't work.
How to think of this problem ? What are the clues to look for in order to find the wrong parameter ? And how to fix it ?

The activation function you are using is not differentiable, because it has a step at every half-integer value of $x$. This could be a problem for training.
I would expect a better result using the identity as the activation function.
Other possible causes could be too small number of epochs or too small batch size. Your chosen value $1$ means Stochastic Gradient Descent, whose convergence tends to be more chaotic.
However, the weights you get aren't too far from the expected $w_1=w_2=1$, and their average is very close to it, so it looks like the code is probably right.

But i am not using the derivative of the activation function to calculate the next weights. Do you mean in general when applying gradient descend or also in this  case ? Also I was using the identify function before but that was causing the weights to change a lot. For ex if Y = 13 and YBar = 12.6 it would keep learning for it to be accurately 13 which takes forever with floating numbers.

Mmm... The step in the round function gets a big change for some small changes in the input, which I bet makes convergence less smooth. If it takes forever that looks like a larger learning rate could help.

And that flat region in the round function also has the effect of killing data points: those whose sum is not yet right but under the rounding operation it looks like correct. I think it has the effect of reducing the effective batch size.

can you clarify more what you mean by "those whose sum is not yet right but under the rounding operation it looks like correct" ?

Note this network should only deal with adding integers not floating numbers

Sorry it's clear. some benificial correction to the weights is in ignored when because these flat regions in the rounding function, which is similar to reducing the batch size

I advise to change the cost function to a mean squared error (as commonly done in regression problems): $\mathcal L(y,\hat{y}) = \frac{1}{2B}\sum_{i=0}^{B-1} (y^{(i)}-\hat{y}^{(i)})^2$; where you average the squared differences over the batch size $B$. Also it's common practice to divide by $1/2$ to remove the factor of $2$ when taking the derivative of this.
Moreover, since the range of the data can vary a lot, use some weight decay to reduce the magnitude of the weights and prevent over-fitting to a given range of values: this may also help to reach the optimal of $w_1=w_2 = 1$.
Finally, prefer larger batch sizes (e.g. 64, 128, ...) since the noise in the gradient reduces, and generate at least 20x more data samples.

Although the cost is not squared, the weight update expression corresponds to the MSE.

Valid point for the weight decay and larger data set/batch size. But why does the cost function have any effect ? I am not using gradient descend for the need to have a differentiable cost function and this is a simple learning process which is not complex to make use of the benefits of mse.

You use MSE implicitly because it is the summation of your cost expression, the difference of y's.


text:
How do I classify whether a document is legal or not given a set of keywords that appear only in legal documents?
Let's say that I want to classify whether a document is a legal document or not. I have a list of keywords that will be presented only in legal documents.
What is the proper way or algorithm to calculate probability based on this list?

The way this question is posed, it seems trivial: if any/all (depending on interpretation) of the keywords are present in the document, it is a legal document. Else it is undetermined. If this is not what you intended, can you clarify your question?

Maybe this is what you are looking for:
https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm
Basically you would

build and store a finite-state machine that resembles a trie with additional links between the various internal nodes using the given keywords.
for the candidate document, go through it with the above finite-state machine.

Then based on the result

If there is any match, then the document is valid.
Otherwise it is possible that the document is invalid. However, in terms of the probability, I do not have any good idea yet.

This is a pretty classic case of document classification via the Naive-Bayes based Bag of Words Algorithm. It'll give you a prior probability based on your keywords. Try this link for more information. In your specific case, the "Bag" is already pre defined which makes things easier.
Another related question is this which may spawn more ideas for your use case. More generally though, if you format your problem as text classication using a feature vector constructed as some set of counts based on your keyword database then you can use pretty much any ML algorithm which gives you a baysian prior or some output probability.


text:
Keras model accuracy not improving beyond threshold
I am currently working on a public project for the National Weather Model. We are experimenting with using a recurrent neural network to replace the output of a quadratic formula that is in use. The aim of the experiment is to get a speedup in the computation by using a neural network to essentially mimic the output of the quadratic formula. We have achieved an accuracy of about +-.02 but would like to see that improve to +-.001 or so in order to make the outputs indiscernible from a usage standpoint. Despite changing or increasing the training data size, validation data size, number of layers, size of layers, optimizer, batch size, epoch number, normalizations, etc. we cannot seem to move past this level of accuracy. We have changed and tested every standard metric we can find on how to improve the model, but nothing improves the accuracy beyond that threshold.
The main question we have is whether or not Keras is rounding at some point between each layer or has some limiting factor on the backend limiting the model's significant figures in the output. The training data resolution should allow for a finer level of accuracy, but as stated before, any changes made the model cannot improve past what has been achieved. Any insight on what is holding the model back would be greatly appreciated and could help with applying this method elsewhere. The Github has a readme file explaining what is occurring in each file and how to run the model as this is still a work in progress. I would be happy to dive deeper into any aspect of the model as well.
https://github.com/NOAA-OWP/t-route/tree/testing/src/lookup_routing


text:
Is there a neural network that accepts both the current input and previous output?
I am quite new to neural networks. I am trying to implement in Python a neural network having only one hidden layer with $N$ neurons and $1$ output layer.
The point is that I am analyzing time series and would like to use the output layer as the input of the next unit: by feeding the network with the input at time $t-1$ I obtain the output $O_{t-1}$ and, in the next step, I would like to use both the input at time $t$ and $O_{t-1}$, introducing a sort of auto-regression. I read that recurrent neural network are suitable to address this issue.
Anyway I cannot imagine how to implement a network in Keras that involves multilayer recurrence: all the references I found are linked to using the output of a layer as input of the same layer in the next step. Instead, I would like to include the output of the last layer (the output layer) in the inputs of the first hidden layer.

You originally tagged your post with "recurrent neural networks", so I guess you are aware of them and know what they are used for. If yes, I don't understand the purpose of this question.

Sure, after a couple of days spent searching I came across "recurrent neural networks" and I have read some things about that. I just cannot understand how can such a RNN be implemented in Python. All the information I found are about networks in which the output (e.g. the hidden states) of a layer is used as input of the same layer; instead, I would like to create a recurrent structure that involves more layers.

Well, you should have specified that in your question. I suggest that you edit your post to clarify that you are already aware of RNNs and what your question really is.

Sorry, I think you're right. I'll do it

@piotor Here's an example implementation of an LSTM, a popular type of RNN, in python: https://github.com/nicodjimenez/lstm/blob/master/lstm.py

Note that, on this site, programming issues are generally off-topic. See https://ai.stackexchange.com/help/on-topic. Your last edit seems to make yours a programming issue. So, you probably should ask your question on Stack Overflow. Can you clarify if this is a programming issue or just an engineering/conceptual issue?

You could just do this; concatenate your input_vector with zero's vector that has the size of your output. Then in the first pass you concatenate with the output instaid of the zero's vector. After that repeat.. At the end just compare (compute the loss) your entire output from t0 to t1 to your target and backprop.
You might want to look into recurrent layers, these are layers that have connections back to themselves so that the network can learn what to "remember".
These have some problems with longer sequences, so the "newer" versions try to deal with that. (LSTM and GRU)
You can also use attention mechanisms if you're dealing with sequences. (basically you learn what parts of your input sequence to look at given a certain "query", in your case maybe the last timestep) (generally used in natural language processing) But it's a bit more exotic and complicated.

You want to look at recurrent neural networks.


text:
What is Precision@K for link prediction in graph embedding meaning?
I am trying to re-implement the SDNE algorithm for graph embedding by PyTorch.
I get stuck at some issues about evaluation metric Precision@K.

precision@k is a metric which gives equal weight to the returned instance. It is defined as follows
$$precision@k(i) = \frac{\left| \, \{ j \, | \, i, j \in V, index(j) \le k, \Delta_i(j) = 1 \} \, \right|}{k}$$
where $V$ is the vertex set, $index(j)$ is the ranked index of the $j$-th vertex and $\Delta_i(j) = 1$ indicates that $v_i$ and $v_j$ have a link.

I don't understand what "ranked index of the $j$-th vertex" means.
Beside, I am also confused about the MAP metric in section 4.3. I don't understand how to calculate it.

Mean Average Precision (MAP) is a metric with good discrimination and stability. Compared with precision@k, it is
more concerned with the performance of the returned items ranked ahead. It is calculated as follows:
$$AP(i) = \frac{\sum_j precision@j(i) \cdot \Delta_i(j)}{\left| \{ \Delta_i(j) = 1 \} \right|}$$
$$MAP = \frac{\sum_{i \in Q} AP(i)}{|Q|}$$
where $Q$ is the query set.

If anyone is familiar with these metrics, could you help me to explain them?

These measures are used for evaluating how "good" an embedding of a graph is or how "good" the graph reconstructed from the embedding resembles the original.
Given the embedding and vertex $i$, it seems to be that the rank of the vertices is dependent on the probability of there being a link between vertex $i$ and vertex $j$ in the original graph. If there is a higher probability of there being a link between $i$ and $j$ in the original graph, $j$ has a lower rank.
In other words, $precision@k(i)$ is the proportion of vertices $j$ that vertex $i$ has a link to in the original graph out of the $k$ vertices for which vertex $i$ has the highest probability of having a link to, recovered from the embedding.
This matches up with the common definition of $precision@n$ used in evaluating information/document retrieval, defined as the proportion of relevant documents out of the $n$ best retrieved documents.
The average precision of a vertex, $AP(i)$, is the average of $precision@j$ over all $j$ such that there is a link between vertex $i$ and vertex $j$. Perhaps a more clear definition would have been $$AP(i) = \frac{\sum_{j \in S_i} precision@j(i)}{\left| S_i \right|}$$
where $S_i = \{j \, |\,  \Delta_i(j) = 1 \}$, the set of all $j$ such that there is a link from $i$ to $j$.
$MAP$ for a query set $Q$ is then the mean of the average precision ($AP$) over all vertices in $Q$.

Thank you very much helping me! However I still don't understand why the $precision@k(i)$ has $index(j) \le k$. Meanwhile, $k$ is just a number indicate the top $k$ probability has link  $i$ to $j$ in prediction.

$index(j) \le k$ basically means that this vertex $j$ is one of the vertices for which vertex $i$ has the $k$-highest probability of having a link to in the original graph. Are you not understanding why this metric is created as is, or how the formula for the metric is?

I think I am understanding what you wrote. It's more clear for me. Thank you so much!

No problem. Glad to know I helped!

Sorry but I still don't understand AP, and MAP. Can you provide some simple code for me?
May be for me, code is easier to explain :((

Exactly which part do you not understand? If you are more specific about this, then it would be easier to help you.

I think I understand $precision@k(i)$ which has the $k$-top probability link to vertex $i$. I don't get AP. I think AP should $AP(i) = \frac{ \sum_{i \in S}{precision@k(i)} }{|S|}$

I don't know why is $precision@j(i)$? Is $j$ like $k$ in $precision@k(i)$. If as the formula, I need to check all vertex (node) in graph by a for loop?
And what is exactly $Q$ query? Like Q = [1,2,10,100] as value of $k$?

@TruongHoang With that definition of $AP$, the argument of the function is being used as the index in the summation (which wouldn't make sense). Regarding $precision@j(i)$, yes - the $j$ is like the $k$ in $precision@k(i)$. For your last question, $Q$ is the set of vertices that you want to find the average precision for (it may be the whole graph).

I understand the confusion and I wanted to refer to this (older post) because the metric really is unclear in the context of the SDNE paper.
Perhaps I can try to explain it for future readers, in hopes that this makes sense. All this is my own interpretation, of course.
SDNE is an autoencoder setup that outputs both node embeddings ($y_i$ vector for focal node $i$) and a reconstruction of the ties of $i$ denoted by $\hat{x}_i$ with the original being $x_i$. Note that $y_i$ is the input to the decoder component, and thus the reconstruction is a function of the embedding.
In SDNE, $x_i$ are the inputs and the "labels", hence autoencoder.
Now, the notion of precision comes from information retrieval. However, for networks, the problem setting differs. We do not retrieve documents repeatedly, instead we literally predict an entire adjacency vector (especially if one uses transformer layers and such). For that reason, the "ranking" part needs to be reformulated to make any sense.
Let's take a naive view and see what would make substantive sense.
In the context of a reconstructed network, precision should mean the following:
"What percentage of reconstructed ties are in the real network?"
Whereas recall would mean.
"What percentage of real ties in the network are found in the reconstruction?"
So we have our reconstruction $\hat{x_i}$ and our ground truth vector $x_i$ - typically rows of the adjacency matrix of the network $\hat{X}$ and $X$ respectively. Let's denote these networks as $\hat{X}$ and $X$ as well as there won't be any confusion (in the paper the authors distinguish the network $G$, its adjacency matrix $S$ and, finally, the inputs and outputs $X$ as subset of $S$.)
The vectors denote ties between $i$ and $j$. With some abuse of notation, we could write for unweighted networks $(i,j) \in X \Leftrightarrow x_{i,j}=1$
Precision would be:
$$\frac{|(i,j) \in \hat{X} \cap (i,j) \in X|}{|(i,j) \in \hat{X}|}
\Leftrightarrow \frac{|\{j| x_{i,j}=1 \cap  \hat{x}_{i,j}=1\}|}{|\{j| \hat{x}_{i,j}=1\}|}$$
and recall would have the denominator with $x_{i,j}=1$ instead of $\hat{x}_{i,j}=1$.
The only difference to the precision@k metric in the paper comes from the ranking. As mentioned above, it is not immediately apparent from the paper how a reconstruction would yield probabilities that we can use for a rank- especially if ties are binary.
However, SDNE does not predict binary ties, even if these appear in the original graph. Instead, it applies a sigmoid function and thus gets some value that is proportional to the likelihood of a tie between two nodes. Long story short, each element in $\hat{x}_i$ is akin to a probabilistic prediction across possible neighbors.
To get the $index(j)$ we can thus rank the values of $\hat{x}_i$ from highest to lowest.
Let the top $k$ of $\hat{x}_i$ be above some cutoff value $t_i(k)$.
We can write precision@k as
$$\frac{|\{j| x_{i,j}=1 \cap  \hat{x}_{i,j} \geq t_i(k)\}|}{|\{j|\hat{x}_{i,j} \geq t_i(k)\}|}=\frac{|\{j| x_{i,j}=1 \cap  \hat{x}_{i,j} \geq t_i(k)\}|}{k}$$
If our network were weighted, we could do a similar ranking for $x_i$. In any case, this solves the first issue.
Now, the main problem comes from the description of $AP(i)$.
Both in the paper, and in the previous answer given, there is an obvious mistake: Where precision@k takes an integer $k$ as parameter, we are to sum over $j$ in $AP(i)$. That is, we are told in the other answer (and in the paper) that
$$AP(i) = \frac{\sum_{j \in S_i} \text{precision@}j(i)}{|S_i|}$$
with $S_i=\{j|x_{i,j}=1\}$
This of course makes no sense. $j$ comes from the node set. Nodes could be numbers, but could also be things like $v_i = $"Apple" and $v_j=$"potato". Obviously, the measure precision@"apple"$(i)$ can not be derived from the above definition.
So, we need to find an interpretation that works. Note first, that the denominator is the number of neighbors of $i$ in the network. Thus, the above sum should maximally yield $|S_i|$.
Furthermore, the authors want to sum over neighbors $j$, employing some sort of precision measure for each. Consequently, whatever is summed up, should sum up to 1 for each $j \in S_i$.
Let's consider an embedding that predicts everything perfectly.
Note that then precision@k is $1$ for every $k$. That leaves us with the conclusion that the measure must be
$$\frac{1}{|S_i|}\sum_{j \in S_i} \frac{\sum_k \text{precision@}k(i,j)}{|k|}$$
where $\text{precision@}k(i,j)$ denotes some node wise measure of precision. In any case, the measure collapses to
$$\frac{\sum_k \text{precision@}k(i)}{|k|}$$
for each $k$ where $\text{precision@}k(i)>0$.


text:
Implementing Gradient Descent Algorithm in Python, bit confused regarding equations
I'm following the guide as outlined at this link: http://neuralnetworksanddeeplearning.com/chap2.html
For the purposes of this question, I've written a basic network 2 hidden layers, one with 2 neurons and one with one neuron. For a very basic task, the network will learn how to compute an OR logic gate so the training data will be:
X = [[0, 0], [0, 1], [1, 0], [1, 1]]
Y = [0, 1, 1, 1]

And the diagram:

For this example, the weights and biases are:
w = [[0.3, 0.4], [0.1]]
b = [[1, 1], [1]]

The feedforward part was pretty easy to implement so I don't think I need to post that here. The tutorial I've been following summarises calculating the errors and the gradient descent algorithm with the following equations:
For each training example $x$, compute the output error $\delta^{x, L}$ where $L =$ Final layer (Layer 1 in this case). $\delta^{x, L} = \nabla_aC_x \circ \sigma'(z^{x, L})$ where $\nabla_aC_x$ is the differential of the cost function (basic MSE) with respect to the Layer 1 activation output, and $\sigma'(z^{x, L})$ is the derivative of the sigmoid function of the Layer 1 output i.e. $\sigma(z^{x, L})(1-\sigma(z^{x, L}))$.
That's all good so far and I can calculate that quite straightforwardly. Now for $l = L-1, L-2, ...$, the error for each previous layer can be calculated as
$\delta^{x, l} = ((w^{l+1})^T \delta^{x, l+1}) \circ \sigma(z^{x, l})$
Which again, is pretty straight forward to implement.
Finally, to update the weights (and bias), the equations are for $l = L, L-1, ...$:
$w^l \rightarrow w^l - \frac{\eta}{m}\sum_x\delta^{x,l}(a^{x, l-1})^T$
$b^l \rightarrow b^l - \frac{\eta}{m}\sum_x\delta^{x,l}$
What I don't understand is how this works with vectors of different numbers of elements (I think the lack of vector notation here confuses me).
For example, Layer 1 has one neuron, so $\delta^{x, 1}$ will be a scalar value since it only outputs one value. However, $a^{x, 0}$ is a vector with two elements since layer 0 has two neurons. Which means that $\delta^{x, l}(a^{x, l-1})^T$ will be a vector even if I sum over all training samples $x$. What am I supposed to do here? Am I just supposed to sum the components of the vector as well?
Hopefully my question makes sense; I feel I'm very close to implementing this entirely and I'm just stuck here.
Thank you
[edit] Okay, so I realised that I've been misrepresenting the weights of the neurons and have corrected for that.
weights = [np.random.randn(y, x) for x, y in zip(sizes[:-1], sizes[1:])]

Which has the output
[array([[0.27660583, 1.00106314],
   [0.34017727, 0.74990392]])
array([[ 1.095244  , -0.22719165]])

Which means that layer0 has a weight matrix with shape 2x2 representing the 2 weights on neuron01 and the 2 weights on neuron02.
My understanding then is that $\delta^{x,l}$ has the same shape as the weights array because each weight gets updated indepedently. That's also fine.
But the bias term (according to the link I sourced) has 1 term for each neuron, which means layer 0 will has two bias terms (b00 and b01) and layer 1 has one bias term (b10).
However, to calculate the update for the bias terms, you sum the deltas over x i.e $\sum_x \delta^{x, l}$; if delta has the size of the weight matrix, then there are too many terms to update the bias terms. What have I missed here?
Many thanks

There seems to be a mismatch between the weights you provide and your network diagram. Since w[0] (the yellow connections) is meant to transform $ x \in \mathbb{R}^2 $ into the layer 0 activations which are $ \mathbb{R}^2 $, w[0] should be a matrix $ \in \mathbb{R}^{2 \times 2} $, not a vector in $\mathbb{R}^2 $ as you have. Likewise, your w[1] (the red connections) should be a vector $ \in \mathbb{R^2} $ and not a scalar. Finally, if you are indeed scaling the output of layer 1 (the blue connection), then you'll need an additional scalar value. However, the blue connection confuses me a bit as usually the activated output is used directly in the loss function, not a scaled version of it. Unless the blue connection stands for the loss function.
In short, I believe if you change the shapes of your weight matrices to actually represent your network diagram, your update equations will work. I'll go through the network below to make sure I illustrate my point.
$ x \in \mathbb{R}^{2} $, an input example
$ W^0 \in \mathbb{R}^{2 \times 2} $, the yellow connections
$ W^1 \in \mathbb{R}^2 $, the red connections
$ z^0 = xW^0 \in \mathbb{R}^{2} $, the weighted inputs to the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.
$ a^0 = \sigma(z^0) \in \mathbb{R}^{2} $, the output of the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.
$ z^1 = a^0 W^1 \in \mathbb{R} $, the weighted inputs to the layer 1 nodes. The dimensions of this should match the number of nodes at layer 1.
$ a^1 = \sigma(z^1) \in \mathbb{R} $, the output of the layer 1 nodes and thus the output of the network. The dimensions of this should match the number of nodes at layer 1.
Weight Updates
As you say before your edit, $\delta^1$, as the product of two scalars $\nabla_a C$ and $\sigma'(z^1)$, is also a scalar. Since $a^0$ is a vector in $\mathbb{R}^2$, then
$\delta^1(a^0)^T$ is also a vector in $\mathbb{R}^2$. This matches what we expect, as it should match the dimensions of $W^1$ to allow the element-wise subtraction in the weight update equation.
NB. It is not the case, as you say in your edit, that the shape of $\delta^l$ should match the shape of $W^l$. It should instead match the number of nodes, and it is the shape of $\delta^l(a^{l-1})^T$ that should match the shape of $W^l$. You had this right in your original post.
Bias Updates
This brings us to the bias updates.
There should be one bias term per node in a given layer, so the shapes of your biases are correct (i.e. $\mathbb{R}^2$ for layer 0 and $\mathbb{R}$ for layer 1). Now, we saw above that the shape of $\delta^l$ also matches the number of nodes in layer $l$, so again the element-wise subtraction in your original bias update equation works.
I also tried using this book to learn backprop, but I had a hard time connecting the variables with the different parts of the network and the corresponding code. I finally understood the algorithm in depth only after deriving all the update equations by hand for a very small network (2 inputs, one output, no hidden layers) and working my way up to larger networks, making sure to keep track of the shapes of the inputs and outputs along the way. If you're having trouble with the update equations I highly recommend this.
A final piece of advice that helped me: drop the $x$ and the summations over input examples from your formulations and just treat everything as matrices (e.g. a scalar becomes a matrix in $\mathbb{R}^{1 \times 1}$, $X$ is a matrix in $\mathbb{R}^{N \times D}$). First, this allows you to better interpret matrix orientations and debug issues such as a missing transpose operation. Second, this is (in my limited understanding) how backprop should actually be implemented in order to take advantage of optimized linalg libraries and GPUs, so it's perhaps a bit more relevant.

Thank you for the comment. I actually realised this mistake too and updated the weight variables to as you said. So for layer0, a 2x2 matrix is used to represent the weights of that layer and for layer1, a 2x1 matrix is used for the weights. This has solved the issue with updating the weights as far as I can tell (delta is a vector with the same shape as the weight vector now). The issue I have is with updating the biases now. If I understand correctly, each neuron should have only 1 bias term (layer0 has a 2x1 vector of biases, layer1 has a 1x1 vector of biases).

But if delta has the shape of the weights, then how are the bias terms updated? I'll update my question to ask this in more detail. Thank you for the response.

delta does not have the shape of the weights. I've edited my answer to clarify this and to include an explanation of bias updates.

THank you for the reply again. Sorry, so for the NN I've specified, $W = [\R^{2x2}, \R^2]$ and $B = [\R^2, \R^1]$. Then $\sum_x \delta^{x, \mathbf{L}} (a^{x, \mathbf{L-1}})^T = [\R^{2x2}, \R^2]$ and $\sum_x \delta^{x, \mathbf{L}} = [\R^2, \R^1]$
Have I understood that correctly? Because you need as many update terms as you have weights and biases?

Yes, exactly. The weight and bias updates are element-wise subtractions, so the updates need to have the same shape as the thing they're updating. Again, I highly recommend deriving the update equations for your network by applying the chain rule so you can check the shapes at each step. You're very welcome and best of luck with your implementation. Also, please accept my answer if you're satisfied.


text:
Why would my transformer model always predict the same pregnancy code?
I'm trying to predict pregnancy codes with a basic transformer model architecture. These pregnancy codes are like following prg001, prg002 to prg030. Prg001 would be antenatal screening and prg030 would be maternal outcome of delivery.
The source is codes in one year (2022 for example) and the target is the codes in the following year (2023 for example).
My dataset has about 42k rows. About half (22k rows) have empty target (no code because the pregnancy ended in the previous year). I use pytorch's Transformer module.
The model is basically predicting zero code no matter what sequence im testing. For example, its not predicting that if a person has prg008 in the prior year, they would have prg030 (the code that represents delivery) in the target because they would give birth in the following year.
#Hyperparameters
max_seq_length = 20
max_gen_length = 20
learning_rate = 0.001
weight_decay = 0.01
dropout = 0.1
d_model = 128
nhead = 4
num_layers = 3
dim_feedforward = 512
batch_size = 256
num_epochs = 10

I'm using crossentropy loss. Any suggestion on how I can improve this? Most of the code is below:
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import numpy as np
import math
from typing import List, Tuple
from torch.nn.utils.rnn import pad_sequence
import os
import time

class PreTokenizedPRGDataset(Dataset):
    def __init__(self, preprocessed_file: str):
        """
        Dataset that loads pre-tokenized data from disk
        """
        with open(preprocessed_file, 'rb') as f:
            self.data = pickle.load(f)
            self.data = self.data

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        item = self.data[idx]
        return (
            torch.tensor(item['src'], dtype=torch.long),
            torch.tensor(item['tgt'], dtype=torch.long),
            item['id']
        )

class PRGDataset(Dataset):
    def __init__(self, sequence_list, tokenizer):
        self.tokenizer = tokenizer
        # Get special token IDs
        self.bos_token_id = tokenizer.bos_token_id
        self.eos_token_id = tokenizer.eos_token_id
        self.pad_token_id = tokenizer.pad_token_id

        # All sequences
        self.data = []

        for idx, seq in enumerate(sequence_list[:50000]):
            # Convert list of tokens to IDs
            seq_src_ids = torch.tensor(tokenizer.encode(seq[0]))
            seq_tgt_ids = torch.tensor(tokenizer.encode(seq[1]))
            indiv_entpr_id = int(seq[2])
            # create source sequence
            seq_ids = torch.cat([
                torch.tensor([self.bos_token_id]),
                seq_src_ids,
                torch.tensor([self.eos_token_id])
            ])
            # Create target
            tgt_ids = torch.cat([
                torch.tensor([self.bos_token_id]),
                seq_tgt_ids,
                torch.tensor([self.eos_token_id])
            ])
            # # indiv_entpr_id
            # indiv_entpr_ids = torch.cat([indiv_entpr_id], dim=0)
            self.data.append((seq_ids, tgt_ids, indiv_entpr_id))

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx]

def collate_fn(batch):
    """Custom collate function to handle variable length sequences"""
    src_sequences = [item[0] for item in batch]
    tgt_sequences = [item[1] for item in batch]
    indiv_entpr_id = torch.tensor([item[2] for item in batch])

    # Pad sequences in batch
    src_padded = pad_sequence(src_sequences, batch_first=True, padding_value=0)
    tgt_padded = pad_sequence(tgt_sequences, batch_first=True, padding_value=0)

    return {
        'src': src_padded,
        'tgt': tgt_padded,
        'indiv_entpr_id': indiv_entpr_id
        # 'src_lengths': torch.tensor([len(s) for s in src_sequences]),
        # 'tgt_lengths': torch.tensor([len(t) for t in tgt_sequences])
    }

class PositionalEncoding(nn.Module):
    def __init__(self, d_model: int, dropout: float, max_len: int = 20):
        super().__init__()
        self.dropout = nn.Dropout(p=dropout)

        # Create positional encoding matrix
        position = torch.arange(max_len).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))

        pe = torch.zeros(1, max_len, d_model)
        pe[0, :, 0::2] = torch.sin(position * div_term)
        pe[0, :, 1::2] = torch.cos(position * div_term)

        # Register as buffer (not parameter)
        self.register_buffer('pe', pe)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """
        Args:
            x: Tensor, shape [batch_size, seq_len, embedding_dim]
        """
        x = x + self.pe[:, :x.size(1), :]
        return self.dropout(x)

class PRGTransformer(nn.Module):
    def __init__(self, tokenizer, d_model, nhead, num_layers, dim_feedforward, max_seq_length, max_gen_length, dropout):
        super().__init__()
        self.tokenizer = tokenizer
        self.vocab_size = len(tokenizer.get_vocab())
        self.d_model = d_model
        self.pad_token_id = tokenizer.pad_token_id
        self.bos_token_id = tokenizer.bos_token_id
        self.eos_token_id = tokenizer.eos_token_id
        self.max_gen_length = max_gen_length

        # Token embedding
        # ***** very important to add padding_idx so model can ignore padding *****
        self.embedding = nn.Embedding(self.vocab_size, d_model, padding_idx=self.pad_token_id)

        # Positional encoding
        self.pos_encoder = PositionalEncoding(d_model, dropout, max_seq_length)

        # Dropout
        self.dropout = nn.Dropout(dropout)

        # Transformer
        self.transformer = nn.Transformer(
            d_model=d_model,
            nhead=nhead,
            num_encoder_layers=num_layers,
            num_decoder_layers=num_layers,
            dim_feedforward=dim_feedforward,
            dropout=dropout,
            batch_first=True,
            norm_first=True
        )

        # Output layer
        self.output_layer = nn.Linear(d_model, self.vocab_size)

        # Initialize parameters
        for p in self.parameters():
            if p.dim() > 1:
                nn.init.xavier_uniform_(p)

    def create_masks(self, src: torch.Tensor, tgt: torch.Tensor) -> tuple:
        src_mask = None
        tgt_mask = self.transformer.generate_square_subsequent_mask(tgt.size(1)).to(tgt.device)

        # Create padding masks if needed
        # True is where the pad tokens are
        src_padding_mask = (src == self.pad_token_id)
        tgt_padding_mask = (tgt == self.pad_token_id)

        return src_mask, tgt_mask, src_padding_mask, tgt_padding_mask

    def forward(self, src, tgt):
        src_mask, tgt_mask, src_key_padding_mask, tgt_key_padding_mask = self.create_masks(src, tgt)

        # Embed tokens and add positional encoding
        src_embedded = self.embedding(src) * math.sqrt(self.d_model)
        src_embedded = self.pos_encoder(src_embedded)
        # src_embedded = self.dropout(src_embedded)

        tgt_embedded = self.embedding(tgt) * math.sqrt(self.d_model)
        tgt_embedded = self.pos_encoder(tgt_embedded)
        # tgt_embedded = self.dropout(tgt_embedded)

        # Transformer forward pass
        output = self.transformer(
            src_embedded,
            tgt_embedded,
            src_mask=src_mask,
            tgt_mask=tgt_mask,
            src_key_padding_mask=src_key_padding_mask,
            tgt_key_padding_mask=tgt_key_padding_mask
        )

        return self.output_layer(output)

    @torch.no_grad()
    def generate(self, src: torch.Tensor) -> torch.Tensor:
        """
        generate sequences until EOS token is predicted
        """
        self.eval()
        device = src.device
        batch_size = src.size(0)

        # Start with BOS token
        decoded = torch.full((batch_size, 1), self.bos_token_id,
                           dtype=torch.long, device=device)

        # Generate tokens until max length (-1 for bos token)
        for _ in range(self.max_gen_length-1):
            output = self(src, decoded)
            next_token = torch.argmax(output[:, -1:, :], dim=-1)
            decoded = torch.cat([decoded, next_token], dim=1)

        # Find first EOS token position for each sequence, or set to sequence length if no EOS
        eos_mask = (decoded == self.eos_token_id)
        has_eos = eos_mask.any(dim=1)
        # where there's no eos, set position to sequence length
        eos_pos = torch.where(has_eos, eos_mask.float().argmax(dim=1), torch.full_like(has_eos, decoded.size(1) - 1, dtype=torch.long))

        # Create a mask: 1s before EOS (inclusive), 0s after
        mask = torch.arange(decoded.size(1), device=device)[None, :] <= eos_pos[:, None]

        # Apply mask and pad with zeros
        padded = torch.where(mask, decoded, torch.zeros_like(decoded))

        return padded

def train_model(model, train_loader, valid_loader, learning_rate, num_epochs, weight_decay, device, tokenizer, save_dir='checkpoints'):
    criterion = nn.CrossEntropyLoss(ignore_index=0)
    optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate, weight_decay=weight_decay)
    scheduler = torch.optim.lr_scheduler.OneCycleLR(
        optimizer,
        max_lr=0.001,
        epochs=num_epochs,
        steps_per_epoch=len(train_loader)
    )

    train_losses = []
    valid_losses = []
    best_valid_loss = float('inf')
    patience = 3        #number of epochs to wait before stopping

    for epoch in range(num_epochs):
        epoch_start = time.time()
        # print('epoch ', epoch)
        model.train()
        train_loss = 0
        # correct_sequences = 0
        # total_sequences = 0

        for batch_idx, batch in enumerate(train_loader):
            batch_start = time.time()
            src = batch['src'].to(device)
            tgt = batch['tgt'].to(device)

            # ***************Create teacher forcing input**************
            # 1. Create shifted version of target (shift right by 1/remove last token)
            tgt_input = tgt[:, :-1]
            # 2. Target output: remove first token
            tgt_output = tgt[:, 1:]

            # zero gradients out (instead of accumulating) for each batch because the previous gradients from the previous batch have already been used.
            optimizer.zero_grad()
            output = model(src, tgt_input)
            # print(f'output {output.size()}', output)
            loss = criterion(output.reshape(-1, output.size(-1)), tgt_output.reshape(-1))
            loss.backward()

            # Gradient clipping
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=0.5)

            optimizer.step()
            scheduler.step()

            train_loss += loss.item()

        train_loss /= len(train_loader)         # Average training loss for the epoch
        train_losses.append(train_loss)

        # validation
        print('Validation')
        model.eval()
        valid_loss = 0
        total_code_accuracy = 0
        total_sequences = 0
        with torch.no_grad():
            for batch_idx, batch in enumerate(valid_loader):
                src = batch['src'].to(device)
                tgt = batch['tgt'].to(device)
                indiv_id = batch['indiv_entpr_id'].to(device)

                # ***************Create teacher forcing input with SEP token**************
                # 1. Create shifted version of target (shift right by 1/remove last token)
                tgt_input = tgt[:, :-1]
                # 2. Target output: remove first token
                tgt_output = tgt[:, 1:]

                output = model(src, tgt_input)
                # print('output', output)
                # Reshape output and target for loss calculation
                loss = criterion(output.reshape(-1, output.size(-1)), tgt_output.reshape(-1))
                valid_loss += loss.item()

                if batch_idx % 100 == 0:
                    # Calculate accuracy
                    predictions = model.generate(src)

                    for pred, target in zip(predictions, tgt):
                        # print('pred', pred)
                        # print('target', target)
                        # remove padding tokens
                        pred = pred[pred != 0]
                        target = target[target != 0]

                        # Convert to sets of codes for comparison
                        pred_codes = set(pred.cpu().numpy())
                        target_codes = set(target.cpu().numpy())

                        accuracy = len(pred_codes.intersection(target_codes)) / len(target_codes)
                        total_code_accuracy += accuracy
                        total_sequences += 1

                    print(f'Batch {batch_idx} time: {time.time() - batch_start:.2f}s')
                    print(f'Epoch {epoch+1}, Batch {batch_idx}')
                    print(f'Loss: {loss.item():.4f}')
                    print(f'Sample indiv_entpr_id:     {indiv_id[0].cpu()}')

                    # create tensor of tokens to exclude
                    src_mask = (src[0] != tokenizer.pad_token_id) & (src[0] != tokenizer.bos_token_id) & (src[0] != tokenizer.eos_token_id)
                    pred_mask = (predictions[0] != tokenizer.pad_token_id) & (predictions[0] != tokenizer.bos_token_id) & (predictions[0] != tokenizer.eos_token_id)
                    tgt_mask = (tgt[0] != tokenizer.pad_token_id) & (tgt[0] != tokenizer.bos_token_id) & (tgt[0] != tokenizer.eos_token_id)

                    sample_input = tokenizer.decode(src[0][src_mask])
                    sample_generated = tokenizer.decode(predictions[0][pred_mask])
                    sample_target = tokenizer.decode(tgt[0][tgt_mask])

                    print(f'Sample Input:     {sample_input}')
                    print(f'Sample Generated: {sample_generated}')
                    print(f'Sample Target:    {sample_target}\n')

            valid_loss /= len(valid_loader)     # Average validation loss for the epoch
            valid_losses.append(valid_loss)
            valid_accuracy = (total_code_accuracy / total_sequences) * 100

        print(f'Epoch {epoch+1}/{num_epochs}, Train Loss: {train_loss:.4f}, Valid Loss: {valid_loss:.4f}, Valid Accuracy: {valid_accuracy:.2f}%')
        # Save best model and early stopping
        if valid_loss < best_valid_loss:
            best_valid_loss = valid_loss
            epochs_without_improvement = 0

        else:
            epochs_without_improvement +=1
            if (epochs_without_improvement >= patience):
                print(f'Early stopping at epoch {epoch+1}')
                break
        print(f'Epoch time: {time.time() - epoch_start:.2f}s')
    # Save checkpoint when we get better validation loss
    save_checkpoint(model, optimizer, scheduler, epoch,
                train_losses, valid_losses, best_valid_loss,
                save_dir)
    return train_losses, valid_losses

def evaluate_model(model: PRGTransformer, tokenizer, test_sequences: List[torch.Tensor], batch_size, device: torch.device):
    model.eval()
    with torch.no_grad():
        test_dataset = PRGDataset(test_sequences, tokenizer)
        test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=True, collate_fn=collate_fn)

        correct_sequences = 0
        total_sequences = 0
        for batch_idx, batch in enumerate(test_loader):
            src = batch['src'].to(device)
            tgt = batch['tgt'].to(device)

            # Calculate accuracy
            predictions = model.generate(src)

            # Only compare up to actual sequence lengths
            for pred, target, source in zip(predictions, tgt, src):

                print(f"\nInput:     {tokenizer.convert_ids_to_tokens(source)}")
                print(f"Generated: {tokenizer.convert_ids_to_tokens(pred)}")
                print(f"Expected: {tokenizer.convert_ids_to_tokens(target)}")
                print(f"Correct?   {torch.all(pred == target).item()}")

                correct_sequences += torch.all(pred == target).item()
                total_sequences += 1

        test_accuracy = (correct_sequences / total_sequences)
        print(f'Test Accuracy: {test_accuracy:.2%}')

def main():
    torch.manual_seed(42)
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    print(f"Using device: {device}")

    # Load tokenizer
    tokenizer = AutoTokenizer.from_pretrained('/path/prg_tokenizer')

    # Hyperparameters
    max_seq_length = 20
    max_gen_length = 20
    learning_rate = 0.001
    weight_decay = 0.01
    dropout = 0.1
    d_model = 128
    nhead = 4
    num_layers = 3
    dim_feedforward = 512
    batch_size = 256
    num_epochs = 10

    # Create train dataset and dataloader
    train_dataset = PreTokenizedPRGDataset(train_file)
    train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, collate_fn=collate_fn, num_workers=4,
    pin_memory=True)

    # Create valid dataset and dataloader
    valid_dataset = PreTokenizedPRGDataset(valid_file)
    valid_loader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=True, collate_fn=collate_fn)

    # Initialize model
    model =PRGTransformer(tokenizer=tokenizer, d_model=d_model, nhead=nhead, num_layers=num_layers, dim_feedforward=dim_feedforward, max_seq_length=max_seq_length, max_gen_length=max_gen_length, dropout=dropout).to(device)

    # Train the model
    save_dir = '/path/model_checkpoints'
    print("Starting training...")
    train_losses, valid_losses = train_model(model, train_loader, valid_loader, learning_rate, num_epochs, weight_decay, device, tokenizer, save_dir)

    return train_losses, valid_losses

if __name__ == "__main__":
    train_losses, valid_losses = main()

Hi. If your source code is open, it might be a good idea to share the link to it, if it's too big, or include it in the question if it's small (20 lines say). It would also be a good idea if you share more info about the tools/frameworks you're using to train this model.

@nbro hi i updated the question. I use pytorch's transformer module for this. Thanks for any input!


text:
Why does loss and accuracy for a multi label classification ann does not change overtime?
I have run into a strange behavior of my multi label classification ANN
model = Sequential()
model.add(Dense(6, input_shape=(input_size,), activation='elu'))
#model.add(BatchNormalization(axis=-1))
model.add(Dropout(0.2))
#model.add(BatchNormalization(axis=-1))
model.add(Dense(6, activation='elu'))
model.add(Dropout(0.2))
#model.add(BatchNormalization(axis=-1))
model.add(Dense(6, activation='elu'))
model.add(Dropout(0.2))

# model.add(keras.layers.BatchNormalization(axis=-1))
model.add(Dense(6, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer='nadam',
              metrics=['accuracy'])
history = model.fit(X_train, Y_train,batch_size=64 ,epochs=300,
                    validation_data = (X_test, Y_test), verbose=2)

The result is quite strange, I have a feeling that my model could not improve any more. Why does the loss and the accuracy does not change overtime ?
P/S For clarification, I have 6 output and the value of each output is 0 or 1 that is
output 1: can be 0 or 1
output 2: can be 0 or 1
output 3: can be 0 or 1
output 4: can be 0 or 1
output 5: can be 0 or 1
output 6: can be 0 or 1

Hi ! How many samples do you have in X_train ? Can you try to visualize the outputs on your test set (for instance, are your predictions constants ?) ? Maybe one problem is that you put a lot of Dropout, and your network is already narrow so instead of preventing overfitting this could hinder the learning...

about 4000 training samples, and validate 1000 samples.The output for each test set data point is a binary output vector that is
[out1 out2 out3 out4 out5 out6]
Particularly
output 1: can be 0 or 1
output 2: can be 0 or 1
output 3: can be 0 or 1
output 4: can be 0 or 1
output 5: can be 0 or 1
output 6: can be 0 or 1

Okay. Why did you use Dropout and Batch norm ? It may be useful to visualize the actual predictions of your trained net. You know, using for instance plt.scatter(Y_test, model.predict(X_test))

(As you have multi label classification, you may plot plt.scatter(Y_test, model.predict(X_test)) for each particular label)


text:
What are the best techniques to perform text simplification?
I'm evaluating the state of the art techniques to translate legal text to simple text, what are the best approaches for a non-English language (Portuguese)?

Maybe you should explain more in detail what you mean by "text simplification" or "simple text".


text:
How to use a conv2d layer after a flatten?
I am not familiar with Deep learning and Pytorch. And I want to know how to deal, in general with such a situation. So, I was wondering if I used a pretrained model (EfficientNet for example) if I want to change the _fc attribute and use conv2d in it, how can I recover a 2D structure? Because the pretrained model flattens it just before _fc.
for example, the pretrained model outputs a flattened feature vector of 1280 elements what I did is the following:
self.efficient_net._fc = nn.Sequential(
                nn.Linear(1280, 1225),
                nn.Unflatten(dim=1, unflattened_size=(1, 35, 35)),
                nn.Conv2d(1, 35, kernel_size=1),
                ...,
                )

I didn't have a specific height and width to recover in the 2D structure, so I assumed that h = w = some size and I use a linear layer whose output is equal to the square of the "some size". In the example above 35² = 1225. I am not sure if the unflatten is the correct way to do this. Then I added the conv2d. My code works but it doesn't give good results which probably means that the 2D structure I recovered does not capture any meaningful information.
Can anyone enlighten me with general knowledge about how things are done in my situation, or give me some comments? Thank you!

This seems to be a programming issue, so it seems to be off-topic here, and it would be more appropriate for Stack Overflow. Can you confirm that this is just a programming issue?

@nbro, I am asking how people deal with this issue in general, I am not asking that someone corrects my code (my code works as I mentioned).
I am more into how to do things properly. Asking for experienced people how they deal in general with such a situation. I put my code to show how I dealt with it and because I don't know whether my reasoning is correct or not. I asked for a general approach

I don't know pytorch but in tensorflow when you have a pretrained model, you have access to all the layers. So you can just cut the network from before the flatten layer. I think you can do so in pytorch

@amin Thanks for the answer, that would be actually nice to discard the flattening layer from the pretrained model. I will look into it.

To answer the question in the title, your enclosed method is a valid way to use 2d convs after a flattened feature vector. However, the bad results you experience could come from the structure of your model or from the way you train it. Regarding you last question, it is very hard to give you an advice without knowing your intentions in detail. Regardless, here are my two cents.
First of all, in the case you actually want to use this approach, to have a pretrained model and add your layers after one of its layers, you might want to keep the parameters of the original network intact at least until your newly initialized layers get trained properly. To achieve that, you need to use the gradients for updating only the parts you added as described in this comment. (You definitely should read the other comments there as well to get a better picture.)
Secondly, it might be worth reconsidering if you really want to add your layers after the very last layers of the pretrained network. Depending on your goals, using the output of some prior layers as the input to your layers might be more beneficial to you. (Just do not forget to keep the parameters of the pretrained model intact as I advised in the prior point.)
Lastly, the structure of your layers should also be reconsidered. 2d convs with a kernel size of 1x1 in this situation seems strange to my limited experience, but without knowing what you want to do, its hard to give any solid advice in this regard.
Therefore you might be better off splitting your question into smaller parts and work your way through them one by one.
(My reputation is not high enough otherwise I would have left a comment.)

Hello thanks for your answer. My question was more or less general, I did not have a particular goal, just wondering if it was doable, the way I did it. But your advice are appreciated, all you said makes sense, at least to me, thanks again!


text:
How to fight with unstability in self play?
I'm working on a neural network that plays some board games like reversi or tic-tac-toe (zero-sum games, two players). I'm trying to have one network topology for all the games - I specifically don't want to set any limit for the number of available actions, thus I'm using only a state value network.
I use a convolutional network - some residual blocks inspired by the Alpha Zero, then global pooling and a linear layer. The network outputs one value between 0 and 1 for a given game state - it's value.
The agent, for each possible action, chooses the one that results in a state with the highest value, it uses the epsilon greedy policy.
After each game I record the states and the results and create a replay memory. Then, in order to train the network, I sample from the replay memory and update the network (if the player that made a move that resulted in the current state won the game, the state's target value is 1, otherwise it's 0).
The problem is that after some training, the model plays quite well as one of the players, but loses as the other one (it plays worse than the random agent). At first, I thought it was a bug in the training code, but after further investigation it seems very unlikely. It successfully trains to play vs a random agent as both players, the problem arises when I'm using only self play.
I think I've found some solution to that - initially I train the model against a random player (half of the games as the first player, half as the second one), then when the model has some idea what moves are better or worse, it starts training against itself. I achieved pretty good results with that approach - in tic-tac-toe, after 10k games, I have 98.5% win rate against the random player as the starting player (around 1% draws), 95% as the second one (again around 3% draws) - it finds a nearly optimal strategy. It seems to work also in reversi and breakthrough (80%+ wins against random player after the 10k games as both players). It's not perfect, but it's also not that bad, especially with only 10k games played.
I believe that, when training with self play from the beginning, one of the players gains a significant advantage and repeats the strategy in every game, while the other one struggles with finding a counter. In the end, the states corresponding to the losing player are usually set to 0, thus the model learns that whenever there is the losing player's turn it should return a 0. I'm not sure how to deal with that issue, are there any specific approaches? I also tried to set the epsilon (in eps-greedy) initially to some large value like 0.5 (50% chance for a random move) and gradually decrease it during the training, but it doesn't really help.

The AlphaZero paper mentions an "evaluation" step that seems to deal with the  the problem similar to yours:

... we evaluate each new neural network checkpoint against the current best network $f_{\theta_*}$ before using it for data generation ... Each evaluation consists of 400 games ... If the new player wins by a margin of > 55% (to avoid selecting on noise alone) then it becomes the best player $\alpha_{\theta_*}$ , and is subsequently used for self-play generation, and also becomes the baseline for subsequent comparisons

In the AlphaStar they've use a whole league of agents that was constantly played against each other.

When in an environment with competing agents, from the perspective of each agent, the environment becomes non-markovian. That occurs because each agent is constantly adapting its own strategy to other's actions, so a transition that occurred to a pair (s,a) before, resulting in a positive reward, might result in zero or negative reward in future iterations of the game.
I didn't see mentioned, but I imagine that you are using some DQN variation to train the network, since you use a replay buffer. To use this framework, you assume that the environment, from the perspective of the agent, follows a MDP. But, as I argued above, some tuples from the replay buffer might not represent valid data for training, so the corresponding network that is trained with it becomes unstable.
A solution might be use the idea of centralized training with decentralized execution, in conjunction with some policy gradient (PG) algorithm, like REINFORCE or Actor-Critic. Since PG are on-policy algorithms, the data used to train the network is generated by the current policy, so you don't have the replay buffer issue. On the other hand, since is on-policy, it's sample inefficient. The centralized training might help to increase the sample efficiency (it's in fact a good solution to partial observable environments, but from what I understand is not the case with the your game). An additional solution to the sample inefficiency is to use off-policy PG, using, for example, past policies, with respective experience, in a importance sampling framework.
Some related references:
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments: https://arxiv.org/abs/1706.02275
Off-Policy Policy Gradient: https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#off-policy-policy-gradient


text:
How to properly resume training of deep Q-learning network?
I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a certain number of episodes and save the resultant model. Later, I will load up the previously saved model and resume training.
However, what I'm noticing is that when training resumes, the average rewards goes back to very low again, compared to what it achieved at the end of the previous training session. What I'm currently doing is to load up the previously saved model into the prediction and target models, and I keep all hyperparameters unchanged.

Is this behaviour expected?
If not, how do I properly resume training of a deep q-learning
network?
Do I start off with the epsilon value at the end of the
previous session, currently I reinitialize that as well?

Do I start off with the epsilon value at the end of the previous session, currently I reinitialize that as well?

You should probably re-start with $\epsilon$ at the value you left off at. Using high values of epsilon may cause the neural network to forget some of what it learned from close-to-optimal policies in favour of learning possibly useless values of states and actions that are not important to a more highly-trained agent.
Also, you should either save, or wait to refill experience replay memory before restarting training updates to the neural network. Working from a small memory may also cause the neural network to overfit to specific samples and generalise less well - at least temporarily until the memory fills up again.
From your description, I am assuming that you are monitoring the average reward per training episode. This is metric that is easy to collect, but that has a problem in off-policy RL. The problem is that you are seeing the results from your behaviour policy, not your learned policy. When using $\epsilon$-greedy in Q learning, with a high value of epsilon, then the behaviour policy is likely to perform badly.

Is this behavior expected?

I would expect it if you re-set $\epsilon$ to a high value.

If not, how do I properly resume training of a deep q-learning network?

I recommend that you look at the following things:

Re-start with $\epsilon$ at or close to where you left off. Perhaps allow starting epsilon to be passed in to the script as an argument.

Measure performance of the greedy policy at checkpoints - e.g. run 100 test episodes at the end of every 500 training episodes - and use plots of that to decide how well your agent is performing. You can still plot training performance, but it should not be your guide to how well the agent is learning.

Save experience replay so far at checkpoints too, and reload it on re-start. Alternatively, allow for experience replay to fill significantly before allowing update steps in the training loop.

You could consider these in priority order. Note that just the first one may appear to "fix" your problem, but that is because you are not yet properly measuring your agent's performance.

Thank you for your suggestions and the detailed explanation. As you have suggested, re-start at previous epsilon  has helped solved the problem. Regarding the performance measurement, would it be correct to say that as epsilon becomes quite small, the reward obtained in training episode should be quite close to test result?

@TianxunZhou Yes, the expected reward for $\epsilon$-greedy policy will become close to expected reward for greedy policy as $\epsilon$ becomes close to zero. It is not always clear how close to zero you need to get before that is true - that depends on the environment.


text:
How to source training data in ML for information security?
A company entrusts a Data Scientist with the mission of processing and valuing data for the research or treatment of events related to traces of computer attacks. I was wondering how would he get the train data.
I guess he would need to exploit the logs of the different devices of the clients and use statistical, Machine Learning and visualization techniques in order to bring a better understanding of the attacks in progress and to identify the weak signals of attacks... But how would he get labelled data?
He might get the logs of attacks received before, but that might not have the same signature with the attacks that are going to come later? So it might be difficult to create a reliable product?

You have implicitly assumed that supervised learning is being used, given the assumption that labels are needed. But this might lead to the following potential problems:

Log file data tends to be huge, and it may be infeasible to label due to the time/expertise required;
Then there's the class imbalance problem, in that attack examples are far far rarer than "normal" behaviour, and this can mess up supervised models during both learning and evaluation stages;
And finally even if the data is labelled, a supervised model is unlikely to be useful in detecting completely novel attacks because it has not been trained to recognise these.

I think a far easy way to approach these kinds of problem would be unsupervised learning: model normal patterns and behaviours in the logs, and then flag any deviations from normality. It may be an attack or it may be new normal behaviour. In the latter case, the model can be updated. There are various approaches here such as clustering, outlier detection and possibly self-supervised learning that might be useful. Dimensionality techniques might also be useful to visualise clusters of "normal" behaviour that can be compared to abnormal patterns.

We do a lot of work in this space.  You can absolutely do a great job using supervised learning.  Evaluation of reconstruction loss is just one example of how this can be accomplished using limited training data or possibly even only using examples of "good" activity.  It's all about feature selection.


text:
Language Processing: Determine if one paragraph is relevant to another paragraph
Context: I want to determine if someone's written review contains content that is relevant to a paragraph that they are reviewing.
To do so, I am trying to determine if one paragraph is relevant to another paragraph. I initially tried to use TF-IDF to calculate the relevancy, but I think TF-IDF works well for determining if one paragraph is relevant to a whole set of paragraphs. I only want to determine if two paragraphs are relevant with each other.
What would be a good approach for this problem?

A very simple approach can be:

Calculate tf-idf vector for sentence 1 and 2.
Calculate vector similarity (Cosine similarity) of these 2 vectors.

This is a general approach and works for any representational vector.
For a more complex one, check semantic similarity with BERT post from Keras blog.


text:
Recommended Time serie forecasting model for Fibonacci levels classification
I have a set of time series data which gives me fibonacci levels and the duration at which the value is at this level.
Data structure to look like:
Date / Duration (minutes) / Level

201201 / 380 / 2
.....

210422 / 400 / 4

I'd like to create a NN model (LSTM maybe) that would forecast the next level, the probability it reaches it and this for several steps ahead (1 step = 400 minutes).
Which time series forecasting model would you recommend ?
Thanks in advance.

Could you explain what a Fibonacci level is? I'm not sure what you are trying to achieve with your model. What is the desired output?

Fibonacci levels in stock price (for example) is defined between the minimum and the maximum price during a given period. The range (min, max) is chunck in levels. Below min and above max levels are called extension levels. So, let's say that I look at 1 year back. It gives me my level and I start classifying and I count the duration in minutes price stays in each level. I am now trying to have a time series prediction model which tells me the probability that in X minutes, it should be at the level Y with a given probability. Hope it's clear, if not I am happy to explain more.

So basically, instead for predicting the actual value of the stock price, you want to know in which range the price will fall in and for how long?

Exactly this with a given probability

It seems like you are trying to produce two outputs here. This usually makes the models more complex. What if you only predicted the Fibonacci levels for each time step? Then count how many time steps it stays at that level.
As for producing the Fibonacci levels themselves, you can look into categorical time series where the values are categories instead of real numbers.
You have a many-to-one problem. So, yes the model takes two inputs, Duration and Level, but only produces one output, the Level.
The times septs in a model will always be the same. You have to choose a range that is appropriate and all your predictions will be based on that. It's important to note that finding the best time step size is a trial and error process. For example, you could first try using 10 time steps and if you find the model needs more you retrain it with 20 and compare the results.

So, I have basically two input for each time step (duration, level) and the output is the next level only ? 2nd question, does the time step have to be equal ? Thanks for your help

Thanks a lot. Could you please clarify one thing more. Do you confirm that a step is a duration regadless of the duration: i.e input vector (300 minutes, Level 2) output (Level 3) and 1 step is this 300 minutes, but could be 222 or anything else ? or one step is a given duration ? Do you recommend Rocket as time series forecasting for this exercise ?

@user3311337 Yes you are correct about the steps. However, in time series a time step is how far back you are considering. For example, using one week to forecast the next day is a time step of either 7 days or 1 week. Depends on your data. As for Rocket, I never used it so I can't recommend it.


text:
Why disentangling the features of variation in representation?
Consider the following excerpt from abstract of the research paper titled Better Mixing via Deep Representations by Yoshua Bengio et al.

It has been hypothesized, and supported with experimental evidence,
that deeper representations, when well trained, tend to do a better
job at disentangling the underlying factors of variation.

In general, as per my current knowledge, we want to preserve the factors that contribute to variation in the final representation. But, the abstract is contrary to that. Where am I going wrong? Why there is a need to disentangle the factors of variation?


text:
Why readout operation in message passing graph neural nets have to be invariant to node permutations?
I am reading the paper Neural Message Passing for Quantum Chemistry by Justin Gilmer et al. And I have a question regarding this passage

The message functions $M_t$, vertex update functions $U_t$, and
readout function $R$ are all learned differentiable functions.
$R$ operates on the set of node states and must be invariant to
permutations of the node states in order for the MPNN to be
invariant to graph isomorphism.

It is not clear for me, why MPNN have to be invariant to graph isomorphism. Could you please share your thoughts on it?

It is necessary for GCNs / MPNNs to be invariant to node permutations in order to generalize properly. Firstly, node permutation means that you reindex the nodes. The edges also use this new indexing, i.e. the graph remains completely the same. It is just the ordering with which you store the graph in your memory. Would the MPNN we sensitive to that, the same graph may be classified differently simply by assigning different indices to the nodes.  That would not be nice.
The above is related to isomorphic graphs, because if two graphs are isomorphic, they have the same structure. This is good for the same reason: Two graphs with the same structure should not yield different MPNN predictions.
Here it is important to note that once you consider nodes with features, there are two definitions for graph isomorphism:

Structure and Feature preserving: This is basically the same as node permutations and I'm sure this is also what Gilmer et al. mean in their paper.
Structure-preserving with similar node feature distribution: If you have the same struicture with equivalent node features - in some cases you'd your MPNN to be invariant to that, in others you don't. It depends on your use case.

Therefore, the invariance to node permutations is generally desirable, whereas the invariance to graph isomorphism can be, depending on the exact definition of isomorphism you mean and your use case.