text:
How to tell an agent that some actions in the action space are currently not available in gym and the design of action space
I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task.
However, there is a constraint that one task should be served by only one server.How should I design the action space?
So when I design the decision-making action space,I don't know which design is better:
        self.action_space = spaces.MultiDiscrete([2]*(M*N)) #action space: [a_11,a_12,...,a_1N,a_M1,a_M2,...,a_MN]  M*N discrete vector,a_mn in [0,1]
        self.action_space = spaces.MultiDiscrete([M]*N) #action space: [a_1,a_2,...,a_N] N discrete vector, a_n in [0,M-1]

In first way, the variable a_mn stands for if the n-th task  is offloaded to m-th server or not. While in the second way,a_n represents which server should be chosen by the n-th task.
If the first way is better, how can I implement the constraints mentioned before, since there is lots of infeasible actions in the action space?
And I was confused in both situation if it is possible to use the DQN to produce the Q value of all action from this action space?
Thank a lot

maybe https://towardsdatascience.com/action-masking-with-rllib-5e4bec5e7505?

It may be easier to allow infeasible actions to exist in your problem. Instead, set them to transition to a terminal state returning a highly negative reward. This should prevent such actions from being selected at evaluation, however, it depends on whether it would be ok to select these sort of actions in training.

Thanks a lot. Do you mean the first way is better, even there is lots of actions infeasible in the first action space?


text:
Is there a neural network architecture specialized for mapping lower-to-higher-dimensional data?
I am building a neural network that takes in a set of 86 parameters (primarily architecture-related, such as building floor area, kitchen size, number of a certain type of furniture, etc.) and outputs a 4D mulitdimensional array of shape (6, 2532, 39, 5) containing vertices of path data that are meant to output to a CAD file of the building. As of the current moment, I am using the most primitive method to do this: a basic densely-connected neural network with the architecture:
Input(86),
Normalize(),
Dense(64, activation="relu"),
Dense(128, activation="relu"),
Dense(128, activation="relu"),
# required for the reshape
Dense(2962440),
Reshape((6, 2532, 39, 5))

However, this densely-connected neural network produces losses in the trillions, making it hardly viable as a deployable model. This is to be expected - MLPs aren't meant to map lower-dimensional features onto higher-dimensional labels. My "cheating" way of making a Dense layer of high enough dimensions to do a reshape isn't truly providing information that the neural network can use to learn. Yet I cannot think of a neural network architecture specialized for low-to-high dimensional training - most architectures are instead specialized for high-to-low dimensional training, like CNNs, which reduce 2D/3D data down to 1D.
Is there any neural network architecture suitable for the task I have outlined?

UPDATE: I found that significant loss reduction could be accomplished by rescaling my train labels to between [0, 1]. However, the model now performs poorly on any input data other than its own training dataset, as the rescaling means that the neural network is highly sensitive to any variation in input data.

probably can take inspiration from GAN encoders

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why are some Neural Networks more forgiving on Quantization?
I know this might be a bit general question and concerning a rather active research field, much beyond my expertise, but I do believe there're some answers.
The use of NN parameters quantization can span from post-training static/dynamic quantization (PTQ) to quantization-aware training (QAT). Generally the target is cutting down FP-32 weights to UINT-8 whilst retaining overall accuracy; the benefits in performance are often sensible, yielding zero to negligible (depending on application) to few percentage drop in accuracy.
However, I hear these statements are true depending on which NN is being quantized (source). Some models indeed are very forgiving, whereas others are not even using more aggressive strategies.

(extracted from PyTorch source)

There're some networks that are very forgiving: you can do PTQ and
the end result is as accurate as the FP value. [...] In some NN
they're slightly more demanding, they're slightly less forgiving on
the approximation [...]

My question are:

Which categories of DNN are more suited for quantization?
What could be one rationale why this would be? Does the size of the network (i.e. number of parameters) has any role, for instance?
What is introducing such demands (this will depend on actual model indeed) in the forward path? Is it to be found in the activation/actual MVP or what?

My guess would be depending on achievable weights sparsity -- however, this in general (AFAIK) can be tuned forcing training constraints, so it wouldn't really answer the question.

IMHO, depends more on the task than on the architecture, even though I can imagine that architectures that tend to overfit might have a hard time being quantized

Regarding post-training quantization, how well you quantize your model depends on the range in which you model weights are.
if you have the following weights:
$[-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53]$,
you quantize and de-quantize them you get:
$[-0.10, -0.23, 0.08, -0.38, -0.28, -0.28, -2.11, 0.33, -0.53]$.
Two minor errors on the 0.01 precision level.
However if you have the following weights with an outlier:
$[-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53, -67.0]$,
and you again quantize and de-quantize, then you get:
$[-0.00, -0.00, 0.00, -0.53, -0.53, -0.53, -2.11, 0.53, -0.53, -67.0]$.
Currently, one of the famous quantization algorithms for large language models is LLM.int8(). This algorithm exploits a very interesting phenomenon. It turns out that such kind of outliers are very systematic with language models with > 6.7B parameters, so they can be separated from the rest of the parameters. They quantize the regular values (which are around 99.9% of the model weights) and the outliers are not quantized. You can read more here, this is the blog of the lead author of the LLM.int8() paper.
Now, if you want to quantize your own model you have to check if you have such outliers, and combine and quantize the weights accordingly. What is so important about the LLM.int8() paper is that they provide an algorithm for any large language model. They claim that the irregularities that they found are present in every large language model.

So you're saying that although some outliers may spoil quantization, there're indeed solutions; you mention PTQ may fix this, but for sure weight calibration alone will avoid that altogether. So I don't see how this answer my questions.

The rationale is that simpler tasks involve less complex decision boundaries, and sparse networks have more zero-valued weights that can be efficiently represented in lower precision. Larger networks with more parameters may face challenges during quantization, as there is more information to compress into fewer bits.
Ultimately, the quantization suitability depends on a combination of factors, and experimenting with different architectures

For sure shallower nn will have more ease 'compressing', however also large DNNs (VGG et al) and ResNet do quantize down quite nicely. There're quite a number of strategies you might take to do so, yet AFAIK even this for some NN won't apply. So what you say does not scale, and also does not ansswer my questions.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
GPT beam search length (number of tokens)
Background: I'm currently trying to use GPT to give me numerical scores, and looking for tips on prompt design, see my previous StackExchange post.
To craft good prompts it seems important to have a good understanding of how the generative model works...
Question:
How many tokens ahead does GPT 3.5 look with its beam search feature?
Extra context: I found it hard to find good references for beam search, a decent starting point seemed to be huggingface blog post.
I tried asking BingChat about GPT-3.5 beam search length: BingChat replied that it was 10 tokens but could only give a 'reference' to an OpenAI API page which did not seem to support the claim. I couldn't find any other results online.
Why I care? Suppose I have a long theatre review and want to score how impressed the critic was by the quality of acting on a scale of: -5 extremely unimpressed to +5 extremely impressed.
My prompts currently ask the model to finish the reply with a sentence in the form:
"Overall the critic was very impressed by the quality of acting - score 4."
But perhaps by asking the model to continue generation I can make the prompts more reliable. E.g. I could ask the model to subsequently explain the score with a quotation from the text; e.g. along the lines of
"Overall the critic was very impressed by the quality of acting - score 4/5 - and indeed that 'Mark Strong's performance stole the show'"
Knowing beam search length would really help me design prompts like these (which ask for the continuation of text after a numerical score to improve reliability).


text:
CNN-Regression insensitive to input data
I'm currently training a CNN + multiple target regression model that does the following
input: $ \dim x = (L, 2), \text{where} \ x_i \in (-0.1, 0.1) $
output: $\dim y = (M), \text{where} \ y_i \geq 0, y_{i-1} \leq y_i$
with $10< L, M < 100$
The basic architecture is (each step with ReLU activation) :
input $\rightarrow$ CNN1(3x2) $\rightarrow$ maxpool(2x2) $\rightarrow$ CNN2(3x2) $\rightarrow$ maxpool(2x2) $\rightarrow$ flatten $\rightarrow$ hidden1 $\rightarrow$ hidden2 $\rightarrow$ M outputs
The issue is that the model is completely insensitive to the variations in the input data, i.e., regardless of the input value, the model would predict an output that is almost the average of all training outputs.
Some hyperparameters:
batch = 20
epoch = 5
lr = 0.05
decay = 0.02
momentum = 0.9

The training errors are not really lowering after the first few epochs, so I think it could be stuck in a local minima.
Here I'm plotting 10 random predictions on the test set data

Looks to me that there is not enough distinction in the inputs for the model to recognize, or that the input simply looks like noise to the model. I've tried some rudimentary data augmentation techniques, exponentiating the differences, normalization, etc. to no avail.
The training data is generated by diagonalizing matrices parametrized by input values, and the lowest $M$ eigenvalues are the output values.
Since my background is not in the field, I would appreciate any advice and suggestions. Thanks in advance.

for how many epochs are you training it? with which optimizer/learning rate? not clear what you are doing, but are you sure that there is enough information in the input to predict the output?

@AlbertoSinigaglia Please see the parameters above. As the answer below suggested, a larger learning rate sometimes results in weights becoming NaN, presumably some gradient explosion (?)
The inputs generates a Hermitian matrix (they alone do not generate the matrix, but the additional information needed for such generation is identical across all cases), and I'm simulating the process of solving (diagonalizing) that matrix, but that's a good question. I will look into that more closely

What loss function did you use? This could be the key here

It seems like the model is falling into a bad local minimum, where the model is stuck in an easy-to-learn "good-enough" solution. Increasing the initial learning rate could fix this.
See this post for more information on this.


text:
Can a robot that perceives the environment, conducts experiments, and writes OOP classes as its final result be considered AGI?
In my blog post, I provide the blueprint for "Artificial Conceptual Intelligence" (ACI), a type of robot that can:

Perceive objects in the environment
Manipulate the environment to obtain new perceptions
Classify such objects into OOP classes

For instance, given a glass of water, an ice cube and a glass of alcohol, the robot:

Perceives the three objects as separate from each other
Manipulates them with experiments (such as to change their temperature)
Writes OOP classes such that two of the objects belong to the OOP class Water, while the other belongs to the OOP class Alcohol.

Can this type of robot be considered an Artificial General Intelligence?
Note: I am using the definition of AGI used by this website, which states that it is: "a hypothetical machine characteristic permitting it to learn any arbitrary intellectual ability up to the limits of the available computing resources."

Welcome to AI Stack Exchange. The question needs to be more self-contained. Please don't rely on the link. Put a summary of the contents of the link in the body of the question, so that someone would not need to read the link to come to an initial opinion. In addition, put the main point of the question as the title, so that someone would not need to read the question to understand what expertise was required E.g. "Is an embodied robot that can navigate and converse and follow instructions an AGI?"

@NeilSlater Alright. Working on it.

Please, reformulate the question so that it won't just lead to opinions.

@nbro I am not sure that it should lead to opinions. Whether anything is an AGI or not is not a matter of opinion. But if you can explain more why you think it should, I’ll take it into account.

I’m using the AGI definition used by this website which says: “a hypothetical machine characteristic permitting it to learn any arbitrary intellectual ability up to the limits of the available computing resources.”

@RashadSaleh It may lead to opinions because people use different definitions of AGI or not even use definitions but just intuitively think what an AGI may be or not.

@nbro OK I edited the question to explicitly point out which definition is being used. Thanks.

(Edited)
No, unless it self-learns mixing the ice with alcohol, offer it to someone, and invite to dance. Otherwise it is not 'general'.
To be precise, AGI needs to be able to (according to Wiki):
1. reason, use strategy, solve puzzles, and make judgments under uncertainty;
2. represent knowledge, including common sense knowledge;
3. plan;
4. learn;
5. communicate in natural language;
6. if necessary, integrate these skills in completion of any given goal.

Now, take OP's case as example. What is missing (i.e. what can human do which this robot cannot)?
I'd say if this robot is truly AGI if it can:

Look at the table and understand what the 3 liquids are, and what they can be used for.

Learns how to make a cocktail out of them. If not enough ingredients, go buy some. (solve puzzles, and make judgments under uncertainty)

Make the cocktail.

Invite a lady to drink.

Flirt with the lady.

Invite the lady to dance, and dance.

Plan and execute all above-mention steps

These are all tasks we can (well, theoretically at least) do. OP's robot clearly cannot do some tasks:

it may learn how to, but cannot make the cocktail (but let's say that's fine by instructing others to);
cannot flirt (communicate in natural language) to get the lady to dance (integrate skills in completion of given goal).
Most importantly, we don't know if, given the goal 'dance with lady', it can draw this whole plan by itself.

We can build a cluster that holds all the OOPs in this world and make it write more OOPs alone, but that doesn't make it AGI as far as it doesn't know how to plan and achieve goals other than writing OOP.

It does self-learn which experiments to conduct and how to conduct them.

So it can only conduct experiment, nothing else.

As explained in the question, that's 1 of 3 things it does.

No I read it right, all it can do is to classify and write a summary. AGI requires doing any intellectual task that human can perform, which is more than that.

ACI excels at performing conceptual tasks. What intellectual tasks that are not conceptual are you thinking of?

reason, use strategy, solve puzzles, and make judgments under uncertainty;
represent knowledge, including common sense knowledge;
plan;
learn;
communicate in natural language;
integrate these skills in completion of any given goal.

directly from Wiki

All of these are conceptual tasks.

@NeilSlater good point, I expand my argument.


text:
MDP to model finding an optimal sequence of actions with no other state data
I would like to build an RL agent who's objective is to find an N length sequence of actions to a reward where N is not known and the states are indistinguishable from each other (at least for it's sensors).  All actions are available in all states.
For example it's goal may to find the sequence [1, 2] in the action space {1, 2, 3, 4, 5}.  Or even [1, 2, 1, 2] where the chief reward only comes from the state at the end of the fourth transition.  I expect the sequences to be much longer and there to be rewards at some states along many sequences but one chief reward at the end of one or more "correct" sequences.
I'm not sure how to model this as an MDP.  Each action in a sequence feels like it belongs in it's own state.  But given I am not sure how many items are in a sequence I'm not sure what to do other than have an empty table large enough to support most sequences and proceed with POMDP.
I am attempting to avoid anything deep (like Neural networks).

By "the states are indistinguishable from each other" do you mean that there is literally no data available to the agent other than the history of its actions? Is there actually some hidden state that the agent cannot see? Is there a termination signal when the agent "completes" a sequence (or a special action to declare "I have completed the sequence")?

The agent knows it's history and has a reward signal.  At this time I am assuming it is otherwise in the dark.  The agent has no idea what states exist.  A sufficiently high reward marks a completed sequence.  It can be assumed that once this state is reached, the same sequence entered again will arrive back at this state.

Does the agent know (or is told) when it receives a "sufficiently high" reward? If it misses it, can it in theory continue forever, generating an infinite low-scoring sequence?

Is there any similarity between "good" sequences, that could be learned? Or are the good sequences effectively randomly scattered in sequence space as far as the agent is concerned?

We can assume to know what sufficiently high means in this case.  There is likely similarity between good sequences.  Even, if the sequence was [1, 5, 3] x 5, at each 3 in that sequence it might receive a small reward. [4, 1, 5, 3, 2] may give a small reward on both 3 and 2.  Finally [2, 2, 3] may give a reward where [2, 2, 3] x 5 does not give a sufficiently high reward.

What happens when the agent does not receive a "sufficiently high" reward? Can any assumption be made about maximum sequence length, and the agent give up?

The theoretical existence of a sufficiently high reward following a sequence of N would necessitate that there is a finite number M where N <= M.  If the agent interacts with an environment M times, I could accept a simplifying assumption that it goes back to the beginning with no reward.

By "I am attempting to avoid anything deep." you mean you prefer to avoid neural networks?

That is correct.  I recognize that a NN can estimate a complex state/transition/Q value relationship.  I'm looking for other solutions.

I'm not sure how to model this as an MDP.

You have defined a fixed set of actions, and I think from your statements that the environment is deterministic (same sequence of actions from start state returns the same reward). In addition, there is no observational state data available to the agent.
The reward depends on the sequence of actions somehow, and reliably. That makes the full history of actions the only thing that can be used to describe the state, and it does so perfectly. Aside: The rest of the advice here applies even if the reward function or unobservable state is not deterministic - although your problem may become impossible to solve in that case depending on degree of randomness.
Your state is therefore the vector of actions so far, and your state space size is $(|\mathcal{A}|+1)^M$ where $M$ is the maximum sequence length you want to consider. The $+1$ is because shorter, truncated sequences are full states. With this state, you have a full MDP model: An environment with states, actions, rewards and transition rules for states (append action to state vector).
Without anything to base a belief model on for the agent to guess at some internal state, I don't think there is any opportunity to use a formal POMDP model.
You could approximate value functions, possibly usefully, by using a deep recurrent neural network, fed actions one at a time as a sequence, and that might learn to generalise predicting action values from sampling trials. How successful this is will depend on how the rewards are structured in practice. You have stated that you would prefer not to do this, but it might be something that makes the scale practical.
Other approximation schemes are possible - anything that reduces down the action history to a smaller or simpler space could work. For a discrete approximation you could use a table for Q values, for a continuous one you could use a linear regression to learn weights that predict Q values. For example, a continuous summary that is a vector the size of the action space, where taking an action multiplies the whole vector by a decay factor, then adds 1 to the element matching the action would work in a linear model. However, although this definitely summarises the sequence into a more manageable space, you have nothing from the environment to base this on, it would just be a guess at a useful summary. From your examples I don't think such a summary would have a linear relationship to action values, so you would again need a neural network for deep RL in order to make good use of it.
It is far from clear whether RL is a useful system to discover optimal sequences in your environment. Other than RL, simple combinatorial searches could be effective. Which is better will depend on how reward patterns are distributed. Most RL will tend to search between randomly or focused close to best rewards seen so far. It is not clear from your description whether that would be the most effective search strategy. Exhaustive breadth- or depth- based searches could easily be more effective.


text:
Unable to understand Figure 3.13 Artificial Intelligence: a Modern Approach
I'm currently studying the functionality of BFS and the AIMA book shows
.
I am unable to replicate them using some trivial calculations.
For example, if the BFS generates $10^6$ nodes per second and I have $10^6$ nodes, it would be obvious that it would take 1 second, not 1,1.
Please enlighten me, because I don't know if I have to take a different approach.

The table's "Nodes" column uses rough approximations, as do most of the values in the table, because it is trying to give an intuition about scaling, and not accurate predictive values for any real-world code.
The true count of nodes value used for the calculations before approximation is based on a branching tree model with branching factor 10, and a variable search depth.
So the true number of nodes in each case for depth $d$ and branching factor $b$ is $\sum_{i=1}^d b^i$ - summing the number of nodes at each depth of the tree. For $b=10, d=6$ that is $1111110$ nodes (decimal).
If the table was updated to have another digit, then the author would need to write e.g. $1.1 \times 10^6$ in the nodes column for depth of $6$. I guess they felt they had made their point about the scaling that was going on from the pattern in the rows for depths of $2$ and $4$, and wanted to be concise.


text:
Can Q-learning rewards and next states be non-deterministic?
I am working in a team to develop a Q-learning based approach for hyperparameter tuning. I have a disagreement with one of my teammates on how they defined this problem. They defined it as follows:

The states are the values of the metric we are optimizing (ex. F1 score: 0, 0.1, 0.2, ..., 1.0).
The actions are to change one of the hyperparameters. So we have as many actions as we have hyperparameters.
The reward is the difference of the metric between the current state and the previous state. We evaluate the model we are optimizing on the current hyperparameter configuration.
Each of the mentioned actions have a sequence of values that we use whenever that action is picked. Ex. The batch size can have values [16,32,64]. When we choose the batch size action, we set it to 16. Next time we choose it, we set it to 32. Next time 64, then we come back to 16.

I have various issues with this formulation. Mainly:

The rewards and Q-values are non-deterministic. Choosing action a in state s, can be either a good or a bad decision, depending on the previous actions we chose, and whatever value this current action is going to return next. From my understanding, this means that the Q-values are useless now. Choosing an action might be good or bad, non-deterministic.
With this reward scheme, the first action will get a high reward no matter what, because there's no previous metric to subtract from it.
I still can't help but feel that this state space is problematic. Choosing action a in state s will also yield an arbitrary next state depending on the evaluated hyperparameter configuration.

Now I have tried getting my point of view across, but no matter how much I argue this person thinks i am wrong. Their use of ChatGPT unfortunately fuels this arrogance as it says yes to every one of their questions, and they treat it as their bible.
Is my argument reasonable? Is there really nothing wrong with this approach? I don't know where else to ask this question

We cannot really help you with your colleague's insistence on ChatGPT being correct. They are appealing to "authority" - incorrectly - but will view that you are, too. I recommend you remove that part of your question, as it is a separate issue. There are plenty of resources, including on this site, that warn against using ChatGPT in the way your colleague is - in brief you need to be a better expert than the question you are asking in order to filter out the "hallucinations". Never use ChatGPT to learn or plan a complex subject, the chances of failure are very high (more likely than success).

More practically, on the RL side of your question, it is not clear whether your project is to tune one model, or to create a generic "smart hyperparameter tuner" that could tune multiple models based on observing current F1 score and tweaking hyperparameters? Please update with more details of the goal of the metalearning process (i.e. what are you trying to achieve in this application of RL)?

There's a lot being asked here and I don't know that I'm tracking well enough to comment on this formulation.  But I will try to clarify some RL theory and answer the title question.
If the transition from one state to the next state is completely non-deterministic, it is not an MDP and a Q-table will likely just give some metric about the underlying random process but not be a reliable predictor.
However, there is likely some non-determinism in most realistic processes (sensor inaccuracy, under/overshoot action, action of unmeasured factors, etc) and Q-learning can still be used.  Distributional RL even provides an insight into the non-determinism for greater policy accuracy.
Lastly commenting on your issue 1, just because the current state depends on previous states does not make it non-Markovian or non-deterministic as long as for a certain history and action there is a consistent next transition.  In these cases it becomes an n-step MDP.  Typical treatments involve a translation to belief states and proceed with Q-learning from there.

define completely non-deterministic

RL definitely allows non determinism from the transition of the MDP and the reward, indeed they are modeled as random variables

I’d say that practically completely non-deterministic begins where the tools or evaluation methods used to measure it are no longer sufficient.  For a Q-table this can be a function of the granularity of the states for example.

"where the tools or evaluation methods used to measure it are no longer sufficient" can't get what you are saying... no matter the Rand Var you are using as R/S', maximizing the expectation works just fine

To go to an extreme, if one wanted to solve woods101 but the representation/q-table had only a single state, no amount of expectation maximization would solve it.  It would be practically non-deterministic.  This is an extreme example but is true for any problem where granularity or dimensionality of the representation is not sufficient.


text:
Can all RL algorithms learn with discrete state spaces?
This question come to mind when i was planing to do a benchmark of RL algorithms to my Environment.
In fact, Q-Learning, SARSA actually only handles with discrete state spaces because they are tabular methods, but Deep RL algorithms like PPO, DDPG and other algorithms that can handle continuous state spaces, can they actually handle discrete state spaces by discretizing the continuous state space into bins and then input them to the Neural Networks ?

Two things in this post need to be corrected.  Q-learning is not married to an estimator.  For instance, DQN stands for Deep Q network, a type of Q-Learning that utilizes neural networks.  On the other hand, tabular utilizes a table for estimation.  Second, tabular does not necessarily mean only discrete state spaces.  An algorithm can discretize the space and put it into a table.
A neural network doesn't necessarily have an issue taking in a discrete input.  When the input is discrete, a common form of input pre-processing is min-max scaling, which often improves network performance.  This fact is especially true when some input features are on different scales.

For your anwser, can i conclude that effectively all RL algorithms can handle discrete state spaces ?

Yes.  Discrete shouldn’t be a problem for any algorithm.

Absolutely, if my post answers your question, please remember to up vote and accept.

Sorry i can't upvote your anwser. "Thanks for the feedback! You need at least 15 reputation to cast a vote, but your feedback has been recorded."


text:
Can DQN learn with discrete state spaces?
For example in Cart Pole v1 gym environment the state space is continuous, but we discretize it to apply the Q-Learning algorithm because Q-Learning is a tabular method and only works with discrete state and action spaces.
In some examples of DQN use for solving Cart Pole gym environment, they use directly each state variable as input for the neural network (NN), therefore my doubt is: Can this input state variables for the NN be discretized in bins allowing to solve the cartpole environment ?
Thanks.

You could discretise the continuous output into bins and then use a tabular lookup table but you would either a) need to use very coarse discretisations, at which point you'd probably lose too much information to be able to solve the problem well b) require a very fine discretisation that would lead to a combinatorial blow up in number of states required to store in the lookup table.

So what you're telling me is that is possible to discretize the state space and send that "bins" to the NN, though there're some problems of lossing information to effectively learn the goal ?

oh, sorry, I didn't realise that you wanted to pass the discretised input into the NN. Why would you want to do this? I thought you wanted to learn in a tabular way.

To use DQN algorithm. In a case that have to much discretized states and actions, i think that DQN should perform better than a Q-Learning algorithm.

Binning of inputs are of course possible but it is a balancing act that may never outperform a network operating on continuous values.
Coarse binning may lose information necessary to accurately solve the problem.  Imagine backup sensors on a car.  If your goal is to get as close to the wall behind you without touching it, you can do it with a beeping sensor (distance binning sensor) but you won’t be as accurate as with a camera (continuous).
On the other end of the spectrum with many bins one tempts the curse of dimensionality.  This says for an increase in input dimensionality an exponential number of examples may be required to correctly learn.

Ok i understood, thanks. Relatively to this "On the other end of the spectrum with many bins one tempts the curse of dimensionality. This says for an increase in input dimensionality an exponential number of examples may be required to correctly learn.". Discretizing the state variables is not increasing the number of state variables is reducing the information of each variable.

I wrote this very early in the morning for me.  When I read descritizing my brain went to one hot encoding of the continuous input into bins which does tempt the curse.  On the other hand, turning the continuous input into a step wise function (ie, round(x,0)) does not tempt the curse, you are correct.  This nestles up to NN quantization and it's trade-offs.


text:
In Sutton-Barto a confusing point regarding $\epsilon$-soft policies in the proof for optimality of MC control without exploring starts
Sutton-Barto, page 102:
In the second paragraph, we have:

Consider a new environment that is just like the original environment,
except with the requirement that policies be $\epsilon$-soft “moved
inside” the environment."

Later in this paragraph, it is written as:

The best one can do in this new environment with general policies is
the same as the best one could do in the original environment with
$\epsilon$-soft policies.

Is there an inconsistency between the two statements because if we are allowed general policies in the new environment why do we need the requirement of $\epsilon$-soft policies “moved inside” the new environment?

He's trying to turn the requirement "this policy is $\epsilon$-soft" into a feature of the environment. That way every policy in the new environment could be turned into an $\epsilon$-soft in the old.

But if  we are allowed general policies in the new environment  why do we need this requirement because general policies include $epsilon$-soft policies?

The point of $\epsilon$-soft policies is to encourage exploration (important for finding good policies; I think it can be shown that training an RL agent w/ $\epsilon$-soft policies will be exponentially more likely to converge to the highest-reward policy). The point of making a new environment is so he doesn't need to put this requirement on the policy anymore.


text:
Is this video’s description of evolutionary learning correct?
In this video : How AIs, like ChatGPT, Learn, narrator describes a form of machine learning, specifically evolutionary algorithms or evolutionary machine learning.
Stating that, Initially, the builder bot creates student bots almost randomly, and the teacher bot tests them using images of bees and threes. The student bots that survive the tests at first are just lucky.
However, by combining these lucky bots, keeping only the best-performing ones, and making random variations, the process gradually evolves more capable bots. Over time, this leads to a student bot that isn't merely lucky but genuinely begins to distinguish bees from threes.
This iterative process of testing, selecting, and modifying continues until a highly effective bot emerges, even though neither the builder, the teacher, nor humans understand how it works.
How correct this description of evolutionary machine learning actually is?

It is a simplified but ultimately correct description of evolutionary computation.
Technically a large population is not needed.  (1+1)-ES makes a modification and decides if it is better or worse, replacing the one learner if it is.
But broadly the strategies can be seen as gradient-free hill climb optimization and they can be quite effective learners.

Yes, that is how evolutionary algorithms work in general. They are very suitable for hyperparameter optimization, but I don't think they are much used for training the actual models. When the models start having millions or billions of parameters, they converge much faster by using gradient descend algorithms.
Also, the illustrated bot brains resemble more decision trees than neural networks.

What you say about sample efficiency is correct but it’s important to note that GAs work on non-differential problems where gradient descent cannot be used.  Further, decision trees are only one model which can be evolved.  NEAT evolves neural networks and others evolve rule lists, support vectors or almost every other model.

Getting through this paper (1)  I feel that the definition you ask about describes well only a subfield of evolutionary machine learning (EML) and EML is a broader field than that.
The authors of the paper I mentioned frame EML as a bit fuzzy term that is a bag for other subfields. I think that your definition suits Evolutionary Reinforcement Learning.

(1) Telikani, A., Tahmassebi, A., Banzhaf, W., & Gandomi, A. H. (2021). Evolutionary machine learning: A survey. ACM Computing Surveys (CSUR), 54(8), 1-35.

Nothing in the original post describes RL.  It’s a simple classification problem.

@foreverska you might be right, I've been anchored with the 'bot' term. So, coming back to the paper, this case would be classified into a category of Evolutionary Neural Networks and Deep Learning.


text:
Multiple text inputs to neural network
What is the best setup for a neural network if I want to classify records containing multiple text fields?  For example, a record has a title and an abstract:
title:    Neural Networks for Pattern Recognition
abstract: This book provides the first comprehensive treatment of feed-forward neural networks [...]

I've seen other people concatenating all text fields (with or without boundary symbols) before the feed it to the classifier.  So in the example, the input to the classifier might become just the single text
Neural Networks for Pattern Recognition <B> This book provides the first comprehensive treatment of feed-forward neural networks [...]

Are there better ways to handle this situation of having multiple text inputs?

In the end I embedded each text individually with a shared embedding layer and passed these embeddings on to the latter stages of the neural network.  This is similar to existing work, e.g. openalex, and performed slightly better than concatenating the inputs.


text:
Calculating i_delta in upper confidence bound algorithm
I wanted to apply upper confidence bound algorithm.
There is a 2 dimensional dataset in it Each user in the dataset reviewed (((all))) movies with 0 or 1 not some movies only
And i want to get the best movie
My teacher said to calculate i_delta we should divide by num_selections[movie](the number of positive reviews) is it wrong? i know that we should divide by the number of times that the users reviewed the film not the number of selection in the code because all users reviewed the all films with 0 or 1 not some users reviewed with 1 and ignore the others so it is 0
The full code
num_users = len(dataset['User'])

for user in range(num_users):

    selected_movie = 0

    max_upper_bound = 0

    for movie in range(num_movies):

        if num_selections[movie] > 0:

            average_reward = sum_rewards[movie] / num_selections[movie]

            delta_i = np.sqrt(1.5 * np.log(user + 1) / num_selections[movie])

            upper_bound = average_reward + delta_i

        else:

            # Initialize with a high upper bound for unexplored movies

            upper_bound = 1e400

        if upper_bound > max_upper_bound:

            max_upper_bound = upper_bound

            selected_movie = movie

    # Simulate user selection and reward (replace 'your_reward_column' with the actual column name in your dataset)

    reward = dataset.iloc[user, selected_movie]

    num_selections[selected_movie] += 1

    sum_rewards[selected_movie] += reward

So should I write this line
delta_i = np.sqrt(2 * np.log(user + 1) / num_selections[movie])

Or this
delta_i = np.sqrt(2 * np.log(user + 1) / user

If you don't understand read this
if the case is that there are 5 ads and we want to know the best to publish and each user enter to the website he saw one ads only and if he click on it the value will be 1 and if not the value is 0
Be careful that each user saw (((one))) ad only here
So which line i should write
The shape of the dataset
ad1 ad2 ad3 ad4 ad5
User1  0   1   0   0   0
User2 ....................


text:
How to Represent Boardless Board Game as Input to RL Model?
I am currently doing my thesis project by creating an Imitation Learning (IL) agent that learns to play the board game Hive, which lacks a traditional 2D board. Pieces are placed relative to one another along their hexagonal edges, and my state space representation for the game currently is a set of Cartesian Co-ordinates for a piece in play, along with an enum representing the piece (i.e., 'white Beetle 1' etc.) - where the first piece in play starts at (0, 0). It is also worth noting that the game isn't merely 2D either - pieces may, in some cases, stack on top of one another.
How do I create an input, and output state for this model in PyTorch given there is no fixed board state in this game? I wish to input game states as an array, where each move is a new input - in essence like a timeseries.
In the paper 'AZ-Hive case study' (found here), the game is represented by a 24x24 tile grid, with pieces represented as integers, with integers being added to one another to represent a stack of pieces. Is there perhaps a quick fix here by transposing my Cartesian co-ordinates onto a Numpy array of 28x28 zeroes and the enumerations for each piece as their value?
Having asked chatGPT, it suggested an input dict of fixed length such as the following snippet as an example:
    {
      "player_turn": "black",
      "pieces": [
        {
          "type": "ant",
          "position": (0, 0),
          "connections": [(0, 1), (1, 0)]
        },
        {
          "type": "grasshopper",
          "position": (0, 1),
          "connections": [(0, 0), (0, 2), (1, 1)]
        },
        {
          "type": "beetle",
          "position": (0, 2),
          "connections": [(0, 1), (1, 2)]
        }
      ]
    }

I am, however, unsure of this input as it relies on far too many dimensions for me to get my head around as an input.
I wish to create a PyTorch model that can capture some of the intricacies and strategies of the game, whilst not having to rewrite my current game representation entirely, as my move validator etc. is extensive. Any feedback in input and output of this model is greatly appreciated!

well, in the first place, you can even merely output a graphical representation of the game, just like a picture taken of the game on the table, and thus using any pretrained VC model...
However, this would make your game training definitely slower, as the input size is bigger, so it requires usually more FLOPS (however, this is not true as the "hardness of learning the game")
In the second place, usually you have to exploit some structure of the game if you don't want to fall in the previous case.
I've never seen that game, so I can't really help on that, however, I would suggest you to give a look at Graph Neural Networks, because as soon as you solve the problem for the state representation, you will probably also have a problem on the action representation, which I see it being easily solved with the output structure of a GNN, so, personal opinion, I would take this as a great opportunity to learn them (they indeed manages graphs, that is what ChatGPT is suggesting you to do)

Thanks, the GNNs are a great pointer! I'm a bit late on in the project to add learning a new tech to my implementation I think but for future versions of my agent I will definitely  represent the state space as a graph given the game itself is best represented as one.

I'm familiar with Hive, and the 24x24 hexagonal board space should work fine. Most games I've played have used less than 12x12. The pieces can interact with each other diagonally. To conceptualize, using a cube grid with offset rows might be more accessible.
You'd want the first piece placed to occupy the center of the board and then expand outward.


text:
A Quest for Maximizing Expected Value: Optimizing Algorithmic Trading Parameters
I am engaged in algorithmic trading, employing specialized models that utilize various parameters to signal trading opportunities. Once I get a signal, I could execute a trade and everything is determine in advance, i.e. I know the entry point, the exit point and I have a stop loss if the price goes against me.
While the existing models are functional and able to generate these signals, I'm facing a challenge in optimizing the parameters to maximize the expected value of my trading strategy. I'm aware that this optimization is likely an iterative process, but I currently lack a systematic method to tweak the parameters effectively.
One example of this challenge is the identification of a "gap-up short pattern." A threshold such as a 70% gap in pre-market trading affects the expected value, and I suspect that there may be an optimal percentage that can enhance my results further. However, I'm uncertain how to explore different thresholds methodically to find this optimal value.
I'm keen on exploring optimization techniques that are tailored to my situation, taking into consideration my data constraints and the nature of the trading models. My goal is to find a robust yet practical approach to iteratively fine-tune these critical parameters, thereby maximizing the expected value without risking overfitting or adding undue complexity.
I have not yet embarked on any specific optimization method, and I'm actively seeking guidance on the best course of action to address this specific challenge.
EDIT I have thought using a reinforcement learning approach, but I am not sure at all it would be possible. I have about 2000 to 3000 signals per models so the data seems a bit limited. I thought using a Bayesian Optimization approach. What do you think about that?

Can you please formulate the title as a specific question?


text:
Does this learning scenario have a name? If so, can someone point me to relevant literature?
I am faced with a problem which I bet was already solved before, but that I had never seen. Perhaps by discussing it abstractly, someone can point me to relevant literature.
It goes like this: I have a dataset of images $I_j$ and numerical features $f_{1,j}, f_{2,j}, ..., f_{k, j}$. In production, I don't have access to the features $f_{i,j}$, which I know to be more relevant to my learning problem than the images.
However, theses features $f_{i,j}$ are also very hard to obtain, and there is a known correlation (informal) between the images and the numerical features $f_{i,j}$. So the whole point of using the images is to avoid using the $f's$.
Know comes the question: is there any useful way I could learn a representation of the images that "exposes" the maximum information about the features $f_{i,j}$? Perhaps some form of autoencoder with a regularization term that penalizes uncorrelated codings?
Any ideas are welcome. Thanks!
Edit:
In the problem that I have, the images $I_j$ are $(64,64)$ images of tumor nuclei, and the $f_{i,j}$ are genetical information about each cell, such as the expressivity of certain biomarks (HER2, ER, etc.).

can you please provide a practical example of what $f_{i,j}$ is?

@AlbertoSinigaglia Done.

It looks like the features you are wanting to learn are explanatory for some other classification that you don't say? And that you want to train a model that predicts the classification from an image as the primary task, whilst also having these features expressed within the model? Could you please clarify

@NeilSlater A first example could be to classify a cell as HER2+ or HER2- based only on the image data. (During training the autoencoder would have access to reliable HER2 score for each nuclei)

I'm not really an expert on this field so i still don't get what are the $f$, but I think that you can formulize this as a contrastive problem, using biomarks to attract of pull away latent vectors


text:
RL framework to optimize my custom multi-agent simulator
I have built a custom discrete event simulator with multiple agents and want to optimize the system using RL frameworks that support multi-agent configurations. I will use custom policies. Which framework should I use? I've looked into SB3, CleanRL, Tianshou, SKRL, RLlib, Acme, and MARLlib, and here's what I found:

SB3 and CleanRL don't offer direct support for multi-agent systems.

RLlib is very functional but has a steep learning curve and hard to customize and some say the documentation is poor.

Tianshou seems good, but its community is small.

Acme doesn't use the PyTorch backend, which I prefer.

I haven't delved deeply into SKRL or MARLlib, but they appear promising.

I prioritize ease of use and documentation. What framework do you suggest? And why?
I’d appreciate any helpful starting advice/resource to approach my problem as well.

Disclaimer: I have only experience with the first three frameworks.
Among those three, I would suggest RLlib for the multi-agent situation:

Distributed training based on Ray library
Out-of-box multiagent solution
Great documentation. You may want to check an example how to train  multiple agents: https://applied-rl-course.netlify.app/en/module2


text:
What is x, y, p(x), p(y) in generative model domain?
Background
Generative modeling
Generative modeling aims to model the probability of observing an observation x.
$$
p(x) = \frac{p(y\cap x)}{p(y|x)}
$$
Representation Learning
Instead of trying to model the high-dimensional sample space directly, we describe each observation in the training set using some lower-dimensional latent space and then learn a mapping function that can take a point in the latent space and map it to a point in the original domain

Context:
Let’s suppose we have a training set consisting of grayscale images of biscuit tins

we can convert each image of a tin to a point in a latent space of just two dimensions. We would first need to establish two latent space dimensions that best describe this dataset(height, width), then learn the mapping function f that can take a point in this space and map it to a grayscale biscuit tin image.

Question:
What is:
P(y|x): is it possibility of point lying in Latent space given it is a point in (high dimensional) pixel space?
Or for that matter, what do x, y, P(x), P(y), P(x|y) really mean/represent here?
P.S.: this is referred from the book Generative Deep Learning 2nd edition By David Foster, O'Reilly Publication.

"What is: x, y, P(x), P(y), P(x|y), P(y|x) in this example?" I don't really see where you are referring to them in the question, so I'll answer with the general interpretation in ML:

$P(x)$: also known as data distribution, it's the density that describes some data... take your example, if your bins are described by height and width, you can assume that maybe the height is described by $H\sim U(1,10)$ (uniform) and the width by $W \sim U(1,5)$, and supposing that they are independent, then the joint distribution of $H$ and $W$ is $P(x)$ (now, if you are dealing with images, you have to consider the joint distribution of all pixels, so it's a very high dimensional distribution)
$P(y)$ also known as target distribution. This can also be considered as "prior", since in discriminative task, you try to model $P(y|x)$... it describes your belief of the target distribution given no additional information... take for example a task where you try to regress the weight of a person, then you know that generally speaking, weight is distributed as a Gaussian (look up statistics online about this and you will see that Bell curve) centered at some weight
$P(y|x)$, your conditional probability that you want to learn with discriminative tasks. It means _"Given these informations $x$, how likely it $y$ to be some value"... take the weight regression task, say generally that the weight is distributed as $N(50,10)$, so given no additional information, the best guess is to say that a person will weight 50kg... however, if I additionally tell you that that persone is 190cm tall, then you know that that person will weight more than the average most likely, so your belief on his weight might shift to $N(70,5)$... this "shift" is the information brought by $x$, thus $p(y|x)$ is just a classification/regression etc etc
$P(x|y)$ is the other way around.. if we consider images, $Y$ might be for example a the digit you want to see in the image, and $x$ the image you want to generate, then $p(y|x)$ is just "which images are more likely to be correct given that I want to see the digit $y$ printed on it"... you can consider that $p(x)$ is already a distribution and $p(x|y)$ just takes the interesting part of it where there is $y$ printed on the image

so, in your example, P(x) is just the distribution of the images, P(y) is the distribution of some classification with those bins (say if a cookie will fit in it, it's your general belief wether a cookie fits in a random bin or not), P(y|x) is the "given this image of a bin will a cookie fit in that", and P(x|y) is "generate a picture of a possible bin such that this y cookie fits in"

Edited/rectified the question to better reflect the intent. Thanks for taking time to explain. Could you please update the answer accordingly. I wish to understand what the probability terms represent in the tin bucket example.

@Prakhar I still don't quite get what you are referring to with those letters, however I have a sense that you are probably referring to Variational Autoencoder, where y would be z/h depending on the book you are reading?


text:
How word2vec de-embeds the special names in language models which output text
I am new to nlp field. I have some questions about word2vec embeddings. as I know they have a fixed size dictionary of vocabs. so definitely there some words which is not in that predefined dictionary of vocab, for i.e. we may get an special name (like people's name) which doesn't exist in the vocab and they are called out-of-vocabulary (OOV) words.
So how embedding and de-embedding for a translation language model work? I want the language model of the example to include the de-embedding part; as we know in classification language models we may only concern about the embedding part as the output is not an embedding vector, and does not need to be converted to a string(word)
I don't know what ways are used to embed those OOV words? I guess one way is to use some main word2vec and another complementary fastText for OOV words, or instead some complementary generic way of embedding like nn.embedding in pytorch. This won't make any problem in the classification models like sentiment analysis or Text classification.
Question 1: what are other methods that can be used for embedding an OOV word while we want to use word2vec?
but in the models which Their output is eventually gonna be a word (like translation or text summarization), I have some concerns as below:
Note we know the output of a translation language model is a sequence of embedding vectors to correspondent output word.
Note the way that we convert those embedding vectors to words, usually is by finding the closest embedding Vector in the embedding Matrix, either word2vec or fastText or de-embedder of nn.embedding. Also note this way eventually result, some Vector and from there, by its index, we would get a word.
as we have two embedding models, so when we are the step which have received the language model output embedding vector, we need first distinguish, this output embedding vector Should be de-embedded from which embedding model (word2vec or fastText)? note as I said eventually the next action (finding closest embedding) would result some word.
so I guess one way maybe is find the closest vectors for 2 embedding models, also we their similarity score to output embedding vector of the language model, and know to de-embed from which one.
question 2: so how this problem to distinguish to which embedding model to de-embed from is handled? I guess there should be more sophisticated methods in Practice so if there are please tell me?
question 3: do u know a model which have implemented my solutions (one for embedding by complementary embedding model, 2nd one distinguishing by similarity score)?

as I get its not probably the case and the tokenizers usually have the alphabet in the so if the word or its subwords are not in their vocab, they break it to alphabet which exists in the vocab

I'm not sure I got 100% of the question, but word2vec is rather simple to understand so I'll give my two cents and if something is missing or not clear I'll edit the answer to improve it.
1 For OOV vectors, as you pointed out, one option is indeed to initialise some random embeddings. The nice aspect of this method is that these missing embeddings can be trained as extra parameters for the model while keeping the already trained vectors fixed. Another alternative is to get all zero vectors, meaning that we force the model to ignore certain words (it might be that these OOV are not of interest for specific use cases). Using vectors coming from other models is unlikely to work because of a problem called "vector alignment", i.e. embedding spaces from different model do not share the same learned geometric structure, meaning that two numerically equal vectors could be present in both spaces but be associated with totally different words.
2 Here I think there is a little misunderstanding going on regarding how word2vec works. First of all, word2vec is a training method for embedding, it does not necessarily refer only to pertained vector of fixed size. Specifically, the word2vec papers proposed two ways of training embeddings, continuous bag of words and skip-gram. The picture below show the difference, which I won't explain since there are already lot of answers about it in the community. The point I want to focus is that both approaches use simple classic supervised machine learning tasks. You have an input and you want to predict an output. In the original paper the desired output was a continuous vectors representation of words to leverage as features for further NLP tasks. This is different from encoding/decoding and the similarity is actually closer to transfer learning. It is also relevant to point out that you can get very good embedding also with classic matrix factorisation techniques, the problem is that these techniques can handle small amount of data, the deep learning formulation  proposed in word2vec allow to train embeddings of ay dimension on any amount of data, hence the success of this approach.
3 As far as I'm aware there is no such model, and I doubt someone tried such approach for at least two reasons: 1 the same alignment problem already cited above, 2 the quick arise of transformers that happened not long after word2vec (attention is all you need was published in 2015, word2vec in 2013) which canalised all the attention (pun intended) in the NLP community to those new architectures.

thanks for the answer, but I have understood that OOV is not the case anymore because they dont let this to happpen. with having alphabet letters, even for other languages in the fixed vocab, they prevent OOV words to exist, because it would finally be breaked to some combination of subwords and letters.


text:
Why use a "square root" in the scaled dot product
In attention settings, typically when the both Query Q and Key K are of the same dimension d we can compute their attention score in the following manner:
$$\frac{Q^T K}{\sqrt{d}}$$
The justification being that the dot product tends to blow up in higher dimension, and $\sqrt{d}$ helps to "keep things reasonable". I also read that it serves to reduce the standard deviation to 1, however chatGPT contradicts that statement, and I can't find resources that support this claim.
Therefore what is the mathematical justification behind $\sqrt{d}$ and not another form such as $\log d$, $d^5$, $e^{\pi d}$ or simply $|Q||V|$ (a cosine similarity!) if we want to keep things "bounded"?

In the official paper the justification for this scaling is given in footnote $[4]$ on page 4.
Here is a more lengthy explanation of the same thing:
Let’s assume that the keys $K$ and queries $Q$ have zero mean and unit std. The attention score between some query $i$ and some key $j$ is:
$$ \alpha = q_i k_j^T = \sum_{n=1}^{d_k} q_{in}k_{jn} $$
We know that the variance of a sum of random variables equals the sum of the variances of those variables. Thus, for the variance of the attention score we get:
$$ \text{Var}(\alpha) = d_k, $$
since $ \text{Var}(q_{in}k_{jn}) = 1 \quad \forall n.$
The standard deviation is:
$$ \text{std}(\alpha) = \sqrt{d_k} $$
After you compute the attention scores you need to apply softmax in order to obtain the attention probabilities. The problem is that the softmax function saturates very quickly. This means that if one of the attention scores is slightly bigger than the others, then the softmax function will put almost all of the weight on that element. See for example this:
x = torch.FloatTensor([1, 1, 1, 5])
y = torch.softmax(x, dim=-1)
# y: [0.0174, 0.0174, 0.0174, 0.9479]

Since the variance of the attention scores is $d_k$ then almost certainly one random score will be much higher than the others. The softmax will output attention probability of $1.$ for that element and zeros for the others and you will get no gradient propagation.
What you want to do to ensure stable gradient flow is normalize your attention scores. That is, make them with zero mean and unit std. We already assumed that the keys and queries have zero mean and unit std, so the only thing you need to do in order to achieve this is to scale the attention scores with $\sqrt{d_k}$.
But is it safe to assume that our keys and queries have unit variance?
Well, yes! The architecture of the Transformer includes LayerNorm layers that are used exactly for this reason: normalizing the inputs before applying the self-attention layer. In addition the weights of the self-attention layer are initialized using Xavier initialization and so the computed keys, queries and values will also have unit std. In my opinion this scaled dot-product works so well only when combined with LayerNorm and Xavier initialization. I have written a very detailed blog post about the Transformer architecture, if you want to read it here is a link:
https://pi-tau.github.io/posts/transformer

"you will get no gradient propagation." I think this is not entirely correct. You will get no backprop only for the element with $p \to 0$.

@adosar If the std is too large, then the softmax will output p=1 for one element and p=0 for all the rest.

But where these 1's and 0's appear during backprop? If we can't answer this mathematically, we can't state there will be no gradient during backprop.

During the forward pass (not the backward) we will get a score of 1 for one random entry and all the others will be 0. This happens because the standard deviation of the computed attention scores is quite large ($sqrt(d_k)$). This is the mathematical explanation. The simple explanation is that the attention scores will vary by a lot. Now, since the softmax func involves an exponentiation operation, then calculating the attention prob for the largest number will be close to 1. and the other probs will be close to 0. This is just fact you can see the example with [1, 1, 1, 5]

Now, since the forward pass for the self-attention layer produces 0s everywhere and 1 at a random position, then when you compute the derivative you will get a gradient of zero. Thus, no gradient flow.

Thanks for the elaboration. With regards to the gradient flow: the derivative will be zero since the derivative of softmax is 0 when the output is 1 or 0 (i.e. we are in the saturating regime)?

Yes, exactly. You can see this from the graph of the softmax function. It is almost horizontal for inputs that produce output of 0 or 1. This means that the slope of the graph (i.e. the derivative) is 0.

The answer by @pi-tau is upto mark. Here is a TL-DR version of it.
Assuming the keys and queries have mean of $0$ and standard deviation of $1$, the variance of the attention score (the sum of the dot products) is $d$, and the standard deviation is $\sqrt{d}$. To ensure stable gradient flow, you want to normalize your attention scores, i.e., make them have mean of $0$ and standard deviation of $1$. Since we already assumed that the keys and queries have $0$ mean and unit standard deviation, the only thing you need to do to achieve this is to scale the attention scores with $\sqrt{d}$.
This scaling helps to maintain stable gradients and prevents the softmax function from saturating and becoming unresponsive to small changes in its input. Hence, it's a crucial part of the self-attention mechanism in Transformer models.

From the attention is all you need paper (section 3.2.1):

Dot-product attention is identical to our algorithm, except for the scaling factor of $\frac{1}{\sqrt{d_k}}$. [...] While for small values of $d_k$ the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling [...]

Then

We suspect that for large values of $d_k$ the dot product grow large in magnitude, pushing the softmax function into region where it has extremely small gradients. To counteract this effect, we scale the dot products by $\frac{1}{\sqrt{d_k}}$.

As you can read the scaling is to fight vanishing gradients, in fact you can interpret it as a sort of temperature that smooths the logits $QK^\top$ making them more uniformly distributed (since a peaked distribution would cause gradient instabilities), therefore increasing the gradient magnitude.

The motivation of this particular expression and not another one (like the product of the norm $\frac{1}{|Q| |K|}$ or even $1/d$) is not very well explained


text:
Software stack for practical learning of an ML-based inverted pendulum system?
Some background first: My programming background is Python and C. Also, "classical" control systems. I.e PID control, analogue, feedback-based correction etc.
Inverted pendulum is a textbook control problem. Traditionally it is solved through use of some type of algorithm, most notably PID.
Recently I've discovered that a number of successful solutions have been implemented that balance the pendulum using Machine Learning frameworks instead of a pre-programmed algorithm.
Unfortunately, the links listed are intended for people with some ML background, and could be completely opaque to a beginner, even a beginner with a programming background.
Is there a complete set ("stack") of software that is beginner-friendly with (relatively) detailed documentation that can be used for practical learning of ML-based motor/system control (i.e. learning through tinkering)?

Asking for software or tools is off-topic here. Having said that, I think you won't find things much easier than gymnasium (previously gym), which you can use to solve the inverted pendulum problem using reinforcement learning, which is the specific type of machine learning that is often applied to this problem. I'd recommend that you learn a bit about RL before getting into trying to solve problems with RL with tools, although you may also be able to actually play with the code examples, but you won't appreciate them fully without knowing the RL basics.

It would be like trying to use Django without knowing Python. You may be able to play with some code examples (especially if you have some programming or web development experience), but at some point you will need to learn Python properly. The gynmsium docs have a very brief intro to RL, but that's probably not enough if you want to learn things properly. For books on the topic, see https://ai.stackexchange.com/q/22010/2444


text:
Why does my loss function fluctuate so much?
I have a loss function that I'm trying to maximise using a neural network.
While it does appear to increase and plateau over the training, it does so in a very "noisy" manner, spiking up and down, as opposed to smoothly. Below is an example (y-axis is loss, x-axis is number of epochs).

I'm having a hard time diagnosing why this might be. Is it my learning rate that's too high (1e-4 with Adam optimiser)? Is it due to my neural network architecture (a MLP of structure input_dim --> 512 --> 256 --> 128 --> 1) being too complex/not complex enough? Or could it be another factor?
EDIT:
In response to the comment by Edoardo, I am trying to implement MINE (Mutual Information Neural Estimation). MINE computes a lower bound on the mutual information between two variables. The lower bound it computes is the supremum, over a class of functions, of a particular expression (more details can be found here https://arxiv.org/pdf/1801.04062.pdf). The family of functions is chosen to be a neural network of a particular architecture parametrised by parameters $\theta \in \Theta$ (each element in $\Theta$ corresponds to a unique neural network, i.e. function, in the family), and taking the supremum over  of the expression in question all possible neural networks is essentially equivalent to optimising the parameters to maximise the expression.

Why are you maximising rather than minimising the loss function? Just asking cause it's a very odd thing to do (maybe your doing reinforcement learning and dealing with a reward function rather than a loss function?). So if you have a specific reason for that it will definitely help to know it in order to answer the question.

@EdoardoGuerriero Thanks for your reply. I've added a paragraph to elaborate on why I'm maximising a "loss function". In retrospect, I guess my use of loss function was misleading - perhaps I should have said "objective function" instead.


text:
Calculating mutual information between layer outputs and targets in a neural network
I've seen in several papers that it is possible to calculate the mutual information between a layer's outputs and the desired outputs. For example:

Source: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7517266/#B9-entropy-22-00727
How exactly is this calculated? I'm trying to calculate this quantity and track how it evolves during training for a simple feed-forward neural network used in regression. The mutual information should increase with training.
I am currently using this Python package for information theory calculations: https://github.com/gregversteeg/NPEET
This is my dataset:
X = torch.FloatTensor(500, 1).uniform_(-3, 3)
Y_no_noise = X**2 -X - 1
noise = torch.randn(X.size())
Y = Y_no_noise + 0.2*noise

And this is my network:
class MLP(nn.Module):
    '''
    Multi-layer perceptron for non-linear regression.
    '''
    def __init__(self, nInput, nHidden, nOutput):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(nInput, nHidden), # Input layer
            nn.ReLU(),
            nn.Linear(nHidden, nHidden), # Hidden layer
            nn.ReLU(),
            nn.Linear(nHidden, nHidden), # Hidden layer
            nn.ReLU(),
            nn.Linear(nHidden, nHidden), # Hidden layer
            nn.ReLU(),
            nn.Linear(nHidden, nOutput) # Output layer
        )

    def forward(self, x):
        return(self.layers(x))

model = MLP(1, 8, 1)

I assemble the activations from a layer into a $n \times d$ array, where $n$ is the number of training examples (500) and $d$ is the number of neurons.
The targets/desired outputs are in a $n \times 1$ array.
I then use the mutual information function, mi() provided in the package, passing in the two arrays as arguments.
However, the mutual information graph I obtain doesn't seem to have any meaningful pattern/trend as I would expect:

The presence of negative mutual information values here is also concerning.
I would be very grateful for ideas of where I might be going wrong in my calculation and how I can get the correct trend of mutual information increasing.
EDIT: For reference, the entirety of my code is as follows:
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision.models as models
import torchvision.datasets as datasets
import torch.optim as optim
import torchvision.transforms as transforms
import torch.utils.data as data
import matplotlib.pyplot as plt
import copy
plt.style.use("ggplot")
from pyhessian import hessian
import sys
import seaborn as sns
from scipy.stats import gaussian_kde
from scipy.integrate import quad
from scipy.integrate import nquad
from sklearn.decomposition import PCA
from sklearn.feature_selection import mutual_info_regression
from npeet import entropy_estimators as ee

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

X = torch.FloatTensor(500, 1).uniform_(-3, 3)
Y_no_noise = X**2 -X - 1
noise = torch.randn(X.size())
Y = Y_no_noise + 0.2*noise

X,Y= X.to(device), Y.to(device)
dataset = data.TensorDataset(X, Y)

train_size = int(0.7*len(dataset))
test_size = len(dataset) - train_size

train_dataset, test_dataset = data.random_split(dataset, [train_size, test_size])

x_train = [x[0] for x in train_dataset]
x_train = torch.stack(x_train)
x_test = [x[0] for x in test_dataset]
x_test = torch.stack(x_test)

y_train = [x[1] for x in train_dataset]
y_train = torch.stack(y_train)
y_test = [x[1] for x in test_dataset]
y_test = torch.stack(y_test)

x_train = x_train.to(device)
y_train = y_train.to(device)
x_test = x_test.to(device)
y_test = y_test.to(device)

y_train_for_mi = y_train.cpu().detach().numpy()
y_train_for_mi = y_train_for_mi.tolist()
#print(len(y_train_for_mi))

nInput  = 1
nHidden = 8
nOutput = 1

class MLPcondensed(nn.Module):
    '''
    Multi-layer perceptron for non-linear regression.
    '''
    def __init__(self, nInput, nHidden, nOutput):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(nInput, nHidden), # Input layer
            nn.ReLU(),
            nn.Linear(nHidden, nHidden), # Hidden layer
            nn.ReLU(),
            nn.Linear(nHidden, nHidden), # Hidden layer
            nn.ReLU(),
            nn.Linear(nHidden, nHidden), # Hidden layer
            nn.ReLU(),
            nn.Linear(nHidden, nOutput) # Output layer
        )

    def forward(self, x):
        return(self.layers(x))

model = MLPcondensed(nInput, nHidden, nOutput)
model.to(device)

# Select layer for which mutual information is to be calculated
layer_index = 8 # Goes 0, 2, 4, 6, 8

loss_fn = nn.MSELoss().to(device)
optimiser = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
epochs = 500
batch_size = 32

train_loader = data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

tl = []
mutual_info_list = []

for epoch in range(epochs):
    train_loss = 0
    #print(f"Epoch {epoch+1} of {epochs}")

    activations_overall = []

    for batch, (X, y) in enumerate(train_loader):
        model.train() # Training mode
        X, y = X.to(device), y.to(device) # Move data to GPU
        predictions = model(X)
        activations = model.layers[0:layer_index+2](X)
        #print(activations)
        individual_tensors = torch.split(activations, 1, dim=0)
        for tensor in individual_tensors:
            individual_array = tensor.squeeze().detach().cpu().numpy()
            activations_overall.append(individual_array)
        loss = loss_fn(predictions, y)
        train_loss += loss # Accumulate the loss over the batch
        optimiser.zero_grad()
        loss.backward()
        optimiser.step()

    tl.append(train_loss/len(train_loader))

    activations_overall = np.array(activations_overall)
    activations_overall = activations_overall.tolist()

    mutual_info = ee.mi(y_train_for_mi, activations_overall)
    mutual_info_list.append(mutual_info)

final_loss = [tl[-1]]
final_loss_numpy = torch.stack(final_loss).cpu().detach().numpy()[0]
#print(f"Final training loss: {final_loss_numpy}")
train_loss_curve = torch.stack(tl)
plt.plot(train_loss_curve.detach().cpu().numpy(), label="Train loss")
plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.legend()
plt.show()

plt.plot(mutual_info_list, label="Mutual information")
plt.xlabel("Epoch")
plt.ylabel("Mutual information")
plt.legend()
plt.show()

EDIT: Response to nbro's comment
The mutual information between two variables X and Y measures the information that they share. It measures how much knowing one of these variables reduces uncertainty about the other. It is a non-negative, symmetric quantity.
I wish to use treat the activations of a layer as a single multivariate variable, and the desired output as the other variable. The mutual information between these two, I believe, can be used to measure how learning progresses in a layer because as training progresses, the activations should tell you more about the desired outputs. So, I expect the mutual information to steadily increase (and then maybe plateau) with training duration/epochs. This idea is quite widely discussed in the literature, especially in the context of the information plane.

https://www.semanticscholar.org/reader/267980e417f1d01a897e87fa409f64e2a76b96cd

https://towardsdatascience.com/information-theory-of-neural-networks-ad4053f8e177#:~:text=Mutual%20Information%20along%20the%20Training%20Phase&text=With%20successive%20layers%20the%20mutual,output%20is%20low%20as%20well.&text=As%20we%20train%20the%20Neural,of%20information%20about%20the%20output.

When I say "desired outputs", I mean the targets given my input. My regression dataset consists of 1D inputs and 1D outputs (the "desired outputs").

What exactly are you expecting to see in your plots? Could you also briefly remind us what the mutual information is in general, and tell us how you want to interpret it in this case. One thing that is unclear to me is "the desired outputs". Are you referring to the labels of the network? But would that make any sense? In theory, it requires 2 random variables. What would they be in this case? Finally, it might be a good idea to provide the source of your first plot.

@nbro Thanks for the reply. I've added more detail in response to your questions.

NPEET uses a K Nearest Neighbors estimation for mutual information and they note in section 4.3 of their provided pdf doc that there is a chance for negative values in the estimated mutual information. They note their own example has this issue. They mention their shuffle.test function may help resolve the negatives.
https://github.com/gregversteeg/NPEET/blob/master/npeet_doc.pdf
In section 6.4 of the doc they detail further on shuffle.test for potentially resolving negative estimates.
The pdf doc notes that the mutual information follows the calculation from the following linked paper in equation 8
https://journals.aps.org/pre/abstract/10.1103/PhysRevE.69.066138
I see you are purposefully comparing the activations to the desired output, rather than the predicted output. As a sanity check it may be useful to see if the output mutual information behaves as expected.
The docs say these measurements default to base 2 thus are in bits and your mutual information estimates all < .15 bits as absolute values. I suspect this noise in your measurements is due to the estimation technique and there is less than .15 MI in the layer you compare the targets to.
I believe that'd be the weights not the activations unless you mean the output of the activation functions. I have not run your code and do not know exactly which layer or part of the network you assign as the activations in the code with line activations = model.layers[0:layer_index+2](X)
Your example has relatively small dimensions so I doubt it'd be a sample data amount too small vs measure being estimated, though it may be worth trying more samples just to check.


text:
How does AI 'see' the images it generates- from what perspective?
I've been using AI image generation for a while now, and I've noticed how profoundly AI doesn't seem to see the image as a whole, sometimes generating an image with parts of fingers floating near objects supposed to be being held, VERY warped perspective, and of course, extra fingers and occasionally legs. I also notice that  by negative prompting 'extra/distorted limbs' (or similar)it often solves the problem, so it appears that there must be some sort of awareness of what is happening in the image. Stable Diffusion being open source and thus available for analysis, why would AI specifically have this as a common problem?

There are lots of image-generating technologies, making this question very broad. Perhaps pick a single  popular and accessible image generation model? Stable Diffusion might be a good choice to get a factual answer, since it is open-source, so no-one has to guess at what proprietary systems might be in use. It is also one of the more popular recent releases of image generators

@neilslater The problem is, I'm using a forum, NightCafe, which hosts many, from Dall-e to Stable Diffusion. I have noticed the 'dismembering' on SD, so I suppose that'd work. An edit is on it's way. As well as some further points. thanks!

Stable Diffusion (and the others) don't "see" anything. Simply put, they are iteratively denoising pure noise until it "makes sense". It's not like it created a 3D space or layers of foreground and background. It is guided by the prompt because it was trained to use that as part of the denoising context, but it doesn't really "understand" anything.

Presently generative AI for images does not understand the meaning of fingers as physical 3D anatomical structures that have a range of mobility/posability. It only has a bin of imagery to draw from that is suggestive of fingers. Kind of like how you can't look at your hands in a dream and make sense of their detail (one method people use to go "lucid"). To correlate the relationship between a 3D mesh and a 2D image, a dataset of 2D/3D pairings must be provided. If trying to teach a machine what the physical world feels like, touch sensor data is needed etc, to expand domain of experience.

Let's start from the problem associated with generating details in images, probably as old as computer vision itself.
Consider an image of a person of classic size 512x512 pixels, one finger in that image will probably consists only of 10 of those pixels. Whatever loss you might use, the contribution coming from the reconstruction of 10 pixels will be most of the time irrelevant compare to the contribution coming from other elements in the image. Let's also remember that most architectures today rely on convolutions, meaning that those 10 pixels will be averaged together trough some pooling operation, making them count even less. This should give the overall idea about way in general even state of the art models fails with high details.
Diffusion models add even more complexity to the overall picture because they start from random noise and they iteratively subtract a different noise distribution learned during training to produce something that make sense. So not only they have the hard task to learn details, they also have to learn to distinguish between noise that's actual noise to remove, and details that sometimes do resemble noise but that we want to keep.
To understand the connection between noise and details it useful to know that in the literature image details are often referred to as high frequency functions. It's actually an on point definition rather than just an analogy. The frequency of a wave relates to its wavelength, basically how much space it need to complete a full cycle, in an image you can think of the frequency of an element exactly the same way, i.e. how much space in the image the element need to end or repeat itself (I know it more complicated than this, but we care about the gist). This very nice paper shows how it become suddenly easy for simple architectures to reconstruct high details when random Fourier transformation are used for feature extraction (basically high frequency random functions). The image below is taken from the paper. This unfortunately comes with huge increase in computational power and it not a real feasible solution for large models like stable diffusion.

One last consideration should be done about the negative prompts. Unlike purely visual models, text to image models try to learn a semantic latent space, i.e. they try to associate specific latent vectors with elements that the output image have (during training) or should have (during inference). Moreover, this latent space for the most part have nice geometric properties, like vectors of word of opposite meaning have opposite direction. Now, stating that something should be in the generated image gives absolutely total freedom to a model about extra elements that could be potentially present. The weights of a model are always used all at once, the text prompt simply causes some weight vectors in the latent space to have more impact than others. This is why negative prompt are an easy solution to remove artefacts (to some extent), you're just constraining the model more. The model has already learned that most of the times a hand has 5 finger, but since a hand it's a rather detailed element to generate and because it's not explicitly sated in the prompt that we want 5 fingers, a model has more than enough space for random manoeuvre to end up generating aberrations.


text:
How to generate a 3D model from only 1 image?
I'm posting this on the AI stack exchange because even though this can be solved with a "regular" complex and sophisticated algorithm, it seems that trying to generate something for which all information isn't available for is usually solved with AI techniques.
The idea is to just have an image of some random object (e.g. a watch) and be able to generate a polished 3D model of it even though we don't have the full 360 view of it. This means there would have to be some inferences made, because the program would have infer what the rest of the object may look like.
I would also like to know if this is an extremely complex endeavor, or maybe much easier than I think due to the recent advances in AI? There's a company, 3DFY.ai, that can turn a text prompt into a 3D model, so I would think that it's something along the lines of that, but using an image instead of text for reference.
When doing a google search for the question that I'm asking, I get results that I'm not looking for, which are:

Photogrammetry - this requires taking a picture from different angles of an object, which isn't always possible if you don't have access to the object.
Depth Estimation - from what I've seen, it just looks like the image gets extruded a bit to give somewhat of a 3D appearance

What kind of disciplines would this require besides AI? Computer Vision?

I don't know how hard that would be to program with an AI but an AI would not just create a 3-d model from your single picture. Rather it would look through the vast collection of pictures in its training data looking for similar pictures (hopefully of similar objects) and then create a 3-d image from this large collection of pictures of (hopefully) the same object from different angles.
If your single picture is of a reasonably common object this could still work very well but the moment you give it a picture of something not well represented in the training data it will fail miserably.
So a picture of a watch will probably give you a good 3-d model but I would expect the model to fail on a tower made out of wooden toy bricks because every such tower looks different in 3-d.

Thanks for the reply. In this case, probably using AI isn't really the solution. The reason I assumed AI would be the solution for something like this is because I see things like Nvidia's DLSS or Photoshop's generative fill that use AI.

I'm not sure if you're looking for an explanation on how one could train such model or for the tools to use it.
If looking for tools :
paper: https://huggingface.co/papers/2309.16653
tool: https://huggingface.co/spaces/jiawei011/dreamgaussian
also: https://research.nvidia.com/labs/dir/magic3d/
There are also some other companies offering this service but I have not tested them.
If you would like to build your own tool I would suggest:

Building a encoder-decoder like model
Having a set of simple object with 3D model and multiple images to do photogrammetry

Also state of the art seems more advanced in text-to-3D then 2d-to-3d so I would suggest to explore that first

The image-to-3d models: https://huggingface.co/models?pipeline_tag=image-to-3d