text:
AI and Machine Learning Prediction Algorithms for predicting outcome results of Hypothetical poll
Can artificial intelligence and Machine Learning Prediction Algorithms assist in deciding the Outome Results of a Hypothetical Online Poll?
Poll: Selecting favorite American President till date.
Example: Input dataset for All Presidents of USA till date.
This includes Text, Images, Videos etc. I.e. Multimedia.

This cannot be done as described, unless you find a way to create a feasible and measurable link or association between the input data and output data, that exists already in your training data, and that could be analysed statistically.
Basic supervised machine learning works by being given example inputs and outputs - so called "ground truth" - that already answer your question for some subset of a population. Your suggested input dataset of text, images, videos about US presidents does not appear to include any poll results that could be learned from.
A variation of your dataset might be used to predict poll preferences. If you had some simple summary data about people who have voted in similar polls in the past (e.g. demographics such as age, political party membership, location) and also summarised data about each president (approval ratings, number of terms etc), then you may be able to train a machine learning system to learn a little how personal preferences work in the context of your poll. You could then use it to predict poll results for unseen people, and unseen presidents, provided you have the same summary data for the new people and new presidents.
This is quite similar to recommendation systems, so if instead of structured data about people and presidents, you had results of multiple similar polls from multiple people, then you might be able to train a system that predicted outcome of a new poll if you knew what a person chose in one or more previous similar polls.
Processing that included unstructured data such as text articles, images and videos is harder, but not impossible. The trick would be to reduce this data down into simpler numerical format - e.g. you could use a language model to extract sentiment (positive or negative articles) from texts. You would still need to have some previous poll results in your training data in order to help predict what these summaries might mean for a new poll.
What you should not do is pour lots of unstructured data into some summary machine, and expect it to output anything usefully predictive about a specific function that you want to know about. The chances of the results being correct is no better than you simply guessing at the start.

Thanks. So input dataset would include additional "you had some simple summary data about people who have voted in similar polls in the past (e.g. demographics such as age, political party membership, location) and also summarised data about each president (approval ratings, number of terms etc), then you may be able to train a machine learning system to learn a little how personal preferences work in the context of your poll. You could then use it to predict poll results for unseen people, and unseen presidents, provided you have the same summary data for the new people and new presidents"?

@PrashantAkerkar: Yes if you had that kind of data, you could build a supervised learning model that predicted how much a person might prefer certain choices in a poll, based on things you know about the person and the presidents they should choose between. The quality of such a model would vary a lot, depending on what data you had, and how much of it. The usual, least work method of collecting this data in case like your example would be to actually run the poll, and then maybe adjust for demographics of the responses.


text:
For an image (of any object), how to find its location in the other image(s) which contains it, given there are no labels or annotations for any image
Problem Statement:
I am given 2 sets of images. All the images in both sets are without annotations and labels.
First set : a set of images of the grocery store shelves (captured in the grocery stores).
Second set: a set of close-up images of the products kept on those store shelves.
What I am trying to achieve:
Goal: For every product image, I want to find the location of that product in all shelf images in which it appears.
For example,
Input Product image

Output Corresponding Shelf image

My approach:

For each product image, first find all the shelf image(s) which contain that product.
Then predict a bounding box by finding the location of the product in the shelf image.

I am using YOLOv5 for this task but I am not sure how should I start off given that I have to do it without annotations or labels.
I have come across terms like Zero-shot learning, self-supervised Object Detection, etc. but I haven't been able to figure out their use case as a starting point.
There's a similar question asked on StackOverflow but I am not sure the answer to it solves the problem.

Can you please put your specific question in the title? "Object Detection without annotations and labels" is not a question and it's also not very specific.

@nbro Done! If I may ask, does that make it more clear now? For a detailed description of the problem, please read the explanation down there that I have provided along with the newly edited title.

For something like this you may just want to compare the image features directly, rather than trying to fanangle your data into a class based segmentation model.  There are a number of very fast feature detection algorithms, like SIFT, SURF, ORB, and BRIEF, that don't necessitate the use of a neural network at all.
OpenCV - Feature Matching
The features locked away in the hidden layer of a CNN are gold for this sort of thing as well, and chopping the head off of a pretrained neural network to get at those conv layers, and then using those in a manner similar to SIFT features, may improve accuracy while accomplishing the same end.
Paper - Deep Semantic Feature Matching
Either way, you are going to be doing some kind of similarity search here, rather than a classification task.  You want to use methods that see if the important bits of image A exist in image B, and any attempt to do that via classification models would create so many classes as to be unusable.  Any method used to accomplish this task is going to have to rely on similarity of image features in one way or another.

You can formulate your problem as a template matching (TM) one. This task is similar to object detection, but you don't need to label anything. There are 2 images: the template (in your case, the template is the image of a product) and the image where you want to search the template (the image with the shelf of products). There are different TM algorithms. For example, OpenCV offers a simple one with different similarity measures.
From my limited experience with TM, it works well only if the objects in both the template and the other image are in a plane parallel to image plane. This is more or less your case (at least, in the images you're showing us), so I expect that TM will work reasonably well with those 2 specific images, but you should first resize the first image so that there are no padding around the object (the chocolate bar) - in a way, this is like labeling. If this feasible, then you should try TM.

I am given a Hint: Use Visual Similarity. Maybe I am wrong but I believe TM isn't of much use here because time spent resizing all the product images is as good as labeling, so please usher me to a path like where I have a pre-trained model on a similar dataset and now I would like to freeze some layers and fine-tune the model on a new dataset which comes with a problem that it does not have labels or annotations. So how should I proceed from there? Note : You can assume I have no experience with self-supervising the pre-trained model, or maybe something better that you can suggest?

I don't know who gave you that hint, but template matching uses some kind of "visual similarity". There might be a tool that attempts to automatically resize the image to fit the object in it. So, resizing might be less laborious than labelling and maybe you don't have to do a lot manual work. Moreover, you probably don't have to be super precise when resizing. You can still leave some padding/spacing around the object and it might still work. Having said that, it might be possible to use a pre-trained model to do your task, but I don't know which one - I am not an expert in this topic.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Eval loss when fine-tuning in an unsupervised way/pretraining?
I'm fine-tuning the base Mixtral 8x7B model (4-bit quantized) with Lora on my own data, following these guidelines: https://www.stochastic.ai/blog/xfinance-vs-bloomberg-gpt
I'm first fine-tuning it in a self-supervised way, and then using the SFTTrainer on some conversational data as a final step. I'm not a deep learning engineer so I have no idea what I'm doing really, but I tried a few different hyperparameters at first to see which got the train loss down quickest.
I then chose the best out of them and trained for about 8 epochs over my text data (5MB of text data). train loss was still quite high (0.6), the eval loss kept increasing. However, I assume the eval loss at this stage with my setup is irrelevant for this NSP self-supervised stage of training.
Then finally, I used the SFTTrainer on conversational data for like 100 epochs (only 100 Q&A pairs, but high-quality and some with extensive answers of about 4k tokens). The train loss got extremely low but the eval loss kept rising. Weirdly I got pretty good results when testing the final model.
My first question is how do I know when to stop the first stage (self-supervised) of training, do I need to create some advanced validation loss function to see if the model is accurately learning patterns in the unstructured text data? Secondly, why would my eval loss in the second stage of training keep rising even though with each checkpoint, the performance seems to be getting better when I tried the model out after training?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How are gradients backpropogated in ALBERT?
I was reading the ALBERT paper and saw that they use the same parameters in each layer hence reducing the number of unique parameters. From what I could gather it seems if the all the layers have say parameters W then parameters of different layers would be updated differently which would destroy the parameter sharing.
So one way I can think of is say we have only one set of parameters W and after each layer completes it's weight update W changes to W' and the preceding layers now use W'.
Is this the right way to think about it or does something else happen under the hood?

It might be a good idea to provide the link to the paper.

After researching a bit I've reached the conclusion that it must be behaving just like other networks with shared parameters do i.e. they accumulate gradients in the backpropagation stage and update it once they've backpropagated through all layers. This answer clearly explains it in the case of CNNs. I'm sure this is the case after going through their codebase which does not have any special implementations for handling backpropagation and hence must follow the traditional route which DL libraries automatically use!

ALBERT (A Lite BERT) is a variant of the BERT (Bidirectional Encoder Representations from Transformers) model, which is a type of transformer-based language model for natural language processing tasks such as language understanding and machine translation. Like BERT, ALBERT uses self-attention mechanisms and transformer blocks to process input sequences and generate output representations.
In ALBERT, gradients are backpropagated (also known as "backprop") during the training process in order to update the model's parameters and improve its performance. This is done using the standard backpropagation algorithm, which is a widely-used method for training neural networks.
During backpropagation, the model computes the gradients of the loss function with respect to the model's parameters, and then uses these gradients to update the parameters in the opposite direction of the gradient. This helps to reduce the loss and improve the model's ability to make predictions.
In the case of ALBERT, the model's parameters include the weights and biases of the self-attention mechanisms and transformer blocks, as well as any other trainable parameters in the model. The gradients of the loss function with respect to these parameters are computed using the chain rule, and are then used to update the parameters using an optimization algorithm such as stochastic gradient descent (SGD) or Adam.
I hope this helps! Let me know if you have any other questions.


text:
What is intended for 1x1 convolution for input images?
I'm reading this about Self-Attention GANs : https://sthalles.github.io/advanced_gans/
I'm trying to better frame what is intended about 1x1convolution for input images. Does this simply mean that we produce feature maps by multiplying a scalar filter to each single pixel of the image? It's not clear since in the reported picture the 1x1conv part resembles a vector.

I had not read this work, but I am quite curious since it seems to me that this would just create three sets of layers that are just scaled representations of each other.  Though, that could be the purpose given the different intentions of $f(x), g(x), and h(x)$ in self attention models.

Anyway I think that my question is kinda trivial: saying we have feature maps of dimension $(N, K,K)$ where $N$ is the batch size, then convolving them with $1 \times 1$ filters (single-channel) will return transformed tensors with the same dimensions $(N,K,K)$ and the image is a bit misleading from my perspective. Is this correct? Also, convolving with $1 \times 1$ filters is what it's usually named 'Point convolution' ?

1x1 convolutions is a very simple trick generalized by the Inception module published by Google in 2014 in the famous paper Going Deeper with Convolutions.
The most common use case is to modify the output channels of the input feature maps. This is mostly used when you have a net architecture with multiple branches that needs to be aggregated into one result (such as in your image or such as in the Inception module).
Here is an example:

Input Feature map: $F_{in}: B \times H \times W \times C_{in}$
Convolution: $W: 1 \times 1 \times C_{out}$
Output Feature map: $F_{out}: B \times H \times W \times C_{out}$

In plain english: 1x1 convolutions modify the output feature maps channels $C$ without altering the resolution $H \times W$.
When is this useful? For when you need to expand or shrink the feature maps channels. Examples: Inception, SE Blocks, Bottlenecks Blocks, Detectors Head (RetinaHead)...
Here is a more in depth article

many thanks! So is this the same as 'point convolution' ?

Yes, exactly. In some papers they say point convolution or fully connected layers, but in the code they are implemented with 1x1 conv


text:
Definition of negatives in NT-Xent loss
I'm trying to understand few details about NT-Xent loss defined in SimCLR paper(link). The loss is defined as
$$\mathcal{l}_{i,j} = -\log\frac{\exp(sim(z_i,z_j)/\tau)}{\sum_{k=1}^{2N}\mathbb{1}_{[k\neq i]} \exp(sim(z_i,z_k)/\tau)}$$
Where $z_i$ and $z_j$ represent two augmentations for the same image. What I don't understand is: at the denominator, I understand that we want to exclude the point $z_i$ using the indicator function but shouldn't we exclude also $z_j$? Otherwise we will have $k=j$ for some $k$. Essentially, why we do let the positive sample at the denominator?

To train the model, you take $N$ samples then build $N$ pairs by means of applying two different augmentations. So the total num. of samples is $2N$. Now, you take one pair $(i,j)$ and consider that to be positive. The remaining $N-1$ pairs (so $2(N-1)=2N-2$ samples) are negatives.
For each positive pair $(i,j)$ you compute the similarity of the positive pair at the numerator, and normalize it by the sum of similarities between the anchor $i$ and the negative $k$: if you reason on pairs, instead of single samples, you have that $k\neq j$. Maybe in the paper is not so clear, but $i$ and $j$ are positive because they are two different views of the same input sample (say $x$), instead $(i, k)$ and $(k, j)$ are negatives because $k$ is a view from another input (say $x'$, where $x\neq x'$.) So, when computing the denominator you do not include $j$. I found a blog post that explains that too.
Lastly, the loss is also computed on $(j,i)$ pairs, so you sum it and divide by 2 times the batch size $N$.

mmmh ok but to have normalized probabilities shouldn't I also include the index $i$ itself? Thanks for your answer, but could you elaborate a bit more please? I would like to have this point clear.

@JamesArten I've edited my answer, I hope is clearer now

For the denominator we need both the positive and negative pairs.
The blog post you cited explicitly says this, where did you read that you do not include j in the denominator?

If you look at this part: $$\frac{\exp(sim(z_i,z_j)/\tau)}{\sum_{k=1}^{2N}\mathbb{1}_{[k\neq i]} \exp(sim(z_i,z_k)/\tau)}$$
you will notice that it's similar to:
$$\frac{\exp(x_i)}{\sum_{i=1}^{N}\exp(x_j)}$$
Which is basically a softmax function.
It's essentially a way to normalize the loss function so that all the probabilities sum up to 1. In the SimCLR case, you only have to exclude k=i, because of course the similarity of a sample with itself will always be 1.
Essentially this normalization is the reason why the positive samples (similarity close to 1.0) are pushed away from negative samples (similarity close to 0).


text:
Could it make any sense to choose a larger dimension for the latent space of the VAE with respect to the original input?
Could it make any sense to choose a larger dimension for the latent space of the VAE with respect to the original input?
For example, we may want to learn how to reconstruct a relatively low-dimensional input (let's say $20$ dimensions), then could I define my encoder and decoder to have $64,256,512...$ hidden neurons before bringing back the reconstruction?
EDIT:
Well I've thought about that and I think it would still be reasonable as in latent-variable models we are actually assuming that our original observations are generated from unseen 'hidden' variables. And (I think) the lower dimension of the latent space is only assumed for an original dimensionality-reduction purpose.

I recommend that you put your specific question in the title. Here, your question is very specific and it's not just "Dimension of the latent space in VAE". Anyway, one thing is not clear to me. It's clear that you're suggesting to improve the dimensionality of the layers of the encoder, but what about the decoder? Let's say the encoder produces a latent vector of dimension 512, which is bigger than the input, then the decoder should do what? Should it do like the usual role of the encoder and convert this latent vector to a lower-dimensional reconstructed input? Is this what you mean?

Apologizes, I've edited the question title with a more specific one. Anyway yes, shortly speaking we would invert the encoder/decoder role in such a situation. Well I've thought about that and I think it would still be reasonable as in latent-variable models we are actually assuming that our original observations are generated from unseen 'hidden' variables. And (I think) the lower dimension of the latent space is only assumed for an original dimensionality-reduction purpose.

I would recommend that you add this useful info in your comment to your post directly (because comments are temporary so they might get deleted later)!

I am not sure about the VAE in particular, but the convnext presented here uses the "inverse bottleneck" (i.e. internal representations being higher dimensional than inputs) as one of the core changes that leads to an increase in its performance when compared to prior convolutional networks. This and some other changes make the convnet be more competitive against the visual transformer (that is pretrained in a large dataset).
But for really low-dimensional data (as in your example) and for VAEs, I am not sure if the same idea would prove useful.
P.S. I wanted to add this as a comment but I do not have enough reputation to do so.


text:
Out of distribution detection (OOD) in the context of regression problems
I'm working in a regression setting to predict a scalar value $y$ from an input $\textbf{x} \in \mathbb{R}^D$ and I'm interested in understanding whenever my model is fed with something that it is outside the (unknown) training distribution $p(\textbf{x})$. For simplicity we can assume I'm using a simple neural network $f_\theta:\mathbb{R}^K \rightarrow\mathbb{R}$ to predict a single (scalar) property value, training my model with an initial dataset $\mathcal{D} = \{(\textbf{x}_i \, , y_i)\}_{i=1}^N$, that is, my task is specifically about regression.
What I'd be interested in achieving would be that, feeding my neural net with a new input $\tilde{\textbf{x}}$ I could retrieve somehow a confidence score telling me if the new input $\tilde{\textbf{x}}$ lies outside the spectrum of observed instances in training dataset.
A way of doing that would be of course estimating the probability of training dataset $p_\theta(\textbf{x})$ and see if the new material $\tilde{\textbf{x}}$ is in a low-likelihood region of $p$. People have used such approach for images (https://arxiv.org/pdf/1912.03263.pdf) but generative models are hard to train.
Instead, I was looking at recently proposed papers using energy-scores for detecting out of distribution samples (paper1, paper2) but the examples seem to refer specifically to classification settings.
As I'm not too familiar with energy-based models, is there a way such frameworks may be applied to regression settings?

Though I'm not familiar with energy-based methods, have you tried other "simpler" approaches? for example, assuming a mixture of Gaussians and using Mahalanobis distance as anomaly score, or trying to use kNN on the predicted values of $f_{\theta}$ to find a separating threshold based on mean distance from the neighbors?

Many thanks for your suggestions! As I’m not so into ood I was just searching for what is out there, so I appreciate your comment a lot! Could I ask you to further elaborate on these approaches?

I would suggest going over some of the known AD surveys (here's a pretty new one here - https://arxiv.org/abs/1901.03407). If you're looking for something shorter you can check out this repo (https://github.com/Hadar933/Deep-Committee-kNN), try briefing through the introduction section.


text:
how to go from mathematical problem to neural network (and back)?
I am a little confused on how, you can find online papers that describe complex Machine Learning formulas in a mathematical/probabilistic way, and, in the other hands, easy tutorials that teach you how to use frameworks to create neural networks, without mentioning the maths behind.
What is not clear is, what is the correlation between these two worlds? What are the "parameters" that make you understand i.e. how many layer to code, what kind of perceptrons to use, etc?
to make an example:
Let's take this formula, which in Wikipedia Italy is described as "the standard learning algorytm":

And suppose that size of w and x is 4. , and g(x) and f(x) are, for examples, linear functions.
What next? Where do I start coding a neural network that solves this problem?
It would seem more logical to me to code this "directly" without defining perceptrons, convolution, layers etc.

Hi @Barsaas. Please include your main question in the title

Basic feed forward neural nets (MLPs) are essentially just computing sequences of matrix multiplications (with nonlinear activations in between), so this is in fact easy to code "directly" like you mentioned. The more difficult part is computing the gradients with respect to the parameter matrices (usually with backpropagation). However there really isn't anything that fancy behind the basic neural network model, it really is just sequences of simple blocks.
You certainly want at least one hidden layer usually, because without it you'll just have a generalized linear model. Neural networks are useful for creating arbitrarily nonlinear models, which can be achieved by adding more layers or more neurons per layer.
If you want some clarification about how to code a neural net, I recommend taking the classic Andrew Ng ML course on Coursera.
With regard to the amount of layers, number of neurons, etc, these are hyperparameters -- there is no known way to determine the correct values for them without experimentation.


text:
Deep Q-Learning "catastrophic drop" reasons?
I am implementing some "classical" papers in Model Free RL like DQN, Double DQN, and Double DQN with Prioritized Replay.
Through the various models im running on CartPole-v1 using the same underlying NN, I am noticing all of the above 3 exhibit a sudden and severe drop in average reward (with a sudden and significant increase in loss) after achieving peak scores.
After reading online, I can see that this is a recognized problem but I cant find a suitable explanation. Things I have tried to mitigate:

adapt model architecture
tune hyperparams like LR, batch_size, loss function (MSE, Huber)

This problem persists, and I cannot seem to achieve any sustained peak performance.
Useful links I found:

What could be causing the drastic performance drop of the DQN model on the Pong environment?

Example:

till ~250 episodes in Double DQN with PR (with annealing beta), performance steady goes up in both increase in reward and decrease in loss
after that stage, the performance dips suddenly in both decreased average reward and increased loss as seen in output below

Episode: Mean Reward: Mean Loss: Mean Step
  200 : 173.075 : 0.030: 173.075
  400 : 193.690 : 0.011: 193.690
  600 : 168.735 : 0.015: 168.735
  800 : 135.110 : 0.015: 135.110
 1000 : 157.700 : 0.013: 157.700
 1200 :  99.335 : 0.013: 99.335
 1400 :  97.450 : 0.015: 97.450
 1600 : 102.030 : 0.012: 102.030
 1800 : 130.815 : 0.010: 130.815
 1999 :   89.76 : 0.013: 89.76

Questions:

what is the theoretical reasoning behind this? Does this fragile nature mean we cannot use the above mentioned 3 algorithms to solve CartPole-v1?
if not, what steps can help mitigate this? Could this be overfitting and what does this brittle nature indicate?
any references to follow up with regarding this "catastrophic drop"?
I observe similar behavior in other environments as well, does this mean that the above mentioned 3 algorithms are insufficient?

Edit:
Taking from @devidduma's answer, I added time based LR decay to the DDQN+PRB model and kept everything else same. Here are the numbers, they look better than before in terms of the magnitude of the performance drop.
   10 : 037.27 : 0.5029 : 037.27
   20 : 121.40 : 0.0532 : 121.40
   30 : 139.80 : 0.0181 : 139.80
   40 : 157.40 : 0.0119 : 157.40
   50 : 225.10 : 0.0107 : 225.10 <- decay starts here, factor = 0.001
   60 : 227.90 : 0.0101 : 227.90
   70 : 227.00 : 0.0087 : 227.00
   80 : 154.30 : 0.0064 : 154.30
   90 : 126.90 : 0.0054 : 126.90
   99 : 154.78 : 0.0057 : 154.78

Edit:

after further testing, pytorch's ReduceLROnPlateau seems to be working best with patience=0 param.

Does LR decay actually solve your problem? From your result it seems the situation is mitigated but it is hard to see whether the policy can converge to the optimal one in a long term.

This is a case of overfitting the Q function leading to compounding errors when selecting actions.

You have been training your policy for too long on the same data distribution.
Overfitting Q functions will then lead to data distribution mismatches more often in action selection and compounding errors will happen earlier than before.

You should probably train until 400 up to 600 episodes and then stop training the policy. Consider the following slide on compounding errors:

Whenever a wrong action is selected, because of overfitted Q value function, the agent can not generalize well on how to recover from that mistake. Eventually, compounding errors increase quadratically in time.

It will only get worse for your agent once the wrong action is picked.
In Temporal Difference learning methods like TD-0, SARSA or Q learning, finite-state and finite-action MDP's converge to the optimal action-value if the following two conditions hold:

The sequence of policies $\pi$ is Greedy in the Limit of Exploration (GLIE)
The learning rates $\alpha_t$ satisfy the Robbins-Munro sequence such that:

$ \sum^{\infty}_{t=1} \alpha_t = \infty $
$ \sum^{\infty}_{t=1} \alpha^2_t < \infty $

You can infer that by any means we should use a decaying learning rate in order to satisfy the Robbins-Munro sequence. There are three types of decaying learning rates:

time-based decay
step-decay
exponential decay

In the limit of exploration, they guarantee convergence to the optimal policy, for any Temporal Difference learning based algorithm. I would suggest you use a time-based decaying learning rate, which is the default choice in Keras when you set the decay parameter.
$ lr = 1 / (1 + decay\_factor * iteration) $
One iteration here means one epoch. You probably train your neural network every step taken by the agent, so one iteration means one epoch, one step taken by the agent.
In Keras you can set the decay like:
model.compile(loss='mse', optimizer=Adam(lr=self.learning_rate, decay=decay_rate))
I would suggest a value of $0.001$, that should be a good starting point.

that makes sense! What alternatives do we have other than early stopping? Does this problem of compounding errors exist in other algorithm classes which use temporal difference learning like SARSA and so one?

@Virus i edited the answer to elaborate a little longer in order to answer these questions

@Virus I just gave a new idea you could try that could fix the problem you were facing. Let me know.

thanks for adding the theory background to your answer! Its exactly what I was looking for, I have added the new performance numbers after a bit of tinkering with the ideas you mentioned. The results are promisingly better, with the drops being smaller but still there. I start the time - based decay after epoch 50, and kept it at 0.001 which works well.


text:
Selecting features for a neural network: is it redundant to have a feature that is an average (or max, or min) of some other features
I'm trying to create a neural network that would able to look at the current price of a crypto asset and classify between a "BUY", "SELL" or "HOLD". So far for my input features, I've decided to go with the past 40 opens, closes, highs, lows, turnover, and volumes (240 features + the current price so 241 total features).
Would it be redundant/not ideal if I had another feature that was the average of the past 40 opens for example? What about the max/min of the past opens?
My thinking was that with only the raw prices data of the past 40 days, the neural network would be able to "detect" and create the most optimum features like the average or max in the hidden layers. And therefore, having the avg. or the max/min of some existing features would be unnecessary or perhaps worsen the performance of the model.
Or is there no clear answer and would this be something I'd only be able to figure out by testing against data?
Thanks for your help!!

Hi Neil. Ah okay, sorry about that. I edited the question down to exclude the last bit.

Or is there no clear answer and would this be something I'd only be able to figure out by testing against data?

That is the general rule you should always consider when looking at feature engineering (which is what you are proposing), as well as for many architecture choices.
It is very hard to tell in advance what a change to a machine learning system will do for whichever metrics you are interested in. You may have some experience that applies, or find similar experiments online that you can take inspiration from. But you will want to test everything, and should take care to use good practice when evaluating different options - e.g. a cross validation dataset (sometimes called a development dataset).

Would it be redundant/not ideal if I had another feature that was the average of the past 40 opens for example? What about the max/min of the past opens?

One aspect of multi-layer neural networks, is that they can in theory learn useful internal features in the hidden layers from raw data. These internal features are unlikely to be exact copies of mean values or min/max values, or anything else you would construct manually. However, they can be similar enough in end result that manually derived features that you think of will not make much difference.
So you would think that derived features would not be useful in nerual networks. In practice though they can be, because the convergence process to find the best internal features is not perfect. Smart feature engineering can improve the performance of a neural network classification or regression supervised learning. Sometimes you can find "golden" engineered features that relate really well to your target variable, and that boost results significantly.
A couple of things to bear in mind:

A "scattergun" approach of trying a large number of derived features might seem attractive, but there is a risk of overfitting the training data. If you try enough times you may find something that works purely by chance but only for the training data set.

Nonlinear combinations that make conceptual sense given the problem domain can be worth looking at. For instance if you want to predict house prices, and your raw data was house width and depth, then floor area width * depth might be a useful feature.

Feature engineering is still something of an artform. Automated systems using the scattergun approach with filtering are competitive, but domain insight can still win.
If you have vast amounts of data and the CPU time cost is not an issue, you may want to forgo feature engineering due to the theoretical redundancy. It seems possible to make a giant neural network using latest features such as skip connections and batch normalisation, feed it raw (but normalised) data, and press "go" to get a state-of-the-art result. From that perspective, feature engineering is for when you don't have "big data" or deep pockets for heavy processing - for many of us that still means feature engineering is a standard approach on every project.

+1. my empirical study even showed that derived features can replace raw features (empirical results are not affected, and even improving, when raw features are omitted). I use derived features that are intuitively useful/impactful in my domain, e.g., instead of raw coordinates of points, I use distance between points. instead of only raw price, maybe the derived feature can also be a feature of how close the last few stock prices to a shape. that's a thing right?

@Sanyou Yes, if you have good insight into your problem, making derived features using your knowledge can work very well, especially if you have smaller datasets. With stock trading, some of the "shape" rules may not be as useful as people hope though - there's a lot of pseudoscience and wishful thinking that you can have a "system" that will gain money. I don't trade stocks though, so I cannot really say which ideas are worth making derived features for in practice.


text:
Clustering key, value pairs with similar meanings
I have a dataset of key-value pairs that describe specific medical laboratory mappings. Example of this data:

Group Name
Specific Test Name

Blood Test
Red Blood Cells

Blood Test
Hemoglobin

Complete Blood Test
RBC

Complete Blood Test
WBC

Blood Test Platelet
WBC

Blood Test Platelet
Platelet Count

PSA Total
PSA

As it can be seen, the top 6 rows are all talking about a blood test, while the last row is talking about an unrelated PSA test. In row #1 and row #3, "Red Blood Cells" and "RBC" mean the same thing.
I am trying to leverage machine learning and pattern-recognition to model this data and put the top 6 rows in one group (even though they spell the same thing differently) and the last row in a different group.
So far, I have tried to use K-Means clustering with TF-IDF and have also looked into Self Organizing Maps, but I have not been successful in either one of them. For K-means specifically, I am struggling with converting the textual data into integers so they can be consumed by the algorithms.
My question is -

Is it possible to achieve what I want in terms of "intelligently clustering" the 6 rows together?
If so, how do I go about it?

Can you please put your specific question in the title?


text:
Can you use a graph as input for a neural network?
We want to try and distinguish real voices from (deep)fake voices using the graphs generated by a discrete fourier transform (generated from .wav audio files). We know from each image if it is a real or a fake voice, so it's a supervised classification problem. An image would look like this:

We think that real voices generate a graph with clear spikes, whereas fake voices have more noise resulting in less clear spikes. For this reason, we thought of using a CNN to take such an image as input (with x and y-axes ommited), and classify it as real or fake. Our concern is that it's actually a graph and not an image of an object, so we're not sure if this would be a good approach. We could also use the arrays generated from the fourier transform, but we're not sure how we could use that as input as we want to classify if it's real or fake, and not predict y for each x.

There no problem with the use of the data in form of an array to classify, whether the audio belongs to a real or fake voice. Just use 1d convolutional neural network with downsamplings or some global pooling operations, such that in the final layer the temporal extent of the signal has length 1. This would be the logit for binary classification.
However, as far as I understand, you get rid of phase after the Fourier transform, but it can be useful for the prediction. Probably, a better approach would be to use mel_spectrogram https://en.wikipedia.org/wiki/Mel-frequency_cepstrum for this problem.

Thanks! Wouldn't using a 1D CNN require time-series data? As you said, at this point we only have the frequencies vs. amplitude left.
We're aware that using mel spectrograms could(possibly) perform better in such a CNN, but we were wondering if other methods could also work while making sense (so if it makes sense to use graph-like images for a CNN, or if there's some factor(s) as to why not)

A 1D CNN can be used, whenever you want to detect local patterns in one dimensional data. This can be used for time series but also in your example. 2D CNNs are used for images, and you basically have an image with one dimension, where the amplitude describes the pixel value. Therefore 1D CNN would be a good fit I think. You could also try working with recurrent cells (LSTM or GRU), but this will probably be much slower, and might not even work better.


text:
What Are the Use Cases for Machine Learning or even NLP for Bitcoin Market Making?
My Background
Machine learning and software engineering
Got an Offer for
Data and AI engineer in a bitcoin market making firm/exchange.
Context
I already talked to one software engineer in the team, he said the team's major focus is on post trade compliance. I am trying to do more research on this topic, but some pointers on what exact problems constitutes the intersection  between big data, ML and Bitcoin market making will be sincerely appreciated.
It is a new team, and I want to arrive with at least some vague ideas on what kind of problems will be feasible and beneficial to solve, using big data and ML in a bitcoin market making context, assuming we have past tick data and transaction data available. Some ideas I have are basically user clustering (risk hungry, or low risk), or somehow helping institutional traders (who trade using our exchange) on volatility or sentiment etc. But still too vague, so any pointer, blog, tutorial will be helpful.
The answer, of course, need not be exclusively focussed post trade context, any problem ideas or typical data engineering challenges faced by bitcoin market makers will be helpful to understand.

ML models and pipeline designs are highly context specific, thus it's better for one question to focus on one use case. If you have other concerned cases in mind, you'd better ask a new question per SE site rules.

Since handling post-trade compliance in financial applications involves monitoring and analyzing large volumes of transaction data to ensure adherence to regulatory requirements, this task often requires sophisticated and robust ML models capable of detecting rare and various complex anomaly patterns, like the ML use cases of credit card fraud detection. Bear in mind training data for these types of problems are often technically challenging due to several special issues such as class imbalanced data which often needs special calibration or loss reweighting schemes for real world production deployment as discussed in a recent post.
As a start you could first read the brief summary paper of Tiwari et al. (2021) "Credit Card Fraud Detection using Machine Learning: A Study" for some general but important design ideas of your own ML pipelines and models since they're very similar use cases of ML.

In this paper, we present a comprehensive review of various methods used to detect credit card frauds. These methodologies include HMM, Decision Trees, Logistic Regression, Support Vector Machines (SVM), Genetic algorithm, Neural Networks, Random Forests, Bayesian Belief Network... The study shows that the Fraud Bayesian Network Classifier followed by probability threshold is more beneficial than
Naïve Bayes, Tree Augmented Naïve Bayes, Support Vector Machines and Decision Trees on the PagSeguro dataset when precision, recall and economic efficiency are taken into account. Bayesian Learning when taken with Dempster-Shafer Theory resulted in 98% True Positives and less than 10% False Positives... When dataset is highly imbalanced and independent of the rate of frauds Bagging Ensemble Classifier is one of the most stable approaches and has a high fraud catching rate... A lot of financial resources were saved when Cost-Sensitive Decision Tree approach was used.

In summary it's very hard to choose the best ML model for such use cases as each instance may have its own characteristic. For example, for a person with a long enough personal transaction history HMM is well suited for anomaly detection, but some people may just start their transactions without any history. You must consider your use case's own characteristic to choose your model in addition to some common knowledge similar to your problem.


text:
A comprehensive list of machine learning algorithms
I could not find a comprehensive list of machine learning algorithms online.
Can we try to make one here and improve and update it as needed?
There are challenges in the algorithm definitions and how to group and organize them, but my preliminary list (built from e.g. 1, 2, 3), is given in my answer below.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

Supervised learning

ANOVA
Averaged one-dependence estimators
Artificial neural network

Convolutional neural network
Extreme learning machine
Feedforward neural network
Logic learning machine
Long short-term memory
Recurrent neural network
Self-organizing map

Bayesian networks
Boosting
Case-based reasoning
Conditional random field
Decision tree algorithms

C4.5 algorithm
C5.0 algorithm
Chi-squared automatic interaction detection
Classification and regression tree
Conditional decision tree
Decision stump
Decision tree
ID3 algorithm
Iterative dichotomiser 3
Random forest
SLIQ

Ensembles of classifiers

Bootstrap aggregating
Boosting

Gaussian process regression
Gene expression programming
Group method of data handling
Inductive logic programming
Information fuzzy networks
Instance-based learning
K-nearest neighbour
Lazy learning
Learning vector quantization
Linear

Elastic-net
Lasso
Linear discriminant analysis
Linear regression
Logistic regression
Multinomial logistic regression
Naive bayes classifier
Ordinary least squares
Passive aggressive algorithms
Perceptron
Polynomial regression
Ridge regression / classification
Support vector machine

Logistic model tree
Minimum message length

Analogical modelling
Nearest neighbour algorithm

Ordinal classification
Probably approximately correct learning
Quadratic classifiers
Random forests
Ripple down rules
Symbolic machine learning

Semi-supervised learning

Active learning
Co-training
Graph-based methods
Generative models
Low-density separation
Transduction

Unsupervised learning

Association rule learning

Apriori algorithm
Eclat algorithm
FP-growth algorithm

Auto-encoders
Cluster analysis

BIRCH
Conceptual clustering
DBSCAN
Expectation-maximization
Fuzzy clustering
Hierarchical clustering
K-means clustering
K-medians
Mean-shift
OPTICS algorithm
Single-linkage clustering

Dimensionality reduction

Canonical correlation analysis
Dynamic mode decomposition
Factor analysis
Feature extraction
Feature selection
Independent component analysis
Linear discriminant analysis
Multidimensional scaling
Non-negative matrix factorization
Partial least squares regression
Principal component analysis
Principal component regression
Projection pursuit
Sammon mapping
T-distributed stochastic neighbour embedding

Expectation-maximization algorithm
Generative topographic map
Information bottleneck method
Manifold learning
Vector quantization

Reinforcement learning

Deterministic policy gradient
Learning automata
Proximal policy optimization
Q-learning
Soft actor-critic
State–action–reward–state–action
Temporal difference learning
Trust region policy Optimization

Other

Bayesian belief network
Bayesian knowledge base
Deep belief networks
Deep boltzmann machines
Deep neural networks
Discrepancy modelling
Gaussian naive bayes
Generative adversarial network
Hierarchical temporal memory
Knowledge-enhanced machine learning
Markov models
Multinomial naive bayes
Neural style transfer
Physics-informed machine learning
Sparse identification of nonlinear dynamics
Transformer


text:
Identifying rotating and resizing letters with background noise
I'm trying to complete a captcha, and here is what it looks like:

Between captchas the calligraphy of the letters is the same, but the letters may be resized and rotated. And the background noise (the small dots and lines over and around the letters) will be different. Any Hangul letter may appear.
Edit 1: I can generate any number of new captchas with an answer for each of them. But to be clear, the answers that are generated are for entire captchas, that is, multiple Hangul letters arranged in a specific order as the answer for each captcha, not for individual letters.

What type of machine learning is best for this problem?
How do I extract good data from the image above for this problem?

Update 1: Unfortunately no one has given any suggestions for how to solve this yet. My idea at the moment is to mimic the model in this paper: https://www.ics.uci.edu/~xhx/publications/HHR.pdf

Do you have access to the original Captcha generator? That is the best source for training data, as it will generate pairs of images and correct answers. If you don't have access, could you please clarify "academic purposes"? For most academic purposes, it will not matter which precise generator you use, so an answer would point you to use a different generator that you do have access to - you would learn the same things by creating an image processor that could crack a different Captcha . . .

Thank you for your reply. I do have access to the captcha generator and I can generate any number of new captchas with an answer for each of them. But to be clear, the answers that are generated are for entire captchas, that is multiple hangul letters arranged in a specific order as the answer for each captcha, not for individual letters. I edited my post to include this information.


text:
Backpropagation not working as expected
I'm new to neural networks and I try to make a model that is guessing if a point is below or above relative to a function output. The idea is inspired from this video https://youtu.be/DGxIcDjPzac .
What am I doing wrong?
In the gif below I start the training but it seems that is not working. The blue line is the function (y = x + 50) and all the points above it should be green, but aren't. In order to simplify the example and to debug easier, I picked a simple function such that I can use only a perception for the model.
I also made a method backPropagationDebug(...) to display for the points that are predicted wrong all that matrices in each step, but I couldn't find what's wrong.

public void backPropagation(double[][] input, double[][] expected) {
    double[][][] outputs = getOutputs(input);

    double[][] currentOutput = outputs[outputs.length - 1];
    double[][] currentError = Matrix.subtract(expected, currentOutput);

    for (int i = brain.length - 1; i >= 0; i--) {
        final double[][] layer = brain[i];
        final double[][] previousOutput = outputs[i];

        final double[][] layerTranspose = Matrix.transpose(layer);
        final double[][] previousError = Matrix.multiply(layerTranspose, currentError);

        /* FIST BIT */
        double[][] errorSigmoid = Matrix.copyOf(currentError);

        for (int k = 0; k < errorSigmoid.length; k++) {
            errorSigmoid[k][0] *= - derivativeActivationFunction(currentOutput[k][0]);
        }

        /* SECOND BIT */
        final double[][] slopeMatrix = Matrix.multiply(errorSigmoid, Matrix.transpose(previousOutput));

        /* UPDATE THE WEIGHTS */
        for (int k = 0; k < layer.length; k++) {
            for (int l = 0; l < layer[0].length; l++) {
                layer[k][l] = layer[k][l] - learningRate * slopeMatrix[k][l];
            }
        }

        currentOutput = previousOutput;
        currentError = previousError;
    }
}

The backpropagation steps are inspired from this formulas:

(From: Make Your Own Neural Network By Tariq Rashid)
The code is on github: https://github.com/StamateValentin/Artificial-Intelligence-Playground/tree/7a7446b7faedd7673bc53a62304ff3a5180d77eb
The resources I used are in the README.md file.

Please include any mathematics using LaTeX; just pasting images of mathematics is frowned upon.

Welcome to the site. What is your specific question? Please put it somewhere it can be recognized easily.

I added the question.

@Valentin, Thank you, that helps. Also, if you don’t put @[my username] at the beginning of your comment, then I won’t be notified that you responded. (More information about the comment system)


text:
GAN performs worse after 50 epochs than after 2
I am training GAN on SVHN dataset (house numbers in Google Street View images, dimensions: 3x32x32 - 3 color channels).
The problem is that it performs worse after some training (e.g. after 50 epochs) than after only 2.
Could you please check out my code? Maybe you will be able to notice what can I improve.
I have already tweaked betas in ADAM optimizer (it helped a little bit, because before that, with default settings, d_loss went to 0 after 5 epochs). I also added an extra discriminator training step.
You can find the code below:
batch_size=128

# the input data is scaled to a mean of 0.5 and a standard deviation of 0.5:
transform = transforms.Compose([
                transforms.ToTensor(),
                transforms.Normalize(mean=(0.5,), std=(0.5,)),
                transforms.Lambda(lambda x: x.view(-1))])

# wypłaszczanie do loadera (view) albo konwolucyjne
# lambda
train_dataset = SVHN(root=".", split='train', download=True,
    transform=transform)

test_dataset = SVHN(root=".", split='test', download=True,
    transform=transform)

train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, drop_last=True, num_workers=0)
test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=True, drop_last=True, num_workers=0)

# Initialize random noise

def noise(size):
    n = torch.randn(size, 100)
    return n.to(device)

# Define the generator model

class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
                                # take a 100-dimensional input (random noise)
                                nn.Linear(100, 256),
                                nn.LeakyReLU(0.2),
                                nn.Linear(256, 512),
                                nn.LeakyReLU(0.2),
                                nn.Linear(512, 1024),
                                nn.LeakyReLU(0.2),
                                nn.Linear(1024, 3*32*32),
                                nn.Tanh()

                            )

    def forward(self, x): return self.model(x)

# Define the discriminator model

class Discriminator(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
                                nn.Linear(3*32*32, 1024),
                                nn.LeakyReLU(0.2),
                                nn.Dropout(0.3),
                                nn.Linear(1024, 512),
                                nn.LeakyReLU(0.2),
                                nn.Dropout(0.3),
                                nn.Linear(512, 256),
                                nn.LeakyReLU(0.2),
                                nn.Dropout(0.3),
                                nn.Linear(256, 1),
                                nn.Sigmoid()

                            )
    def forward(self, x): return self.model(x)

# define generator training - input is fake data
def generator_train_step(fake_data):
    # reset the gradient so the parameters will update correctly
    g_optimizer.zero_grad()

    # predict the output of the discriminator on fake data
    prediction_fake = discriminator(fake_data)

    # torch.ones, because we want 1s outputted by the discriminator when training the generator
    error = loss(prediction_fake, torch.ones(len(real_data), 1).to(device))
    error.backward()

    g_optimizer.step()
    return error

# define discriminator training:

# discriminator as an input takes real data and fake data
def discriminator_train_step(real_data, fake_data):
    # reset the gradient so the parameters will update correctly
    d_optimizer.zero_grad()

    #print(real_data.shape)
    #print(fake_data.shape)

    prediction_real = discriminator(real_data)

    # calculate loss on real data (expected 1, so torch.ones)
    real_loss = loss(prediction_real, torch.ones(len(real_data),1).to(device))
    real_loss.backward(retain_graph=True)

    prediction_fake = discriminator(fake_data)

    # calculate loss on fake data (expected 0, so torch.zeros)
    fake_loss = loss(prediction_fake, torch.zeros(len(fake_data), 1).to(device))
    fake_loss.backward(retain_graph=True)

    d_optimizer.step()
    return real_loss + fake_loss

lr = 1e-3

discriminator = Discriminator().to(device)
generator = Generator().to(device)
d_optimizer = optim.Adam(discriminator.parameters(), lr=lr,  betas=(0.5, 0.999))
g_optimizer = optim.Adam(generator.parameters(), lr=lr,  betas=(0.4, 0.9))
loss = nn.BCELoss()
num_epochs = 50
log = Report(num_epochs)

for epoch in range(num_epochs):
    N = len(train_loader)
    for i, (images, _) in enumerate(train_loader):
        real_data = images.view(len(images), -1).to(device)
        fake_data = generator(noise(len(real_data))).to(device)
        fake_data = fake_data.detach()
        d_loss = discriminator_train_step(real_data, fake_data)
        fake_data = generator(noise(len(real_data))).to(device)

        d_loss = discriminator_train_step(real_data, fake_data)

        g_loss = generator_train_step(fake_data)

        log.record(epoch+(1+i)/N, d_loss=d_loss.item(), g_loss=g_loss.item(), end='\r')
    log.report_avgs(epoch+1)
    z = torch.randn(10, 100).to(device)
    # 10 pictures of dimensions of 3x32x32:
    sample_images = generator(z).data.cpu().view(10, 3, 32, 32)
    grid = make_grid(sample_images, nrow=4, normalize=True)
    show(grid.cpu().detach().permute(1,2,0), sz=10)
log.plot_epochs(['d_loss', 'g_loss'])

In case you would like to see the results of training this GAN, I enclose an example generated images after epoch 2:
EPOCH: 2.000    d_loss: 4.936   g_loss: 1.919   (130.46s - 3131.14s remaining)))

and after epoch 50:

EPOCH: 50.000   d_loss: 1.253   g_loss: 1.038   (3487.47s - 0.00s remaining))

And here is the plot of discriminator loss and generator loss:

Did you normalize your images to [-1, 1]?

@ArayKarjauv oh, I forgot to paste that part of code. But yes, I used transform of transforms.Compose([ transforms.ToTensor(), transforms.Normalize(mean=(0.5,), std=(0.5,)), transforms.Lambda(lambda x: x.view(-1))]) when loading data.

It is difficult to say what is wrong with your implementation without debugging it, but GANs are known to be very sensitive to each hyperparameter, which makes the training highly unstable. For example, you don't have any normalization layers. Take a look at this paper. It contains some tricks that can improve the training. But I would suggest switching to the Wasserstein GAN or adding $R_1$ regularization


text:
How to output an integer/discrete number n with a single output neuron?
Say I have a game with 4 base actions [left, right, up, down] and then a value n, which determines how many times the chosen action is repeated.
For example, action = left, n = 3 -> go left 3 times. In this game $(left,1)*3 \neq (left,3)$ as negative reward is handed out at every single time step (this is for research purposes, so it cannot change).
I would like to test how a DDQN and a DQN algorithm are affected, as I increase the number of actions available (increase $n$).
My question is; Is there a smarter way to implement this, other than increasing the depth of the output layer? I.e len(output_layer) = n?
I was thinking of whether or not, there was a way for a single neuron to determine n and then have 4 other neurons that determine the best action? Would this even have any positive effects? (such as less training time, faster computation, better generalization, etc.)
If yes, how would this typically be done?

Predicting the correct amount of repetitions for an action sounds like a regression task. Turning it into a classification task using a model with n output nodes will lead to several drawbacks, the biggest ones being:

Having to choose a priori a finite max amount of actions n
Turning the data into really sparse vectors, especially for large n.

So a better choice in my opinion would be to treat this problem as a regression one, i.e. using a single output node trained on predicting continuous values, and round the predictions to convert real values to integers.
Of course the task of choosing an action is a classification one, but it is possible to combine classification and regression into a single model, using Multi Task Learning, the idea being to train a model with one head (input layer) one body (some hidden layers) and 2 tails (output layers), one trained on classification and one trained on regression. The image below is an example of such model (the t subscript in the output layers stands for task). There are several other options to accomplish the same results, so check this survey as well.

Last but not least, if multitask learning is not suitable for you, you could simply train 2 individual models, one for action selection and one for repetition prediction. If going down on this route I would suggest to perform classification first and then repetition prediction, since the output of the classification model would be relevant for the regression one.


text:
How to tell if a model is generative vs. predictive?
How does one tell if a given model is generative AI or predictive AI?
Do generative models have more outputs than inputs and vice versa for predictive models?

It's hard nowadays to draw a distinct line between the two with the advent of conditional generation and normalizing flow
Usually, we say that is generative if you try to model the joint probability $p(x,y)$, whereas discriminative models try to model the marginal $p(y|x)$
However, generative model aims at sampling from such learned distribution, where usually instead discriminative models are only interested in some statistics of it (such as the mode)

I would lead with and emphasize sampling, a lot of sequence generators are predictive models of distribution, plus a separate sampling routine

@NeilSlater yup, though they start from the hypothesis of modelling $p(x)$ and they factor it as $\prod p(x_i | x_{j<i})$, and so it falls back to the usual classification problem


text:
Does 1-bit quantization (layers with boolean tensors) machine learning exist?
Does 1-bit quantization machine learning exist?
Pytorch's docs on "Quantization" define it as:

techniques for performing computations and storing tensors at lower bitwidths than floating point precision.

torch.bool tensors exist with Pytorch, but what's the use for them if most layers use floats? I see the smallest bitwidth besides torch.bool is torch.quint4x2 (unsigned 4-bit integer), but only one sort of layer (nn.EmbeddingBag sparse layer) supports it.

I don't have the time for a longer answer, but essentially no. There is 1.58-bit (ternary) networks, but only for inference (the "forward" pass). You need more bits to do gradient descent (the "backward" pass), otherwise the gradients end up pretty much random. I've tried implementing networks purely with logic gates (e.g. 3-input majority, where one input is the weight), but it couldn't train past a single layer.

@programjames The 1-bit  BitNet model was the precursor to the 1.58 bit models. If you read The Era of 1-bit LLMs paper, where they talk about 1.58 bit models, they clearly reference the original BitNet paper that introduced 1-bit quantizations.

1-bit quantization does exist, at least at the inference stage: a common approach is to constrain weights (and sometimes activations) to be -1 or +1. I'd recommend this survey paper for a good overview.
However, as @programjames mentioned, binarization during training is much more difficult (at least with traditional SGD) as SGD explores through "small and noisy steps", which isn't really possible with such low precision.
During inference, however, binary networks can have some nice properties, like being able to perform convolutions through XNOR and bit-counting operations.

I thought Adam was better than SGD for sparse or noisy gradients.

XNOR-Net seems interesting, as XNOR can be used to find the similarity between bit sequences.

Yes, 1-bit LLMs are a thing (although, technically, they are 1.58 bit LLMs because they use a ternary system using {-1, 0, +1}). This paper explores practical implementations of 1.58 bit LLMs.
1-bit quantization of LLMs is explored in this 2023 paper about BitNet models. They use BitLinear instead of the standard nn.Linear. This git page has the full implementation.


text:
How does the paper implement NEAT without a global set tracking Innovations?
I have been reading this paper on NEAT and trying to implement the algorithm in C#. For the most part, I understand everything in the paper however, there are 2 things I don't understand that confuse me.

In the paper it states:

A possible problem is that the same structural innovation will receive different innovation numbers in the same generation if it occurs by chance more than once. However, by keeping a list of the innovations that occurred in the current generation, it is possible to ensure that when the same structure arises more than once through independent mutations in the same generation, each identical mutation is assigned the same innovation number. Extensive experimentation established that resetting the list every generation as opposed to keeping a growing list of mutations throughout evolution is sufficient to prevent innovation numbers from exploding.

This implies that no global list/set is used to track innovations. If you have a global set to track them, then you wouldn't need to use a list during the evolution of the current generation because as soon as an innovation is created, it would be added to the set. This would be seen by the next evaluated Net/Genome.
From what I have read everyone uses a global list to track the innovations. This makes sense to me. I am just very confused as to how they did it in the paper considering they did extensive testing to figure out they only need a list for the evaluation of the current generation.


text:
Is "node embedding" in GNN analogous to "hidden layer" of FFN?
So in Graph Neural Network (GNN) we have node embeddings which is a feature vector that describes the node, is it analogous to hidden layer of Artificial neural network such as feed-forward neural network? After all hidden layer stores set of weights and biases to extract some "feature", which begs the question if node embeddings are analogous to a hidden layer?

In my understanding embedding is a static vector, but hidden layer is a function. But this is a good question as hidden layer is indeed extracting a feature, while node embedding represents features of the node.

Embeddings are vectors. Layers are functions.
So, node embeddings (e.g. produced by TransE) are analogous to word embeddings or code embeddings, i.e. they are vector (and lower-dimensional) representations of the original objects, such that similar objects have similar embeddings, for some notion of similarity.
Your question is analogous to "Are convolutional layers analogous to feature maps?". So, you could use neural networks with layers to learn embeddings, but this does not have to be the case. For example, TransE does not "really" use a neural network (i.e. a sequence of layers that compute a non-linear function of the inputs), but it just optimizes the $L_2$ norm of a simple function of the vector representation of the nodes and edges, although it uses gradient descent.

makes so much sense, so that means embeddings are linked to model they are learned with? I thought embeddings are more universal as a space that separate two dissimilar things but here it appears that to use the embedding we need, the network on which it has been learned on, as well


text:
Alpha beta pruning - rules for updating alpha/beta value
I have been working on a problem to which I've applied alpha-beta pruning. While I got most of the answers right, there is one part I'm not quite getting:

Note that I've only provided a part of the tree I'm working on. Node $B$ starts with the following values:
$B$

$v = \infty$
$\alpha = - \infty$
$\beta = \infty$

Now, we push the alpha and beta values down to node $D$ from parent node $B$, and calculate its value:
$D$

$v = -\infty$
$\alpha = -\infty$
$\beta = \infty$

As leaf node $J$ has a value of $-7$, we push that back up to parent node $D$, changing node $D$'s value to $-7$ (as it is better than the old value of $-\infty$), and we also change the $\alpha$ value of node $D$ (as it is also better than the old value of $-\infty$).
New $D$

$v = -7$
$\alpha = -7$
$\beta = \infty$

We now push the value of $-7$ back up to parent node $B$, changing node $B$'s value to $-7$ (as it is better than the old value of $\infty$), and we also change the $\beta$ value of node $B$ (as it is also better than the old value of $\infty$).
New $B$

$v = -7$
$\alpha = -\infty$
$\beta = -7$

We now traverse down to node $E$ (and we don't prune it because node $B$'s value is NOT <= its $\alpha$ value), and we push the alpha and beta values down from node $B$, and calculate its value:
$E$

$v = -\infty$
$\alpha = -\infty$
$\beta = -7$

As leaf node $K$ has a value of $0$, we push that back up to parent node $E$, changing node $E$'s value to $0$ (as it is better than the old value of $-\infty$). Now, this is the point where my confusion lies. According to my understanding, at this point we would also set the $\alpha$ value of node $E$ to $0$ (as it is better than the old value of $-\infty$). However, the answer I received to this question specifies that we do NOT change the $\alpha$ value of node $E$, and rather leave it as $-\infty$.
Can someone please explain to me why this is the case?
UPDATE
I did not originally include the full subtree - this is it:

In this instance, only node M should be pruned. However, my question still stands as to why the answer did not update the alpha value of node E, as no pruning happened in that part of the tree.

First, allow me to draw it for better visualization:
1. (α=-∞,β=∞) from B ➡ D

             B (α=-∞,β=∞)
          ↙ / \
(α=-∞,β=∞) D   E
           |   |
        -7=J   K=0
--------------------------
2. (v=-7) J ➡ D α=max(-7,-∞)=-7

             B (α=-∞,β=∞)
α=max(-7,-∞)/ \
(α=-7,β=∞) D   E
      ↖    |   |
        -7=J   K=0
--------------------------
3. (α=-7) D ➡ B β=min(∞,-7)=-7

               β=min(∞,-7)
             B (α=-∞,β=-7)
         ↗  / \
(α=-7,β=∞) D   E
           |   |
        -7=J   K=0
--------------------------
4. (α=-∞,β=-7) from B ➡ E

             B (α=-∞,β=-7)
            / \       ↓
(α=-7,β=∞) D   E (α=-∞,β=-7)
           |   |
        -7=J   K=0
--------------------------

5. (v=0) K ➡ E β=min(0,-7)=-7

             B
            / \ β=min(0,-7)
(α=-7,β=∞) D   E (α=-∞,β=-7)
           |   |    ⬈?
        -7=J   K=0
--------------------------

Update: I found this simulator. It behaves exactly like you describe:
http://homepage.ufp.pt/jtorres/ensino/ia/alfabeta.html

Enter Tree Structure: 2 3 1 1 1 1 1
Enter Values -7 0 -4 -10

I noticed that it tries to update β on node K->E. As β=min(0,-7), it won't change.
It's possible to check their internal code by inspecting the page, to debug even further.

Thanks a lot for the explanation Andre! I have taken note in my textbook that if a node will be cut, you don't have to update the alpha and beta values, but in this case, E is not being pruned. I will edit the post with some more information - apologies for not clarifying

OK, I'll check your updates later.

Updated with an interactive live example with the exact behavior as you stated.


text:
Convolutional Neural Network (CNN) with Tree architecture to organize the number of classes
At the moment, I have around 1.000 classes with accuracy and loss that are acceptable. In the long term, there could be more than 100.000 classes. The main problem is that every time a new class is needed, the model needs to be rebuilt.
For this, I made a POC with a Siamese Network with the goal that new classes can be added without the need to rebuild. The results were not what I expected, and probabilities are a must. As far as I know, this could not be done with this network. The conclusion was that this was not the best option for this case.
Before I start implementing, I would appreciate some feedback and second opinion on the following architecture:
The next thing I would do is build a hierarchy chain of CNN’s. The structure is already available in a database and I could automate the build of the CNN’s to a certain level.
The first CNN could have 4 “main” classes. Based on the probability, the next layer will be determined.
Then the second CNN would have 50 to 200 classes. Based on the probability, the next layer will be determined as well.
Then the last layer would be a CNN with up to 1.000 classes. In case there are more, this could be divided even further.
This way, I could gradually build up the model without the need to rebuild everything (last layer). And the first and second layer only needs to be rebuilt if the accuracy and probability start dropping.
I found a paper with a similar proposal, but could not find feedback or experiences of others. Is this something that is feasible? What could be the problems I will face with a structure like this? Or would you tackle this problem in another way?

How about embeddings instead of classes? I don't know what problem you are working on, but this approach can be applied to NLP or images, where you first extract the latent features using an encoder and then compare those vectors, for example using cosine similarity.


text:
What exactly is the population in the problem of finding the best path in a network of nodes using genetic algorithms?
I have 17 nodes in my network with 3000 different paths in total. I have to select the path with highest available bandwidth, using genetic algorithm. I'm confused about the approach! Should I have all paths as the population, or should I create a population same size as the nodes(17).

Should I have all paths as the population,

No, this is not usually possible for more realistic problems where a population that covered all possibilities would be far too large to manage.

or should I create a population same size as the nodes(17).

No, there is no need to link the population size to other properties of the problem so directly.
If your path must pass through all 17 nodes (like a travelling salesman problem) then your genome coding might have 17 elements to it, and could simply be the path through the nodes. That's not the only way to address even the TSP, and may not be the case here. However, I mention it because it is common that numerical features of the problem will influence the design of the genomes.
The population size is a hyperparameter for the solution, along with mutation rate, rules for recombination etc. It is something you will want to experiment with.
With 3000 combinations to assess, a direct search of all combinations would be fast and effective (and probably easier to code). My understanding is therefore that this is a learning exercise. Your eventual goal might be to have the genetic algorithm find a good solution with less than 3000 evaluations of the path. Finding a good solution in any number of iterations is also a reasonable start to demonstrate you have understood the basics of genetic algorithms.


text:
Is it possible to train an RL agent using images?
I have an image which consists of a start and an end point, the journey has some obstacles which have to be avoided.

Is it possible to train an RL agent using such images to find the best path avoiding objects.
Or what algorithm should be used in order to find the best path avoiding objects, where the input is an image.

For example a picture of a person on a field track and there are obstacles in between from the start to the end point. I want to predict the series of actions that are required to reach the final position.

Could you give an example of the image? There are some good basic pathfinding algorithms that don't need machine learning, but that do need clean data to operation. If we are dealing with natural photos here, you may have a couple of choices, depending on what other data you have.

Also, for RL you will need well-defined actions, and be able to follow through with them to get feedback on state. For any machine learning at all, you will need some way to collect and/or assess data for correctness. It is not clear from your question what your experiment set up is, so which methods you have a good chance to apply. Could you please explain more about your task - e.g. if the person is in one place, and receives an instruction from the AI, what happens next, what feedback does AI get (in training, and for finished model or agent)

Alright the main problem statement I have is an image of a road with potholes, if I could somehow add some variables to indicate the position of the potholes to the state space of the environment, would i be able to train an agent to avoid those pot holes using a generic action space of up, down, left and right.

If possible I would like to map the image to a grid of a fixed size so as to map the position of the agent according to the grid co-ordinates which would act as the state space, presence of a pothole etc. The reward function would then be decided accordingly.

From your description I think Edoardo's answer is a reasonable fit. I am still not 100% certain about the details - I am not sure if your output is a plan for multiple actions starting from a single photo, or something more interactive. Either way, you will need some way to simulate actions being taken in the environment - at the very least for training, and possibly for real if the problem is for planning multiple actions in advance based on a single photo.

A quick search about reinforcement learning applied to video games will lead you to countless tutorials that describe exactly what you're asking for.
With images the way to go is usually deep reinforcement learning. A convolutional neural network (or any other deep learning architecture) is used to process the image and compress it to a latent vector used as the "environment" seen by the agent.
Given that you can then apply whatever reinforcement learning algorithm (sarsa, q-learning, monte carlo tree search, etc.) to train the agent itself on a specific task, in this case reaching the end pixels area without hitting obstacles.
If you're familiar with python a good starting point is OpenAI Gym, and I would say in particular the Super Mario tutorial, since conceptually the game is basically the same as you're task of interest.

Alright, Thanks for the information. I shall look into this!!


text:
Why is Lisp such a good language for AI?
I've heard before from computer scientists and from researchers in the area of AI that that Lisp is a good language for research and development in artificial intelligence.

Does this still apply, with the proliferation of neural networks and deep learning?
What was their reasoning for this?
What languages are current deep-learning systems currently built in?

First, I guess that you mean Common Lisp (which is a standard language specification, see its HyperSpec) with efficient implementations (à la SBCL). But some recent implementations of Scheme could also be relevant (with good implementations such as Bigloo or Chicken/Scheme). Both Common Lisp and Scheme (and even Clojure) are from the same Lisp family. And as a scripting language driving big data or machine learning applications, Guile might be a useful replacement to Python and is also a Lisp dialect. BTW, I do recommend reading SICP, an excellent introduction to programming using Scheme.
Then, Common Lisp (and other dialects of Lisp) is great for symbolic AI. However, many recent machine learning libraries are coded in more mainstream languages, for example TensorFlow is coded in C++ & Python. Deep learning libraries are mostly coded in C++ or Python or C (and sometimes using OpenCL or Cuda for GPU computing parts).
Common Lisp is great for symbolic artificial intelligence because:

it has very good implementations (e.g. SBCL, which compiles to machine code every expression given to the REPL)
it is homoiconic, so it is easy to deal with programs as data, in particular it is easy to generate [sub-]programs, that is use meta-programming techniques.
it has a Read-Eval-Print Loop to ease interactive programming
it provides a very powerful macro machinery (essentially, you define your own domain specific sublanguage for your problem), much more powerful than in other languages like C.
it mandates a garbage collector (even code can be garbage collected)
it provides many container abstract data types, and can easily handle symbols.
you can code both high-level (dynamically typed) and low-level (more or less startically typed) code, thru appropriate annotations.

However most machine learning & neural network libraries are not coded in CL. Notice that neither neural network nor deep learning is in the symbolic artificial intelligence field. See also this question.
Several symbolic AI systems like Eurisko or CyC have been developed in CL (actually, in some DSL built above CL).
Notice that the programming language might not be very important. In the Artificial General Intelligence research topic, some people work on the idea of a AI system which would generate all its own code (so are designing it with a bootstrapping approach). Then, the code which is generated by such a system can even be generated in low level programming languages like C. See J.Pitrat's blog, which has inspired the RefPerSys project.

I'd like to add here, that tensorflow and theano are based on computational graphs, which is actually symbolic programming, but within imperative language.

David Nolen (contributor of Clojure and ClojureScript; creator of Core Logic a port of miniKanren) in a talk called LISP as too powerful stated that back in his days LISP was decades ahead of other programming languages. There are number of reasons why the language wasn't able to maintain its initial reputation.
This article highlights some key points why LISP is good for AI

Easy to define a new language and manipulate complex information.
Full flexibility in defining and manipulating programs as well as data.
Fast, as the program is concise along with low-level detail.
Good programming environment (debugging, incremental compilers, editors).

Most of my friends in this field usually use Matlab for Artificial Neural Networks and Machine Learning. It hides the low-level details though. If you are only looking for results and not how you get there, then Matlab will be good. But if you want to learn even low-level detailed stuff, then I will suggest you go through LISP at-least once.
The language might not be that important if you have an understanding of various AI algorithms and techniques. I will suggest you read "Artificial Intelligence: A Modern Approach (by Stuard J. Russell and Peter Norvig". I am currently reading this book, and it's a very good book.

AI is a wide field
AI is a wide field that goes far beyond machine learning, deep learning, neural networks, etc. In some of these fields, the programming language does not matter at all (except for speed issues), so using Lisp would certainly not be an issue there.
Planning systems
In search or AI planning, for instance, standardised and commonly used languages, like C++ and Java, are often the first choice, because they are fast (in particular C++) and because many software projects like planning systems are open source, using a commonly used language is likely to get more feedback or contributions. On the other hand, Common Lisp ecosystem is welcoming, the language is also standardised, and there is a LONG history in AI research using Lisp.
I am only aware of one single planner that is written in Lisp. Just to give some impression about the role of the choice of the programming language in this field of AI, I'll give a list of some of the best-known and therefore most-important planners:
Fast-Downward
This is probably the best-known classical planning system, which is written in C++ and some parts (pre-processing) in Python.
Fast-Forward
Together with Fast-Downward, this is the classical planning system everyone knows. It's written in C.
VHPOP
This is one of the best-known partial-order causal link (POCL) planning systems. It's written in C++.
SHOP and SHOP2
This is the best-known HTN (hierarchical) planning system. There are two versions: SHOP and SHOP2. The original versions have been written in Lisp. Newer versions (called JSHOP and JSHOP2) have been written in Java. Pyshop is a further SHOP variant written in Python.
PANDA
This is another well-known HTN (and hybrid) planning system. There are different versions of the planner, PANDA1 and PANDA2, which are written in Java. PANDA3 is written primarily in Java, with some parts being in Scala.
These were just some of the best-known planning systems that came to my mind. More recent ones can be retrieved from the International Planning Competitions (IPCs), which take place every two years. The competing planners' codes are published open-source (for a few years).

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What would motivate a machine?
Currently, within the AI development field, the main focus seems to be on pattern recognition and machine learning. Learning is about adjusting internal variables based on a feedback loop.
Maslow's hierarchy of needs is a theory in psychology proposed by Abraham Maslow that claims that individuals' most basic needs must be met, before they become motivated to achieve higher-level needs.

What could possibly motivate a machine to act?
Should a machine have some sort of DNA-like structure that would describe its hierarchy of needs (similar to Maslow's theory)?
What could be the fundamental needs of a machine?

Interesting question, and welcome to AI!  (I have a few thoughts on the subject, related to game theory, and other contributors have talked about goal oriented learning in relation to algorithms.)

The current method to implement motivation is some kind of artificial reward. Deepmind's DQN for example is driven by the score of the game. The higher the score, the better. The AI learns to adjust its actions to get the most points and therefore the most reward. This is called reinforcement learing. The reward motivates the AI to adapt its actions, so to speak.
In a more technical term, the AI wants to maximize utility, which depends on the implemented utility function. In the case of DQN, this would be maximizing the score in the game.
The human brain functions in a similar fashion, although a little more complex and often not as straight forward. We as humans usually try to adjust our actions to produce a high output of dopamine and serotonin. This is in a way similar to the reward used to control AIs during reinforcement learning. The human brain learns which actions produce the most amount of those substances and finds strategies to maximize the output. This is, of course, a simplification of this complex process, but you get the picture.
When you talk about motivation, please don't mix it up with consciousness or qualia. Those are not required for motivation at all. If you want to discuss consciousness and qualia in AI, that's a totally different ball game.
A child isn't curious for the sake of curiosity. It gets positive reinforcement when exploring because the utility function of the child's brain rewards exploration by releasing rewarding neurotransmitters. So the mechanism is the same. Applying this to AI means defining a utility function that rewards new experiences. There is no inner drive without some kind of reinforcing reward.

in regards to the edit i think a good example of "a utility function that rewards new experience" would be the novelty search fitness functions proposed by ken stanley to be used in his neat algorithm.

This is an interesting question actually.
There's a quite realistic idea about "where can the curiosity originate from" in the book "On intelligence" written by Jeff Hawkins and Sandra Blakeslee.
It's based on such statements:

Mind creates its own model of the world it exists in.
It makes predictions about everything all the time (actually Jeff Hawkins states that this is the main characteristic of intelligence).
When prediction about something wasn't followed by appropriate behavior of the world, then this thing gets very interesting to the mind (the model is wrong and should be corrected) and needs more attention.

For example, when you look at left human eye your brain predicts that it's a human face and there should be second eye to the right. You look to the right and see a.. nose! What a surprise! It now takes all your attention and you have this motivation to make more observations about such a strange thing that did not fit into your model.
So I'd say that AI might do something certain according to its model or behave randomly while the predictions it is making about the world are true. But once some prediction is broken the AI gets a motivation to do error-correction to its model.
In a simple case a machine starts at a total randomness just doing everything it can with its output. While it has no model or a random model when it detects some kind of order or repeated patterns it is getting "interested" and adds it to the model. After a while, the model becomes more sophisticated making more complex predictions and detecting higher level mistakes in a model. Slowly it gets to know what to do to observe something interesting to it, instead of just remembering everything.

Thank you for the contribution! I have come to basically the same conclusions... now thinking of a way to implement it :)

This answer makes an important point. Error correction on prediction models would provide a great incentive for an intelligent AI to learn and act in curious manner.

I asked professor Richard Sutton a similar question, in the first lecture of the reinforcement learning course. It seems that there are different ways to motivate the machine. In fact, machine motivation seems to me like a dedicated field of research.
Typically, machines are motivated by what we call an objective function or a cost function or a loss function. These are different names for the same concept. Sometimes, they are denoted by
$$L(a)$$
The goal of the machine is then to solve either a minimization problem, $\min_a L(a)$, or a maximization problem, $\max_a L(a)$, depending on the definition of $L$.

I've spent some time thinking about this in the context of games.
The problem with reward functions is that they generally involve weighting nodes, which is useful but ultimately materially meaningless.
Here are two materially meaningful rewards:
COMPUTATIONAL RESOURCES
Consider a game where an AI is competing not for points, but for processor time and memory.
The better the algorithm performs at the game, the more memory and processing it has access to.  This has a practical effect--the more resources available to the automata, the stronger its capabilities.  (i.e. it's rationality is less bounded in terms of time and space to make a decision.)  Thus the algorithm would be "motivated" to prevail such a contest.
ENERGY
Any automata with a sufficient degree of "self awareness", here specifically referring to the knowledge that it requires energy to process, would be motivated to self-optimize its own code to eliminate unnecessary flipping of bits (unnecessary energy consumption.)
Such an algorithm would also be motivated to ensure its power supply so that it can continue to function.

I think we give ourselves too much credit by already referring to our algorithms and machines as actually thinking and acting on motivations. In my opinion we still have a bit to go before we can actually refer to a human creation as thinking or being able to have motivations more then basic physical ones.
By that I would say that a Machines' or AI algorithms' motivations are similar to a car engine. Simple and basic, the "motivations" of a car engine to run are just the first and second laws of thermodynamics, namely the conservation of energy and the exchange between energy types, and the always increasing level of entropy in a closed system.
By having a really specific design, we can insert fuel in the system and create a lot of potential energy which will "motivate" the engine to transform it in other types of energy (heat, sound, etc.)
An AI algorithm is exactly the same, it's just that now we're playing with electricity. By putting multiple levels of abstractions from the actual level of electrons moving through wire, up to your python Deep Learning algorithm training to learn how to recognize images of dogs. The concept is similar in my opinion, for now we do not have machines that are complex enough to have higher-level motivations, or even develop them by themselves.
As the other answers pointed out, specific algorithms, namely reinforcement learning try to emulate those "needs" and "motivations", but in the end in my opinion, for now, they are still just emulations. Similar to other Deep learning algorithms, the same basic concept described at the beginning applies, trying to minimize the error by emulating different concepts that we know, as conservation of energy, following the path of least resistance, maintaining the laws of entropy, etc.

Good verses bad?  From an imaging standpoint, humans/biologicals tend to notice details not “imagined”.  A “good” feedback learning loop might be refining the correctness of the imagined. Using efforts to reduce the number or amount discrepancies between imagined or predicted environment and perceived reality as measured against “beneficial” outcomes.  I’m not sure what a Ai wood think of as good/bad in the abstract, in my limited considerations of imagination in humans, I seem most useful in recognizing things that be mor meaning than the general background.  As the imagination becomes more able to predict a more “real” outcome, it is more useful to the individual in predicting a more correct outcome and greater usefulness in defining things that deverge form the expected.  Sort simplistic I realize, but intelligence might be as simple as refining.the ability to imagine a result and judging the more meaningful of the descrpencies .

Good/Bad are entirely human concepts, humanity lived for millions of years without those, so there may not be useful words in evaluating... "urge for action", these concepts are a higher level of abstraction


text:
What is the Bellman operator in reinforcement learning?
In mathematics, the word operator can refer to several distinct but related concepts. An operator can be defined as a function between two vector spaces, it can be defined as a function where the domain and the codomain are the same, or it can be defined as a function from functions (which are vectors) to other functions (for example, the differential operator), that is, a high-order function (if you are familiar with functional programming).

What is the Bellman operator in reinforcement learning (RL)?
Why do we even need it?
How is the Bellman operator related to the Bellman equations in RL?

A few papers related to this topic are Feature-Based Methods for Large Scale Dynamic Programming (by John N. Tsitsiklis and  Benjamin Van Roy, 1996), An Analysis of Temporal-Difference Learning with Function Approximation (by John N. Tsitsiklis and  Benjamin Van Roy, 1997) and Least-Squares Policy Iteration (by Michail G. Lagoudakis and Ronald Parr, 2003).

A few more related papers that I found are Generalized Markov Decision Processes: Dynamic-programming and Reinforcement-learning Algorithms (by Csaba Szepesvári and Michael L. Littman, 1997) and $\epsilon$-MDPs: Learning in Varying Environments (by István Szita, Bálint Takács, András Lörincz, 2002).

The notation I'll be using is from two different lectures by David Silver and is also informed by these slides.
The expected Bellman equation is
$$v_\pi(s) = \sum_{a\in \cal{A}} \pi(a|s) \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_\pi(s')\right) \tag 1$$
If we let
$$\cal{P}_{ss'}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{P}_{ss'}^a \tag 2$$
and
$$\cal{R}_{s}^\pi = \sum\limits_{a \in \cal{A}} \pi(a|s)\cal{R}_{s}^a \tag 3$$
then we can rewrite $(1)$ as
$$v_\pi(s) = \cal{R}_s^\pi + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^\pi v_\pi(s') \tag 4$$
This can be written in matrix form
$$\left.
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}=
\begin{bmatrix}
\cal{R}_1^\pi \\
\vdots \\
\cal{R}_n^\pi
\end{bmatrix}
+\gamma
\begin{bmatrix}
\cal{P}_{11}^\pi & \dots & \cal{P}_{1n}^\pi\\
\vdots & \ddots & \vdots\\
\cal{P}_{n1}^\pi & \dots & \cal{P}_{nn}^\pi
\end{bmatrix}
\begin{bmatrix}
v_\pi(1) \\
\vdots \\
v_\pi(n)
\end{bmatrix}
\right. \tag 5$$
Or, more compactly,
$$v_\pi = \cal{R}^\pi + \gamma \cal{P}^\pi v_\pi \tag 6$$
Notice that both sides of $(6)$ are $n$-dimensional vectors. Here $n=|\cal{S}|$ is the size of the state space. We can then define an operator $\cal{T}^\pi:\mathbb{R}^n\to\mathbb{R}^n$ as
$$\cal{T^\pi}(v) = \cal{R}^\pi + \gamma \cal{P}^\pi v \tag 7$$
for any $v\in \mathbb{R}^n$. This is the expected Bellman operator.
Similarly, you can rewrite the Bellman optimality equation
$$v_*(s) = \max_{a\in\cal{A}} \left(\cal{R}_s^a + \gamma\sum_{s' \in \cal{S}} \cal{P}_{ss'}^a v_*(s')\right) \tag 8$$
as the Bellman optimality operator
$$\cal{T^*}(v) = \max_{a\in\cal{A}} \left(\cal{R}^a + \gamma \cal{P}^a v\right) \tag 9$$
The Bellman operators are "operators" in that they are mappings from one point to another within the vector space of state values, $\mathbb{R}^n$.
Rewriting the Bellman equations as operators is useful for proving that certain dynamic programming algorithms (e.g. policy iteration, value iteration) converge to a unique fixed point. This usefulness comes in the form of a body of existing work in operator theory, which allows us to make use of special properties of the Bellman operators.
Specifically, the fact that the Bellman operators are contractions gives the useful results that, for any policy $\pi$ and any initial vector $v$,
$$\lim_{k\to\infty}(\cal{T}^\pi)^k v = v_\pi \tag{10}$$
$$\lim_{k\to\infty}(\cal{T}^*)^k v = v_* \tag{11}$$
where $v_\pi$ is the value of policy $\pi$ and $v_*$ is the value of an optimal policy $\pi^*$. The proof is due to the contraction mapping theorem.

why the limit given by applying the bellman optimal operator is the value function of the optimal policy?

@piero:  good question.  The operator T acts on the Value function V(s) which is itself a function over states.  The operator T acting on V gives the V of the next state.  If T keeps acting on an initial V ad infinitum, then it gives the V of the "final state", which is the end value of the entire policy.

Actually the domains of T is $\mathbb{R}^{S \times A} \rightarrow \mathbb{R}^{S \times A}$  where S = state space, A = action space.