text:
understanding the distribution shift problem in direct preference optimization (DPO)
I'm having trouble understanding this paragraph of the DPO paper:

Why does it matter so much that the preference data distribution aligns with the reference model output distribution? My understanding is that during training, the parameters of the sft (supervised fine-tuning) are updated such that chosen responses ($y_w$) have a higher probability of being generated, and rejected responses ($y_l$) have a lower probability of being generated, and the reference model is just there to prevent the sft model from straying too far from the original parameters. But I fail to understand how the wrong reference distribution could hinder this process. Could someone please help me?

I had a problem concerning this and I think I can shed some light onto your question.
We had a dataset of preferences generated for a really bad model, so naturally $y_w$ and $y_l$ were bad responses. If we were to use such preferences to train an already good SFT model, we would be looking into $\pi(y_w\vert x)$ or $\pi(y_l\vert x)$ for our good model, which is probably an extremely small number!
In a sense we are telling it "you should prefer to generate this bad answer rather than this atrocious one". This isn't helpful, and in my opinion will probably make the model worse.
This can also happen if the model that the DPO data is coming from is radically different from the one we're fine tuning. If they produce really different text, $\pi(y_w\vert x)$ will again be extremely small, and we'd be comparing sentences that our network is highly unlikely to produce.
Because of this I think it's highly desirable to evaluate responses coming from the same distribution, or at least a "similar" one. Hope this clarifies stuff for you.


text:
Does the fixed context in attention mechanism is accquired after getting the decoder hidden layer of the first hidden state?
here, the fixed context vector (ci) is used for the decoder model, why the decoder model also used by the attention weights. On the first (c1), does that mean the decoder does not have context ? (i = 1 ; c1 - 1 = c0). Thanks

references : Jurafsky, D., & Martin, J. H. (2023). Speech and language processing: An Introduction to Natural Language Processing(pp.205-208), (3rd ed. draft), Computational Linguistics, and Speech Recognition. Prentice Hall.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What are examples of simple gradient based NLP models?
I am looking for a simple example of gradient-based methods for NLP. More specifically I am looking for post-hoc local explanations gradient-based methods, that is to say, which explain a single prediction by performing additional operations (after the model has emitted a prediction). Here is an example of a gradient-based method for "Finance"? (I'm not very sure of what domain it is as it is not NLP, nor Computer Vision, nor time series ...):

Let us consider a minimal example, where a linear regression is used to estimate the future capital asset $y_{c}$, based on two investments $x_{1}$ and $x_{2}$. Let assume the assumptions above are met and the model parameters are estimated as follows:
$$\mathbb{E}\left[y_{c} \mid x_{1}, x_{2}\right]=1.05 x_{1}+1.50 x_{2},$$
We can derive immediately a global interpretation of this model. Every dollar invested in fund $x_{1}$ will produce capital of $1.05 \\\$$, while every dollar invested in $x_{2}$ will produce a capital of $1.50 \\\$$, independently of the values $x_{1}$ and $x_{2}$ might assume in a concrete scenario. Notice that this explanation is purely based on the learned coefficient $w_{1}=1.05$ and $w_{2}=1.50$. These, sometimes called partial regression coefficients, are themselves candidate attribution values to explain the influence of the independent variables of the target variable:
$$R_{1}(x)=1.05 \quad R_{2}(x)=1.50$$
Notice also that the coefficients are the partial derivatives of the target variable with respect to the independent variable, therefore this attribution is nothing but the model gradient.

I thought about transformating it to an NLP example like:
Let us consider a minimal example, where a linear regression is used to estimate the toxicity of a text $y_{c}$, based on the occurence of words $x_{1}$ and $x_{2}$. Let's assume the assumptions above are met and the model parameters are estimated as follows:
$$\mathbb{E}\left[y_{c} \mid x_{1}, x_{2}\right]=1.05 x_{1}+1.50 x_{2},$$
We can derive immediately a global interpretation of this model. Every occurrence of word $x_{1}$ will produce a toxicity of $1.05$, while every occurrence of word in $x_{2}$ will produce a capital of $1.50$, independently of the values $x_{1}$ and $x_{2}$ might assume in a concrete scenario. Notice that this explanation is purely based on the learned coefficient $w_{1}=1.05$ and $w_{2}=1.50$. These, sometimes called partial regression coefficients, are themselves candidate attribution values to explain the influence of the independent variables of the target variable
That look pretty dumb, isn't it? Do you have any better example
Update : does the ablation model use gradient?
So it seems my model isn't so dumb, but Ant gave the idea of using the ablation model. I didn't know about this technique but it seems straightforward: you iterate over your text and get rid of a given word in part of the text and see how much the text gains or loses toxicity. It looked very gradient-friendly.
However, I tried to create an ablation algorithm that looks at the k next words and tries to get the toxicity of the word by doing the difference between the text predicted toxicity $\hat y$ and the same one when the text is being ablated $\hat y_{ablated}$. I inspired myself from Warren Freeborough's example for time series in his paper page 5.

I get an en error, or sensibility, matrix $E_{avg}$. But I don't see where any gradient has been used. At most I could say that the model can use gradient. For instance, if it is an RNN, it uses a gradient method, isn't it? But I don't see where ablation uses gradient.

Why does it look dumb? If you think the model makes sense, then the explanation seems reasonable to me.
I would not call this gradient-based explanation, however, since the features in this case are discrete, not continuous (you cannot take a   derivative wrt to $x_1$ and $x_2$, if those are discrete features)
You can however look at ablations (i.e. change in scores if the word is not present), which would be the analogous of gradients for discrete features. The interpretation is: if that word is present, the score increases by this much. Why has this text been classified as toxic? Because it contains the following toxic words, each sorted by their "toxicity".
Of course that would be a very simple model; words do not occur in isolation and looking at their mere presence completely disregard the context in which they appear. However, that's a problem of the model, not of the explanation of the model.
The more sophisticated models these days use as base architecture the transformer, which is able to differentiate words based on the context in which they occur. There are some explanatory models for transformer-based architectures, but they will not simply assign a numerical score to a word (as that cannot accurately reflect the predictions of the model, since it ignores context).

Thanks for your comment, Ant! However I don't get how "gradient-friendly" the ablation model would be. The change in scores, if the word is not present, would be very "binary" and therefore not very differentiable. Unless you "ablate" word after word and look at the change in the score? I have another question, do you know which NLP methods use gradient and activation methods?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Examples of self-explainable models used in NLP other than prototype-based
I am looking for all the methods used in NLP that are self-explainable, or explainable by design. That is to say, the ones that use the predictive model itself to explain the entire model's predictive reasoning (a.k.a. directly interpretable model).
I have only a few example in mind:

prototype-based methods, models that reveal the network's reasoning by providing prototypical explanations. It may allow users to directly manipulate a network component according to their preferences like Rasa or, more recently, Proto-Trex.
Human-in-the-loop ML, where the co-creation allows to interpret and update the model logic directly by enabling interaction with rule generators themselves. But as the first of these are created with deep learning, is it truly part of explainable AI? An example of this is HEIDL, which did reference to related work such as Chorus


text:
Torch CNN not training
I am completely new to CNN's, and I do not quite know how to design or use them efficiently. That being said, I am attempting to build a CNN that learns to play Pac-man with reinforcement learning. I have trained it for about 3 hours and have seen little to no improvement. My observation space is 3 channels * 15 * 19, and there are 5 actions. Here is my code, I am open to any and all suggestions. Thanks for all your help.
from minipacman import MiniPacman as pac
from torch import nn
import torch
import random
import torch.optim as optimal
from torch.autograd import Variable
import matplotlib.pyplot as plt
import numpy as np
import keyboard

loss_fn = nn.MSELoss()
epsilon = 1
env = pac("regular", 1000)
time = 0
action = random.randint(0, 4)
q = np.zeros(3)
alpha = 0.01
gamma = 0.9
tick = 0
decay = 0.9999

class Value_Approximator (nn.Module):
    def __init__(self):
        super(Value_Approximator, self).__init__()
        # Convolution 1
        self.cnn1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=5, stride=1, padding=2)
        self.relu1 = nn.ReLU()

        # Max pool 1
        self.maxpool1 = nn.MaxPool2d(kernel_size=2)

        # Convolution 2
        self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=2)
        self.relu2 = nn.ReLU()

        # Max pool 2
        self.maxpool2 = nn.MaxPool2d(kernel_size=2)

        # Fully connected 1 (readout)
        self.fc1 = nn.Linear(384, 5)

    def forward(self, x):
        # Convolution 1
        out = self.cnn1(x)
        out = self.relu1(out)

        # Max pool 1
        out = self.maxpool1(out)

        # Convolution 2
        out = self.cnn2(out)
        out = self.relu2(out)

        # Max pool 2
        out = self.maxpool2(out)

        # Resize
        # Original size: (100, 32, 7, 7)
        # out.size(0): 100
        # New out size: (100, 32*7*7)
        out = out.view(out.size(0), -1)

        # Linear function (readout)
        out = self.fc1(out)

        return out

approx = Value_Approximator()
optimizer = optimal.SGD(approx.parameters(), lr=alpha)

while time < 50000:
    print("Time: "+str(time))
    print("Epsilon: "+str(epsilon))
    print()
    time += 1
    state = env.reset()
    tick = 0

    epsilon *= decay

    if epsilon < 0.1:
        epsilon = 0.1

    while True:
        tick += 1
        state = np.expand_dims(state, 1)
        state = state.reshape(1, 3, 15, 19)
        q = approx.forward(torch.from_numpy(state))[0]

        if random.uniform(0, 1) < epsilon:
            action = env.action_space.sample()
        else:
            _, action = torch.max(q, -1)
            action = action.item()
        new_state, reward, terminal, _ = env.step(action)
        show_state = new_state
        new_state = np.expand_dims(new_state, 1)
        new_state = state.reshape(1, 3, 15, 19)

        q_new = approx.forward(torch.from_numpy(new_state).type(torch.FloatTensor))[0]  # " find Q (s', a') "
        #  find optimal action Q value for next step
        new_max, _ = torch.max(q_new, -1)
        new_max = new_max.item()

        q_target = q.clone()
        q_target = Variable(q_target.data)

        #  update target value function according to TD
        q_target[action] = reward + torch.mul(new_max, gamma)  # " reward + gamma*(max(Q(s', a')) "

        loss = loss_fn(q, q_target)  # " reward + gamma*(max(Q(s', a')) - Q(s, a)) "
        # Update original policy according to Q_target ( supervised learning )
        approx.zero_grad()
        loss.backward()
        optimizer.step()

        #  Q and Q_target should converge
        if time % 100 == 0:
            state = torch.FloatTensor(show_state).permute(1, 2, 0).cpu().numpy()

            plt.subplot(131)
            plt.title("Imagined")
            plt.imshow(state)
            plt.subplot(132)
            plt.title("Actual")
            plt.imshow(state)
            plt.show(block=False)
            plt.pause(0.000001)

        if keyboard.is_pressed('1'):
            torch.save(approx.state_dict(), 'trained-10000.mdl')
        if keyboard.is_pressed('9'):
            torch.save(approx.state_dict(), 'trained-10000.mdl')

        if terminal or tick > 100:
            plt.close()
            break

        state = new_state

torch.save(approx.state_dict(), 'trained-10000.mdl')

Are you implementing DQN algorithm? If yes, where did you study the algorithm from? Or did you wing it and implemented it like regular Q-learning with NN and hoped for the best. As far as I can see you didn't implement experience replay nor target network. You should read original DQN papers here and here  , also there are numerous question asked about DQN on this site so you should dig through for more information.

@Brale_ thanks for your reply! I implemented target network, and I’m pretty sure I did it right because it converges ok, and I didn’t think that experience replay was neccessary, is it? And I found a lot of solutions like making the network larger on here but none of them worked. I have also tried many different network configurations.

experience replay is a must have so you should implement that

@Brale_ ok just to make sure if I wasn’t seeing any improvement whatsoever that could be attributed solely to not having experience replay?

not solely to that but a part of it yes


text:
How do I format task features with a one-hot task identification vector to ensure separate weight matrices for each task in multi-task RL?
I am on Lecture 2 of Stanford CS330 Multi-Task and Meta-learning, and on slide 10, the professor describes using a one-hot input vector to represent the task, and she also explained that there would be independent weight matrices for each task
How is the input to a multi-task network encoded to allow the features of all the tasks to be associated with different weights?
Would you have an input vector containing all the features for every task, and then multiply the input vectors by the task ID vector? Is there a more efficient way to approach this problem?
In other terms, here’s what I’m thinking:
network_input[i] = features[i] * task[i]

where features is a 2d matrix of feature vectors for every task, and task is a one-hot vector corresponding to the task number. Is that multiplicative conditioning?

I provided a link to the supposed lecture 2 of that course. Make sure that's the correct link. Also, maybe, to confirm, you could say which slide says what you say the professor says.


text:
Is there research about teaching AI to "analyze the problem and design a solution"?
Update on 2019-05-19:
My question is about teaching AI to solve the problem, not letting AI teach a human developer to solve a problem.

Original post:
I'm a software developer but very new to AI.
Today my friend and I chatted about the development of AI. One topic was about implementing the capability of "given a problem, analyzing the problem and designing a solution".
Since we are both software developers, we used a simple example problem in our discussion to see how AI might possibly find a solution:
Print the following three lines on the console:

*
***
*****

My friend and I thought we may use some formal method to describe WHAT we want but NOT how we implement it. It's the AI's job to figure out the solution.
Then we came to the question I'm asking here: Since my friend and I are both outsiders of AI research, we don't know if there is any existing research (we believe such research must have existed somewhere) that teaches AI to analyze the problem (which is formally defined) and design a solution using the given tools.
For us human beings, our analysis of the problem and designing might look like the following:

Let me choose a programming language. For example, C.
Let me see what tools I have in the chosen programming language. Oh, here they are:

putchar(ch) which prints a single character on the console.
printf(str) which prints a string on the console.
for-loop; if-else; support of subroutines; etc.

I see the result has three lines of characters: line 1, 2, and 3.
I see the numbers of '*' in the three lines are an arithmetic progression and there is a connection of line number and character number: given the line number i, the character number is 2*i-1, where i is 1, 2, and 3. This is repetition and I can use a for-loop.
Each line is the repetition of '*' so I may implement a function to do this.

void print_line(int N) {
  for (int i = 0; i < N; i++) {
    putchar('*');
  }
  putchar('\n');
}

int main(int argc, char * argv[]) {
  for (int i = 1; i <=3; i++) {
    print_line(2 * i - 1);
  }
  return 0;
}

Alternatively, I may design a naive solution of using printf() three times and hard-code each string:
printf("*\n");
printf("***\n");
printf("*****\n");

We think an AI that can do this may follow a similar analyzing and designing approach as a human developer does. In general, we think:

This AI should have a toolbox using which it can solve some problems (possibly not all problems). In my example above, this toolbox may be a programming language and its corresponding library.
This AI should have the knowledge about some concepts (such as console and string in the example above) and their relationships.
This AI should have the knowledge that connects the toolbox and the concepts, so the AI knows how a tool can manipulate the properties of a concept.
Most importantly, this AI should have the capability of figuring out one or more paths that connect the input to the desired output, using the toolbox. This process, we think, needs the capability of "analysis" and "design".

Excuse us if the description is still vague. My friend and I are both new to AI so, in fact, we don't even know if "analysis" and "design" are the proper words to use. We will be glad to clarify if needed.
BTW, we did some quick search about such AI:

Bayou by Rice University doesn't look like understanding the problem, either.
DeepCoder uses Deep Learning and I doubt whether it understands the problem, either.
The AI-Programmer uses genetic algorithms to generate the desired string in BrainFuck. But this AI doesn't look like understanding the problem. It looks like a trial-and-error with feedback.

Software development is a quite complex task, which involves a lot of abstract concepts. I doubt there's any AI system that is capable of solving any non-trivial programming task satisfactorily. Programming is likely an AI-complete problem.

@nbro Thanks for the information! I'll take a look at the "AI-complete problem". Meanwhile, I just learned about the research field "Program synthesis" which seems to be related to my question. I'll take a look, too.

@nbro Yes, I agree with you that it is quite complex and probably there isn't any AI that can do it so far. This is why when my friend and I were discussing this, we thought we human developers can define the problem formally, instead of using natural language. So the AI doesn't have to process natural language to understand the problem. However, figuring out a solution from even a formally specified problem still seems so difficult for a machine.

There are two main problems with this question: You have to define the problem in a useful way (You just write (which is formally defined)) and you use the word "understanding" which bears some problems. There is no research to this because it might be too broadly-defined. There is research to developing models to solve more than task though.


text:
Correcting 'bad' translations in a sequence-to-sequence neural machine translation model
In working with basic sequence-to-sequence models for machine translation I have been able to achieve decent results. But inevitably some translations are not optimal or just flat-out incorrect. I am wondering if there is some way of "correcting" the model when it makes mistakes while not compromising the desirable behavior on translations where it previously performed well.
As an experiment, I took a model that I had previously trained and gathered several examples of translations where it performed poorly. I then took those examples and put them into their own small training set where I provided more desirable translations than what the model was outputting. I then trained the old model on this new small training set very briefly (3-6 training steps was all it took to "learn" the new material). When I tested the new model it translated those several examples in the exact way I had specified. But as I should have anticipated the model overcompensated to "memorize" those handful of new examples  and thus I noticed it started to perform poorly on translations that it had previously been excellent.
Is there some way to avoid this behavior short of simply retraining the model from scratch on an updated data set? I think I understand intuitively that the nature of neural networks would not lend itself to small precise corrections (i.e. when the weighting of just a few neurons change the performance of the entire model will change) but maybe there is a way around it, perhaps with some type of hybrid reinforcement learning approach.
Update:
This paper speaks of approaches to incrementally improving neural machine translation models

The basic process you're describing sounds a lot like Boosting, which is also well covered in Chapter 18 of Russell & Norvig, or like active learning.
You're correct that training the model with emphasis on a specific subset of the data is likely to lead to mistakes somewhere else. Boosting gets around this in a clever way.

It begins by training a model as normal.
The new model's performance on the training set is measured.
A new dataset is generated, where examples that were misclassified by the previous model are more highly weighted (this is intuitively similar to adding an extra copy of them to the dataset).
A new model is trained on the weighted dataset. The model suffers a greater penalty for making the same kinds of errors as the previous model(s), and so tends to make different errors.
Steps 2-4 are repeated for some time.
An ensemble of models is output, rather than a single model.

To classify new points, the most common score from all ensemble members is used.
I'm less familiar with NLP work using this approach, but it seems like the same basic idea could be used fruitfully here: the notion is to train many models that make independent errors, and then trust their collective decisions.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Using three image datasets with different image sizes to train a CNN
I've just started with AI and CNN networks.
I have two NIFTI images dataset, one with (240, 240) dimensions and the other one with (256, 132). Each dataset is front a different hospital and machine.
If I want to use both to train my model. What do I have to do?
The model needs to have all the train data with the same shape. I've thought to reshape all the data to have the same shape, but I don't know if I'm going to lose information if I reshape the images.
By the way, I have also a third dataset with (232, 256).

I'm a beginner as well but I suppose you could resize all of the images to a specific dimension (e.g 200x200). As far as I know, resized images do not lose information. Again, I'm not a professional.

Yes, the standard way of handling this is to simply resize the images. If you can still interpret the image yourself, it's normally safe to say your CNN will be ok as well

Resizing the images will work.But if you significantly reduce the image dimensions information IS lost. Here is a simple example. Suppose you have a 500 X 500 image. That has 250,000 pixels. Assume in these images the object of interest( lets say a bird in a forest) only occupies 10% of the pixels in the image(25,000 pixels). Now assume you reduce the image to 100 X 100 and thus have 10,000 pixels. Your object of interest (the bird) now only occupies 1000 pixels. These are the pixels the neural net will learn from.
Now a better way to do this is to first crop your original 500 X 500 images to maximize the percentage of bird pixels in the cropped image. For example assume the resulting cropped image comes out to be say a 200 X 200 image but in that image the subject of interest(the bird) occupies 50% of the pixels (20,000 pixels). Now if you reduce the cropped image down to a 100 X 100 the object of interest will occupy(5000 pixels).
Cropping images is of course a hassle. For some types like images for example images of people, and you are just interested in the faces there are routines available that will automatically crop the image to just the face. In general you will have to crop the images yourself.
If you want to build high accuracy classifiers you have to start with a "robust" data set.
The larger the percentage the object of interest takes up in the images the better your classifier will perform.


text:
What are non-held-out data or non-held-out classes?
I'm Spanish and I don't understand the meaning of "non-held-out". I have tried Google Translator and online dictionaries like Longman but I can't find a suitable translation for this term.
You can find these term using this Google Search, and in articles like this one:

"computing SVD on the non-held-out data" from here.
"The training set consists all the images and annotations containing non-held-out classes while held-out classes are masked as background during the training" from Few-Shot Semantic Segmentation with Prototype Learning.
"A cross-validation procedure is that non held out data (meaning after holding out the test set) is splitted in k folds/sets" from here.

What is non-held-out data and held-out data or classes?

Held-out simply means "not included" particularly in the sense of:
This part of the data was not included in this specific training run.
Depending on the context of all of these text non-held-out data/classes means the data that actually was included in a particular modeling exercise.
Consider this excerpt from your first example:

For instance, Owen and Perry (2009) show a method for holding out data, computing SVD on the non-held-out data, and selecting k so as to minimize the reconstruction error between the held-out data and its SVD approximation.

It actually means:

For instance, Owen and Perry (2009) show a method for excluding data, computing SVD on the remaining data, and selecting k so as to minimize the reconstruction error between the excluded data and its SVD approximation.

So it simply talks about a particular way of train-test-validation splitting the data.


text:
Data prepared to linear regression. Can I use it with backpropagation?
I'm studying a Master's Degree in Artificial Intelligence and I need to learn how to use the Java Neural Network Simulator, JavaNNS, program.
In one practice I have to build a neural network to use backpropagation on it.
I have created a neural network with one input layer with 12 nodes, one hidden layer with 6 nodes and one output layer with 1 node.
I'm using Kaggle's titanic competition data with this format following Dataquest course Getting Started with Kaggle for Titanic competition:
Pclass_1,Pclass_2,Pclass_3,Sex_female,Sex_male,Age_categories_Missing,Age_categories_Infant,Age_categories_Child,Age_categories_Teenager,Age_categories_Young Adult,Age_categories_Adult,Age_categories_Senior,Survived
0,0,1,0,1,0,0,0,0,1,0,0,0
1,0,0,1,0,0,0,0,0,0,1,0,1
0,0,1,1,0,0,0,0,0,1,0,0,1
1,0,0,1,0,0,0,0,0,1,0,0,1
0,0,1,0,1,0,0,0,0,1,0,0,0
0,0,1,0,1,1,0,0,0,0,0,0,0
1,0,0,0,1,0,0,0,0,0,1,0,0
0,0,1,0,1,0,1,0,0,0,0,0,0

If you want see the same data better in an Spreadsheet:

But they preprocess the data to use it with linear regression and I don't know if I can use these data with backpropagation
I think something is wrong because when I run backpropagation in JavaNNS I get these data:
opened at: Sat Feb 17 17:29:40 CET 2018
Step 200 MSE:   0.5381023044692738  validation: 0.11675894327003862
Step 400 MSE:   0.5372328944712378  validation: 0.11700781497209432
Step 600 MSE:   0.5370386219557437  validation: 0.11691717861750939
Step 800 MSE:   0.5370348711919518  validation: 0.11696104763606407
Step 1000 MSE:  0.5369724294992798  validation: 0.11697568840154722
Step 1200 MSE:  0.5369697016710676  validation: 0.11665485957481342
Step 1400 MSE:  0.5370053339270906  validation: 0.11684215268609244
Step 1600 MSE:  0.5370121961199371  validation: 0.11670833992558485
Step 1800 MSE:  0.5370200812483633  validation: 0.11673550099633925
Step 2000 MSE:  0.5367923502149529  validation: 0.11675956129361797

Nothing changes, it is like it doesn't learn anything.
How many hidden layers does the network have with how many nodes on each hidden layer?
Maybe the problem is that the data have been prepared to be used in Linear regression and I using it with Backpropagation.
I have only created the neural network, I haven't implemented the backpropagation algorithm because it is already implemented in JavaNNS.

Can you please tell us which gradient descent algorithm you use and which learning rate?

@MolnárIstván The learning rate is 2.0 but I don't know which gradient descent algorithm is JavaNNS is using.

The data has been prepared in Dataquest tutorial to be used with Linear Regression. Can I use it with backpropagation?

Well, I pretty sure that 2.0 for learning rate is way too high. Try 1e-2, 1e-3 or 1e-4 (it differs which gradient descent algorithm you use) . Besides, I think you have some misunderstanding about backpropagation. It is for adjusting the weights in your network through the loss. Well of course the prepocessing can be bad in a lot of cases, but here I think it should work.

@MolnárIstván I have talke with my professor and he said me that there isn't enough data to train the network. With 12 inputs and only 718 cases, it only covers the 17% of the combinations. I don't understand why I have to use JavaNNS. The train data is part of a Kaggle competition and it works with linear regression.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Neural network to get input attributes using only the output value
I have an idea about how to use neural networks but I'm not sure if it is possible or not.
In supervised learning we have a set of attributes labeled with an output value. I can use these set to train my network.
Now I have a network trained to get an output value from an random set of attributes but, can I use this trained network to get the input attributes using only the desired output?
I will have N input values and only 1 output value. I've thought that I can use the weights for that network into a new one with 1 input value and N output values but I'm not sure if I can do that.

Now I have a network trained to get an output value from an random set of attributes but, can I use this trained network to get the input attributes using only the desired output?

It depends:

If you are happy to find any inputs, even non-realistic ones, that get your desired output, then you can use your trained network, with a minor modification. Freeze all the weights, and allow back-propagation to determine the gradient of the input (which should now be a variable to optimise, not source data). Start with a noise input, back-propagate the error to find gradient to make the input better at creating your desired output, then take a gradient step towards it in the input data. This is essentially how Deep Dream works. Like Deep Dream, you will not necessarily get realistic input values, but will get semi-random ones that cause your network to predict a specific class.
If you want the newly generated input to be a best guess at something from the original dataset, then you have to look at one of more advanced models:

Restricted Boltzmann Machines (RBMs).
Variational Autoencoders (VAEs)
Generative Adversarial Networks (GANs). These have become popular recently, and there are many variations on the basic idea, including VAEGANs that combine VAEs with GANs.

These network types are quite advanced, and can be tricky to understand and train successfully. You will want to spend some time researching each type.
To generalise terribly: A GAN will tend to generate realistic "noise" in the generated items, but at the expense of overall structure and cohesion (images tend to look distorted but with realistic textures). A VAE will tend to produce smooth, coherent inputs, but at the expense of lack of fine detail (VAE images tend to look smoothed and/or blurred).
If not sure what to try, probably GAN is a reasonable choice, since there are lots of tutorials available, and recent advances with image generation can look very impressive.

Thanks. I've thought about this question because I'm using genetic algorithm to optimize a function. I know which is the desirable output, 0, and I have tested a lot without successful. Maybe I can use a neural network to find the input to the genetic algorithm to get 0 as output. Thanks again.

NN isn't symmetric and the result may haven't any seems.
You can just use loss like this (target_output - model_output)**2 - MSE
differentiate this loss  according to input variables and use optimizer to solve this task. Try to search about adversarial models, the idea is very similar.

could I ask you to give this answer an edit?  I would do it, but I'm not sure of the meaning of the first sentence...


text:
What are the features get from a feature extraction using a CNN?
I've just started to learn CNN and somewhere I have read if I remove the last FCL I will get the features extracted from the input image but... what are those features?
Are they numbers? Labels? An image location (x,y) where there is a line.
I want to use these features on a one shot network, but I can't imagine how to use them if I don't know what they are.

You get what we call high-level features, which are basically abstract representations of the parts that carry information in the image you want to classify.
Imagine you want to classify a car. The image you feed your network could be a car on a road with a driver and trees and clouds, etc. The network, however, if you've trained it to recognize cars, will try to focus on parts of the image regarding a car. The final layers will have learned to extract an abstract representation of a car from the image (this means a low-resolution car-like shape). Now your final FC layers will try to classify the image from these high-level features. In this example, you would have an FC layer that learns to classify a car if this this abstract car-like figure is present in the image. Likewise, if it isn't present it won't classify it as a car. By accessing these high-level features, you essentially have a more compact and meaningful representation of what the image represents (based always on the classes that the CNN has been trained on).
By visualizing the activations of these layers we can take a look on what these high-level features look like.

The top row here is what you are looking for: the high-level features that a CNN extracts for four different image types.

Do activations really tell you something about the features? How come that a number (the activation) can tell you something regarding a high-level feature? Which high-level feature? I think your answer would improve if you add more details about this.  Pedagogically, I think it is better to only say that a CNN only transforms the image with non-linear transforms and you should emphasize that this extraction of high-level features is just an interpretation of the inner workings.

I agree with you, an activation by itself can't tell you anything. But if you see how this activation changes you can get a better understanding of how that CNN works (and that's what I think the question was about).

So, the convolutional layers reduce the input to get only the more relevant features from the image, and then the fully connected layer classify the image using those features, isn't it? I think I've just understood how a CNN works.

@VansFannel on a very high-level yes.

@Djib2011 and on a very low or low level? Where can I find information about how CNN works? I only find on a very high-level description about them and how to implement them with TensorFlow. Thanks!

@VansFannel I'd recommend Stanford's excellent course on Convolutional Neural Networks.

@Djib2011 Thanks. I've watched all the course. Maybe, I haven't watched in the right way.


text:
How to transform inputs and extract useful outputs in a Neural Network?
So I've been trying to understand neural networks ever since I came across Adam Geitgey's blog on machine learning. I've read as much as I can on the subject (that I can grasp) and believe I understand all the broad concepts and some of the workings (despite being very weak in maths), neurons, synapses, weights, cost functions, backpropagation etc. However, I've not been able to figure out how to translate real world problems into a neural network solution.
Case in point, Adam Geitgey gives as an example usage, a house price prediction system where given a data set containing No. of bedrooms, Sq. feet, Neighborhood and Sale price you can train a neural network to be able to predict the price of a house. However he stops short of actually implementing a possible solution in code. The closest he gets, by way of an example, is basic a function demonstrating how you'd implement weights:
def estimate_house_sales_price(num_of_bedrooms, sqft, neighborhood):
  price = 0

  # a little pinch of this
  price += num_of_bedrooms * 1.0

  # and a big pinch of that
  price += sqft * 1.0

  # maybe a handful of this
  price += neighborhood * 1.0

  # and finally, just a little extra salt for good measure
  price += 1.0

  return price

Other resources seem to focus more heavily on the maths and the only basic code example I could find that I understand (i.e. that isn't some all singing, all dancing image classification codebase) is an implementation that trains a neural network to be an XOR gate that deals only in 1's and 0's.
So there's a gap in my knowledge that I just can't seem to bridge. If we return to the house price prediction problem, hows does one make the data suitable for feeding into a neural network? For example:

No. of bedrooms: 3
Sq. feet: 2000
Neighborhood: Normaltown
Sale price: $250,000

Can you just feed 3 and 2000 directly into the neural network because they are numbers? Or do you need to transform them into something else? Similarly what about the Normaltown value, that's a string, how do you go about translating it into a value a neural network can understand? Can you just pick a number, like an index, so long as it's consistent throughout the data?
Most of the neural network examples I've seen the numbers passing between layers are either 0 to 1 or -1 to 1. So at the end of processing, how do you transform the output value to something usable like $185,000?
I know the house price prediction example probably isn't a particularly useful problem given that it's been massively oversimplified to just three data points. But I just feel that if I could get over this hurdle and write an extremely basic app that trains using pseudo real-life data and spits out a pseudo real-life answer than I'll have broken the back of it and be able to kick on and delve further into machine learning.

This is a good question which I wrestled with myself when first trying to code an ANN.
Below is a good general-purpose solution, and it's the one I implemented in my code for trying to predict well-behaved numerical data. If your data is not well-behaved (i.e. fraught with outliers) then you may need to do more work normalizing the inputs and outputs. Some of the more advanced methods are described here.
Note: I will assume that you are using f(x) = tanh(x) as your activation function. If you aren't, you should still be able to reason through how to normalize your data after reading this.
How to prepare the input data:
The basic idea is that you want a significant variation in each input parameter to be reflected by a significant variation in the activation of the neuron those inputs are being fed into. By looking at a plot of the derivative of the tanh(x) actiavtion function, you'll see that the region of significant slope is within a distance of one or two from the origin. This means that whether the input to the activation function is 2000 or 3000 (values of x for which the derivative is negligibly small), the output of the activation will be almost identical...so your neuron's state will be independent of the difference between 2000 and 3000, and your network will never produce any predictive power from values in that range.
So if you want to input the square footage of the house into a neuron, you need to normalize the square footage so that the network can tell the difference between 2000 and 3000. One way to do this so that all of the significant variations in your data are 'noticed' by the neuron is to z-score-normalize the inputs.

Gather all of your square footage values (from your training set) and calculate the mean and standard deviation. Store the mean and standard deviation---you'll need this information to normalize new square footage values when testing.
Normalize the vector of square footage values by subtracting the mean and then dividing the result by the standard deviation (all operations element-wise of course). Subtracting the mean centers your data at the origin, and dividing by the standard deviation makes sure most of it is between -1 and 1, where the neuron's output is most sensitive to its input. This is called z-score normalization because each input value is replaced by its z-score.
Do the above for each input variable.

Now, when you put each input value through a neuron, the output of the neuron is an activation between -1 and 1 (look at the image of tanh(x)). Since this is already in the 'sensitive' range of the activation function, you don't need to worry about altering the output of the input-layer neurons before sending them to the first hidden layer. Just give any hidden layer neurons the outputs of the previous layer directly---they will be able to handle them just fine.
When you reach the last layer (the output neuron(s)), what you get is again another activation between -1 and 1. You have to convert this back into a value for the house in question, whether that value will be used as a prediction in a test set or to calculate error during training. However you do this, you just have to be consistent and use the same de-normalization procedure in training and testing. One way to think about it is: when the output neuron(s) returns 1, that means the network is returning the maximum possible house value as its prediction. What should the highest value the network can estimate be? The right approach here simply depends on your application. This is what I did:

Calculate the mean of [the/each] output variable and store it.
Calculate the maximum deviation of the output variable from the mean.
Python: MaxDev = max([abs(DataPoint-numpy.mean(TrainingData)) for DataPoint in TrainingData])
When the network returns output(s) between -1 and 1, multiply the output by MaxDev and add it to the mean.

Two basic quick checks you can do to see if your normalization-renormalization scheme is suitable (these are necessary, but perhaps not sufficient conditions):

If all the input values are average (e.g. average no. of bedrooms, average sq.feet, etc), is the network's output equal to the average of the output variable (e.g. house value) as well? (It should be.)
If all the input values are unusually high/low, is the network's output unusually high/low as well? (This only works if all the inputs are positively related to the output...if some of them are inversely related related, you will have to think a bit more).

Observe that the scheme presented here satisfies these two conditions.
Notice that this scheme would allow your network to only predict house values inside the range of house values in your training data set. Depending on the application, this behavior can be desirable or undesirable.
For example: you may want to make it impossible for your network to predict negative house values. Think about how you would do this. De-normalize the output so that -1 is mapped to 0.
If you want to set no limit on the values your network can predict, then you can run the network's output through a function that maps the [-1,1] range to all real numbers...like arctanh(x)! As long as you do this during training your network will adjust its weights to accommodate this.
I hope this was helpful. Let me know if you have further questions. My ANN module is in Python, by the way, so I might have language-specific advice.

This was very useful! Every blog/tutorial I come across seems to avoid (almost deliberately) describing this process, but yes that all makes sense. It'll take a while for me to digest properly but I'll be back if I have any follow up questions. Much obliged!

So couple of questions. If my Sq. Feet training data was { 2000, 800, 850, 550, 2000 } then my z-score inputs for { 1900, 1500, 600 } would be (if I've calculated correctly) { 1.0496, 0.4134, -1.0177 }. So one of those values is > 1 and one is < -1, what would I do with those? Input them into the input layer nodes regardless or round them to 1 & -1? Why does 1900 & 600 produce those values when they are within the 550 - 2000 range? Is this just a trick of the data because there is such a small data set?

With regards to output layer renormalization, do I have it correctly that you'd plot the output to the min and max values? So if the minimum value was $0 and the maximum $100 and the output was zero (assuming -1 to 1) then that would translate as $50?

Remember that the inputs don't need to be strictly between 1 and -1. All you need for the inputs is that most of the data is in that range. A value greater or less than one means that point is more than one standard deviation away from the mean, so that point is closer to the higher end of the data. It should be a bit rare for your data to go outside of [-1, 1], even more rare for it to go outside of [-2, 2] and extremely rare to go outside of [-3, 3]. Look at tanh(x) and you will see the sensitive range isn't just strictly between -1 and 1, but goes a bit further out than that.

Regarding the output denormalization, that min-max denormalization is what I did in my implementation, and your interpretation is correct, but you don't necessarily have to do that. You could make it so that 1 corresponds to twice the maximum house value--that way your network would be able to predict house values above what you trained it on.

"By looking at a plot of the derivative of the tanh(x)"  Dont understund, why should I be lookning at derivative of base function, when we input is mulitplied by by weight and then passed to base funcion eg. RELU. Could u explain?

Because the weight only scales the derivative by a constant value, it doesn’t affect the insight that was sought here


text:
Ideas on how to make a neural net learn how to split sequence into sub sequences
How can I train a neural network to recognize sub-sequences in a sequence flow?
For example: Given the sequence 111100002222 as an input sample from a stream, the neural network would recognize that 1111 , 0000 , 2222 are sub sequences (so 111100 would not be a valid subsequence) and so on for ~ 50 to 100 different subsequences.
There is no particular order in which the subsequence would appear in the flow.
No network architecture restriction.
Subsequences are of variable length.
General concepts, ideas, and theory are welcome.

It is indeed an algorithm problem. However, the 'learning' process of knowing which subsequence goes well together is , in my opinion, a problem where NN are particulary useful.

Is this sequence like a class label for some other related sequence that you are trying to model?  If that's the case, add some detail to you example and maybe we can help you.  If not, you could easily implement what you ask using regex in your language of choice.

Yes it is ! We could replace the example above with , let’s say , sentences without spaces.
ex : If i feed into the neural netwok :
‘ILikePizza’ , a correctly trained neural network would be able to ouput or classify ‘I like pizza’ , recognizing those 3 different words (sub sentences) in the character flow. Whether it’s classification or even a regression of some sort would be valid.

You example in the question is rather useless: Splitting on character change does the job. You can learn it, but there's no reason to. Your second example is better, but ILikePizza is equally trivial, just split before each an uppercase character. Still nothing useful to learn. Should it handle ilikepizza as well? Provide a few more inputs. provide something to learn from. +++ Can you split strcprstskrzkrk (I can; it's an actual sentence)? In case you can't, would you expect the network to be able to? What other input should it get?

Let's take a global approach to be able to understand the examples better : given an entire book where there is NO spaces anywhere, only characters (including special characters, like commas), in what way can we make the network learn the 'word boudaries' of this book .
Ex :  Offmythrone,jester.Thekingsitsthere. = Off my throne, jester. The king sits there.   The whole purpose of this network would be to learn boundaries between concatenated inputs, in other words, when to seperate between a constant input flow, no matter what is the input context.

Other short examples :   **Walktall...myfriends |  WhatcanIsay?Youguys...arethebest.  | I'vecomeupwithanewricepee  | KingsofLucis,cometome!  | MAGITEKENGINE!IT'SCLOSE!**

The problem in the original question is akin to that of inducing a context-sensitive grammar (CSL), except that it is harder because a CSL is assumed to be composed of fixed-length subsequences. It is probably closer to the problem of inducing a Reber grammar, but that in turn seems like an overkill.
LSTMs are known to be able to learn both CSL and Reber grammars. However, I doubt that this is what you really need because of the following comment:

[...] given an entire book where there is NO spaces anywhere, only characters (including special characters, like commas), in what way can we make the network learn the 'word boudaries' of this book.

This is called morphology induction, and it is a much harder problem than that of simple Reber grammar induction. Note that finding word boundaries is a special case of the problem of finding morpheme boundaries. There have been many attempts to solve this (also see this survey paper for more details and references).
Most approaches developed seem to rely on statistical principles (like MDL) and do not use neural networks (a counterexample using LSTMs). My intuition is that the extreme morphological variability across languages (ranging from Finno-Ugric languages with highly inflectional morphology to Sino-Tibetan languages with hardly any morphology at all) makes it hard to train neural networks in a language-agnostic way. However, you might have better luck if you focus on a single language.
Hope that helps.

Another approach could be to predict the class of a sequence and not the break point. Assuming that each sequence is part of a class, you can use a LSTM. Inputing the multiple sequences (111100002222 ) and let predict the class for each sequence (c1,c1,c1,c1,c0,c0,c0,c0,c2,c2,c2,c2)

That would be a good idea, but what about variable length repetitive sub-sequences ? If i follow the 'word' example , how would the network be able to learn between 'scholar' / 'scholarship' / 'scholar' & 'ship' ?

Therefore, you have to annotate your data that scholar is 1111111 and scholarship is 11111112222. Is this your real use case or just an example? Then I can come up with a better idea ...

I guess, supervised learning should work rather well: You'd feed the network with a fixed substring and it'd determine if the middle character is the first letter of a word, or a last one, or neither or both.
So 2*n+1 inputs (fed e.g., with the string "ingsits") should output a 1 on the output determining if the middle letter (here: "s") is the first one of a word and a 0 on the output determining if it's the last one (taken from "Thekingsitsthere"). Each input character should probably be 1 hot encoded.
You'd probably want to use more context characters than in my example. OTOH you can use a simple MLP with no temporal complications. It'll never get perfect as it's impossible, but it get pretty close.
Concerning unsupervised learning I'm skeptical...

(NOTE: I think it will be easier to do it without ANNs...)
But if you insist:

convert the sequence into a fixed-size vectors.
push trough a 2-5 1D-convolution layer with 1 neuron dense layer at the end (sigmoid activation) and another K-points detector for getting the sequence breakage points
create a training set - to find the break-points (12, 23, 34 ...) in the sequence.
train a detector with SGD to find these break-points. - loss functions: cross_entropy.

Then, it should learn to find the breakage points, and based on this you can easily split the sequence.

Mmmm...never thought of using CNN for that kind of things, although, it kinda makes sense. I'm wondering, however, if a K-points detector could overcome the variable length problem ('scholar' / 'scholarship' / 'scholar' & 'ship' ). Maybe using a standard classifier outputting probability of the key points found by the CNN ?

you can pad the sequences, so it will have fixed sizes. Moreover, conv. layers usually can deal with variable size vectors, as long as it is all-convolutional network (no dense layers).

How about this ?
1 - Learn all the basic building blocks of possible sub-sequence
In our words sequence example, that would correspond to phonemes.
(I'm guessing that this step can even be done using unsupervised learning.)
So in the following example :
Hello Laurie, we would have learned 3 phonemes : HE, LO, RI.
2- Learn all subsequence as sequences of 'building blocks'
Using a ClockWorkRNN with timesteps of interval +1 with, let's say, 10-15 timestep (groups), that is fed the next 'phoneme id' in the sequence, we would have a space large enough to record most words (Obviously, the number of timesteps should be size of the biggest word).
This is the subsequences memory RNN.
Its sole purpose is to remember subsequences.
Now, i'm really brainstorming here , taking a very wild guess, but what if :
After training this RNN to a satisfying error rate, we check if the output of the RNN is very different to the next input for a couple of timesteps.
In other word, we see if the neural network has been able to 'guess' the next building block of the subsequence.
If not, then its a point of interest , because there is not a lot of possibilities as of why this would happend : the only one I see is
1 - The RNN is currently receiving another word, thus making this timestep a sub-sequence 'break point'
Do you guys see any points that could prove this theory wrong ?


text:
I'm trying to understand the use model for different Python libraries
I'm new to ML/AI field, and after completing several free university courses from MIT OpenCourseWare and Harvard CS50, I've gained some familiarity with the theoretical foundations of Artificial Intelligence.
Now, I'm attempting to apply this knowledge by experimenting with Python examples I can find online. I've delved into various topics, spanning from Perceptrons and Nearest Neighbors to Reinforcement Learning with Q-Learning and Neural Networks. During this process, I've discovered the existence of numerous ML/AI libraries (such as scikit-learn, TensorFlow, PyTorch, Stable-Baselines, and Gymnasium/Gym) designed to facilitate programming, which is both advantageous and somewhat overwhelming.
What bewilders me the most is that the same AI concepts can be implemented using different Python libraries. For instance, I've come across examples of reinforcement learning implemented using TensorFlow, PyTorch, Stable-Baselines, or just Gym combined with NumPy. This makes me wonder: which one should I choose? Is it necessary to invest my time in learning all of them?
As a newcomer to this field, I'd like to gain clarity on the following:

Are these libraries complementary or do they compete with each
other?

Which two libraries would provide the broadest coverage for
AI applications? (I aim to maintain focus and limit my efforts, so becoming proficient in using only two libraries effectively would be a significant accomplishment. I'm already acquainted with the scikit-learn pipeline and find it quite appealing, but I've also noticed its limitations in the context of Reinforcement Learning and Neural Networks.)

Why two? If you are already familiar and comfortable with scikit-learn, a better question is probably which of the others is the best complement to scikit-learn.

Your choices here are not really any different to choosing an open source library for any other purpose. Each library will have its own idiosyncratic parts, but usually these are minor things compared to:

Is the library maintained, with recent updates, and multiple contributors?
Are there compatibility issues with other parts of your project - software or hardware?
Is there documentation and/or tutorials to get you started?

Do you have to learn all the libraries? No, but if you work on multiple projects in different teams, you will probably be exposed to all the major ones anyway.
When it comes to choosing between PyTorch and TensorFlow, each has its advantages for different kinds of project, but for small learning projects there's little meaningful impact. If you are just starting, pick one but be prepared to learn both pretty early on. They have loads in common anyway, so a lot of learning with one will help with the other.

In short:

Gym is complementary (but optional) to both tensorflow and pytorch. Stable-baselines (be sure to check version 3, which is based on Pytorch) supports gym natively (if I remember correctly.)
Well, TensorFlow, PyTorch and Numpy are general enough to let you implement whatever ML/DL/RL workflow you wish: they are just libreries for numerical computation. With Numpy you would loose GPU (unless using numba) and TPU acceleration, but also the already build layers for neural network stuff. Anyway, I think that nowadays Pytorch is the most comprehensive since many official (and not) implementations of papers are freely available on GitHub and HuggingFace, letting you pick SOTA models easily (think about LLMs.)

I'd like to add JAX to the list of ML frameworks, since it's gaining popularity quite fast. Differently from TF and PT, JAX is designed around the principles of functional programming like stateless functions and immutable objects. These allow to gain a more fine grained control over randomness (and so reproducibility) but also code optimizations for speedy models, although the framework can be difficult to learn initially.

It's good to know that I should focus more on PyTorch, as I was already leaning towards TensorFlow.

I don't have GPU or TPU hardware capability, small-scale ML experiments I'm running locally on my laptop, for large-scale implementation I'll be using AWS cloud and LSF (yet another challenge)

@BorisL. Well, for small ML experiments scikit-learn may be enough if you don't need neural nets, otherwise both TF and PT run also on CPU but expect larger models to run slower.


text:
Are diffusion models still beneficial in highly compressed latent spaces?
Consider for example the MNIST dataset. When we apply diffusion to the pixel space, the image slowly becomes more and more noisy until white noise has been reached (like below). In the last step (t=100), the image does not really depend on the original number. If we have a well-trained unconditional diffusion model based on the original paper, it will thus generate a random number when starting from this image.

Evidently, there is a clear notion of "noise" and thus the concept of "denoising" makes sense. In the "High-Resolution Image Synthesis with Latent Diffusion Models" paper, they suggest to apply the noising and denoising procedure on a latent space to reduce the number of dimensions and thus the computational cost. Makes sense. But i'm struggling to make sense of the concept of "noise" in this latent space, especially if it is a highly compressed representation of the original pixel space. Consider for example the 2D dimensional latent space for the MNIST dataset below.

If we diffuse for example a sample of the number "8", it will travel the latent space and in the end it might end up in the cluster of another number. With sufficient amount of diffusion, the cluster in which it ends up should be randomly distributed. I suppose that denoising from a random starting point will just travel the clusters too. However, it seems to me that the concept of "noise" does not exist in highly compressed latent spaces - every reasonable value of the latent variables corresponds to a number.
Are diffusion models still beneficial in these highly compressed latent spaces? And are such highly compressed latent spaces common in other imaging applications? Or are they more sparse in general?


text:
Is it better to split sequences into overlapping or non-overlapping training samples?
I have $N$ (time) sequences of data with length $2048$. Each of these sequences correseponds to a different target output. However, I know that only a small part of the sequence is needed to actually predict this target output, say a sub-sequence of length $128$.
I could split up each of the sequences into $16$ partitions of $128$, so that I end up with $16N$ training smaples. However, I could drastically increase the number of training samples if I use a sliding window instead: there are $2048-128 = 1920$ unique sub-sequences of length $128$ that preserve the time series. That means I could in fact generate $1920N$ unique training samples, even though most of the input is overlapping.
I could also use a larger increment between individual "windows", which would reduce the number of sub-sequences but it could remove any autocorrelation between them.
Is it better to split my data into $16N$ non-overlapping sub-sequences or $1920N$ partially overlapping sub-sequences?

authors: Максим Цуприк


text:
Will there be some promising techniques that can make AI greener and affordable in the future?
The recent advances in machine learning were mostly achieved by the hardware, and the hardware is said to continue driving the development of AI, but I was still shocked by this thread which reads that the projected future cost for the largest model would be 1B dollars in 2025. And I learned that universities are suffering from an academic AI brain drain partly due to the scarce hardware resources.

Some people proposed the so-called Green AI that encourages sustainable AI development but provides few constructive methods to prevent the trend.
I wonder if the redder and redder AI would be in fact truly inevitable. It seems to me that all companies should build an expensive compute infrastructure to be competitive, but I think the investment would be very risky since most companies cannot get a higher return.
But on the other hand, we human beings have evolved tens of millions of years or billions of years(life) on earth with hundreds of billions of brains that have ever lived on earth as a whole "human brain". The biological wetware seems much much redder than the nowadays hardware and has consumed much much more energy than all the supercomputers. To make machines as intelligent as we humans shouldn't we pay as high a price? It reminds me of the NFL theorem but it should be imprecise in this scenario.
So, will there be some promising techniques on the algorithm side that can make AI greener, affordable and sustainable in the future? If not, could anyone please explain why AI should be unavoidably red and inevitably redder?

As far as I know, green AI and red AI are very recent terms and/or research areas, but their importance will or should definitely be more highlighted in the coming years (for obvious reasons).
I don't know how many people are actively and directly researching this topic, but, in the past, I've already read a research paper on the topic, so a few people are already trying to at least raise awareness about these issues. Moreover, they also suggest that researchers report the number of floating-point operations (FLOPs) performed during the experiments, in addition to the usual performance metrics (such as the accuracy). I don't remember all the details of this paper, but I really recommend that you read it. This another possibly useful paper that people interested in this topic might want to read.
Although I'm currently not doing research on this topic, from an algorithmic perspective, improvements in the following areas will probably contribute to greener AI

zero-, one- or few-shot learning
transfer learning
sample/data efficiency
computational complexity

If you can learn from fewer samples and with less computation, you will probably produce less emissions.
So, I think that making AI greener is definitely possible, as it's possible to use an electric car rather than a petrol/diesel car, although more research on the topics mentioned above (and other topics) needs to be done to make it practically useful.

Zero-, or- or few-shot learning and transfer learning should be all based on expensive pre-training. So far as I know some pre-trained models cannot be employed for specific tasks, please refer to the Reasoning fine-tuning performance part. The pre-trained models are too large to fine-tune.

@LernerZhang You may be right, but if you can re-use some models for different tasks, without training them from scratch, that's already a win. Humans don't learn everything from scratch, so I think that making AI behave/learn more as humans do may actually be very beneficial. I am not an expert in the topic, but, from what I remember having read around, humans or the human brains are a lot more efficient than computers.


text:
How to train/update neural networks faster without a decrease in performance?
I noticed that there are many studies in recent years on how to train/update neural networks faster/quicker with equal or better performance. I find the following methods(except the chips arms race):

using few-shot learning, for instance, pre-taining and etc.
using the minimum viable dataset, for instance using (guided) progressive sampling.
model compression, for instance, efficent transformers
Data echoing, or simply put let the data pass multiple times in the graph(or GPU)

Is there a systematic structure on this topic and how can we update or train a model faster without loss of its capacity?

use validation set maybe?

@Alex What do you mean by validation set? I cannot get your point, sorry. Could you please flesh your comment out into an answer?

Use val data to check if you are overfitting (in addition to other tricks), this will prevent the degradation of the model's performance

@Alex Yes, but let's consider the model that takes N epochs to get the best performance over the develop set, can we speed the training by other methods? Make the N smaller than that for the original model or less time? For instance by reducing the amount of training data or compressing the model size?

what do you mean by 'compressing the model size'?

@Alex Yes, using techniques like knowledge distilling or pruning and etc.


text:
Why did machine learning only become viable after Nvidia's chips were available?
I listened to a talk attended by a panel consisting of two influential Chinese scientists: Wang Gang and Yu Kai, among others.
When asked about the biggest bottleneck in the development of artificial intelligence in the near future (3 to 5 years), Yu Kai, who has a background in the hardware industry, said that hardware would be the essential problem and that we should focus most of our attention on it. He gave us two examples:

In the early development of computers, we compared machines by their chips;
ML/DL, which has been very popular in recent years, would be almost impossible without the employment of Nvidia's GPUs.

The fundamental algorithms existed already in the 1980s and 1990s, but AI went through 3 AI winters and was not empirical until we could train models with GPU boosted mega servers.
Dr. Wang then commented on Yu's opinions, stating that we should also develop software systems because we cannot build an autonomous car even if we combine all the GPUs and computational power in the world.
Then, as usual, my mind wandered off, and I started thinking: what if those who could operate supercomputers in the 1980s and 1990s had utilized the then-existing neural network algorithms and trained them with tons of scientific data? Some people at that time could obviously attempt to build the AI systems we are building now.
But why did AI/ML/DL become a hot topic and practical only decades later? Is it merely a matter of hardware, software, and data?

Many people contributed to machine learning without Nvidia chips. For example Jacques Pitrat

GPUs were ideal for AI boom because:

They hit the right time

AI has been researched for a LONG time. Almost half a century. However, that was all exploration of how algorithms would work and look. When NVIDIA saw that the AI is about to go mainstream, they looked at their GPUs and realized that the huge parallel processing power, with relative ease of programing, is ideal for the era that is to be. Many other people realized that too.

GPUs are sort of general purpose accelerators

GPGPU is a concept of using GPU parallel processing for general tasks. You can accelerate graphics, or make your algorithm utilize 1000s of cores available on GPU. That makes GPU awesome target for all kinds of use cases including AI. Given that they are already available and are not too hard to program, its ideal choice for accelerating AI algorithms.

Machine learning has been around since the 1960's.  They had computers that were less intelligent than the palm-pilots of the 1990's, and they did machine learning.
What most folks call "Machine Learning" is deep neural networks like those that started getting competitive at vision-related tasks in the early 2010's (teens).  The gpgpu hardware standing on video cards (I had friends using the amiga bit-blitter for the task in the early 90's, @TadeuszWestawic) opened up a relative jump in performance, the software to drive that hardware like the methods from Hinton were mature and effective, and then some of the networks started doing amazing things.  Personally I consider Giraffe (Lai, 2015) and AlexNet (Krizhevsky, 2012) to have been watershed moments that motivated the business folks to put resources into the field.

The year 2012 is also generally considered the start of the “deep learning revolution”. The term
“deep learning” refers to a branch of ML that is based on neural networks with many layers (hence the term “deep”). Although this basic technology had been around for many years, it was in 2012 when [KSH12] used deep neural networks (DNNs) to win the ImageNet image classification challenge by such a large margin that it caught the attention of the wider community. Related advances on other hard problems, such as speech recognition, appeared around the same time (see e.g., [Cir+10; Cir+11; Hin+12]). These breakthroughs were enabled by advances in hardware technology (in particular, the repurposing of fast graphics processing units (GPUs) from video games to ML), data collection technology (in particular, the use of crowd sourcing tools, such as Amazon’s Mechanical Turk platform, to collect large labeled datasets, such as ImageNet), as well as various new algorithmic ideas.

Reference: Probabilistic Machine Learning: An Introduction


text:
Does ChatGPT imply that the direction of knowledge graph is unpromising?
In this question I asked about the role of knowledge graphs in the future, and in this answer I found that If curation and annotation are not sufficient, the knowledge base maybe cannot apply in AI.
ChatGPT does not utilize a knowledge graph to understand or generate common sense, then I wonder how knowledge graphs can be utilized in the future. Will they be replaced by LLMs?

It has a reason ChatGPT is not named as KBGPT/Q&AGPT/ExpertGPT, etc, OTOH though lots of curated plausible background knowledge belief network is there it doesn't mean any rational agent can utilize them to the fullest extent either deductively or inductively due to lack of insight of relevance.

@mohottnad ChatGPT is not named KBGPT/Q&AGPT/ExpertGPT but has some basic common sense, you know Siri does not and Watson was very limited.

A couple of days ago, Jordi Torras from Inbenta posted that chatGPT fails at classifying a particular integer as prime, while their chatbot nails it. But the goal of a chatbot is no way factoring integers, is it?
Some weeks ago, Stephen Wolfram suggested some combination of chatGPT and their WolframAlpha, a curated engine for computational intelligence.
A wealth of domains could benefit from integrating preexisting knowledge into the conversational skill of transformers.
As a simple example, take "explain how 30 is 2x3x5", where the verified information plugged as a prompt may be obtained from a curated system and the natural language exposition could be finally written by a conversational system.
I don't foresee knowledge absorbed by LLM, but some form of combination between both techiques. Consider the times tables, the chemical elements, or lots of well known and established knowledge pieces. Is there any advantage in texting all that structured information to afterwards gradient descent train on it? Not to mention algorithms, from Viterbi to  Quick Sort to the Fast Fourier Transform. Those look like specialized intelligence modules to be interfaced by Large Language Models, rather than (re)learned from scratch.

The idea of combining logical or rule-based AI with statistical AI is not new. Even before these GPTs, there were already people claiming that we need to combine both approaches. I agree that ultimately an AI will need to use, and understand when and how to use a tool like a knowledge graph or Wolfram Alpha in order to be reliable. Humans have some kind of database of facts in their heads, which we use when we need. Unfortunately, many people think they can just differentiate end-to-end and get a magical model that works like a human, without even knowing how a human brain works.

To make this answer even more useful, I would recommend that you provide the links to the blog posts that you mention.

It is correct that curation and annotation are crucial to knowledge graph. At the same time, such annotation has been accumulated intensely in few areas like medical and manufacturing (some publicly, and some internally within the organization) - partly accelerated by the need of data interoperability and standardization within the industry.
So while it may not be very ready yet for generic use cases, some form of knowledge graphs/ontologies are already utilized for a long time in domains mentioned earlier.
Besides that, the current active research on knowledge graph generation/inference will potentially increase the breath and depth of the graph in a more scalable way.

Have you ever compared ChatGPT-like LLMs with knowledge graphs in such domains?

If you also want to produce language models that are reliable, you also need to "curate" them (either with RL from human feedback or whatever). I even claim that you need more "curation" to make models like GPTs be more reliable and thus useful than knowledge graphs because they work in ways that are less understandable than knowledge graphs, which may also be built using ML techniques, just to clarify. In the end, a knowledge graph is some kind of graph database, which contains hopefully true facts, but it's not exhaustive.

OTOH, models like GPT are stochastic parrots that seem to be able to produce a lot of stuff, but that may spit out misinformation, and you wouldn't know if you are unaware of the topic, unless you check yourself in a reliable source (which immediately implies that ChatGPT is not reliable, which is clearly true), but at that point why are you using ChatGPT if you can't rely on it? Maybe some form of creative art? Or maybe to fool people? That most useful use of these models is to fool people. It may be good for politicians.


text:
What are other types of knowledge bases except entity-entric KB?
I learned a term entity-centric knowledge base, then I wonder what the non entity-centric knowledge graphs are?
I googled by keys words "entity-centric knowledge base vs" and got some articles with this most related one: A comparison between Entity-Centric Knowledge Base and Knowledge Graph to Represent Semantic Relationships for Searching as Learning Situations, then I thought "Knowledge Graph to Represent Semantic Relationships" would be the answer. But I searched and found that knowledge graphs are just large-scale knowledge bases.
I wonder how entity-centric knowledge base, knowledge graph and knowledge base relate to each other? Maybe ontology-centric knowledge bases?


text:
Showing first layer RGB weights similarly to AlexNet
I would like to show the RGB features learned in the first layer of a convolutional neural network similarly to this visualization of the same layer's features from AlexNet:

My learned weights are in the range [-1.1,1.1]. When I use imshow in python or imagesc in Matlab, the weight values are clipped to [0,1], leaving only positive weights intact, everything else black (obviously).
Negative weight values could be informative, so I don't want to clip them. Rescaling the weights to [0,1] works fine for grayscale features, but not for RGB features as it is unclear how negative values of a channel should be visualized. In the above picture 0 furthermore seems to map to the middle of the range (gray).
How are such RGB features visualized so that they look similarly to above AlexNet visualization?
(Sorry for the beginner's question.)

It seems Alex has just used the Matlab function mat2gray, as described here: https://www.mathworks.com/help/vision/ug/image-category-classification-using-deep-learning.html
The visual outcome of the features is very similar. mat2gray will simply scale the weights between 0 and 1 (no clipping).
Leaving the (slightly adapted) example code of Mathworks here for future reference:
layer1weights = mat2gray(layer1weights) ;
layer1weights = imresize(layer1weights,5) ;

figure
montage(layer1weights)
title('First convolutional layer weights')


text:
Different useful approaches of implementing real-time AI?
I identify myself as a human agent. It is time to think about oncoming senior research and due to small experience in gamedev(as well as in AI field), some questions are raised. What are the most suitable approaches to implement real-time simple AI agent in an action game? I've heard something about cognitive architecture like ACT-R.
By design, entity's AI can have several mutually exclusive states.

This is an existing AI of game, which has states, events and schedules. However, the code is complicated and not flexible. Also, it does not use any cognitive architecture, which I consider as a drawback.
https://www.youtube.com/watch?v=9jO-P3kXlCI
Please, using your experience suggest any modern techniques, which can copy such behaviour as in image or video.
Thank you for your perception.

You're on the right track though here;concerning Artificial Intelligence community principles,we don't recommend resources.also your question should be reasonably scoped. If you can imagine an entire book that answers your question, you’re asking too much.

Thank you. Now I removed useless text. Can you read it again?

I think it would help if you could be a little more clear about what exactly you're asking.  Maybe describe the game and what your agent should do, in text form? A lot of people will probably be put off by the need to click through to a video in order to properly evaluate the question.

About 15 years ago, John Laird's group at Michigan used the Soar rule-based architecture to play several FPS games effectively (Quake II, Descent III):
http://ai.eecs.umich.edu/people/laird/games_research.html
Here's Laird's overview article from 'Computer':
https://www.researchgate.net/profile/John_Laird/publication/2955463_Laird_JE_Using_a_computer_game_to_develop_advanced_AI_Computer_34_70-75/links/54d0f59a0cf20323c21a1bd7/Laird-JE-Using-a-computer-game-to-develop-advanced-AI-Computer-34-70-75.pdf


text:
How to understand 'losses' in Spacy's custom NER training engine?
From the tid-bits, I understand of neural networks (NN), the Loss function is the difference between predicted output and expected output of the NN. I am following this tutorial, the losses are included at line #81 in the nlp.update() function.
I am getting losses in the range 300-100. How to interpret them? What should be the ideal output of this losses variable? I went through Spacy's documentation, but nothing much is written there about losses. Also, please let me know the links to relevant theories to understand this in general.

A critical goal of training a neural network is to minimize the loss.  Loss is not explained for spaCy because it is a general concept for machine learning and deep learning.  Loss is not specific to spaCy and although there are some finer details I don't believe that is your inquiry.
In general, to understand loss functions, I recommend the following resources:

Loss and Loss Functions for Training Deep Learning Neural Networks by Jason Brownlee.  Jason writes some great articles and tutorials and I highly suggest reading his other articles and books.
Introduction to Loss Functions
Loss Functions in Machine Learning for Beginners by John Kaller

If you like videos watch:

Loss Functions Explained by Siraj Raval.  Siraj has many great videos on machine learning.


text:
How to construct Transformers to predict multidimensional time series?
There is plenty of information describing Transformers in a lot of detail how to use them for NLP tasks. Transformers can be applied for time series forecasting.  See for example "Adversarial Sparse Transformer for Time Series Forecasting" by Wu et al.
For understanding it is best to replicate everything according to already existing examples. There is a very nice example for LSTM with flights dataset https://stackabuse.com/time-series-prediction-using-lstm-with-pytorch-in-python/.
I guess I would like to know how to implement transformers for at first univariate (flight dataset) and later for multivariate time series data. What should be removed from the Transformer architecture to form a model that would predict time series?

There is an implementation of the paper ("Adversarial Sparse Transformer for Time Series Forecasting"), in Python using Pytorch, here. Although it has the training and evaluation functionality implemented, it appears to be lacking a function for running a prediction.  Maybe you can fork it and extend it.
UPDATE
There is also a paper, "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting", by Zhou et al., which does forecasts on univariate and multivariate data.  Their code is here.


text:
Is there a technique for analyzing the relationship between time-series clusters?
I have two time-series datasets (temperature and speed of the vehicle). I will use Agglomerative Hierarchical Clustering and DTW to cluster both datasets.
I am looking for a technique (like regression model) to find the relationship between two time-series clustered data. I am curious to find the relationship between changes in temperature and vehicle speed. Does anyone have an idea?

I have no clue. Good question.


text:
Why doesn't the high precision of neural network weights improve accuracy?
Consider the following paragraph from the subsubsection 3.5.2: A dtype for every occasion chapter named It starts with a tensor from the textbook titled Deep Learning with PyTorch by Eli Stevens et al.

As we will see in future chapters, computations happening in neural
networks are typically executed with 32-bit floating-point precision.
Higher precision, like 64-bit, will not buy improvements in the accuracy of a model and will require more memory and computing time.
The 16-bit floating-point, half-precision data type is not present
natively in standard CPUs, but it is offered on modern GPUs. It is
possible to switch to half-precision to decrease the footprint of a
neural network model if needed, with a minor impact on accuracy.

The paragraph is discussing the precision of weights of a neural network. Three types of floating-point numbers are discussed: 64-bit, 32-bit, and 16-bit (half precision). It has been said that it is recommended to use 32-bit numbers. If we decrease to half-precision then it will demote accuracy but if we increase to 64-bit then there will be no impact on accuracy.
I am confused about how the improvement in precision from 32-bit to 64-bit does not improve the accuracy but the improvement from 16-bit to 32-bit can improve accuracy?

First, I have not read and do not have that book.
That said, I would interpret that statement in the context of the intractability of guaranteeing that the optimization function will find global minima in the loss surface.
In other words, higher precision values will do nothing to improve whether we have descended into a global or local minimum.  On the other hand, using a half precision value could result in a decrease in accuracy since the learning rate and weights may not have the precision to "settle" into the lowest point of a local minimum.  If you consider it in this context, a 64 bit value would not greatly improve the accuracy over a 32 bit value, while a 32 bit value would potentially have a noticeable improvement over 16 bit values.


text:
Why is the WMT16 dataset favoured for evaluating machine translation models?
The Workshop on Statistical Machine Translation has released translation challenges each year from 2004 on, which feature a dataset of sentence pairs in a variety of languages.
Even though the conference has been taking place each year, with ever growing dataset sizes, the dataset from 2016 is still being used frequently as an evaluation dataset in scientific papers when evaluating machine translation models and it has by far the most downloads on the huggingface hub.
I would be grateful if someone could shed light on the reason for which the WMT16 dataset is so strongly favoured by the machine translation community.

That doesn’t answer the question. WMT16 featured seven languages, whereas e.g. WMT19 featured ten.

My guess is the reason for the stats on Huggingface Hub is in fact pretty random: something like that someone uses the particular dataset in a tutorial.
Usually, it is considered a good idea to use the same data as previous work a paper is compared with. WTM16 was the first one where neural MT models clearly dominated, this might be part of the reason. There were bigger datasets in later years, but until the WMT16 data will be considered really small, some people will certainly prefer more direct comparability with previous work.
The WMT14 dataset is also frequently used in papers. The original Transformer paper presented the results on WMT14 data and since that time people tend to compare their results with them.
Also that the data is bigger does not necessarily have to be a good thing. University labs do not have as big computational resources as big companies and they need to trade-off the number of experiments and the training data size. Big, but not huge dataset such as WMT16 might be a good choice.

Could it be also because in 2016 WMT became a conference and hence more "trustworthy" as a citation source? Until then WMT was a workshop. Maybe the older the papers, the more famous and studied they get :D
https://machinetranslate.org/wmt


text:
Can in principle GPT language models learn physics?
Does anyone know of research involving the GPT models to learn not only regular texts, but also learn from physics books with the equations written in latex format?
My intuition is that the model might learn the rules relating equations and deductions, as they can learn statistically what correlates with what. I understand that the results can also be a little nonsensical, like the sometimes surreal paragraphs written by these models.
Have there been any attempts to do this?

It's very unlikely that any machine learning model can really understand physics (in a "complete" way that a human can). Maybe you can use these models to find correlations between mathematical symbols and descriptions, but a full or complete understanding of a difficult subject like physics is unlikely to be accomplished with a single model. However, note that neural networks (or deep learning systems) have already been used to solve math problems (at least, people have tried to do this). See, for example, this question.

There is an interesting paper that tests GPT, BERT and ELMo to "show that the Transformer can be mathematically interpreted as a numerical
Ordinary Differential Equation (ODE) solver for a convection-diffusion equation
in a multi-particle dynamic system".  See https://web.stanford.edu/~yplu/pub/TransformerODE.pdf


text:
how to recognise handwriting and convert into text?
I'm trying to develop a project that recognises handwriting and converts into text. What are the algorithms and tools to be used?

I suggest you take a look at the work of Alex Graves - https://www.cs.toronto.edu/~graves/handwriting.html

This feels a bit much like a "do my homework for me"question to me.  Could you make this question a little more specific?

I need to know about handwriting recognition through image processing i.e recognising patterns of handwriting in the image. I need some suggestions. Whether it is feasible or not?

What you described sounds to me like Optical Character Recognition(OCR).
If you want to implement your own, I would say read through how an open source OCR like Tesseract was implemented.
Otherwise just google for OCR and you will find a list of OCR engines, both comercial and open-source. To list but a few:

Tesseract: https://github.com/tesseract-ocr/tesseract
Matlab Computer Vision Toolbox's OCR: http://www.mathworks.de/de/help/vision/ref/ocr.html

Most OCRs makes use of a combination of a segmantation and image classification(usually done with Convolutional Neural Networks).
See here for more information.

Thanks for the info :) . But I want to recognise using image processing. How can I do that?


text:
Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?
BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of the input length of the text, which makes it more cumbersome to use as input to downstream neural networks that take only fixed-size inputs.
Are there any transformer-based neural network architectures that can encode a piece of text into a fixed-size feature vector more suitable for downstream tasks?
Edit: To illustrate my question, I’m wondering whether there is some framework that allows the input to be either a sentence, a paragraph, an article, or a book, and produce an output encoding on the same, fixed-sized format for all of them.

This might help and partially answers your question. You can only try to reduce the size of the obtained state with some convolution, but I don't think this is being done yet.

BERT does provide a fixed-size output. The encoding of the special [CLS] token, which is always prepended to every input example, is meant to encode the entire sentence. Since it is a token, its encoding is always a fixed-length vector of length H, e.g. H=768 for BERT-Base. Specifically, the [CLS] encoding is passed through a "pooling layer", which is just a HxH fully-connected layer, with linear activation. This is called pooled_output in the TF Hub module.

As mentioned, use the CLS token. Else you could just define some max_length, and pad to it when its to low and then use the mean of the tokens. But, look at sentence-bert for e.g https://www.sbert.net/#usage - it generates fixed sized sentence embeddings with varying input sizes :)

One way you could do it is by using SentenceTransformers.

SentenceTransformers is a Python framework for state-of-the-art
sentence, text and image embeddings. The initial work is described in
our paper Sentence-BERT: Sentence Embeddings using Siamese
BERT-Networks.

To install it via pip
pip install -U sentence-transformers

To generate sentence embedding
from sentence_transformers import SentenceTransformer

# We are using "paraphrase-MiniLM-L6-v2" model here, You can find list of model [here][2]
model = SentenceTransformer('paraphrase-MiniLM-L6-v2')

# Our sentences we like to encode
sentences = ['This framework generates embeddings for each input sentence',
    'Sentences are passed as a list of string.',
    'The quick brown fox jumps over the lazy dog.']

# Sentences are encoded by calling model.encode()
embeddings = model.encode(sentences)

#Print the embeddings
for sentence, embedding in zip(sentences, embeddings):
    print("Sentence:", sentence)
    print("Embedding:", embedding)
    print("")

But remember, SentenceTransformers models have an input limit as well, 512 tokens usually. If your text is more than that, then it might not be a suitable method.

Thanks for you answer. Well, if it has an input limit, the input isn’t truly of arbitrary length (in practice I guess the input also becomes of fixed length, and that you just use a padding token to fill it out after the text has ended). I was wondering whether there is some framework that allows the input to be either a sentence, a paragraph, an article, or a book, and produce an output encoding on the same, fixed-sized format for all of them.

Not till today.

Building transformer models, in general, for arbitrary context retrieval is difficult because the architecture and training simply aren't designed to handle this. This is even more of an issue if you want to use the model to embed documents, as you typically need to do some specialized training to get meaningful embeddings.
There has, however, been work on long-context embedding models more generally: e.g., M2-BERT with a maximum of 32k tokens. i.e., it can support documents with length <= 32k. This, however, isn't a transformer model. Comparing to other models in their paper, transformer-based embedding methods only seem to have a maximum context length of 8k.

Ok, so is M2-BERT a state space model, kind of like RWKV and Mamba?

I don't think RWKV is a state space model, but it would be more similar to Mamba.