text:
High Fluctuations in Validation Curve
Below I attach an image of accuracy curves. I got a lot of suggestions regarding some improvement in below curves. Following are my experiments in order to make the curve stable-->

I used lr = 4.44e-5
Weight decay = 3e-6
Used BatchNorm and Dropouts in architecture

I feel that weight updates are very sensitive to validation data, but not very sensitive to train data. I would appreciate any suggestions for making validation curve stable (currently just concerned with stability, not with overfitting or under-fitting)
My model is 1.3 M parameters large. Total dataset = 13,000 hyper-spectral images of wheat seads.
Test_size = 20%
Train size = 55%
Validation size = 25%
Here is my model architecture :
class HSIModel(nn.Module):
  def __init__(self , config, n_res_blocks = 12):
    super(HSIModel, self).__init__()
    self.config = config
    self.in_channels = self.config['in_channels']
    self.n_res_blocks = n_res_blocks
    self.squeeze_channels = 512
    self.model = self.get_model()
    self.layer_lr = [{'params' : self.base_model.parameters()},{'params': self.head.parameters(), 'lr': self.config['lr'] * 1}]

  def get_model(self):
    self.head = nn.Sequential(
                  nn.Flatten(1) ,
                  Dense(0.4, 512, 128),
                  # Dense(0.15, 256, 128),
    )
    li = [ResidualBlock(256) for i in range(self.n_res_blocks)]
    self.base_model = nn.Sequential(
        # BandAttentionBlock(self.in_channels),
        SqueezeBlock(self.in_channels, 100),
        SqueezeBlock(100, 512),
        nn.Dropout(p=0.2) ,
        XceptionBlock(512, 256),
        XceptionBlock(256, 256),
        nn.Dropout(p=0.2) ,
        # XceptionBlock(256, 128),
        nn.Sequential(*li),
        # XceptionBlock(128, 256),
        # SeparableConvBlock(128, 256),

        SeparableConvBlock(256,512),
        nn.MaxPool2d(kernel_size = (3,3) ,stride = (2,2)) ,
        nn.Dropout(p=0.15) ,
        nn.AdaptiveAvgPool2d((1,1)) ,
    )
    return nn.Sequential(
                  self.base_model,
                  self.head,
                  Dense(0, 128, self.config['num_classes']),
                        )

  def forward(self, x):

    return self.model(x)

Below I also attach the code file where modules used in above architecture are written. Please also comment on architecture.
Pls help out, if you need any other information, pls mention
The modules used in above model have their usual architecture.

what about the loss

Similar behaviour

Hi @SarvagyaPorwal and welcome to AI Stack Exchange! If possible, please outline a specific question for this post. It may help you get answers more quickly.


text:
Are there neural networks that compute weights dynamically based on geometric attributes of neurons?
I am interested in exploring neural network architectures where the weights are not stored but are computed dynamically based on certain attributes or "dimensions" of the connected neurons.
For instance, consider a fully connected network where each neuron has three geometric attributes (x, y, z). The weight between two neurons with attributes (x1, y1, z1) and (x2, y2, z2) could be computed using a function like: w = 1/((x2 - x1)^2 + (y2 - y1)^2 + (z2 - z1)^2) (let's call this function 'invsq').
This approach seems like it could potentially reduce the memory requirements for storing weights at the cost of a more complex computation to determine the weights dynamically.
Has there been any research or are there existing neural network architectures that employ a similar method for computing weights? I am open to learning about related or alternative approaches if the exact type of network I'm describing doesn't exist.
I've come across some articles discussing dynamic weight strategies and alternative methods for weight computation in neural networks, but none that specifically address geometric attributes of neurons as a basis for weight computation.
Thank you in advance for any insights or references you can provide!

It's an interesting idea. One major flaw with the current invsq function is that it is always positive. You need to fix that, but the general idea of a function based on distance between neurons in a reduced dimension space (compared to product of layer sizes) might have some interesting properties, and is trainable via back propagation of gradients provided the distance-related function is differentiable. I don't know if there is any related research, but might look into it to see if this can be answered

I see your point, however, as far as i know, the problem with NN are the FLOPS, not the memory... memory is getting cheaper at a much faster rate than the scaling of CPU/GPU

@NeilSlater Appreciate your feedback. The invsq function is a starter idea. I envision a system where each neuron has a unique scale, adjusted during training. Scales could mimic logical gates like NOT (negative scales, making '-1' behave like low power), although OR gates are still in contemplation. This might require a broader network architecture to form multiple circuits. Additionally, the neuron dimensions can remain fixed, and altering the scales could solely drive the learning. Your insights are invaluable as I refine this concept.

@Alberto Thank you for highlighting the concern regarding FLOPS. The idea of dynamically computing weights aims at exploring alternative architectural and operational paradigms for neural networks, which might have implications beyond memory efficiency. The notion of embedding logical operations within the network through adaptable scales and fixed dimensions could potentially lead to novel mechanisms for neural computation, even if it necessitates more complex calculations. Your input sheds light on an important aspect, and I appreciate the discussion.

@DeadbeefDevelopment sure, my comment wanted to be just a note... in my experience, I've never heard of something like this, where neurons have properties and not connections, however, "plain" NN are universal approximators, so they can definitely approximate any network that you would be able to build with this new paradigm

this however doesnt mean that they might not work better (being able to approximate is far from "being better")... might be that this structure induces some properties or regularization that enables generalization easier, or something else

personally, i don't recall anything like this, apart from low rank dense layers (see https://arxiv.org/pdf/2009.03887.pdf) where there is a scalar product between low rank matrix, that are similar to your "xyz" product

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Using nearest neighbor in RANSAC
I found many resources online talking about nearest neighbor concept in RANSAC. For example, figure 2 of this paper, this article and this repo talk about nearest neighbor in the context of RANSAC. But they dont explain it clearly exactly for what they are utilising nearest neighbor concept in RANSAC. May be I am too dumb to get the obvious intuition or may be because they have not put it in pseudocode.
I understand below pseudo code of RANSAC from wikipedia. Can you please explain at what step and for what nearest neighbor is used in RANSAC. Also, it will be best if you can modify below pseudocode to explain the same. Also,  I am using RANSAC for aligning two point clouds. So it will be great if you can explain in that context. If not, thats fine too, as I can derive understanding for my problem context. You may just explain in the context of usual model fitting problem.
Given:
    data – A set of observations.
    model – A model to explain the observed data points.
    n – The minimum number of data points required to estimate the model parameters.
    k – The maximum number of iterations allowed in the algorithm.
    t – A threshold value to determine data points that are fit well by the model (inlier).
    d – The number of close data points (inliers) required to assert that the model fits well to the data.

Return:
    bestFit – The model parameters which may best fit the data (or null if no good model is found).

iterations = 0
bestFit = null
bestErr = something really large // This parameter is used to sharpen the model parameters to the best data fitting as iterations go on.

while iterations < k do
    maybeInliers := n randomly selected values from data
    maybeModel := model parameters fitted to maybeInliers
    confirmedInliers := empty set
    for every point in data do
        if point fits maybeModel with an error smaller than t then
             add point to confirmedInliers
        end if
    end for
    if the number of elements in confirmedInliers is > d then
        // This implies that we may have found a good model.
        // Now test how good it is.
        betterModel := model parameters fitted to all the points in confirmedInliers
        thisErr := a measure of how well betterModel fits these points
        if thisErr < bestErr then
            bestFit := betterModel
            bestErr := thisErr
        end if
    end if
    increment iterations
end while

return bestFit

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How can a RNN with 256 cells accept a input of any size?
I built a 3 layered RNN model with 256 cells each using torch. Input feature size is set to 40. Below give a basic Idea on the model.
 lstm = nn.LSTM(input_size=40,
                        hidden_size=256,
                        num_layers=3,
                        batch_first=True)
 self.linear = nn.Linear(in_features=256,
                            out_features=256)
 self.relu = torch.nn.ReLU()

Now after getting an Idea on how it looks, how can it accept an input of any size of form (x, 40) say (160, 40). It should only be able to accept of size (256, 40). If not
What happens after time step 160? Do the remaining cells take output of previous cell to be its input. Like below
Or is the input mapped from (160, 40) to (256, 60) using some Neural Network before giving it to the RNN.


text:
Can manual feature extraction be considered a part of a learning algorithm?
A learning algorithm is a tuple $(\mathcal{H}, \mathcal{O}, \mathcal{L})$ where $\mathcal{H}$, $\mathcal{O}$ and $\mathcal{L}$ are the hypothesis class, optimizer and loss function respectively.
We can also think of the learning algorithm as the whole pipeline that take us from data to a model.
Suppose we have the following machine learning approach:
$$\text{input, output data} \xrightarrow{\phi} \text{features, output data} \to \text{training a classifier} \to \text{model}  $$
where $\phi$ is fixed, i.e. not learnable from the data. One such example, is the SIFT method if we assume that our input is an image. In this case, can we define the learning algorithm as the whole pipeline that take us from input, output data to the model including the feature extraction step as the learning algorithm? In other words, is the feature extraction step consider a part of a learning algorithm?
My view, possibly wrong
I will argue yes, since we consider CNNs as a learning algorithm. The essential difference is that in the CNN case, $\phi$ is not anymore fixed but learnable from the data.
Moreover, what is the hypothesis class of the learning algorithm when fixed feature extraction methods are used?
If $z=\phi(x)$, does the hypothesis class consists of functions:
$$f \colon Z \to Y$$
or:
$$f \colon X \to Y $$
?


text:
Simultaneous forecasting and classification
I'm working on a project where I need to perform both forecasting (regression) and classification using time series data. The dataset is labelled.
I've been exploring LSTM networks due to their effectiveness in handling sequential data.
Is it possible to use a single LSTM model for both forecasting and classification tasks?

What do you mean by a single model? Multi-Task learning is a good option, it would be a model with multiple outputs.

Do you mean something like both deciding if an image is of a dog, cat, or alligator (classification) AND how tall the animal is (regression)?

@Dave yes this is a similar case.


text:
How to make a RandomForestRegressor learn to differentiatie similar inputs with different outputs?
I'm working on a regression task with Sklearn RandomForestRegressor and I'm having some trouble distinguishing between two similar data with very different expected outputs. For example, each pair of time-stamp (x-axis) and value (y-axis) for the two blue curves shown below are different inputs for the model, and in the context of my data they are considered similar curves (and similar pair).

The expected value to be produced by the model is another value associated with the given time-stamp,as shown in the two green curves below. However, even with the two blue curves being considered similar, the expected output is very different for both.

I tried to give the full curve as input, but the generalization is compromised due to unbalances in my dataset (I mean, the result is always the same profile with small differences to each given curve). Also, all curves are normalized according to mean value (which is necessary for my research). This is just an example, but there are other cases similar to this one in the dataset.
What is happening is that my model is learning well the time-stamps with similar behavior but when there is too much difference between the points, the model is reproducing only the one that was more present in the dataset, resulting in curves that are similar for most points, but very different in other points.
I know that this is an expected behavior of learning models, but I was trying to figure out if there is anything I can do in order to minimize these difference and produce better results.


text:
use AI to creat images from texts
I want to use AI to create art images from a single text available in different languages, with the generated images being different depending on the language of the given text.
example 1:
step 1: the input text is orginal : "It does not do well to dwell on dreams and forget to live."
step 2 ( whatever it means) : AI tech. generates a corresponding image X
example 2 :
step 1: the input original text is translated into French : "Il n'est pas bon de s'attarder sur les rêves et d'oublier de vivre.
step 2 : AI tech. generates a corresponding image Y, diffrent from the image X due to the fact that the text language has changed
I believe one way to achieve this is by training a single AI model that can handle multiple languages and can produce different styles based on each language.
Any ideas on the end to end process and costs ( say for a bilingual corps of 1000 words ) ?
Thanks

To what extend do models such as DallE already do this for you? If you give a prompt in a different language, i reckon they give some other kind of image?


text:
Multi-Task VAE, Decoder Activation Functions?
I'm working on a Multi-Task VAE with one Encoder and two Decoders. The input consists of a vector with parameters which describe a design of a fluid system. The goal is to reconstruct the parameters with one decoder and predict the physical values of the design with the other one.
The model should be able to generate new designs with the correct physical values.
class MTVAE_CNN(nn.Module):
    def __init__(self, task_num, input_shape, input_size=28,z_dim=4, b=1,batch_size=10,c = 1, a = 1):
        super(MTVAE_CNN,self).__init__()

    self.task_num = task_num
    self.log_vars = nn.Parameter(torch.zeros((task_num)))

    self.z_dim = z_dim
    self.a = a
    self.b = b
    self.c = c
    self.input_size = input_size
    self.input_shape = input_shape
    self.batch_size = batch_size
    self.loss_list = []

    self.encoder = nn.Sequential(
        nn.ConvTranspose1d(1, 16, kernel_size=5, stride=3, padding=2),nn.ReLU(),nn.Dropout(p=0.2),
        nn.ConvTranspose1d(16, 32, kernel_size=5, stride=2, padding=2),nn.ReLU(),nn.Dropout(p=0.2),
        nn.Conv1d(32, 64, kernel_size=5, stride=2),nn.ReLU(),nn.Dropout(p=0.2),
        nn.MaxPool1d(7),nn.ReLU(),nn.Dropout(p=0.2),
        nn.Flatten()
        )

    self.decoder = nn.Sequential(
        nn.Unflatten(1,(1,self.z_dim)),
        nn.ConvTranspose1d(in_channels=1, out_channels=16, kernel_size=5, stride=3),nn.ReLU(),
        nn.Conv1d(16, 32, kernel_size=5, stride=3),nn.ReLU(),
        nn.MaxPool1d(7),nn.ReLU(),nn.Dropout(p=0.2),
        nn.Flatten()
    )

    self.predictor = nn.Sequential(
        nn.Unflatten(1,(1,self.z_dim)),
        nn.ConvTranspose1d(1, 16, kernel_size=5, stride=3),nn.ReLU(),
        nn.Conv1d(16, 32, kernel_size=5, stride=3),nn.ReLU(),
        nn.MaxPool1d(3),nn.ReLU(),nn.Dropout(p=0.2),
        nn.Flatten()
    )

    self.conv_out_size = self.calculate_conv_output_size(self.input_shape)

    self.Bn1 = nn.Linear(self.conv_out_size, self.z_dim)
    self.Bn2 = nn.Linear(self.conv_out_size, self.z_dim)

    self.dec_lin = nn.Linear(self.calculate_decoder_output_size(self.input_shape),self.input_size)
    self.pre_lin = nn.Linear(self.calculate_predictor_output_size(self.input_shape),2)

    self.sig1 = nn.ReLU()
    self.sig2 = nn.Sigmoid()

def calculate_conv_output_size(self,input):

    x = torch.randn(*input) # Create a random input tensor
    x = x.unsqueeze(1)
    x = self.encoder(x)

    return x.view(x.size(0), -1).size(1)

def calculate_decoder_output_size(self,input):

    x = torch.randn(*input) # Create a random input tensor
    x = x.unsqueeze(1)
    x = self.encoder(x)
    x = self.Bn1(x)
    x = self.decoder(x)

    return x.view(x.size(0), -1).size(1)

def calculate_predictor_output_size(self,input):

    x = torch.randn(*input) # Create a random input tensor
    x = x.unsqueeze(1)
    x = self.encoder(x)
    x = self.Bn1(x)
    x = self.predictor(x)

    return x.view(x.size(0), -1).size(1)

def reparameterise(self, mu, logvar):
    if self.training:
        std = logvar.mul(0.5).exp_()
        eps = std.data.new(std.size()).normal_()
        return eps.mul(std).add_(mu)
    else:
        return mu

def bottleneck(self,h):

    mu,logvar = self.Bn1(h), self.Bn2(h)

    z = self.reparameterise(mu,logvar)

    return z, mu, logvar

def predict (self,z):

    #y1 = self.predictor(torch.cat([z.unsqueeze(-1), y.unsqueeze(-1)], dim=1))

    y1 = self.predictor(z)

    y1 = self.pre_lin(y1)

    y1 = self.sig1(y1)

    return y1

def encode(self,x):

    x = self.encoder(x.unsqueeze(1))
    #mu = mu_logvar[:, 0, :]
    #logvar = mu_logvar[:, 1, :]

    #x = x.view(x.size()[0], -1)

    mu,logvar = self.Bn1(x), self.Bn2(x)

    z = self.reparameterise(mu,logvar)

    return z,mu,logvar

def decode(self, z):

    y = self.decoder(z)

    y = self.dec_lin(y)

    y = self.sig2(y)

    return y

def forward(self, x):

    z, mu, logvar = self.encode(x)
    y_hat = self.predict(z)
    x_hat = self.decode(z)

    return x_hat, y_hat, mu, logvar

def loss_function(self, x,y,b):

    x_hat, y_hat, mu, logvar = self.forward(x)

    MSE_Reg = nn.MSELoss(reduction = "sum")(y_hat, y)

    MSE = nn.MSELoss(reduction = "sum")(
        x_hat, x)

    KLD = 0.5 * torch.sum(logvar.exp() - logvar - 1 + mu.pow(2))

    kld_loss = b*KLD

    loss1 = self.a*(MSE)

    loss2 = self.c*MSE_Reg

    return loss2,loss1,kld_loss,b

I tried different optimization techniques for multitask learning, grid search for the loss weights, uncertainty weighted loss and increasing the hidden layers of the encoder. The 1DConvolution Layers have been delivering the best results, but theres still room for improvement.
The model should concentrate on representing the designs in the latent space with the information about the values, without focusing too much on the representation of the values.
Is there a way to influence the behavior and the performance of the model with the last decoder activation functions?


text:
Sampling dollar bars for ML model of multiple tickers
I have a Neural Network model that provides predictions for the future returns of a portfolio comprising stocks and cryptocurrencies. The original model operates on standard time bars (sampled at a specific time step, for example 5mins) and generates daily predictions at a specific time, utilising processed OHLC (Open, High, Low, Close) candlestick data as input.
My question is general and the details of the model are not important. The most important thing is that the features are computed from the time bars. For example, the input dataset can be the intraday 5min candles for each instrument, where some processing is involved on the values of the candles.
I would like to extend this Neural Network model to make it work with dollar bars, instead of time bars. Dollar bars are sampled every time a given USD volume is traded. If we choose to sample dollar bars for every 100$, then each candle is still a OHLC candle, but the prices refer to the USD exchanged, not a specific time.
The first questions that comes to my mind are

The original model is time-dependent, issuing predictions daily at specific times. When transitioning to dollar bars, this time sensitivity is lost. How can we establish criteria for determining when the model should make predictions? For a single-instrument model, one possible approach might involve assessing the average daily count of dollar bars within a given sampling size and making predictions accordingly. However, when dealing with a multi-instrument model, how can we adapt this criterion?
What are the most prominent features that can be derived from dollar bars and are suitable for serving as inputs to a neural network? In the earlier example, the input dataset consists of a dataframe containing 5-minute intraday candles. We can reliably anticipate that there will be a 10:00 AM candle for any instrument every day. Consequently, by choosing this candle as a feature, we ensure that the input dataset consistently contains information about the 10:00 AM candle. However, when dealing with dollar bars, there is no assurance that two instruments will share the same dollar bar. So, it seems there is an ambiguity on how I can univocally define the input features for a Neural Network model.

Suggestions? Insights?


text:
Has There Been Research on Using a Neural Network as a Loss Function for Another Neural Network?
I'm intrigued by the idea of employing a separate neural network (which I'll refer to as the "loss network") to compute the loss for a primary network based on its inputs and outputs. The mechanism is as follows:
Primary Network:

Input: Any data sample (e.g., image, text).
Output: Predicted value or classification.

Loss Network (Evaluator):

Input: Inputs provided to the primary network and its corresponding outputs.
Output: Scalar loss value that evaluates the quality or suitability of the primary network's output given its input.

The computed loss would then guide the training of the primary network.
My questions are:

Has there been research or experimentation with such an architecture in the deep learning community?
If so, can anyone point me to relevant literature or papers that discuss this approach and its applications?
Are there known challenges or limitations associated with this method?

I'm keen to understand if this concept has been explored and any insights the community might have. Thanks in advance!

This is somewhat similar to Generative Adversarial Networks where a generator network tries to fool a discriminator network that it's generations are "real'.


text:
How does Google Maps pronounce streets properly?
How does Google Maps pronounce street names properly, or any maps platform for that matter?
Many words don't have straightforward pronunciations given the orthography/spelling, like "chabot (shaBOW)" or even "christina (krisTEENuh)" or many other words, how do they get the correct pronunciation, is it AI or is there a manual step in there to accomplish it?
I would like to make an open source English pronouncer thingy (take English text in JavaScript and write IPA for it), but for some cases it's impossible (tear drop vs. tear paper, tear has a different pronunciation), but other cases it might be possible but I'm not sure how production systems like Google Maps do it, thinking it's AI or something.


text:
Improving forecast for LSTM with additional data
There are two timeseries X and Y. The timeseries X spans the duration [1 Jan, 20 Jan ] and the Y spans [1 Jan, 25 Jan].  I am interested in timeseries forecasting of variable X for the duration [21 Jan, to 25 Jan] for that I also wanted to utilize the values of Y for the duration [21, 25].
What I have done so far-
I have tainted a LSTM model using the two time series, but that does not use future Y values for making the forecast.
 My questions -

Is it possible to incorporate future values of Y to improve forecast for X for the duration [21, 25] using LSTM
If the above is not possible with LSTM, can I perform it using another model.

The following is just an example dataset, the true data have sufficient values to train a timeseries model.

Thanks


text:
From Text-To-Speech to LLMs: Providing "writing style"
I've just recently learned about Text-To-Speech models and how they are trained. Unlike LLMs, to a provided pair (text, speech), a feature vector ${f}$, that was generated by more speech of that speaker and tries to subsume the general voice characteristics, is added. I.e., for a speaker ${S}$, triplets $({text}_i, {speech}_i, {f}_S)$ are used in training (where ${speech}_i$ are speeches from speaker ${S}$). Why? Because speech, compared to text, is much more high-detail-low-content.
I then wondered about training of LLMs. Texts of various sources (scientific, forums, joke books, ...) are used; all trying to fit into one and make one "writer". Hallucinations of LLMs seem to magnify when they are already "on the wrong path", which makes perfect sense: In an article full of nonsense, jokes, wrong data, more of exactly that will follow; so this statistical model of a knowledge base will proceed by giving more of that.
Of course, implicitly, LLMs learn how to write; I may just include in my prompt that "You are a child. Please talk to me as a child." and it will do so. But it was never particularly trained on that.
So the question is: Has there been any work on providing the source, theme, genre, ... of texts provided in training (which should be reasonably possible to add to the training data) as a separate "feature", so as to when prompting that LLM and providing the field of applications, it would generate better results on fewer training data and housr due to less necessary generalization?
(It should be possible to just try this on one's own, e.g., with LoRA, but my expertise and commitment is a bit too low for that, yet.)


text:
Can higher order derivatives be used in diffusion models?
I remember reading an article about predicting both the first and second derivative in a diffusion model and then using the Taylor method to denoise in each step, to achieve faster convergence. But I can't find it.
Is that even a thing?. Could it be used for image generation?


text:
Classes definition for detecting impervious surfaces on aerial photographies
My project is to use deep learning, essentially a UNET segmentation model, to detect impervious surfaces on high resolution aerial photographies.
I wonder if it's better to train the model with many different impervious surfaces classes like buildings, roads, parking lots, cars, etc... or if using a single class that regroup each of them could work the same?
I have to create my labels and masks myself so using a single class without differentiating the categories of impervious surface would be easier and will take less time. However, precision is what matter the most.
Any suggestions? up


text:
I'm trying to build image search like Google Photo-Image with face is given to model & it'll get all the images in database in which he/she is present
When a user upload a selfie, the model search same person in dataset of images of multiple persons and get back all the images in which that person is present.
Step 1: From dataset of images I detect all the faces and save the cropped faces by DeepFace(RetinaFace) and OpenCV.
Step 2: Read all the Cropped faces by OpenCV, resize it, store into a numpy array and standardize it.
Step 3: Feed it to VGG16(Keras) model without including top layer with 0 trainable parameters(use the weights) and extract the features from image numpy array.
Step 4: Feed these extracted features to clustering model(unsupervised)
I tried KMeans and Agglomerative Hierarchical Clustering but not getting as expected.
The clustering is not accurate at all, way more out of the way.
Can anyone help me to address this, how to approach this task, in which step I am making mistakes...
The images/data is not labeled and want to build unsupervised model of clustering.
Any help or suggestions will help me a lot.


text:
Terminology question around "slot filling"
This is a question about terminology.
Assume you have a chatbot with intent classification and slot filling. Assume you have an intent to handle questions about shops in a town. A user asks a questions like "Do you have shops in London" and as a response it should list the shops in London if there are any. Now 'London' is a city, the city is an entity that has to be extracted and in slot filling this is assigned to the parameter 'city' for the operation to find a list of shops.
Now assume the user asks "Do you have shops in my town". Now the entity of the city is missing. With an additional prompt you can ask the user what it his town. Assume you treat this in the very same intent and you want to fill the slot of the parameter 'city' is then the operation where you ask the user for the city part of the 'slot filling'? If not what would be the proper name for this operation?

Where did you find this term "slot filling"? Can you provide a reference?

Actually by exposure in interaction with a group of software engineers that have been referring to this term.
Myself I had a look at https://medium.com/geekculture/intent-classification-and-slot-filling-managing-web-pages-with-natural-language-4d21d5cee4e3 and got the impression "slot filling" should only be used for the automatic ML part, not for the process of asking the user to provide missing information.


text:
How can I feed data files of different sizes to a CNN?
I have 5022 data files. I want to feed them into a CNN model.
However, the lengths of data files vary from, say, 50 to 1015 rows (the number of columns is constant).
In the case of image files, we can resize or downsample. However, the files I am working with are not image files.
What options do I have to feed those data files into CNN without losing any information?
I am thinking about padding the smaller files with zero.
Would that be a good idea?
Crossposted: https://stats.stackexchange.com/questions/630163/how-can-i-feed-data-files-of-different-sizes-to-a-cnn


text:
How to implement a link constraint between the weights/bias of a neural network?
I'm new to the forum, and although I've tried my best, I hope not to write a duplicate post.
Here's the background:
I'm trying to build a neural network from scratch where certain weights are related to each other. For example, I have 2 sigmoid cells of weight and bias (w1, b1) and (w2,b2). I want these two sigmoids to intersect at 0.5 on the ordinate.
After calculation, I obtain the following constraints on the variables:
w_1 * b_2 - w_2 * b_1 =0
However, I don't know how to handle this constraint and integrate it into my neural network.
I have several ideas, but they all seem to have drawbacks:

Add a constraint directly to the loss function? I read about this in an article, but I don't understand how an equality constraint can be found in the loss function. We're not trying to "minimize" this constraint, but to make it true all the time.
Use the projected gradient method, but my constraint space is variable and depends on the weights/bias themselves?
Add a constraint layer? But where do I place it? How can I be sure that I'm still dealing with the same problem?

Have you seen anywhere a method or type of algorithm that uses this kind of constraint?
Thanks a lot!


text:
Which kind of model should I be looking at if I'm trying to estimate volume for 3D objects?
I'm trying to navigate the world of LLMs and understand which classes of models are ideal for which use case. So I'm wondering, for problems related to 3D volume estimation, which class of models are most likely to be able to solve it?
Some sort of multimodal thing, like Flamingo? Or one of the vision / OCR specific classes of models?


text:
how to know to define asuitable posterior for bayesian deep learning
I work with model its purpose for classification task in medical image, i want also to estimate uncertainty , so i work with Bayesian technique , i use a pre-trained model for feature extraction and then add Dense Variational layers to make classification , what is the suitable posterior we an define for medical imaging, because when dealing with dense-variational layer we have to define prior and postrior.
and did that right? ue pre-trained for feature extraction and  then add dense variational layer .
my propose for classification and uncertainty
note; i try normal distribution for posterior didnt work well


text:
In multilayer perceptron neural networks, is normalization required? or is optional? Why the network only seem to work when I use normalization?
I followed the steps in the following article (which teaches how to program a multilayer perceptron neural network from scratch in Python): https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/.
Using the tutorial in article, i testing 3 different datasets: seeds.csv (used as an example of the tutorial in the article itself), and also, Iris.csv and diamonds_filtered.csv, I noticed that the neural network in the article In the various tests I did, the error gradually reduced consistently, in all data sets, but it only works when I use the normalization technique presented in the article. When I run the script without using normalization, the error during training becomes very high and reaches a point that does not decrease, even if many epochs pass. Therefore, the conclusion I reached is that this neural network was only successful using normalization on the dataset.
The datasets I downloaded are here:
https://raw.githubusercontent.com/jbrownlee/Datasets/master/wheat-seeds.csv
https://github.com/selva86/datasets/blob/master/Iris.csv
https://github.com/selva86/datasets/blob/master/diamonds_filtered.csv
I know normalization is used to transform the data to a similar scale. I searched the internet and some sources say that if there are entries in the dataset that have very large or very small values, this can cause convergence problems.
So about normalization, I have a question: in the samples from these 3 data sets that I used (without using normalization), how can I know if there are very high or very small entries? How can I check this better? Is there any step by step I can do to check this?
And i also have another question about this: is normalization required or is optional?
Is it common and expected that a multilayer perceptron neural network can only learn and reduce error if normalization has been performed on the input data?
Why is this neural network only able to reduce error and generalize when I use normalization?


text:
Encoding categorical data with "many" unique values in neural network
I am new to machine learning, in fact, I  am implementing my first deep neural network from scratch without any framework.
The dataset has 3500 rows, and 4 categorial columns of which two have about 160 unique values. The two columns contain the name of Club and Country. The neural network needs to handle such columns, from what I've read many pages, there are some options like

One hot encoding
Embedding
Labeling

I also saw if there are many unique values, One hot encoding will fail. I would like to know what exactly many means. Is this a fixed number? or it is just a ratio between the number of training examples and the unique values.
Thanks

What do you want to see as the output of your neural network? Would you share the first five rows of your table?

To predict wage od players. Link of dataset https://www.kaggle.com/datasets/ultimus/football-wages-prediction/versions/1?resource=download

I see total eight columns in the dataset, which four is the categorial columns, and which two has 160 different values? Is the two are "Club" and "Nation"?

@CloudCho Yes, Club and Nation.

So would your input be club, nation, and etc? Is the output from your neural network the player wage? If yes, I don't think that one hot encoding is a good for the club and the country. You would lose geographic information of the input variables. Your output is continuous value, so house price prediction could be similar (https://www.kaggle.com/competitions/house-prices-advanced-regression-techniques/code?competitionId=5407&sortBy=voteCount)

@CloudCho then what is your suggestion? It's my first coding in AI.

I think it depends on your neural network model choice. How about convert "club" to their annual revenue and "nation" to GDP per capita? I think these values affecting more to player salary.


text:
What are general techniques of structuring an image classification Neural Network for very large numbers of output classes?
I am aware of Neural Networks that have 100K+ classes and I would like to build one myself (yes, I have lots of training data) but I am unsure which technique to use because most of the nets I have previously built struggle when the number of classes exceeds a few thousand. One idea I have is a hierarchical system with multiple nets but that seems like an inefficient architecture (maybe it is currently the best).
For real world context, I was at my local grocery store and they had a new automated shopping cart system that used machine vision (I assume some type of NN) to identify every item in the store (see article that I was able to find). https://bit.ly/3V7Z7VI
I have done some research on techniques and found these threads that both mention hierarchical softmax as a way to achieve large numbers of classes: https://www.reddit.com/r/MachineLearning/comments/3b2odj/most_output_classes_in_a_neural_network/ https://www.quora.com/Can-neural-networks-have-thousands-of-output-classes
This academic article does propose one technique besides a softmax hierarchy (Random Projection): https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123680698.pdf


text:
Book suggestion about deploying real deep learning models in real world
Can you suggest books about deploying machine learning algorithms on robots, especially on real time stream? I don't know how to deal with latency and other challenges that real time inference/stream presents.


text:
What kind of learning architecture can I use to solve a set of nonlinear thermodynamic equations?
I am in the business of solving equations based in thermodynamics. Technically speaking, I am trying to solve for the binodals for a ternary system. The binodal is a curve in a phase diagram.
Fundamentally, I am solving 3 nonlinear equations with 4 variables:
mu_a(pa_1, pb_1) = mu_a(pa_2, pb_2)
mu_b(pa_1, pb_1) = mu_b(pa_2, pb_2)
mu_c(pa_1, pb_1) = mu_c(pa_2, pb_2)

so there is one free variable which parameterizes the curve. Unfortunately, these curves cannot be solved analytically. They can only be solved for numerically.
Using python, I have written a program that painstakingly generates solution points (pa_1, pb_1, pa_2, pb_2). This is an important development in solving for binodals, however, my program is memory-intensive and time-consuming.
My vision is that I can write a neural network to understand the solution of these equations and generate binodal curves, given certain inputs. I have a large number of binodals to train the neural network with, and given the right architecture, it should be possible for it to come up with the binodals given certain inputs. The input would be 6 inputs parameters that characterize the system, and the output would be a 1000x4 matrix (points on the binodal).
Would you have any suggestions for how to go about making a neural network/data-framework of this sort? I do not know much about neural networks apart from them being universal function approximators and robust regression devices. I would appreciate any advice you have for me!


text:
Design approach for image classification regarding genders
I am new to the world of AI and wanted to ask your guidance on how to design a ML model to classify genders based on images.

There will be only one person in the image.
The person could be kids, young or old.
The person could be standing (full body), sitting, or only have his/her upper or lower body in the image.

Based on this I would like to classify the image as male or female.
I currently use Tensorflow and Keras with the below model:
gender_model = tf.keras.Sequential(
        [
            data_augmentation,
            layers.Conv2D(128, (3, 3), activation="relu", input_shape=INPUT_SHAPE),
            layers.BatchNormalization(),
            layers.MaxPooling2D(pool_size=(2, 2)),
            layers.Conv2D(256, (3, 3), activation="relu"),
            layers.BatchNormalization(),
            layers.Dropout(0.5),
            layers.MaxPooling2D(pool_size=(2, 2)),
            layers.Conv2D(256, (3, 3), activation="relu"),
            layers.BatchNormalization(),
            layers.Dropout(0.5),
            layers.MaxPooling2D(pool_size=(2, 2)),
            layers.Conv2D(512, (3, 3), activation="relu"),
            layers.BatchNormalization(),
            layers.Dropout(0.5),
            layers.MaxPooling2D(pool_size=(2, 2)),
            layers.Conv2D(512, (3, 3), activation="relu"),
            layers.BatchNormalization(),
            layers.Dropout(0.5),
            layers.MaxPooling2D(pool_size=(2, 2)),
            layers.Conv2D(256, (3, 3), activation="relu"),
            layers.BatchNormalization(),
            layers.Dropout(0.5),
            layers.MaxPooling2D(pool_size=(2, 2)),
            layers.Conv2D(256, (3, 3), activation="relu"),
            layers.BatchNormalization(),
            layers.Dropout(0.5),
            layers.MaxPooling2D(pool_size=(2, 2)),
            layers.Flatten(),
            layers.Dense(128, activation="relu"),
            layers.BatchNormalization(),
            layers.Dropout(0.5),
            layers.Dense(num_gender_classes),
        ]
    )

gender_model.compile(
    optimizer="adam",
    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
    metrics=[tf.keras.metrics.BinaryAccuracy(name="acc")],
)

I put man and woman images into two separate folders and created the data structures to feed my model. The model prediction has been somewhat disappointing. I am still dealing with overfitting and trying to improve the loss for validation part. However, given the diverse possibilities of input images, I wonder if such an approach is the right one?
For example, should I detect kids vs adults using another model and then feed it to a second classification model? Similarly, classify full body images vs sitting people or upper body vs lower body in separate models, and then use a second model to classify the gender?
I would be grateful if you could provide pointers if you have worked on similar classification problems before.
Thanks,
Doug


text:
What do RNN neural networks lack in nowcasting time series?
I want to write a master thesis on nowcasting GDP? Has this been used and if so I don't fully understand how the neural networks should be built if I forecast quarterly GDP and link that to unemployment how would the neural network update my prediction for 1st quarter GDP when the Feb unemployment is released compared to using only the Jan one. I am now sure where the update element should be placed in an RNN architecture?


text:
Why using a pretrained model with dense variational for uncertainty estimates gives low accuracy?
I try to a train model for real image classification and to estimate uncertainty by using the TensorFlow Probability library. For that, I am using a pretrained model as a feature extractor (the features used from the pretrained model) and then add the DenseVariational layer (I replace the dense layer with it). as a transfer-learning but instead of dense layer i put DenseVariational layer, since training Bayesian model (for conv layer also) is more complex and time consuming than standard deep learning model, so i use DenseNet169 as feature-extractor and then add DenseVariational for FC layer.
the idea for this to estimate uncertainty (to know the Values and reduce the uncertainty if its high) as the model when classify the image is class1 its certain and have high probability for that class1 when classified it as class1(which mean low uncertainty), and probability model one of these method for uncertainty.
The problem is when training the model, it gives me low accuracy (around 30 when using more dense-variational layer, and around 40% when using one dense-Variational layer), but it increases when retrain the model (train model multiple times) bit still the accuracy low. I tried the prior and posterior I found in the examples provides in documentation(tensorflow-probablity).
I also tried different priors and posteriors, but I got the same result, and I don't know why. Maybe it doesn't layer from previous layers, or learns the feature map.
Any notes? Or maybe combining the pre-trained model with probability layers is wrong? I provide the code below
this the code below for prior nd postrior
def prior(kernel_size, bias_size, dtype=None):
    n = kernel_size + bias_size
    # Independent Normal Distribution
    return lambda t: tfd.Independent(tfd.Normal(loc=tf.zeros(n, dtype=dtype),
                                                scale=1),
                                     reinterpreted_batch_ndims=1)

def posterior(kernel_size, bias_size, dtype=None):
    n = kernel_size + bias_size
    return Sequential([
        tfpl.VariableLayer(tfpl.IndependentNormal.params_size(n), dtype=dtype),
        tfpl.IndependentNormal(n)
    ])

kl_weight= (lambda q, p, _: tfd.kl_divergence(q, p) /  # pylint: disable=g-long-lambda
                            tf.cast(NUM_TRAIN_EXAMPLES, dtype=tf.float32))

the model:
base_model = tf.keras.applications.DenseNet169(
    input_shape=(224,224,3),
    include_top=False,
    weights="imagenet",
    classes=3)
for layer in base_model.layers[:-10]:
    layer.trainable = False
x = base_model.output
x =  GlobalAveragePooling2D()(x)
x = tfpl.DenseVariational(units=128,
                          make_prior_fn = prior,
                          make_posterior_fn = posterior,
                          kl_weight= 1/NUM_TRAIN_EXAMPLES)(x)
x = tfpl.DenseVariational(units=32,
                          make_prior_fn = prior,
                          make_posterior_fn = posterior,
                          kl_weight= 1/NUM_TRAIN_EXAMPLES)(x)'''
x = tfpl.DenseVariational(units=3,
                          make_prior_fn = prior,
                          make_posterior_fn = posterior,
                          kl_weight= 1/NUM_TRAIN_EXAMPLES)(x)
model = Model(inputs=base_model.input, outputs=x)

and the loss function is nll(negative-likelihood)
def nll(y_true, y_pred):
    return -y_pred.log_prob(y_true)

On what basis do you think that you can just replace a dense layer with a variational layer and get the results you need, which are unspecified? Have you seen somewhere that you can do this?

well, like transfer learning I think that way, why to train model from scratch with bayesain layer (for conv layer) which is complexs and time consuming  if i can use the feature map?? and  want to have good accurcy and able to estimate uncertainty(to konw the valuses and reduce the uncertainty if its high) as the model when classify the image is class1 its certain and have high probablity for that class1 when classifed it as class1.

And i didn't see it in research, reasearch talk about using monte carlo dropout, wich is add dropout layer in the structure of the model betwwen the conv layer also not berfor FC layer. and actully there's confused and not understandable ideas, and all talk about the same subject, but didn't find the clear to implement this ideas

I've tried to improve the clarity of your post. Can you please check the post's body and title are consistent with your original intention/question? Can you also please add this info you've just added in the comments to your post too? Could also specify exactly which prior and posterior you're using or you tried? You said you tried the ones in the documentation, but you don't specify the link to the documentation or the actual names. Can you also specify what accuracy or "uncertainty estimates" you're having?

Also, what is the pre-trained model? Which loss function are you using? How are you training or fine-tuning? It would be better if you provide the link to the full code. Thanks for providing answers to all these questions.

thank for concern and clarification, i added above the prior and postrior and  i edit my question, for accuracy is 30-40, loss function i added above and can using caterogical-crossentropy but when training the loss became large.

for prior and posterior i found it in tensorflow-probability website and it also explained in course on coursera called  Probabilistic Deep Learning with TensorFlow 2 : Imperial College London


text:
Direct solving of Bellman optimality equation
I'm reading the Sutton and Barto second edition (the 2020 revision). I'm on chapter 3.6 "Optimal Policies and Optimal Value Functions".
It is suggested that the Bellman optimality equations could be solved.

The Bellman optimality equation is actually a system
of equations, one for each state, so if there are n states, then there
are $n$ equations in $n$ unknowns. If the dynamics $p$ of the environment
are known, then in principle one can solve this system of equations
for $v_*$ using any one of a variety of methods for solving systems of
nonlinear equations. One can solve a related set of equations for $q_*$.

And a bit later:

Explicitly solving the Bellman optimality equation provides one route to finding an
optimal policy, and thus to solving the reinforcement learning problem.

I'm wondering what kind of techniques and solvers could be used to solve the Bellman optimality equations for the state-value function. These equations being mostly linear with a $max$ operator kinda remind me of linear programming. But I'm not sure how to formulate the problem this way since the $max$ here are operators, not an objective function.
Note : I know this cannot work for decent-sized MDP, that's not really the question here.

Interesting answer. (I didn't find it because it wasn't tagged [bellman-equations].) I thought of a formulation like this, but it's not obvious to me why we can minimize the $\sum_s v*(s)$ when what we want is to minimize them all independently.

You could ask another question. Btw, if your question here is different, you should edit it to clarify how it's different ;)


text:
Are there any style transfer algorithms that use multiple images (rather than just one)?
The original paper for style transfer only supports one image as an input. However I was interested if there is a model that uses multiple images, preferrably in a way that doesn't require users to provide their own image files, just requiring the model to be trained on images of a single style and then imitating that style without the original image files. Can this be achieved using variants of the algorithms of that paper or should other neural network types be used (eg. GANs, diffusion models...)

Nothing is stopping you from concatenating multiple images into a single one. Although it is a very bad solution, it is technically a workaround for using multiple images. The thing is that the solution in the paper wasn't designed for copying the style of multiple images, but for copying the style of a single image to another in a very fast way. What you're looking for is a latent space generative model. Stable Diffusion Img2Img, models like Pix2Pix, or others of this style can solve this problem.

For text-to-image Diffusion Models there are some popular customization techniques, like Textual Inversion, DreamBooth and Custom Diffusion.
Textual Inversion performs a few-shot finetuning of a text embedding $V^*$ of the model's text encoder, on a set of input images (usually 3 to 5).
Specifically, after finetuning, style stransfer is performed by providing the following prompt, including the learned embedding $V^*$:

Painting of X in the style of $V^*$ art

DreamBooth uses a similar few-shot finetuning strategy, but it instead optimizes the whole model, obtaining better quality samples at the cost of memory-intensive finetuning and a heavy checkpoint (as it includes the whole model)
Custom Diffusion finetunes only a small subset of attention layers, reducing fine-tuning time and cost compared to DreamBooth, and allows the composition of multiple learned concepts, a setting where previous approaches struggled.


text:
Why doesn't diffusion models suffer posterior collapse?
In VAEs, posterior collapse occurs when the approximated posterior $q_\theta(z|x)$ becomes the standard Gaussian prior after training. The forward process of diffusion models transforms the data $x$ into a standard Gaussian at time step $T$, which is similar to approximating the posterior $p(z|x)$ in hierarchical VAEs (see this blog for more details). Therefore, I am wondering whether diffusion models suffer from the issue of posterior collapse, if that makes sense at all.


text:
Understanding the concepts of embedding in Roberta architecture?
I'm reading the implementation file of Roberta architecture, specifically in the RobertaEmbedding class, this class has the comment:
Same as BertEmbeddings with a tiny tweak for positional embeddings indexing.

I'm a newbie to this, I want to ask about the changes in the RobertaEmbedding class, are these changes related to the dynamic masking in Roberta architecture? Thanks

I am not much familiar with roberta or bert, But Embedding is generally used to transform categorical ids to meaningful vectors (so that they can be processed by NN) which are generally learned by back-propagation (for instance use code tf.keras.layers.embedding) because transformers (well, those few i know) do not use rnn, they need a technique to know the order of the words. They do this by using positional embedding which can be done in the same way or y using for instance sin and cos bases. dynamic masking I think is related to masking words ahead so that decoder cannt see ahead.

Can you please formulate the title as a specific question? Thanks.


text:
How the "quantizantion" of models does work?
Consider that a model using FP16 precision is quantized to a lower precision like INT8. Does this reduce the accuracy of the model? From what I know it is designed to reduce the size and required RAM to run the models.

Quantization is an active research area, so there's going to be a number of ways to do quantization.
Huggingface has a nice blogpost with an overview of 8 bit quantization. I'll give a rough summary of some important points here.

Background
With FP16 -> INT8 quantization specifically, you want to project all FP16 numbers into the 256 possible INT8 values. You can't directly round your parameters into integers (otherwise you'll just get a bunch of zeros!) but you can rescale your floating point values to take full advantage of the 256 integers you're limited to.
As a concrete example,

...if my range is -1.0…1.0 and I want to quantize into the range -127…127, I want to scale by the factor of 127 and then round it into the 8-bit precision. To retrieve the original value, you would need to divide the int8 value by that same quantization factor of 127. For example, the value 0.3 would be scaled to 0.3*127 = 38.1. Through rounding, we get the value of 38. If we reverse this, we get 38/127=0.2992 – we have a quantization error of 0.008 in this example.

"Quantization error" here refers to the difference between the original value and the quantized representation. With this quantization method, limited to only 256 possible quantized values, the closest value we can map 0.3 to is 0.2992.
Quantization will pretty much always result in some amount of quantization error---by lowering the precision you're artificially adding noise to your weights in the form of quantization errors.
The interesting question is in knowing what kinds of errors models are robust to. We can then take advantage of this to devise quantization methods that only add this kind of noise that doesn't affect model performance.
As a simple example, consider again the previous quantization method, quantizing floating point numbers to integers from -127 to 127 (by multiplying by 127). If, it turns out that your model's parameters are always between 0 and 1, then this method of quantization would be extremely suboptimal: there's no use dedicating 127 integers to representing negative numbers that you'll never see in practice.
The key takeaway here is that the quality of your quantization depends on taking advantage of knowledge about the range of values that you're quantizing. By quantizing, you're sacrificing precision for values that you know you'll never see in favor of accurately representing values that you will see.
Details
The post, and accompanying paper discuss two methods to do this: absmax and zeropoint quantization. Absmax is the more popular method. It just rescales based on the absolute value of the maximum of the values you're quantizing. For example, if you have the values 0.4, -0.3, -1.2, 0.5. You'd first divide by the max absolute value: 0.4/1.2, -0.3/1.2, -1.2/1.2, 0.5/1.2, giving you values between -1 and 1. You can then rescale to match the range of INT8 by simplying multiplying by 127. This method tends to be worse for asymmetric distributions (e.g., from 0 to 1 as opposed to -1 to 1) as negative integers won't be used. Zeropoint quantization better handles these values.
LLM.int8()
This can work, but there a couple other tricks you need if you want quantization to work consistent, especially across larger models. Again, these methods were originally proposed in the accompanying paper.
If you have a large set of values, you'd be able to quantize more accurately if you were to quantize smaller subsets of your data separately versus quantizing the entire set of data in one go. This is especially important if you have subsets that contain different ranges of values. Here, the authors do this by independently quantizing the rows and columns of matrices being multiplied.
Another trick is that large transformer models tend to have outlier values that are tough to quantize. e.g., if you have a bunch of values between -1 and 1 and single value = 1000, you either need to sacrifice precision significantly by rescaling everything by 1000 or you'll need to get rid of the outlier value. The authors instead propose using fp16 precision for outlier values and int8 for everything else.
Results
The table below shows the results from the paper (for perplexity; lower is better):

With the right quantization method, you can retain most of the accuracy of LLMs despite introducing quantization errors.


text:
Is it possible to calculate a GRU RNN in its entirety by hand on a small dataset?
I want to see whether my code works and want to do it by hand to compare the results. How exactly does the memory cell work in the first example seen?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.


text:
What are the differences between Inception Score and Fréchet Inception Distance?
From the articles I've read about image generation using GANs, the Inception Score measures two things simultaneously: the variety of images (diversity) and the distinct quality of each image. Does Fréchet Inception Distance also measures these things ? If so, what are the differences between them ? Could someone please explain ? Thanks


text:
What are meaning of parameters $\theta$ in this context?
I'm reading the article about generative model from Open AI, here is the section where they explain them:

Our network is a function with parameters $\theta$, and tweaking these parameters will tweak the generated distribution of images. Our goal then is to find parameters $\theta$ that produce distribution that closely matches the true data distribution. Therefore, you can imagine the green distribution starting out random and then the training process iteratively changing the parameters $\theta$ to stretch and squeeze it to better match the blue distribution.

I don't understand what are the meaning of these parameters $\theta$ and how does tweaking these parameters will tweak the generated distribution of images, can someone explain ? Thanks!

As explained in your reference the yellow generative model is a neural network that takes randomly sampled points from a unit Gaussian distribution as input and generates an approximated distribution $\hat p{_\theta}(x)$ to the true data distribution $p(x)$. The unit Gaussian distribution is a simplified assumption here and it can be any latent variables $z$ aka codes of DCGANs or VAEs.
The parameters $θ$ of the generative model are the weights and biases of this generator or decoder neural network for GANs or VAEs, respectively. By adjusting the values of $θ$ you are essentially modifying how the neural network transforms input into the generated synthetic data. Learning in a generative model involves adjusting the parameters $θ$ so that the synthetic data generated by the model becomes more similar to the true data (images) distribution. This is typically done by optimizing an objective function such as ELBO, which includes terms that encourage the generated distribution to match the true data distribution.

So the parameter $\theta$ are just a bunch of weights, what do you mean by weights ?

They're just the commonly referred weights between nodes of fully connected layers and biases of each node in a generic MLP, they're what the backprop algo aims to settle.

Very newbie question, at the start how are these weights initialized ?

There're many tricks, usually they're randomly initialized to some small values to have optimal result from many experiments.


text:
How can the discriminator determine the sample is fake or real?
Based on the articles I've read, the discriminator can identify whether a sample is fake or real. However, the articles don't clarify the conditions used to determine if a sample is fake or real. I came across information suggesting that labeling all the samples generated by the generator as fake before the training process, but I'm not sure if this is correct. Could someone please provide clarification? Thanks

In a GAN, the discriminator starts untrained, and the generator and discriminator are trained alongside each other. The process relies on neither being too strong for the other at any one stage, so that training can continue.
So the reason the discriminator can determine the difference between a real or fake sample is because it has learned from the GAN training process.
You should start with some training data, in the form of real images that you want the generator to produce similar output to.
When you train the discriminator:

For every real image you show it, the target label is "real" (for a simple binary discrimintor the single target output is $p(real) = 1$)

For every generated image you show it, the target label is "fake" (e.g. $p(real) = 0$ for a binary class prediction)

In every minibatch you should train the discriminator on a mix of real and fake images, labelled accordingly.

When you train the generator:

You still show the generator outputs to the discriminator, but now your target is to make the generator better, so you lie to the discriminator and label the generator's image as "real" ($p(real) = 1$). The trick here is that you only do this to get the gradients at the generator output layer. The discriminator is not updated for these training examples, only the generator is.

There are a few variations of this process, but the core idea is always similar, and to answer your main question, the discriminator determines real or fake because this is part of the training process. It is very bad at its job at the start, and you need that to be the case.

In the training process of the generator, I read that backpropagation also goes through the discriminator. So, why did you state, The discriminator is not updated for these training examples, only the generator is.?

@David Because the discriminator is not updated. The reading you have done is also correct - backpropagation is done through the discriminator layers. It has to be so in order to get the gradients to update the generator. What you should not do is use the gradients from the generator training for the discriminator to update it - those are thrown away, only keep and use the generator gradients when training the generator

When training the generator, you lie to the discriminator and label the generator's image as "real" (p(real)=1), then what will be labeled as fake during the generator's training process ?

@David: Nothing is labelled as fake when training the generator. The generator's goal is to make all images as real as possible (as far as the discriminator can tell).

When will the error propagate back to the generator? Like when the generator fails to fool the discriminator (the discriminator classifies the generated images as fake instead of real) ?

@David When you train the generator. There's no need to select only failures, although of course these will give larger error signals, but even success  can be improved by increasing confidence. If you have further questions please start a new question on the site


text:
What does it mean to "learn a distribution", and what does it contain?
When I was reading about discriminative vs generative models, I came across their definitions:

Given a distribution of inputs $X$ and labels $Y:$
Discriminative models learn the conditional distribution $P(Y|X)$.
Generative models learn the joint distribution $P(X,Y)$. If you only
have $X$, then you can still learn $P(X)$.

My questions are:

What does it mean to "learn a distribution" ? Learning what from the distribution ?
What does the distribution contain, and what does it look like?

Learning the distribution: When we talk about learning a distribution, we are essentially trying to capture the underlying statistical properties of the data. In other words, we try to capture the distribution from which the data points in our dataset our sampled. This involves estimating parameters that define the distribution (such as mean, variance, etc.) or learning a model that can generate data points similar to those observed in the dataset.
What does the distribution contain?: The distribution contains information about the likelihood of different values or configurations of the variables in the dataset. For example, in a simple case where X represents the features of a dataset and Y represents the labels, the conditional distribution P(Y|X) would describe the probability of observing a particular label given the input features.
What does the distribution look like?: Depends on the nature of the data and the relationships between variables. It could take various forms, such as Gaussian (bell-shaped), uniform, exponential, etc., depending on the characteristics of the data.

Thank you. What do you mean by "data points"? If I have a dataset of images, what does a data point represent for each of these images ?

One datapoint would be tje combination of (x, y), so the image with label.

Very newbie question, what is "the combination of $(x,y)$", and what do you mean by so the image with label ?

@abcd If you are interested in a binary classification task, classifying whether your images have cats in them or not, then $x$ would be an image and $y$ would be the 'label', which in this example would be a 1 or 0, denoting whether the image contained a cat, or not, respectively.

So you're saying that the data points are just number between $0$ and $1$ for classification task (discriminative model) ?


text:
Is it possible to generate a line graph for these variables?
I have a data set from:
https://archive.ics.uci.edu/dataset/17/breast+cancer+wisconsin+diagnostic

The variables are described in the data set as follows:
1) ID number
2) Diagnosis (M = malignant, B = benign)
3-32)
  a) radius (mean of distances from center to points on the perimeter)
  b) texture (standard deviation of gray-scale values)
  c) perimeter
  d) area
  e) smoothness (local variation in radius lengths)
  f) compactness (perimeter^2 / area - 1.0)
  g) concavity (severity of concave portions of the contour)
  h) concave points (number of concave portions of the contour)
  i) symmetry
  j) fractal dimension ("coastline approximation" - 1)

Is it possible to generate a line plot for any variables from this data set?
I personally think that it is not possible, but how to justify it.


text:
How to optimise a FNN/MLP network with MSE (positive only loss) in C
I can create a FNN/MLP network in C but only g-p loss works, where g = ground truth and p = predicted.
What I don't understand is how MSE a positive only loss value can train a back propagation network. Positive only losses result in weights only incrementing in their pre-initialised sign (- or +) thus they just explode or vanish.

Positive only losses result in weights only incrementing in their pre-initialised sign

No.
In gradient descent, you use the (partial) derivative or the gradient (if you're using vector operations and you have more than one parameter/weight) of the loss function, not the loss function.
So, let's say you have the function $f(x; w) = wx$ that you want to optimize wrt to $w$. Suppose $y$ is the ground-truth label and we use the mean squared error to optimise this function, i.e. $L(x; y; w) = (w x - y)^2$, then the gradient descent step is
$$w \leftarrow w - \alpha \frac{d L}{dw}$$
Now, what is the its derivative of $L$ wrt to $w$, i.e. $\frac{d L}{dw}$? It's
$$2x(w x - y)$$
Now, let's say $x = 1$, $w = 1$ and $y = 2$, then you have that the derivative is
$$2*1(1*1 - 2) = 2 * (-1) = -2$$
Now, let's say $x = 2$, $w = 1$ and $y = 1$, then you have that the derivative is
$$2*2(1 *2 - 1) = 4$$
So, clearly, the derivative can be positive or negative, even if the loss function is always positive. Note that I've used the same $w$ in both examples and the derivatives have different signs.

i don't accept this as a valid answer, because it is too simplistic and doesn't resemble a real working network at all that uses actual activation derivatives. Out of interest why are you so private tor etc why you don't do anything illegal, this is all machine learning?
My point is if you know the answer show me some example C code and I will tell you if you are wrong or right but anyone can spit out meaningless examples in algebra, to actually implement that and prove it works is another thing.

@Torrap Asking for code or to implement something is off-topic here. I initially voted to close your post because of this. We focus on the theoretical aspects of AI. Now, I decided to give you an answer that shows that one of your statements was wrong. So, now you know that, if you use MSE, the weights can become more negative or more positive. It should be clear now that what you stated is not a limitation of MSE. Also, if you don't understand backpropagation or gradient descent, I recommend you read about the topic. There are many articles online or books about it

I don't do anything illegal. Tor can help you to protect your privacy.

I would suggest you don't understand it because I have actually implemented it in C and have it working, as where you do not.

@Torrap Feel free to downvote my answer and wait for other answers. It's not a problem for me. This is not going to affect my sleep. But I don't understand why you asked a question if you understood everything and everything works.

My account is too new to down vote don't worry about it.


text:
What is the best approach to clustering and modeling a network analysis for a large dataset in risk identification?
Need guidance! I want to build a risk scoring system for merchants. The current existing mitigation plans are:

Using the count of hits by the rule engine (i.e., consecutive massive transactions, shared devices, etc.) to determine whether or not that merchant is high-risk.
Flag every merchant with attributes common to fraudsters according to the regulatory compliance list (by business type, province, segment, size, etc.)

However, the result is deemed too naive as it only targets a particular fraud problem defined by the generic behaviors or transactions already known as suspicious; therefore, I want to classify all of the merchants as high, medium, or low risk using a more in-depth measurement:
Semi-supervised learning approach
Data is separated by three categories:

Labeled high risk behaviors (rule engine risk, compliance risk, reported risk) - known risk
Grouped by attribute cluster (location, segment, size) - unknown risk
Network relationship between each merchant (merchants related with other merchants for syndicate indicator, merchants related with customers for collusion indicator) - unknown risk

The issue is that I don't have labeled data to train the weight calculation, so I've been thinking about using unsupervised learning to cluster these merchants. What kind of model would be fitting to tackle this issue? Or should I just use all the merchants flagged by the rule engine as the supervised data (but then it would be the same as simply using the rule engine)?
Would anomaly detection be useful here? I'm debating whether to just group all the merchants with similar behaviors or attributes (regardless of whether they are flagged by the rule engine or not) into clusters using K-Nearest Neighbors or just isolate the anomaly using Isolation Forest (does the data need to be an outlier for it to be indicated as suspicious though? because it's possible that there are a large group of high-risk merchants?
Second, how do I determine the syndicate and collusion? So far, I've only been using network analysis (NetworkX on Python) for requested or reported users to see users related to them. How do I implement this on a larger scale (with hundreds of millions of merchants and users)? Wouldn't that confound the betweenness centrality index? I've been reading about Relational Graph Convolutional Network (RGCN), and I wonder if this is a fitting solution to this.
Or is there another more common solution or parameter that I overlooked? Any advice, guidance, or reading materials would be much appreciated. Thank you!
ADDITIONAL INFO
Feature Engineering (new parameters made for each merchant, 1-11 done, 12-14 in consideration)
by rule engine/FDS risk (count of each FDS hit per category for merchants) - known risk:

count of Loyalty Abuse (integer)
count of Product Loophole (integer)
count of Social Engineering (integer)

by Compliance risk - known risk:

high risk business type (1=low, 2=medium, 3=high)
high risk province according to compliance (1=low, 2=medium, 3=high)
high risk/common location for fraudster (1=low, 2=medium, 3=high)
high risk merchant size (1=low, 2=medium, 3=high)

by Reported risk - known risk:

merchant with reported msisdn (3=high), merchant related to reported msisdn (2=medium)

by Attribute cluster - unknown risk:

segment (string)
subsegment (string)
province (string)

by Network relationship (each transaction) - unknown risk:

merchants related with other merchants (syndicate indication)
merchants related with customers (collusion indication)
coordinate (geographical information of source and target)


text:
Fine tuning or just feature extraction or both using Roberta?
I'm reading a program that use the pre-trained Roberta model (roberta-base). The code first extracts word embeddings from each caption in the batch, using the last hidden state of the Roberta model. Then, the model is trained to align these word embeddings with the image features (pixels) of the image through a type of attention mechanism. Then the models are updated using attention loss function. This iterative process continues until the training is complete, so I guess the word embeddings will be different after each epoch ? This is a multi-modal problem.
When I compare the Roberta model after training with the pre-trained model (roberta-base), I notice that every parameters the trained Roberta model are different, seems like the new model has updated the parameters. I'm not sure whether this is a form of fine-tuning or just feature extraction or both ?

According to this reference:

feature extraction involves creating new features that still capture the essential information from the original data but in a more efficient way.

Feature extraction techniques vary depending on the domain:
Image Processing: Techniques like edge detection filters, Gabor filters, and Histogram of Oriented Gradients (HOG) can be used to extract features from images.
Text Data: Natural Language Processing uses techniques such as Bag of Words, TF-IDF, and word embeddings to extract features from text.

Therefor your using the last hidden state of the Roberta model as new contextualized embedding feature of the input sequence for your downstream fine tuning task is feature extraction, and your whole approach contains both.

The contextualized embeddings in the first epoch are feature extraction because it takes directly from the pre-trained Roberta, but the contextualized embeddings for rest of epochs are different because the model has updated ?

Yes, as the fine-tuning process progresses through subsequent epochs, your model's parameters including those of RoBERTa and any additional layers added for the multimodal task, are updated based on the training data. As a result, the contextualized embeddings produced by RoBERTa in subsequent epochs may differ from those obtained in the initial epoch, reflecting the updated knowledge and representations learned by the model during training.

Thanks for your help


text:
In the VAE, why is $z \sim \mathcal{N}(\mu, \sigma^2)$ equivalent to $z = \mu + \sigma \odot \epsilon$?
In the reparameterization trick of a Variational Autoencoder (VAE), instead of sampling noise $z$ from $z \sim \mathcal{N}(\mu, \sigma^2)$, we can use a different method: $z = \mu + \sigma \odot \epsilon$, where $\epsilon \sim \mathcal{N}(0,1)$. I'm having trouble understanding why these two methods are equivalent ?

Note that $z$ is not noise in the same way that $\epsilon$ is, $z$ here is the latent variable/sample, which is part of the generative process considered in the VAE.

@nbro, So a latent variable is different from noise ? I thought they are the same. Is it accurate to say that a latent variable $z$ depends on the input $x$, while noise $\epsilon$ does not depend on anything ? Am I correct ?

I'll attempt a less formal explanation.
The distribution $\mathcal{N}(\mu, \sigma)$ represents a normal distribution with mean $\mu$ and standard deviation $\sigma$. When we sample from this distribution, we obtain a value $z$.
Now, consider the expression $z = \mu + \sigma \odot \epsilon$. Here, $\epsilon$ is a random value sampled from a standard normal distribution, which is a normal distribution with mean $0$ and standard deviation $1$. By multiplying $\epsilon$ by $\sigma$ and adding $\mu$, we effectively scale and shift the standard normal distribution to match the desired mean $\mu$ and standard deviation $\sigma$.
In essence, this expression generates values ($z$) from a normal distribution with the specified mean and standard deviation, $\mathcal{N}(\mu, \sigma)$. By incorporating $\mu$ and $\sigma$ into the sampling process, we ensure that the resulting distribution maintains the desired characteristics.

I'll give 2 cents.
The point is that Normal distribution can be shifted and rescaled:

if $X$ is a normal gaussian distribution with parameters
$\mu$ and $\sigma^2$, then this $X$ distribution can be re-scaled and shifted via the formula $Z=(X-\mu )/\sigma$ to convert it to the standard normal distribution.

Now, if you write that equation like $X=f(Z)$, you get what you are looking for...
The proof that shows it formally can be found under many other many other answers

From your unanswered comment it seems you're still not clear about the root cause and mechanism of VAE's reparameterization trick, so it would be beneficial to clarify your expressed confusion by the formal analysis of VAE reference surrogate objective function ELBO part right before the reparameterization:

A variational autoencoder is a generative model with a prior and noise distribution respectively... For example, a standard VAE task such as IMAGENET is typically assumed to have a gaussianly distributed noise; however, tasks such as binarized MNIST require a Bernoulli noise.

Maximizing the ELBO... The form given is not very convenient for maximization, but the following, equivalent form, is:
$${L_{\theta ,\phi }(x)=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln p_{\theta }(x|z)\right]-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }(\cdot ))}$$...The distribution of ${q_{\phi }(z|x)}$ and ${p_{\theta }(z)}$ are often also chosen to be Gaussians as ${z|x\sim {\mathcal {N}}(E_{\phi }(x),\sigma _{\phi }(x)^{2}I)}$ and ${z\sim {\mathcal {N}}(0,I)}$

Therefore the prior distribution of $z$ doesn't depend on any input data $x$, and its variational posterior distribution ${q_{\phi }(z|x)}$ is the noise distribution mentioned above and optimized to be close to its prior distribution as shown in the above equivalent ELBO form via chain rule which is convenient for maximization. On the other hand from the original ELBO form $\ln p_{\theta }(x)-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }({\cdot |x}))$ mentioned in my reference whose KL-Divergence term only contains conditional posterior distributions of $z$ conditioned on input data $x$, thus obviously ${q_{\phi }(z|x)}$ can no longer simply be ${\mathcal {N}}(0,I)$ but usually simply assumed to be variance-scaled and mean-shifted as ${\mathcal {N}}(\mu,\sigma^2)$.
In summary, your reparameterized $\epsilon \sim \mathcal{N}(0,1)$ is exactly same as the unconditioned $z$, and the conditioned $z|x \sim \mathcal{N}(\mu, \sigma^2)$ is simply assumed to be scaled and shifted without any necessary theoretical basis.
Finally of course with basic probability knowledge of Gaussian distribution, if you're given a random variable transformation from $\epsilon$ to $z$, you can simply substitute $z$ for the given known probability density function (PDF) of $\epsilon$ and verify $z$ obtains its desired distribution. But this pure math knowledge has nothing to do with AI here and cannot give above VAE related equivalence insight, and one can always ask why not choose another reparameterization, say as $μ^2+σ/2∗ϵ$, where $ϵ$ is chosen to be some other distribution than $\mathcal{N}(0,1)$.

I think this does not answer the question. The question is about why $z$ sampled directly from $N(\mu, \sigma)$ is equivalent to the one sampled using the re-parametrization trick, i.e. why the re-parametrization trick is not e.g. $\mu^2 + \sigma/2 * \epsilon$? This is just the usual confusing explanation of why people decide to use the re-parametrization trick, i.e. "you cannot backpropagate through the sampling operation".

@nbro thanks for your feedback! I thought the last equation in my answer would formally answer the question of statistical equivalence, now I've added further informal explanation at the end to further clarify.


text:
How do transformer-based architectures generate contextual embeddings?
How do transformer-based architectures like Roberta generate contextual embeddings? The articles I've read keep saying that transformer encoders work bidirectionally. Because of self-attention, they can look at every token, unlike RNN/LSTM, which can only process the previous hidden state. I'm not sure how the Transformer accomplishes that.


text:
Why are various deep learning models unsupervised?
While studying the field of deep learning, the questions I had from the beginning have still not been resolved.
In general, supervised learning is known to solve problems by comparing the results output from the data with the labels because the data is labeled.
In contrast, unsupervised learning is understood as analyzing patterns in input data without separate labels and outputting appropriate results for those patterns.
I have a question.
Autoencoders are commonly known as unsupervised learning models.
For example, when a specific vector is used as an input to an encoder model and there is a model in which the corresponding image is generated through a decoder, a loss function is often constructed by comparing the generated image with the correct image for learning.
So, I wonder if this should be classified as supervised learning at the stage of comparing the output results with the correct answer data.
In addition, when considering the transformer model for translation, the translated text output through the decoder is compared with the actual correct translation text to construct a loss function. I am curious as to whether this case also has a tendency for supervised learning.


text:
Why different noise in GAN generate different images?
I understand that noise $z$ serves as the input to the generator. Noise $z$ is essentially a vector of random numbers, typically from Gaussian distribution with chosen size of like $100$. However, I don't understand how different noise can produce different images.
Sorry if this sounds like stupid question, but I've been experimenting with GANs and noticed that if I keep the noise constant, the generated image remains the same, seems like the numbers in the noise vector are linked to generating specific image, but I haven't come across an explanation for this online.

Let's take this apart. GANs stand for Generative Adversarial Networks.
Your question is how GANs are generative (the G part of the name). For this we need to understand what they try to achieve.
In very simple terms, the generator in GANs is a mapping from a probability distribution to another. The original distribution is a Gaussian distribution and the final distribution (we hope) is the data distribution.
The generator is parameterized as a neural network (this is the N part) and the way we learn the mapping from the data is through adversarial training (the A part).
Now for your questions, what you need to understand is that the generator has two important properties:

It's deterministic
Has the capacity to learn very complex mappings

The first property explains your observation that "if I keep the noise constant, the generated image remains the same".
The second property is important for the diversity you observe. It allows small differences in the initial noise to map to very different samples from the learned distribution.
But this comes with a caveat. While the generator is able to learn complex mappings, this is by no means guaranteed. Depending on the effectiveness of the training you might end up with a generator that only generates very few different images (an issue known as mode collapse).
Extra note: The reason that we use a Gaussian source distribution stems from the fact that we know how to easily sample from it. Plus normals are also maximum entropy distributions under fixed variance something that could potentially help with the diversity part.

Thanks, but could you explain how the generator has the ability to learn very complex mappings ?

The generator is a neural network and for these models there exists the universal approximation theorem, which means that if they have enough parameters they can learn an arbitrarily complex function (a mapping is essentially a function). Of course this is only theoretical. In practice we have only finite parameters and also the trickiest part is how to learn these, which is where the adversarial training comes into play.

To further address you comment question, it's GAN's generator network's deep learning ability consisting of multiple layers of nonlinear transformations (e.g., convolutional layers, transposed convolutions, batch normalization, and hierarchical architecture gradually upsampling the latent representation to higher and higher resolutions, etc) to be able to gradually learn and transform from a usually relatively low dimensional Gaussian input noise $z$ into a very complex high-dimensional data output directly. Each layer in the generator captures different aspects of the data generation process such as edges, contours, spatial orientation features, textures, grey-scale or RGB brightness and object shapes which can be usually accounted for by your example size of 100, and it learns to map different regions of the latent space to different features in the output data.
On the other hand, as the other answer rightly pointed out GAN's generator is always deterministic unlike the decoder of variational autoencoder, so intuitively mode collapse is a more common and serious problem for GAN's generator which is its potential limitation.

Thanks for your help

Maybe you should briefly explain what mode collapse is and why we care about it.

@nbro see the accepted answer's explanation of mode collapse - "a generator that only generates very few different images (an issue known as mode collapse)" which is good and I don't want to repeat. If you're not satisfied with that, your comment should be placed under the accepted answer which preceded mine in time.

@cinch Yes, no need to repeat indeed. I read your answer before the other.


text:
Is it possible to separate a learnable function $f(x)$ to the integration of an unlearnable part and a learnable part?
For a general learnable function $f(x)$, is it possible to construct a function $h(x)$
s.t. $x \rightarrow h(x)$ is unlearnable or very hard to learn, while $h(x) \rightarrow f(x)$ is learnable?
Any theoretical results or concrete construction is highly appreciated.

In general it's difficult to pin down a truly unlearnable function especially in the context of deep learning. And for hard to learn functions, I suppose you mean complex nonlinear functions, discontinuous functions, non-differentiable function which hinder gradient based optimizations. Under such assumptions, transfer learning could be such a concrete ML technique to extract features from some existing model which is hard to learn such as GPT and try to learn a relatively easier and related downstream task by fine tuning, usually:

For example, for image classification, knowledge gained while learning to recognize cars could be applied when trying to recognize trucks.

Thank you for the helpful answer. However, I can't quite get why transfer learning can be applied here. I think transfer learning means that, we can separate a "hard to learn" function to a "hard to learn" function and an "easy to learn" function. Here the target is separating an "easy to learn" function to a "hard to learn" one and an "easy to learn" one. Although they are similar, but I can't really get how they can be connected together.

@Marc_12 You only specified "For a general learnable function f(x)" which should be regarded as hard imo. If you already know a priori f(x) is easy, then why you need to decompose it into a hard component in the first place?

Sorry for the confusion. I'm just wondering whether there is a magic function that is very hard to learn, but relatively easy to learn from.

@Marc_12 Then GPT neural network as universal approximated function with billions of parameters is certainly hard to learn in almost every aspect such as money, hardware resources, time, intellectual effort, but relatively easy to learn from such as via ChatGPT anyone can try to solicit their own prompted answer which is your 2nd step h(x)→f(x).

well say $x$ are the initial conditions of a double pendulum, and $h(x)$ is  a model of the differential equation that governs it... learning it will be very hard, because it's a chaotic system, but if you set $f$ being "does the pendolum exist", well the answer is yes always, so you can definitely learn it

Thank you for your helpful answer. I'm wondering whether or not there exists a more general "hard to learn" function that can be applied on input of a general ML model, while still enjoying the property. Thanks!


text:
Can we make a Chatbot that creates a 3d environment or models
for example : (you lived a dream or you want to live in an environment you like )
so whenever you want that you can open UnrealEngine or blender and just using chatbot to build your own dream world , maybe it can be done with virtual reality like metaverse just write what you want to see . I think this idea is doable since virtual realities domination the world few years ago but how it can be done ??
of course I wont ask some one to do that and bring me the task ... I want someone who thinks of ideas or steps like me to do this and where to start and actually how to do this if its possible or not.


text:
Can all good language models be used as a good embedding model and vice versa?
Does solving the language modeling task go hand in hand with finding a good embedding?