text:
How does changing the order of a sequence of rewards affect the cumulative discounted reward
Let's say from time step $t$, the agent interact with the environment for $n$ steps, and gets a reward seuqence $R = (r_{t}, r_{t+1}, \cdots , r_{t+n})$
Now we respectively sort the reward sequence $R$ in both ascending order and decending order to get $R_{asc}$ and $R_{dec}$. By the definition of the cumulative discounted reward which is:
$G_{t}=r_{t}+\gamma\cdot r_{t+1}+\gamma^{2}\cdot r_{t+2}+\cdots=\sum_{k=0}^{\infty}\gamma^{k}\cdot r_{t+k}$
We compute the cumulative discounted reward of these three reward sequences $R, R_{asc}, R_{dec}$, and get $G, G_{asc}, G_{dec}$
What is the result of comparing these numbers $G, G_{asc}, G_{dec}$? Based on my understanding of the idea of discounted reward, my intuition says $G_{asc} < G < G_{dec}$. If that is the case, how do i prove this?

The relation is actually
$$
G_{\text{asc}} \leq G \leq G_{\text{dec}}
$$
Without loss of generality, we can say that the sequence $r_0, r_1, .., r_n = G_{\text{dec}}$ is already sorted in descending order.
Then we know, that
$$
r_0 \geq r_1 .. \geq r_n
$$
Lets look at the difference $G_{\text{dec}} - G$:
$$
G_{\text{dec}} - G = r_0 - r_i + \gamma(r_1 - r_j) ..
$$
Let $\delta_{kj} = (r_k - r_j)$ and lets note, that there always exists a sequence of differences of rewards, weighted by gammas:
$$
\gamma^k \delta_{kj} + \gamma^j \delta_{ji} + ... + \gamma^p \delta_{pm} + \gamma^m \delta_{mk}
$$
It important, that it starts with $\delta_{kj}$ and ends with $\delta_{mk}$, take note of the circular nature of the indexes!
Without loss of generality we can assume, that $r_k$ is the largest element. Then $\gamma^k$ is also the largest and $\delta_{kj} \geq 0$. Then we can say, that
\begin{align*}
\gamma^k \delta_{kj} + \gamma^j \delta_{ji} + ... + \gamma^p \delta_{pm} + \gamma^m \delta_{mk} & \geq \\
\text{(I change }\gamma^k\text{ to }\gamma^m\text{ at first difference)} & \\
\gamma^m \delta_{kj} + \gamma^j \delta_{ji} + ... + \gamma^p \delta_{pm} + \gamma^m \delta_{mk} & = \\
\text{(Now we can combine }\gamma^m \delta_{kj}\text{ and }\gamma^m \delta_{mk}\text{ )} & \\
\gamma^m \delta_{mj} + \gamma^j \delta_{ji} + ... + \gamma^p \delta_{pm}&\\
\end{align*}
So at the end, for sequence, where $r_k$ is the largest element we get:
$$
\gamma^k \delta_{kj} + \gamma^j \delta_{ji} + ... + \gamma^p \delta_{pm} + \gamma^m \delta_{mk} \geq \gamma^m \delta_{mj} + \gamma^j \delta_{ji} + ... + \gamma^p \delta_{pm}
$$
But the resulting sequence is reduced by one element! Now we can repeat the same procedure(*):

Find the biggest element.
Look at the $\delta$s that contains it and combine them.

After multiple steps you will be left with last two differences:
$$
\gamma^k \delta_{kj} + \gamma^j \delta_{ji} + ... + \gamma^p \delta_{pm} + \gamma^m \delta_{mk} \geq \gamma^v \delta_{vw} + \gamma^w \delta_{wv}
$$
Just as before, without loss of generality we can assume, that $r_v$ is the biggest element, then
$$
\gamma^v \delta_{vw} + \gamma^w \delta_{wv} \geq \gamma^w \delta_{vw} + \gamma^w \delta_{wv} = 0
$$
So, in other words we have:
\begin{align*}
G_{\text{dec}} - G &= r_0 - r_i + \gamma(r_1 - r_j) .. \\
&= (\gamma^k \delta_{kj} + ... + \gamma^m \delta_{mk}) + (\gamma^a \delta_{ac} + ... + \gamma^b \delta_{bc}) + ...\\
& \geq  0 + 0 + ...
\end{align*}
And then
$$
G_{\text{dec}} - G \geq 0 \rightarrow G_{\text{dec}} \geq G
$$
For proof of $
G_{\text{asc}} - G \leq 0$, the logic is exactly the same, but all the signs are inversed. We have then
$$G_{\text{dec}} \geq G$$
$$G_{\text{asc}} \leq G$$
Or in other words
$$
G_{\text{asc}} \leq G \leq G_{\text{dec}}
$$

Update
Example of reduction of series of differences:
Without loss of generality, we can say that the sequence $G_{\text{dec}} = r_0, r_1, r_2, r_3, r_4, r_5$ is already sorted in descending order. Let $G = r_5, r_3, r_0, r_1, r_2, r_4$ be (pseudo)random shuffle of $G_{\text{dec}}$. We can re-write the difference as:
\begin{align*}
G_{\text{dec}} - G & = \gamma^0 \delta_{05} + \gamma^1 \delta_{13} + \gamma^2 \delta_{20} + \gamma^3 \delta_{31} + \gamma^4 \delta_{42} + \gamma^5 \delta_{54} \\
 & = \gamma^0 \delta_{05} + \gamma^5 \delta_{54}  + \gamma^4 \delta_{42} + \gamma^2 \delta_{20} + \gamma^1 \delta_{13} +  \gamma^3 \delta_{31}\\
 & = (\gamma^0 \delta_{05} + \gamma^5 \delta_{54}  + \gamma^4 \delta_{42} + \gamma^2 \delta_{20}) + (\gamma^1 \delta_{13} +  \gamma^3 \delta_{31})\\
\end{align*}
Lets take a look at two sums in parentheses:

$(\gamma^0 \delta_{05} + \gamma^5 \delta_{54}  + \gamma^4 \delta_{42} + \gamma^2 \delta_{20})$

$(\gamma^1 \delta_{13} + \gamma^3 \delta_{31})$

Then, by the procedure I have noted with ( * ) in the explanation, we have:
\begin{align*}
\gamma^0 \delta_{05} + \gamma^5 \delta_{54}  + \gamma^4 \delta_{42} + \gamma^2 \delta_{20} & \geq\\
\gamma^2 \delta_{05} + \gamma^5 \delta_{54}  + \gamma^4 \delta_{42} + \gamma^2 \delta_{20} & =\\
\gamma^2 \delta_{25} + \gamma^5 \delta_{54}  + \gamma^4 \delta_{42} & \geq\\
\gamma^4 \delta_{25} + \gamma^5 \delta_{54}  + \gamma^4 \delta_{42} & =\\
\gamma^4 \delta_{45} + \gamma^5 \delta_{54} & \geq \\
\gamma^5 \delta_{45} + \gamma^5 \delta_{54} & = 0 \\
\end{align*}
And
\begin{align*}
\gamma^1 \delta_{13} + \gamma^3 \delta_{31} & \geq \\
\gamma^3 \delta_{13} + \gamma^3 \delta_{31} & = 0\\
\end{align*}
Then, its clear that
$$
G_{\text{dec}} - G = (\gamma^0 \delta_{05} + \gamma^5 \delta_{54}  + \gamma^4 \delta_{42} + \gamma^2 \delta_{20}) + (\gamma^1 \delta_{13} +  \gamma^3 \delta_{31}) \geq 0
$$

This is brilliant! i was thinking this might could also be interpreted as a selection sort, after every sorting, the more ordered sequence gets a increment greater or equal than 0 than the less ordered sequence. But your answer is way more mathematical and authentic

however, i have trouble understanding how do we get to that "After multiple steps you will be left with last two differences"

By "step" I mean reducing the sum of differences by one difference. @juicyliberty I added an example of what it looks like to make it clearer. Hope it helps.

Thank you so much! this sure make things clearer, but actually it is the "circular nature of the indexes" that really bothers me, i want to make sure that "there always exsist a sequence that starts with starts with δkj and ends with δmk", which ensures that the reduction of series of differences can be safely executed. Are there any resources or materials explaining this kind of concept, i'm no expert in mathmatics unfortunately. i'm really sorry for not making my question clearer.

While Im not fully sure how to give a rigorous proof to this, I feel like you can find a solution in graph theory. You can imagine something like a directed graph, where nodes are rewards and edges are given by the $\delta_{ij}$. So if $\delta_{ij}$ is present in $G_{\text{des}} - G$, then you have an edge $ij$ in the graph. We can say, that each node will have two edges - one outgoing and one incoming. Just from this I think you can conclude that it will consist of non-intersecting rings, which is equivalent to the statement we are trying to proof.


text:
Minimum Number of Samples for LLM Benchmark?
I am working on a project to evaluate various fine-tuned LLMs. Unfortunately inference is prohibitively slow, and I don't think I will be able to test my models on the full test set of 40,000 samples. My current plan is to implement an early stopping threshold.
Is there an "accepted" or theoretical minimum number of samples that are required to benchmark an LLM?

The minimum number of test items is a function of the statistical power you want to reach so that you can reliably detect differences of a given magnitude. A small number of test items (a small sample size) can be acceptable for some purposes but be too small for other purposes.
In some fields of research, there are traditionally accepted test set sizes, for example 3000 items. In such fields, you have a good chance of getting your paper published (and pass your PhD exam) with a test set of such a size even if some publications have shown that the size is too small for the task.
If your task is to pick the best model from a large set of candidate models with a limited inference budget, you could first test all models on a small subset of the test set, select the best models (the optimal number of models to select depends on many factors) according to this incomplete test, then test the selected models with more data and repeat the process. Whether the final round uses the full test set and how many models should be in the final is also hard to say. Ideally, you'd test some candidate selection procedures using small, inexpensive LMs and then scale up to your LLMs. An ad-hoc approach like going from 1000 models to 100 models using a test set of 1000 items, then 100 to 10 using 7000 items and finally 10 to 1 using all 40k items can be acceptable in many contexts.
Suggested reading:

Bowman and Dahl (2021) What Will it Take to Fix Benchmarking in Natural Language Understanding?
Card et al. (2020) With Little Power Comes Great Responsibility
Kiela et al. (2021) Dynabench: Rethinking Benchmarking in NLP

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How are POMDPs solved in practice?
In the literature that I've seen so far on how to either exactly or approximately solve POMDPs (Partially-Observable Markov Decision Processes), there seems to be a lot of focus placed on maintaining a distribution over all possible states of what the current state is as well as approximating the value function as a function of this distribution at each step.
Is this level of rigour how things are implemented in practice?
For example, say we have a market with cash and a single other asset that our agent participates in.
First, our agent receives an observation:
$o_0 = (t_0, b_0, h_0)$ where $t_i$ denotes the time of the observation, $b_i$ denotes some actionable observation of the market such as current bid/ask price of the asset, and $h_i$ denotes our holdings of cash and the other asset.
Then, our agent chooses an action $a_0 \in  \mathcal{A}_0$ which is to either sell or buy a specified amount of the asset or do nothing.
Then our agent receives the next observation $o_1 = (t_1, b_1, h_1)$
and also receives a reward, $r_1 = L(h_1, b_1) - L(h_0, b_0)$ where $L(h_i, b_i)$ denotes the value of the portfolio with holdings, $h_i$, as determined by the market representation $b_i$
This is then repeated for the remainder of the potentially-infinite horizon of the POMDP.
If our goal is to maximimize the sum of the rewards over this potentially-infinite horizon and we are able to interact with this environment "live" (as in we are not simulating the dynamics, our agent is directly interacting with the market), how would we go about approaching this in practice if it does differ from the theoretical approaches?

A formal POMDP approach that models a belief state (a distribution over possible states) needs to have a theoretical model for the hidden aspects of the state. This is not always appropriate or possible.
If you have no theoretical MDP that underlies the POMDP (e.g. possible sets of cards an opponent holds in poker), and are limited in not only not knowing hidden data, but also not knowing what structure hidden data might have or the rules governing it, then you need to look at other options.
One approach is to try to extract maximum information that the full observation history can provide. This being the only data that you have, is the best you can do if you lack a theoretical model. This full history can include all observed state data, actions and rewards.
If the data can be summarised logically into a fixed size - perhaps with some theoretical justification in the problem domain - you can use that. If not, you can attempt to learn an abstract summary from experience by making your value function, or your policy function, or a separate "helper" function (e.g. that predicts next observables) into a sequence-based predictor such as an LSTM.
Predicting market behaviour and making investment choices are not known to be easy targets for machine learning models. This is in part because all the low hanging fruit for statistical analysis has been researched and exploited for many years already. There are skilled mathematicians who work in this area professionally, who already understand how to create sophisticated predictive models, and they are your competition for any agent that you could write. It is very hard to create an autonomous agent that can do well in this kind of environment.

Thank you so much for this detailed and insightful response. However, I have a few questions: 1. What do you mean by "summarised logically into a fixed size"? 2. Would you be able to provide a very high level example of the "attempt to learn an abstract summary..." perhaps as applied to this financial example? 3. How do we predict future observables if they depend on our actions taken? do we just add this into our model? Thanks again :)

@QMath I think trying to answer those follow-on questions here would make the answer too long. Apart from 3 - yes you should add action history and current action. Action history may easily affect hidden state, and current action very likely to impact next observables. For other follow up questions, they would be good new questions on the site

That's fair regarding the first two questions and great, that makes sense regarding the third. Thanks again!

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
SOTA model (PAtt-lite) implementation with Pytorch
I'm trying to implement a model guided by the paper of PAtt-lite (weights are on https://github.com/jlrex/patt-lite but no implementation provided yet). Using FER2013+ and RAF-DB. The main classes I've implemented are:

Classes for separable convolutional layers: Depthwise and pointwise.
import math
import torch
import torch.nn.functional as F
from transformers import MobileNetV1Config, MobileNetV1Model, AutoImageProcessor

class DepthWiseSeperable(nn.Module):

    def __init__(self, in_channels, out_channels, stride, padding, kernel_size):

        super(DepthWiseSeperable, self).__init__()

        # groups used here
        self.depthwise = nn.Conv2d(in_channels = in_channels , out_channels = in_channels , stride = stride , padding = padding, kernel_size = kernel_size , groups=in_channels , bias = False)
        self.relu = nn.PReLU(in_channels)
        self.bn = nn.BatchNorm2d(in_channels)

    def forward(self, x):

        x = self.depthwise(x)
        x = self.relu(x)
        x = self.bn(x)
        return x

class PointWiseSeperable(nn.Module):

    def __init__(self, in_channels, out_channels, stride):

        super(PointWiseSeperable, self).__init__()

        self.pointwise =  nn.Conv2d(in_channels = in_channels , out_channels = out_channels , stride = 1 , padding = 0, kernel_size = 1, bias = False)
        self.relu = nn.PReLU(out_channels)
        self.bn = nn.BatchNorm2d(out_channels)

    def forward(self, x):

        x = self.pointwise(x)
        x = self.relu(x)
        x = self.bn(x)
        return x

class SepConv2d(nn.Module):
    def __init__(self, in_channels, out_channels, stride, padding, kernel_size):

        super(SepConv2d, self).__init__()

        self.depthwise = DepthWiseSeperable(in_channels, in_channels, stride, padding, kernel_size)
        self.pointwise = PointWiseSeperable(in_channels, out_channels, stride)

    def forward(self, x):
        x = self.depthwise(x)
        x = self.pointwise(x)

        return x

Since later on we are going to create an instance of MobileNetV1 from huggingface a class is implemented to truncate such net. Paper says:

layers after the depthwise convolution of block 9 are skipped

class Truncated_MobileNet(nn.Module):

    def __init__(self, original_model):
        super(Truncated_MobileNet, self).__init__()
        self.conv_stem = original_model.conv_stem
        self.layer = nn.ModuleList(original_model.layer[:17])  #layers after the depthwise convolution of block 9 are skipped

    def forward(self, x):
        x = self.conv_stem(x)
        for layer in self.layer:
            x = layer(x)

        return x

Here patch extraction module. In the paper they say that pading is needed from 14x14 to 16x16, but I already obtain 16x16 from truncated mobilenet. I adjust the kernel sizes and strides to match dimension with the architecture graphic.
class PatchExtraction(nn.Module):

    def __init__(self, in_channels, out_channels):
        """
        Patch extraction block.

        Args:
            in_channels (int): Number of input channels.
            out_channels (int): Number of output channels.
        """
        super(PatchExtraction, self).__init__()

        # Padding to achieve 16x16 spatial dimensions
        self.padding = nn.ZeroPad2d((1, 1, 1, 1))

        # Depthwise separable convolution output 4x4x256
        self.depthwise1 = SepConv2d(in_channels = 512, out_channels = 256, stride = 4 , padding = 0, kernel_size = 4)
        # Depthwise separable convolution output 2x2x256
        self.depthwise2 = SepConv2d(in_channels = 256, out_channels = 256, stride = 2 , padding = 0, kernel_size = 2)

        # Pointwise convolution output 2x2x256
        self.pointwise = PointWiseSeperable(in_channels = 256, out_channels = 256, stride=1)

    def forward(self, x):

        x = self.depthwise1(x)
        x = self.depthwise2(x)
        x = self.pointwise(x)

        return x

The attention classifier module. I am using residual as I saw that stabilises the training, but I don't know if this way is the best practice.
class AttentionClassifier(nn.Module):
    def __init__(self, in_features, hidden_size, num_classes, d_q):
        super(AttentionClassifier, self).__init__()

        # First fully connected layer
        self.fc1 = nn.Linear(in_features, hidden_size)
        self.relu = nn.ReLU()

        # Linear projections for query, key, and value
        self.fc_query = nn.Linear(hidden_size, d_q)
        self.fc_key = nn.Linear(hidden_size, d_q)
        self.fc_value = nn.Linear(hidden_size, d_q)

        # Second fully connected layer
        self.fc2 = nn.Linear(d_q, num_classes)
        #torch.nn.init.xavier_uniform_(self.fc2.weight)
        self.softmax = nn.Softmax(dim=-1)
        # ScaledDotProductAttention instance
        #self.attention = ScaledDotProductAttention(d_q)

    def forward(self, x):

        # Apply the first fully connected layer
        y = self.fc1(x)
        y = self.relu(y)
        # Linear projections for query, key, and value
        query = self.fc_query(y)
        key = self.fc_key(y)
        value = self.fc_value(y)

        attention_output = F.scaled_dot_product_attention(query, key, value, dropout_p= 0.0, scale = None)
        attention_output = attention_output + x
        # Apply the second fully connected layer
        x = self.softmax(self.fc2(attention_output))

        return x

Joining patch extraction and attention modules. I don't know if input_dim<hidden_dim is the best use of attention.
class Top_model(nn.Module):
    def __init__(self,  num_classes=7):

        super(Top_model, self).__init__()

        self.patchextraction = PatchExtraction(512, 256)

        # Global Average Pooling and classifier

        self.classifier = nn.Sequential(
            AttentionClassifier(in_features=256, hidden_size=512, num_classes=7, d_q=256),
        )
        self.gap = nn.AdaptiveAvgPool2d((1,1))

    def forward(self, x):
        x = self.patchextraction(x)
        x = self.gap(x)
        x = x.view(x.size(0), -1)  # Flattening the tensor
        x = self.classifier(x)

        return x

Finally join all:
class Patt_lite(nn.Module):

    def __init__(self, truncated_model, top_model, freeze_base=True, freeze_top = True):

        super(Patt_lite, self).__init__()

        self.base = truncated_model
        if freeze_base:
            for i, param in enumerate(self.base.parameters()):
                param.requires_grad = False

        self.top = top_model
        if freeze_top:
            for i, param in enumerate(self.top.parameters()):
                param.requires_grad = False

    def forward(self, x):
        x = self.base(x)
        x = self.top(x)

        return x

For the training I'm using Fastai:
# Data augmentation for training
train_batch_transforms = [
     # Quality
     Brightness(p=0), Contrast(p=0), Saturation(p=0, draw = 0),
     # Data Alterations
     RandomErasing(p=0.25),
     # Orientation
     Flip(p=0.5), Rotate(p=0.75, draw=30), Warp(p=0.5),
     #Norm
     Normalize.from_stats([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
 ]
train_item_transforms = [
    Resize(224),
    ToTensor(),
]

test_batch_transforms = [
     Normalize.from_stats([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
 ]
test_item_transforms = [
    #ToTensor, Normalize
    Resize(224),
    ToTensor(),

]

#Set batch size
batch_size = 8

# Create data loaders using the defined transforms
dls = ImageDataLoaders.from_folder(
    path='path to train dataset',
    valid_pct=0.2,
    bs=batch_size,
    valid_bs=batch_size,
    item_tfms=train_item_transforms,
    batch_tfms=train_batch_transforms,
    device="cuda",
    convert_mode='L'
)

#Model instance
configuration = MobileNetV1Config(tf_padding=False)
original_model = MobileNetV1Model.from_pretrained("Matthijs/mobilenet_v1_1.0_224", config=configuration)
truncated_model = Truncated_MobileNet(original_model)

top_model = Top_model(num_classes = 7)
patt_model = Patt_lite(truncated_model, top_model, freeze_base = False, freeze_top = False)
patt_model = patt_model.to(device)

#Optimization and learning
opt_func = partial(OptimWrapper, opt=optim.RMSprop)
learn = Learner(dls, patt_model, opt_func=opt_func, metrics=accuracy)
learn.loss_func = CrossEntropyLossFlat()
learn.fit(n_epoch=50, lr = 1e-4, cbs = [ShowGraphCallback()])

I tried Matthijs/mobilenet_v1_1.0_224 and google/mobilenet_v1_1.0_224, but seems the same.
On the paper they split the training in two parts: top model training with mobilenet freezed and fine tuning of mobilenet. Are the weights of mobilenet correctly loaded?
The first training result in bad performance and the second one do not improve the accuracy or loss.

Any idea? Thanks

Edit:
Option A: Eliminate dot_product_attention (also softmax and residual), so
class AttentionClassifier(nn.Module):
    def __init__(self, in_features, hidden_size, num_classes, d_q):
        super(AttentionClassifier, self).__init__()

        # First fully connected layer
        self.fc1 = nn.Linear(in_features, hidden_size)
        self.relu = nn.ReLU()

        # Second fully connected layer
        self.fc2 = nn.Linear(d_q, num_classes)

    def forward(self, x):

        # Apply the first fully connected layer
        x = self.fc1(x)
        x = self.relu(x)

        # Apply the second fully connected layer
        x = self.fc2(x)
        #x = self.softmax(x)

        return x

After 25 epoch:

Option B: No patch, no dot_product_attention, just MobileNet (freezed) and 2 FC layers.

Option C: same as Option B but with mobilenet unfreezed. Clearly overfitting.

Option D: No patch extraction, attention without residual.
class AttentionClassifier(nn.Module):
    def __init__(self, in_features, hidden_size, num_classes, d_q):
        super(AttentionClassifier, self).__init__()

        # First fully connected layer
        self.fc1 = nn.Linear(in_features, hidden_size)
        self.relu = nn.ReLU()

        # Linear projections for query, key, and value
        self.fc_query = nn.Linear(hidden_size, d_q)
        self.fc_key = nn.Linear(hidden_size, d_q)
        self.fc_value = nn.Linear(hidden_size, d_q)

        # Second fully connected layer
        self.fc2 = nn.Linear(d_q, num_classes)

    def forward(self, x):

        # Apply the first fully connected laye
        x = self.fc1(x)
        x = self.relu(x)
        # Linear projections for query, key, and value
        query = self.fc_query(x)
        key = self.fc_key(x)
        value = self.fc_value(x)

        attention_output = F.scaled_dot_product_attention(query, key, value, dropout_p= 0.0, scale = None)
        #Apply the second fully connected layer
        x = self.fc2(attention_output)
        #x = self.softmax(x)

        return x

class Top_model(nn.Module):

    def __init__(self,  num_classes=7):

        super(Top_model, self).__init__()

        self.patchextraction = PatchExtraction(512, 256)

        # Global Average Pooling and classifier

        self.classifier = nn.Sequential(
            AttentionClassifier(in_features=512, hidden_size=256, num_classes=num_classes, d_q=256),
        )
        self.gap = nn.AdaptiveAvgPool2d((1,1))

    def forward(self, x):
        x = self.gap(x)
        x = x.view(x.size(0), -1)  # Flattening the tensor
        x = self.classifier(x)

        return x

Option E: Same as option D but with  attention_output = attention_output + x

In my opinion it's always important to make sure that the model trains first.
Since your training loss does not really decrease, it's interesting to know if you have tried changing the hyperparameters? For example increasing the learning rate?
I also noticed that in your attention layer you are adding x as a residual element and not the y, that looks like an input to attention head. Is it intentional?

I don't see any problems in the code at first glance, but what can be useful, if you have in fact checked that its not hyperparametetrs problem, is to try simpler base or top model. For top you can just flatten all the features and apply one fully connected layer for example. Like this you can assure yourself that specific parts of your model in fact are working as they are suppose to. You can really do this for all the important "elements" of the model, where the problem can be hidden in your opinion.

The using of x and y was intentional since i tried to increase the hidden dimension, so the hidden dimension was grater than k,q, v and attention output dimension, thus y = self.fc1(x) does not match with attention_output but x matches. I tried also with input_dim = hidden_dim = output_dim and use y as a residual: same performance.

Thanks! I will try it part by part to address the issue.

@vl_knd Do i need a flatten between GlobalAvgPool and the classifier aprt in top model forward? It is enough with the AdaptativeAvgPool2d((1,1)) and x = x.view(x.size(0), -1)??

I would suggest you replace x.view(x.size(0), -1) by torch.flatten (https://pytorch.org/docs/stable/generated/torch.flatten.html)

Can you try doing global average polling on feature maps of mobile net and training your attention classifier first, to see if it works? Without the "patch extraction block"?

Let us continue this discussion in chat.


text:
Find the relationship between data in this plot
Attached image. How would you find the relationship between independent variable (x) and dependent variable (y)? Is it linear or non-linear? What would the function looks like?
P.S. I believe this is a non-linear regression problem as we are trying to fit a model (function) to this data. I just can't figure out the function. It seems like a mix of positive and negative sigmoid? or quadratic?

Here is my code I am trying to recreate the plot I saw before:
import pandas as pd
#create DataFrame
df = pd.DataFrame({'x': [0, 1, 2, 3, 11, 12.2, 12.4, 13.9, 14, 15, 4, 10.7, 12.6, 8, 8.5, 9, 13.5, 13.1, 12.8, 5, 15, 14.5, 6, 6.3, 6.6, 7, 7.4, 5.4, 4.4, 3.5, 2.5, 1.5, 0.5],
'y': [0, 3, 7, 8, 25, 40, 15, 105, -10, -20, 12, 60, 80, 37, 40, 46, 100, 66, 5, 16, 120, 28, 22, 24, 26, 30, 32, 18.5, 14.3, 9.8, 8, 5.2, 1.8]})
#view DataFrame
print(df)
import matplotlib.pyplot as plt
#create scatterplot
plt.scatter(df.x, df.y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.title('Function?')
plt.show()

This does not look like a function. For the same x, it seems the function may produce different $y$s. So, there might be some stochasticity here, or it's just the plot that suggests that it.

@nbro Sorry that is my bad. I drew this plot based on what I remember from the similar plot I saw before. There should not be multiple y values per x, so imagine this is a 1 on 1 function. How would you approach this then? what type of activation function does this look like? It must be one from the famous DL activation function family, e.g. logistic, ploynomial, sigmoid, etc. https://en.wikipedia.org/wiki/Activation_function

It would be great if you removed the wrong dots then.

@nbro I just did. To the best of my knowledge. All I remember is the +120 and -20 on Y axis. and data looked non linear. almost exponential. data was really concentrated up until x=4, then started spreading up and down .

Reminds me of this sort of graph: https://graphicmaths.com/img/fractals/fractals/bifurcation/bifurcation-4.png implying you have a system with behaviour phasing between linear and non-linear behaviour, becoming chaotic. If that's the case, accuracy of predictions for x > 4 will necessarily be inaccurate, because precise prediction is impossible

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.


text:
Image classification for two categories: background and not background
Sorry, I have some background on AI, but very little experience.
I am facing an image classification problem:
I need to detect whether an image is background or not (images are always of the same size, which may vary from 15x15 pixels to 70x70 pixels, depending on the context)
Is there any easy and light (and preferrably standard) algorithm to be applied to detect that?
The model should have to be trained with only 10 (or more) background image examples, and 10 or less not background image examples.
Modification
The algorithm will be applied for detecting empty boxes at a chess board.
Each box might be empty (background) or with a chess figurine (not background).
The model or the image feature, will have to be extracted from a pair  [ image, FEN string ]

The image is a chess board image
The FEN string is a string that defines univocaly the position of the figurines inside the chess board.

The current algorithm is able to split the chess board into 64 separate boxes.
So now is the turn to create a model that is able to decide whether a box is empty or not.
There will be two kind of boxes (the white ones and the black ones).
There are then, at most, 16 figurines per player (32 figurines).
Hopefully, there will be more than 10 examples of empty white boxes, and 10 examples of empty black boxes.
And hopefully, there will be at least one not background example for white and black boxes.
So, the meaning of background in this case is: an uniform texture or flat color at the whole of the image.
And not background, is an image with our background as background, but with any other overlapping shape as foreground.
Update (2024-03-03)
Thanks for your answer, Adrien.
I have just done a look on your proposal.
And, based on canny outputs got from an image set, I have inferred that the idea might work for a flat color background, but it is not so easy to distinguish whether the image is background or not based only basing on a canny output when the background is a texture.
I have calculated the canny outputs using a code I found on the internet some years ago, and configuring the algorithm with different parameters.
You can download the results here: https://frojasg1.com/stackOverflow/20240303.imageClassification/20240303.canny.7z
Any other ideas are welcome!
Update 2024-03-17
They have asked me to attach some examples, and here they are:
The examples
They are organized in folders.
All images in a particular folder, have the same board box size.
There are some images in tiff format (tiff was chosen for avoiding loosing quality for compression).
The name of files say which chess figurine they represent.
There can be black or white figurines, over black or white board boxes, and the letters for those figurines are the initial letters (KQRBNP): King, Queen, Rook, Bishop, Knight and Pawn.
The patterns that represent empty board boxes, have a name with four letters: WWWW for empty white board boxes, and BBBB for empty black board boxes.
The examples in the zip file have been obtained automatically from automatic training of the chess position recognizer I programmed for my application, and I have not checked if all of them have been right identified, I hope so ...
Well, those are the examples.
I hope that helps for achieving the solution

Could you elaborate on the description of your "background" . What is that visual feature that describes the background? Based on the description maybe the problem has to be redefined.

Well, the application for that image classifier, will be to detect, in a chess board context, whether a board box is empty or not.
An empty board box, is filled with a texture, which may be a plain color, or a texture, which is not so easy to detect.
The non background boxes, include a chess figurine, that would made the detection be: not background.

My idea, would be to use some kind of image feature, which would not need many examples to be calculated, and then to use some kind of error measure, and a threshold, for the decision. But I do not know if that is possible, neither which feature, nor error measure algorithms to be used.

You could try something like an edge detector (maybe canny algorithm?) If there is almost to no edge, it's backgroud, if there is some edges, if's a piece.

Thank you Adrien suggesting an idea for solving the problem. I will update the body of my question to answer it more appropiately

Can you provide examples of background and not background?

Hi Marko, sorry for the delay, I have been very busy
I have just zipped the examples I have (they have been obtained automatically, from some automatic training for the chess position recognizer I programmed for my application and I have not checked if all of them are valid).
I will update the text of the question to include a link to the zip with the examples.


text:
Is this right logic flow of Variational Autoencoder?
Object Function
$$
maximize = \Pi_d^D P(X_d) = \Pi_d^D \sum_i^N P(X_d,Z_i)
$$
D : # of data
N : # of latent variable state
$$
NLL = - \sum_d^D \log \sum_i^N P(X_d,Z_i) = - \sum_d^D \log \sum_i^N P(X_d,Z_i) \frac{Q(Z_i|X_d)}{Q(Z_i|X_d)}
$$
Jensen's Inequality
$$
\leq  - \sum_d^D  \sum_i^N Q(Z_i|X_d) \log \frac{P(X_d,Z_i) }{Q(Z_i|X_d)}
$$
$$
=  - \sum_d^D  \sum_i^N Q(Z_i|X_d) \log \frac{P(X_d|Z_i) P(Z_i) }{Q(Z_i|X_d)}
$$
$$
=  \sum_d^D [ D_{KL}(Q(Z_i|X_d)|| P(Z_i)) - \sum_i^N Q(Z_i|X_d) \log P(X_d|Z_i) ]
$$
$$
=  \sum_d^D  D_{KL}(Q(Z_i|X_d)|| P(Z_i)) - E_Z[\log P(X_d|Z_i)]
$$
KL- divergence term (Regularizer)
$Z_i$ is continous, so we assume Z as Gaussain.
$$
D_{KL}(Q(Z_i|X_d)|| P(Z_i)) =  -\int_{Z_i} Q(Z_i|X_d) \log P(Z_i)) + \int_{Z_i} Q(Z_i|X_d) \log [Q(Z_i|X_d) ]
$$
multivariate Gaussian Distribution
if basis is appropriately adjusted, then we can assume that sigma is diagonal matrix which means that offdiagonal term is 0. so covariance matrix $\sum$ is written as $\sigma^2I$
$$
Q(Z|X) \sim N(\mu,\sigma^2I)
$$
$$
D_{KL} = (\mu^2 + \sigma^2 -1)/2 - \log \sigma
$$
Q() can be reparameterize to differentiable, determistic variable
$$
P(Z_i) \sim N(0,I) \\
Z_i \sim Q(Z_i|X_d) = Q_\phi (\epsilon,X_d) \\
\epsilon \sim p(\epsilon) \\
$$
applying amortized inference Q
$$
Q(Z_i|X_d;\phi)  = \Pi_i Q(Z_i;NN_i(X_d;\phi_i))
$$
Reparameterization Trick
$$
Q_\phi (\epsilon,X_k) = \mu + \epsilon * \sigma
$$
Monte-carlo sampling
where $\epsilon^{(d)} \sim p(\epsilon)$
$$\epsilon \sim \mathcal{N}(0, I)$$.
Expectation term (Reconstruction)
$E_i^{(d)}$ is reconstructed data and $E_i$ is original data
Depending upon the assumption of $E_i$ and $p_\theta\left(E \mid E^{(d)}\right)$
If $E_i$ is continuous and $p_\theta\left(E_i \mid E_i^{(d)}\right) \sim \mathrm{N}\left(E_i^{(d)}, 1^2\right)$
$$
\log p_\theta\left(E \mid E^{(d)}\right)=C \times \log \exp \left(-\frac{\left(E_i-E_i^{(d)}\right)^2}{2}\right)=-C \times\left(E_i-E_i^{(d)}\right)^2
$$
$\cdot$ If $E_i$ is discrete and $p_\theta\left(E_i \mid E_i^{(d)}\right) \sim \operatorname{Bern}\left(E_i^{(d)}\right)$
$$
\log p_{\theta}(E \mid E^{(d)})
$$
$$
=\log E^{(d)^{K_i}}(1-E^{(d)})^{1-E_i}
$$
$$
=E_i \log E^{(d)}+(1-E_i) \log (1-E^{(d)}
$$
is this right logic flow?
please let me know about this.
thanks to read :D

what you called amortized inference is actually Amortized inference + mean field assumption... but the rest I would say that is correct, however, I would say that this only explains the math, not the intuition (though this might be a personal take)


text:
Why do adversarial attack transfer well?
I have read (*) that a common technique to attack a black box AI system based on a neural network is to use it to train a surrogate model to make the same classifications as the black box one.
Once this is done, one can look for adversarial examples on the surrogate model (on which the attacker has access to all the weights and can compute gradients).
The key property that makes such attacks successful is transferability: an adversarial example on the surrogate model is likely to be an adversarial example on the black box model.
Question: Do we know why such transferability properties hold and under which conditions?
(*) Although I don't remember where I first read it, this is mentioned in the book Not with a bug but with a sticker although it doesn't get into technical details (this is a book for the general public).

A reference or link to where you have read this information would help a lot for anyone attempting an answer

@NeilSlater Right, I added a reference in my question.

As far as I know, there are only hypothesis on this (https://simons.berkeley.edu/talks/new-perspective-adversarial-perturbations) which pretty much relies on the idea that there are "spurious" correlation in the dataset used for training (check the video in the link for more informations, they have also some papers referenced to read more about it)


text:
weighted multi class classification
i'm working on a multi class classification problem which classifies jellyfish and plastic pollution so basically i have 6 classes (barrel_jellyfish, compass_jellyfish, lions_mane_jellyfish, mauve_stinger_jellyfish, moon_jellyfish, plastic pollution) according to the description of the competition the evaluation is this

Correct Classification Upon making the correct class prediction, no
penalty will be applied.
Classifying Plastic Pollution as any Class of Jellyfish: This error
will incur a heavy penalty. Misclassifying plastic pollution as
jellyfish has more significant consequences in the context of our
mission to raise awareness about the impact of plastic on marine
ecosystems.
Misclassifying any Class of Jellyfish as Plastic Pollution: This error
will incur a moderate penalty. This will emphasize the need to
maintain a balanced approach in the classification task.
Confusing Different Classes of Jellyfish: Errors within the spectrum
of jellyfish classes will incur a minor penalty. While accuracy in
distinguishing between jellyfish types is valued, our primary focus
remains on minimizing the misclassification of plastic pollution.

i made a simple multi class classification neural net which gave me great results (accuracy of around 95%) however i believe penalizing miss classification of plastic pollution will give me better results so far i tried weight class in keras and it didn't help.
here's the model:
def build_model(hp):
    base_models = [keras.applications.Xception(include_top=False, weights="imagenet", input_shape=(224,224,3)), keras.applications.ResNet50(include_top=False, weights="imagenet", input_shape=(224,224,3)), keras.applications.ResNet50V2(include_top=False, weights="imagenet", input_shape=(224,224,3))]
    base_model = base_models[0]
    for layer in base_model.layers:
        layer.trainable=False
    model = Sequential()
    model.add(base_model)
    model.add(layers.Flatten())
    model.add(layers.Dense(hp.Int("units1", min_value=32, max_value=256, step=32), activation=hp.Choice("activation1",["relu", "tanh", "sigmoid"])))
    model.add(layers.Dropout(0.3))
    model.add(layers.Dense(hp.Int("units2", min_value=32, max_value=256, step=32), activation=hp.Choice("activation2",["relu", "tanh", "sigmoid"])))
    model.add(layers.Dense(6, activation="softmax"))
    optimizer = keras.optimizers.Adam(hp.Choice("lr", [0.001, 0.01]))
    model.layers[0].trainable = False
    model.compile(optimizer=optimizer, loss=keras.losses.categorical_crossentropy, metrics="accuracy")
    return model

hp = keras_tuner.HyperParameters()
tuner = keras_tuner.RandomSearch(hypermodel=build_model, objective="val_accuracy", max_trials=5, executions_per_trial = 1, directory="tuner")
tuner.search(train_generator, validation_data=val_generator, epochs=12)

best_hp=tuner.get_best_hyperparameters()[0]
h_model = tuner.hypermodel.build(best_hp)
h_model.summary()
weight_class = {0:0.1, 1:0.1, 2:0.1, 3:0.1, 4:0.1, 5:0.5}
h_model.fit(train_generator, validation_data=val_generator, epochs=12, class_weight=weight_class)

if you believe in my case weighted class is useless let me know what else can improve my model


text:
Extracting features from multiple curves
I am building a model that predicts the SOH of a lithium ion battery.
My data are from 600 battery charge cycles as follows: for each cycle I have 3 curves each of length 128 samples: voltage, current and derivative of voltage. Also I have the real SOH for each cycle. My question is how I can encode these 3x600 curves to a feature space of around 5-6 features per cycle, that I will then feed to a LSTM model. The LSTM will take the features of the 10 previous cycles and will predict the SOH of the next 3. Thanks in advance.


text:
ML Model for Route Optimization in Manufacturing
I am looking for a ML Model for route optimization within a factory. I am starting simple with optimizing 1 aisle (2 rows). We have a bunch of criteria, and it would be dependent on the data at that time to create an optimal route. For now I want to stay within 1 aisle, so I don't know if ant colony is the answer. I am looking for a suggestion of which model should I use? RL, GA etc.
we have a df of shelf, rack, row, item and when it was assigned to come down (starting with just rows 1 and 2)
The driver can access 2 rows of shipping locations at the same time, and the rows' rack numbers and shelf numbers are aligned with each other . For example, the shipping locations at rack 4 and shelf 3 are accessible for row 1 and row 2 at the same time, as those rows are linked to each other, so the machine driver can get items from those 2 shipping locations at the same exact time.
Machine max capacity is 60 cf - The items with the highest priority are items with the earliest dock assignment time but if a load is assigned to a dock at a similar time to the earliest dock assignment, and has significantly fewer items than a load with the earlier dock assignment time, prioritize the smaller load.
Calculate whether the item will fit into the machine in order to be removed from its shipping location. Consider the size of the item and the capacity available in the machine.
When creating the route, prioritize staying within the same rack number higher than staying within the same shelf number.

CSPF provides a good approach- where it's not so complex that you have to tune, based on dijkstra, and it can handle constraints.


text:
How can Transformers handle random sequences?
I have asked ChatGPT the following:

Can you concatenate jfef9230rj2mreg90r23ewfrn02eqwdk and
32ir20r3i2ofg90r32kee?

And without any error the model produces:

jfef9230rj2mreg90r23ewfrn02eqwdk32ir20r3i2ofg90r32kee

My question:
Does ChatGPT rawly tokenizes random sequences, or do they do some preprocessing and replace these sequences by new Tokens like:

Can you concatenate SEQUENCE1 and SEQUENCE2?

where SEQUENCE1 ... SEQUENCE_N is in the vocabulary, and before the Cross-Entropy on the prediction (which is hopefully SEQUENCE1||SEQUENCE2) they resubstitute back the sequences.
I think the copying is quite straightforward, shouldn't a Transformer be able to learn how to copypaste/move around long sequences of bytes without any preprocessing techniques?
In my project I have done good amount of training and it only manages to move around only little segments of the sequence, and I was thinking if this is even the right approach to rawly Tokenize the sequences, or if I just had too little training.

The stage of setting the available token encodings when training LLMs is very early on. It is before, and separate to the token prediction training (the core training process for the models).
It is not possible to change the tokenisation after training by any mechanism. You would need to restart the full training process from scratch, which is very time-consuming and expensive.
There is no special detection or pre-processing of requests based on the type of question and passing to different "modules". Whilst this is theoretically possible, the effort required would be high, and I doubt anyone has gone to lengths required to impress you with ChatGPT being good at a task that can be easily and 100% reliably done in a single line of regular code without any AI.
So the answer to your question is that your random sequences are being tokenised using ChatGPT's existing tokens. It does not create any new placeholder tokens, and there is no pre-parsing to route this kind of request. Instead, it really does perform string concatenation - relatively accurately and reliably - on long strings of tokens, when requested using natural language, using the core model.

In my project I have done good amount of training and it only manages to move around only little segments of the sequence

For a person writing a hobby transformer project, a "good amount of training" is at a very different kind of scale than large companies publishing LLMs that can perform the concatenation feat amongst many other capabilities.
For GPT-3 (a relatively old model, but still capable of performing the open-ended natural language concatenation task), OpenAI spent over four million dollars, and trained on 45TB of data, taking over 9 days to train on large numbers of V100 instances.
If all you want to do is train a transformer model to concatenate inputs, you should be able to do it much more cheaply. If that is your goal, you could try sharing more details of your approach and results in a separate question.


text:
Common technique to induce a parser?
I am interested in running a simple but hopefully rigorous algorithm which “learns” how to parse HTML - ideally optimally fast.
I have never done this before and I’m wondering if there’s a go-to algorithm or library for this.
I would prefer to start with an extremely simple algorithm even if it is not fast or efficient - but effective. Something like, I give it a corpus of perfect HTML with no errors, and it should generate every possible parsing “hypothesis”, and score each one.
Maybe a simple way to represent a successful “parse” (for a first version of the project) is just the insertion of spaces between HTML language elements.
So, let’s say the model may receive HTML like
< html >      text     </ html >
and
<html>text</html> (or something like that, anyway),
through trial and error it should have generated every possible arrangement of characters in ASCII/UTF-8, and build a tree of states depending on what it gets in some given input string. If it gets “a”, maybe the result has any other letter as of identical decision-relevance, but a “<“ as a different branch on the decision tree (or a cyclical decision graph)?
I’m completely new to this and excited to finally be trying out simple, deterministic, optimizing grammar induction, which I have been learning about for the last 6 months and finally would like to try.
I can read up on some resources here for this:
https://nlp.stanford.edu/projects/up-gi.shtml
Thanks.

Just an FIY THIS is what I have been looking for for months - https://en.wikipedia.org/wiki/Symbolic_regression - but could still use help in implementing a simple first run of it.


text:
Do the mathematics of Go imply an AI could solve it, or does a complexity bound imply AI skill will plateau?
Based on the mathematics of Go and the machine learning algorithms used to play it, is there a mathematical limit as to how much of the game-tree space even an AI could learn, because of the inherent complexity / computational bounds (number of possible states of the board) of the game Go? Are we approaching a technological limit given current computer hardware (even massive data servers)?

You need to define timeframe or the question turns comically irrelevant - given technological development it is unlikely there is not a time we can not just solve GO with brute force. Not day, not tomorrow, but what in 100 years, even without a singularity event? Also, even if AI is complexity bound (i.e. does not learn a lot more above a certain size) it likely will get FASTER, enabling AI to do more in a timeframe and be smaller, meaning you suddenly talk of AI swarms.

Consider that solving whatever problem with AI or not is a computational matter. Simply put: you can't break the rules of computational complexity.
Let's consider the game of Go, you can enumerate all the possibile combinations of states which is a very huge number like $10^{350}$ or so (I don't remember exactly, but is something larger than the number of atoms in the universe!) The point here is that Go and similar have an exponential complexity meaning that the number of possibilities increases exponentially at each step in time: these are usually called NP problems in computer science terms, i.e. problems with non-polynomial complexity which are intractable when aiming for a perfect (or globally optimal) solution.
In practice, if you consider a brute-force approach it would take forever to solve Go optimally no matter how many processors you have and how much fast they are: the computational complexity is just intractable, and the situation won't change even in 1000 years! (Also considering a computer machine with infinite memory.)

The effect of increasing the computation power is only limited. For example, say after 10 year you can get a 100x faster machine or even more. Maybe you'll be able to solve Go perfectly but only for small boards say $N\times N$ but not for $(N+1)\times (N+1)$ ones due to the exponential nature of the problem.
So you need an exponential increase in computation to increase the size of a solvable state space just by $1$, or so.
Time allows you to get better equipment but in case of NP problems it won't allow to solve that class of problems for whatever $N$, i.e. problem size, in an acceptable time.

For such NP (or combinatorial) problems, the solution is to lower the computational complexity at the cost of introducing inaccuracies in the solution. Thus not computing a perfect (optimal) solution anymore but, instead, an approximated (or locally optimal) one. The way you can achieve this is, for example, by using search heuristics that help to cut down the search space (by pruning some sub-trees) thus allowing to solve larger $N$, or to approximate more severely like sampling a single (or few) paths in the full tree of possibilities like MTCS (monte-carlo tree search, which also powers AlphaGo) does.

Another example is deep learning, that allows to solve hard but also difficult to code problems by learning from data. DL models yield an approximated solution that is usually (or better said always, in practice) a local optimum.
You can control the goodness of the solution by the number of epochs for example. Therefore, a better model may allow to get the same solution in few epochs or even a better one - but the point is that it's still approximated.
Thus, DL can be seen as a smart way to approximate a brute-force search.

So to conclude better (learning) algorithms allow to calculate better solutions in less time, but the problem remains intractable. Therefore is best to invest time in inventing better/faster/more accurate (approximated) algorithms rather than waiting to get a better CPU.
(As a side note, in theory and in ideal conditions, quantum computers may process an exponential amount of information, potentially solving combinatorial problems in linear or so time: be aware that I'm not a big expert. Indeed, in practice you have few qbits that are partially interconnected and you need to deal with errors and external interference, and as well approximation of operations with the quantum gates. Maybe in the future quantum computers would allow to solve for very large $N$, with good solutions too, but I think that it would still be possible to hit the hardware limit by finding a sufficiently large $N$ that is just too much to solve for.)

The excellent answer I was hoping for. Thank you!

In the world of AI 100 percentage accuracy is almost impossible it is always a continous learning , The models which learns the game typically fall under Reinforment learning which learns the game by doing the mistakes, if a model did a mistake it punish itself with a quantative measure for that mistake and reward itself for a right move.
In a virtual setting the 100 percentage acuracy is only possible if the model makes all the possible mistakes in all kind of setting(placement of coins) on the board ,which was quite impossible to generate that many combinations but with the today's computation power we had reach almost 70% to 75%(guess) of the combinations but still computing for the combination is still a challenge

Take a look at the combinatorial complexity of Go. You may want to re-assess that 70%-75% guess a lot: https://en.wikipedia.org/wiki/Go_and_mathematics#Legal_positions


text:
What is the most abstract mathematical treatment of machine learning?
The essential characteristic of machine learning is that an algorithm can discover the behavior of a system on its own. Neural networks are a foremost example of this.
But what property do neural networks have, for them to be able to discover a range of patterns?
Has the class of all algorithms with that property been studied?
I think any function is trainable if it has a parameter of variation, and a rule by which the parameter varies.
A trivial example is that the equation y = 3 is “trainable” by this rule:
If y < 3, add 1.
If y > 3, subtract 1.

Starting at y = 0, you would succeed in finding the right value for y.
I would like to view the concept of “trainable functions” from the perspective of set theory or category theory, but I am not experienced in them yet.
One core premise I am working with is that all of mathematics can be defined in terms of sets, so any mathematical concept - like “+”, or a function - can be thought of as some set. Therefore, any aspect of a system can vary - the operation, rather than an numerical value - and the rule by which it varies can be of any type, not just numerical adjustment, but just random selection of a different element in the set.
This is as far as I’ve gotten in my thinking. I think the next point of observation is how some “sets with a rule of variation” are limited. If you vary the m in y = mx, you can not discover a function like y = x^2.
The point is to eventually understand why neural networks are so general, compared to other optimization functions.

Welcome to the site!  I'm reaching out to help you formulate a better question since you are new to this site.  This is the type of question that would normally get downvoted because it is verbose, is asking multiple questions, and is asking subjective questions.  Check out the help section to learn how to ask good questions.  As a general rule of thumb, people don't have time.  They certainly don't have time to read long wordy posts, so to get responses, make your posts short, clear, and to the point.

I think I made it into a single question, maybe I’ll need a second edit though. It’s basically, if we identify a key property of neural networks, then study the class of all functions with that property, can we understand why neural networks are more capable than techniques like linear regression, what extra property do they have compared to other optimization techniques. Thanks.

What you are looking for is called Computational Learning Theory. It directly deals with all of these questions, albeit using somewhat different terminology. A good place to get started is here:
https://en.wikipedia.org/wiki/Computational_learning_theory
https://en.wikipedia.org/wiki/Probably_approximately_correct_learning
If you are willing to pay, this book is a decent resource: https://www.amazon.com/Introduction-Computational-Learning-Theory-Press/dp/0262111934


text:
Why policy improvement theorem can't be applied in case of function approximation?
Policy improvement theorem is proven as follows:
$$v_\pi(s) = \sum_{a \in A} \pi(a|s)q_\pi(a,s) \leq \max_{a \in A} q_\pi(a,s) = q_\pi(\pi'(s), s)$$
What step of the proof does not hold for function approximation?
Or maybe the question could be asked differently. As referenced in Sutton's RL book in Chapter 10:

It is no longer true that if we change the policy to improve the
discounted value of one state then we are guaranteed to have improved
the overall policy in any useful sense.

Why?

As referenced in Sutton's RL book regarding policy improvement theorem (PIT):

The policy improvement theorem applies to the two policies that we considered at the
beginning of this section: an original deterministic policy, $\pi$, and a changed policy, $\pi'$, that is identical to $\pi$ except that $\pi'(s)=a \neq \pi(s)$. For states other than $s$, (4.7) holds because the two sides are equal.

Note (4.7) is just your listed inequality, therefore with function approximation generally speaking the inequality (4.7) no longer holds for all states between every policy improvement step via their approximated function of feature vector and parameters, regardless you're using a simple linear function or a neural network approximator. This is because after policy improvement step to update the parameters, unlike the exact tabular cases, the updated parameters would possibly affect all other states' (action) values as well.
--------further clarification after OP's addition----
Convergence is a much harder topic. In some cases such as with Monte Carlo methods using nonlinear approximation, convergence can still be guaranteed because MC methods update the parameters based on the full return which provides unbiased estimates of the value function even with nonlinearity and above mentioned parameters global impact.
However, in other cases such as TD(0) using nonlinear approximation, convergence guarantees are not as straightforward. TD(0) updates are based on bootstrapping where the value estimate is updated using the estimate of the next state's value. Nonlinear approximators can introduce biases and instability, leading to divergent behavior.

Thank you for your answer. I read the same book, and I accept what you are saying. However, I still miss the key point, I still see only "what book says" and no "here it's proven". From page 254: "It is no longer true that if we change the policy to improve the discounted value of one state then we are guaranteed to have improved the overall policy in any useful sense". So apparently, we are getting in one improvement step $\pi'$ which is equal to $\pi$ except $\pi'(s) \neq \pi(s)$

For example non-linear approximation of MC guarantee convergence, but non-linear approximation of TD(0) doesn't. Why? Your answer seems to me applicable to both of them, even one of them has to converge, and other hasn't.

You've significantly expanded your OP to several fronts including convergence difference for various model-free RL methods and a specific discounted case which is not policy gradient method. I've updated for your last comment. Perhaps if I have time will read your discounted case.


text:
What might be the suitable machine learning algorithm to train a model suitable for forecasting a patient's urine output?
The task involves developing a machine learning model trained on urine output trends, clinical parameters, medications, and fluid input of patients to predict their future urine output.
What machine learning model would be most suitable for predicting a patient's urine output, considering the complex relationships between clinical parameters and historical data, and the sequential nature of urine output trends?

All algorithms have inductive biases which determine what they will be good or bad at.  Which model is good on a set of data is determined by the data, not the model.  For instance, in classification, Support Vector Machine works wonders on linearly separable data.  Whether or not the classes are linearly separable and/or by which features is born out by the data.
It's best to start with a bit of data exploration to see which features are important for prediction and one's favorite "simple" model (i.e. Decision Tree Regression) just to see if decent performance can be simply achieved.
Lastly, just because something has dependency in the time domain does not mean one has to jump to something complex like LSTM.  Feel free to include values from previous time steps as features to the present prediction.  If it's a predictor for future values K-nearest neighbor is as likely a candidate as the next.

This sounds like a tabular regression problem to me.
tl;dr: use h2o.ai's automl, it's plug and play quite easy. It will test various different algorithms on your problem and create an ensemble of the best ones (that will likely be better than any individual model).
More context:
The likely best algorithm would be a decision tree ensemble since they are still state of the art in tabular ML problems. But honestly if your problem turns out to be very simple a linear model might even be sufficient. You just need to make sure you provide the right features (e.g. how much water they drink, time since last pee, body size etc...).


text:
How does Regularization Reduce Overfitting?
As I understand, this is the general summary of the Regularization-Overfitting Problem:

The classical "Bias-Variance Tradeoff" suggests that complicated models (i.e. models with more parameters, e.g. neural networks with many layers/weights) are able to well capture complicated patterns in data (i.e. low bias) but are unable to generalize well to unseen data (i.e. high variance). On the other hand, simpler models are able to generalize better to unseen data (i.e. low variance), but unable to capture complex patterns in data (i.e. high bias).

Regularization tries to navigate this compromise by attempting to improve the ability of complicated models to generalize to unseen data. Regularization does this by making "complex models simpler", by strategically reducing the number of parameters in complex models such that they maintain their ability to capture complexity in the data but also generalize to unseen data.

Regularization does this by bringing some of the model parameters towards 0 (L1 Regularization) or by bringing many of the model parameters somewhat towards 0 (L2 Regularization). This "shrinkage" effectively negates the influence of some of the parameters in complex models - and as a result, regularized models tend to have "sparser" solutions (i.e. contain more model parameters with values closer to 0).

Regarding this, I am still not sure if the mathematics behind why sparser models might result in less overfitting is clearly known.
The way I currently see things, Regularization seems to be more of a general heuristic : Countless evidence shows that models overfit less when you add a "regularization penalty term" to the model's Loss Function - and thus deliberately choose model parameters corresponding to a region of the Loss Function that is situated away from the true minimum point. Mathematically, I can understand how this happens.
But are there any mathematical justifications that suggest a sparser model based on a regularized solution is less likely to overfit data compared to a non-regularized solution -  or is this still based on heuristics and anecdotal evidence? Do we have any insights as to how the Mathematics of Regularization acts to prevent Overfitting?

Do not ask people to comment on something, but ask specific questions that can be answered with facts. For this reason, I removed your last part of the post.

What do you mean by "anecdotal evidence"? Maybe you mean "empirical evidence"? Regularisation really can help, it's not just an anecdote.

@ nbro: thank you for your helpful comments! I will remember this in the future!

I guess you mean "why sparser models might reduce overfitting", not "result in"; please edit your post and, as a general remark, kindly review after posting.

it basically restricts or biases your hypothesis (model) space.

I think different mathematical explanations exist for different situations where regularization is useful. The importance of regularization varies by problem as well. It is absolutely necessary when $p>>n$ as I'll mention below. In general it is a way to impose reasonable priors on the model though from a bayesian perspective.
I'm going to put together a quick answer that I hope is somewhat satisfactory. I don't think it is exactly what you are going after though. At a high level I would recommend skimming Hastie (2001), especially section 16.2.2 titled The "Bet on Sparsity" Principle. You can think of sparsity in a bunch of different ways in addition to the number of zero weights in a linear model.

$L_1$ penalty is better suited to sparse situations, where there are few basis functions with nonzero coefficients (among all possible choices).

I think the key here is that sparsity could exist in some basis, not necessarily your model weight basis.
Another even more targeted mathematics heavy book would include Statistical Learning with Sparsity.
Solution identifiability
For example in the case where your parameter space is much larger than your number of samples ($p >> n$), you have an identifiability problem. Infinite numbers of solutions exist, so picking one with small total weight of parameters is just as justified as any other, but perhaps more plausible in most situations from aesthetics. Without regularization in this setting, you would have instability issues where different equally good solutions could be chosen, perhaps based on random initial conditions.
Domain specific knowledge In many cases, you wouldn't expect all of your parameters to be meaningful. Enforcing sparsity will mathematically limit the solution space to one with more zeros, as you point out, or in general a solution space with a fewer number of underlying basis functions being involved in the data generation. In many domains there are a smallish number of factors that are causing a large number of observed variables to be changed, so regularization is imposing that kind of a constraint onto your model. Since the remainder of the variables are not real, or would otherwise make use of too many basis functions to represent your task, you are helping the model out by providing this useful piece of information. There are many extensions on this. For example if you know that your features are spatially correlated you could add in a fused lasso penalty, etc. The rationale here mathematically is probably something along the lines of including more noise terms in your solution results in a lower likelihood of generalizing.

To my knowledge, this is well understood in the setting with a true (sparse) linear model.
In high dimensional regimes and when the true model is sparse, a regularized solution is less likely to overfit the data because with high probability, we will obtain the true support of our model. This follows from the equivalence of L1 solution to solving L0 solution, when the Restricted nullspace property (RNSP) holds. A sufficient condition for RNSP is the restricted isometry property (RIP). Under a subgaussian analysis, one can see RIP is satisfied with high probability meaning solving lasso obtains our true support. Analysis gets more complex in a noisy setting, but intuition can be built from the noiseless case. Low l2 error of coefficients follows with high probability as well.
See https://www.amazon.com/Statistical-Learning-Sparsity-Generalizations-Probability/dp/1498712169 for more information.


text:
Machine Learning Models for Longitudinal Data
Recently, I had the following question about supervised classification models (e.g. random forest) for longitudinal data.
Suppose I have the following data about students passing a fitness test - the students (each student has an "id") who enroll in a school take a fitness test each year and record their height and weight (at the start of each school year, before the fitness test). They can either pass (1) or fail (0) the fitness test each year. The school is interested in knowing which students are likely to fail the fitness test, so they can focus more attention on these students. Naturally, some students might have taken the fitness test more times than other students.
I simulated some data (using the R programming language) to show how the historical data might look like:
score <- c("1","0")
score <- as.numeric(sample(score, 1000, replace=TRUE, prob=c(0.3, 0.7)))
id_sample <- 1:140
id <- sample(id_sample, replace = TRUE, 1000)
height <- abs(rnorm(1000, 150,5))
weight <- abs(rnorm(1000, 75,5))

data = data.frame(id, height, weight, score)
data <- data[order(data$id),]

I then added two variables to this data - one to show how many times the fitness test was taken, the another to show the (cumulative) average number of times the test was passed:
library(dplyr)
data =  data.frame(data %>% group_by(id) %>% mutate(counter = row_number(id)))
data$csum <- ave(data$score, data$id, FUN=cumsum)
data$average <- data$csum/data$counter

Now, suppose some of the students are about to take this test again and we would like to predict what their score will be - some of these students are existing students, but some of these students are new and have never taken the test before (i.e. they have no historical data):
id_sample <- 1:140
id <- sample(id_sample, replace = FALSE, 23)
height <- abs(rnorm(23, 150,5))
weight <- abs(rnorm(23, 75,5))

new_data = data.frame(id, height, weight)
new_data <- new_data[order(new_data$id),]

id_sample <- 141:200
id <- sample(id_sample, replace = FALSE, 5)
height <- abs(rnorm(5, 150,5))
weight <- abs(rnorm(5, 75,5))

#simulating data for students who never took the test before
n_data = data.frame(id, height, weight)
n_data <- n_data[order(n_data$id),]

test_data = rbind(new_data, n_data)

Now, to this test data, (where applicable) I added "longitudinal variables" that take into account the number of times the students took the test and their most recent average cumulative score:
#counter
max = data.frame(data %>%
             group_by(id) %>%
             filter(counter == max(counter)))

colnames(max)[5] <- "max_counter"

max$max_counter = max$max_counter + 1

test_with_counter =  merge(x = test_data, y = max, by = "id", all.x = TRUE)

test  = test_with_counter[, c(1,2,3,7,9)]

 test$max_counter[is.na(test$max_counter)] <- 1

 test$average[is.na(test$average)] <- 0

#formatting
colnames(test)[2] <- "height"
colnames(test)[3] <- "weight"
colnames(test)[4] <- "counter"
data$csum = NULL
data$score = as.factor(data$score)

At this point, there is nothing stopping me from training a supervised classification model (e.g. random forest) to predict the "score" variable for the test data:
#skip cross validation for brevity of question
library(randomForest)
rf <- randomForest(score~., data=data)
pred = predict(rf, newdata = test)

print(rf)

Call:
 randomForest(formula = score ~ ., data = data)
               Type of random forest: classification
                     Number of trees: 500
No. of variables tried at each split: 2

        OOB estimate of  error rate: 23.4%
Confusion matrix:
    0   1 class.error
0 636  79   0.1104895

My Question: Does the approach that I have proposed for supervised classification of longitudinal data sound reasonable (e.g. better than "nothing") - or are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.) ? Or is it better to use some supervised classification model/software implementation that has been specifically designed for longitudinal data (e.g. https://cran.r-project.org/web/packages/LongituRF/LongituRF.pdf)? Thanks!
Note:

This is a rough sketch of the situation I am dealing with - I am also planning to include variables such as "number of days that elapsed since last fitness test".

The sample data in this stackoveflow question is randomly simulated and obviously wont show any longitudinal trends.

I have heard that models such as Random Forest have the ability to recover/model around complex interactions and correlations within the data that otherwise need to be explicitly specified in standard supervised models (https://ishwaran.org/papers/IKBL.AOAS.pdf).

Can you please put your specific question in the title? "Machine Learning Models for Longitudinal Data" is not a question, but the title of a list.

I added "longitudinal variables" that take into account the number of times the students took the test and their most recent average cumulative score:
My Question:
a. Does the approach that I have proposed for supervised classification of longitudinal data sound reasonable (better than nothing)
b. are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.)

You actually have a pretty good grasp of what is going on here.
With regards to your problem there are 2 interactions I see you should keep in mind:
Predictability and Explain-ability
you have some set of n factors [x1,x2,...,xn] and one binary label you are trying to predict Pass/Fail, with your positive label being those likely to fail.
So you are trying to figure out P(Fail | {x1,x2,...xn})
You want be able to catch as many students as possible that are likely to fail so that you can help them, but with a model that is too complex, you lose the ability to explain why they needed the help. This can prevent you from helping future students by being proactive and addressing root causes.
To address your first question:

Does the (LV) approach ... sound reasonable (better than nothing)

This approach makes a reasonable (common sense) assumption that when it comes to fitness, past performance acts as a good Bayesian prior.
To address your second question:

are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.)

Decision trees are not affected by collinearity so they are a great model to address multicollinearity and variance inflation.
Decision trees also offer the added benefit of explainablity, which when it comes to dealing with students helps to mitigate ethical-social issues that might crop up. (why did you help that kid and not my kid)
Random forests allow you to increase the predictability of the model, but lose some explainability. I would recommend starting with a single decision tree first. However, with random forest you can plot feature importance. R-doc: feature importance
Be aware that with decision trees it is very easy to overfit, one of the main ways this happens is by making a tree that is too deep and allowing too few samples at each split R-doc: decision_tree.
You can start by trying to keep the depth small (3-4), and the min samples not too small (10+) and seeing how far that takes you.
You can increase the depth from there. Remember that you only have 200 samples (which is small for this kind of problem). So if the minimum number of samples to split is 10 that gives you only 20 nodes before they become a leaf.
To reach 20 nodes, you only need a depth of 5.
Hope this helps :)

@ J.Kent : Thank you for your answer! I was interested in knowing if the general idea behind adding "longitudinal indicator variables" is a mathematically valid approach - if this approach is mathematically valid, then the choice of model is secondary (e.g. I now have the liberty of choosing whichever supervised classification model I want based on my interpretability requirements)! BTW - what do you mean by "LV Approach"? Thank you so much!

LV is just short hand for "Longitudinal Variable".
Even though I know what you mean, I wouldn't think of this in terms of being mathematically valid. You probably mean it more in the sense of being statistically valid, it's probably better to think of this as being valid modeling approach. Since machine learning in general is about Bayesian inference P( y | x) I would say you are safe.  More specifically, what you are doing is called Feature Engineering

I think there are some things you can do to get it work better.
Suggestions:

add a column to the input giving number of previous tries at the test.  If there was a score to go with it, include the average and standard deviation of those, if you can.
start with 5-fold CV to get a sense of how well it generalizes, and to get a sense of the spread of your losses/fit-performance
look at the h2o.ai random forest tool because you can get several
good things out of it that aren't available in all the 'old stuff'.
The "Flow" interface through the browser gives you good plots and
fit-analysis.  (They have some very nice speed-ups for random forests in there.)

I have been "that kid" who took physicals and struggled, and took it again.
Having an estimated probability of membership can be more useful than a pass-fail because a 51% +/- 1% chance of passing and a 99%  +/- 1% chance in passing can be very different creatures.  If you have tried several times and failed, you might be more likely to fail again.
The h2o.ai "Flow" interface makes "pretty" tables and graphs with very little effort, and that can be nice for write-ups to give to a boss, or instructor.
I would not start with a single tree.  You can learn all the wrong things.  It is a Faustian bargain to pay the price of being much more  incorrect to buy the ability to explain those incorrect actions clearly.  There are tools for explaining RF, and in particular, any single outcome of a forest is a weighted average of single branches, so you can get the bounds on the axes that drove the decision and say "because it is in this window, the answer is that".
References:

https://www.usu.edu/math/adele/forests/ENAR.htm#slide0227.htm
https://julienbeaulieu.github.io/2019/10/16/model-interpretation-with-random-forests-and-going-beyond-simple-predictions/

@stats_noob - https://www.youtube.com/watch?v=06-AZXmwHjo&t=1990s


text:
Are Bayesian Optimization Methods Better Suited Noisy Optimization Problems?
We know that in many applied contexts (e.g. Machine Learning, Loss Functions for Neural Networks), the functions we are trying to optimize are "noisy" by definition (unlike in the classical sense, when we are only required to optimize "deterministic functions") - this is because these functions are based on measurements that we have (i.e. the data), and it is very possible that these measurements can either contain some "measurement error" and these same measurements could take different values if we were to measure them again (i.e. "stochastic").
I have heard that Bayesian Optimization techniques are generally favored for instances where the gradient of the loss function does not exist (e.g. discrete/combinatorial problems) - but I am particularly interested in learning about the effectiveness of Bayesian Optimization Methods on "Noisy Functions". For instance, (my very limited knowledge of this topic tells me that) Bayesian Optimization uses a Gaussian Process to "reconstruct" the surface of the Loss Function based on the data we have , and our confidence in this surface is iteratively updated as we perform the optimization task.
Thus:  Is it because of this fact (i.e. modelling the loss surface of the objective function using a Gaussian Process) that Bayesian Optimization methods are said to be effective at optimizing "noisy functions"? Does doing so somehow result in solutions with a higher level of statistical robustness? In general - why do we say that Bayesian Optimization is more effective at optimizing noisy functions?
Thanks!


text:
Effects of ReLU Activation on Convexity of Loss Functions
I have heard the following argument being made regarding Neural Networks:

A Neural Network is a composition of several Activation Functions
Sigmoid Activation Functions are Non-Convex Functions
The composition of Non-Convex Functions can produce a Non-Convex Function
Thus, Loss Functions for Neural Networks that contain several Sigmoid Activation Functions can be Non-Convex

Using the R programming language, I plotted the second derivative of the Sigmoid Function and we can see that it fails the Convexity Test (i.e. the second derivative can take both positive and negative values):
e = 2.718

eq = function(x){ (-e^-x)* (1+e^-x)^-2  + (e^-x)*(-2*(1+e^-x)^-3 *(-e^-x))}

plot(eq(-100:100), type='l', main = "Plot of Second Derivative of the Sigmoid Function")

My Question: (If the above argument is in fact true) Can the same argument be extended to lack of Convexity of Loss Functions of Neural Networks containing several "RELU Activation Functions" ?

On it's own, the ReLU function is said to be Convex.
Mathematically, we can show that compositions of Convex Functions can only produce a Convex Function.

However, Neural Networks that contain compositions of (only) ReLU Activation functions make it unclear to me how a Loss Functions that contains (only) "RELU Activation Functions" would a Non-Convex.

Can someone please comment on this? If compositions of Convex Functions can only produce Convex Functions - does this mean that the Loss Function of a Neural Network containing only containing ReLU Activation Functions can never be Non-Convex?
Thanks!

References:

https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html
Note: Using some informal logic, I do not think that the Loss Functions of Neural Networks containing RELU Activation Functions are generally Convex. This is because RELU (style) Activation Functions are generally some of the most common types of activation functions being used - yet the same difficulties concerning mon-convex optimization still remain. Thus, I would like to think that Neural Networks with RELU Activation Functions are still generally non-convex.

single layer with relu activation should be convex. But compositions of convex functions are not necessarily convex, so even in the case of a single layer with relu, the loss function may not be convex.

Also, you should note that relu is a nonlinear function - just like sigmoid.

@ Taw: thank you for your reply! what do you mean by "compositions"? do you mean "a loss function composed of several relu functions"?

@ Taw: can you please explain how "Relu is a non-linear function"? it seems to be a piecewise linear function, no?

@stats555 each piece itself is linear, but the function as a whole is non-linear.

@stats555 piecewise linear functions are not linear, hence they are nonlinear. A composition of two functions $f, y$ is defined as $f(y(x))$ where $x$ is input. E.g. call the model y, call the loss function $f$. Then the loss function used in deep learning would look something like $f(\hat y, y(x))$ where $\hat y$ is the target, $y$ is the model, $x$ is the input. My point is, even if $y$ is a single layer with relu, $f(\hat y, y(x))$ is not necessarily convex.

@ Taw: thank you so much for your reply! I spent the whole day reading more about this - I think I finally understand what you mentioned in the first part of the reply. I am just wondering about the last part : "even if y is a single layer with relu f(y_hat, y(x)) is not necessarily convex" : is there some way we have of knowing why this is true?

For instance, I tried writing the loss function for a small neural network containing RELU activations. I then tried to evaluate the Hessian Matrix for this loss function. I was able to fill the Hessian Matrix with the second derivatives with respective to each weight from the loss function - but I am still not sure how we can show that this loss function containing RELU is not necessarily convex. Do you have any ideas about this? thank you so much!

Please, avoid cross-posting the same question. Decide which site you want to post and initially post there. Then, if you don't get an answer there, delete the question from the site where you asked, and ask it on another similar/related site.

You're missing a couple of quite important concepts:

Universal approximation theorem: with enough parameters a neural network can approximate any function.
Basically every loss function is non convex. (There is this little problem in machine learning call local minima about which we like to complain a lot :) )

But no need to trust me, just run a simple experiment and try yourself to approximate a non convex function, like $sin(x)$ with relu:
from sklearn.neural_network import MLPRegressor
import numpy as np
import matplotlib.pyplot as plt

f = lambda x: [[x_] for x_ in x]
noise_level = 0.1
X_train_ = np.arange(0, 10, 0.2)
real_sin = np.sin(X_train_)
y_train = real_sin + np.random.normal(0, noise_level, len(X_train_))
nodes = 1000
layers = 10
regr = MLPRegressor(hidden_layer_sizes=tuple([nodes] * 4), activation="relu").fit(f(X_train_), y_train)
predicted_sin = regr.predict(f(X_train_))

plt.plot(X_train_, real_sin, label="sin target")
plt.plot(X_train_, predicted_sin, label="sin predicted")
plt.legend()
plt.show()

You'll see it's not a task too hard to learn:

PS: of course this is just of a toy example, and if you decrease layers and amount of hidden units the results will become crap, but it still proves that activation surely affects, but not constrain the non linearity of the final function learned by a neural network.

I would disagree, that every loss function is non-convex. Mean squared error and  Crossentropy (Binary or Multilabel) are convex functions.

Convex in their formulation, but when applied in neural networks they lose this property, for details check this thread https://math.stackexchange.com/questions/2402455/convexity-of-mse-in-neural-networks/2402472

one needs to specify for convexity with respect to which variable the function is. I spoke about convexity with respect to the input arguments of the loss function. $\mathrm{MSE} = \frac{1}{2}(y - \hat{y})^2$ is convex with respect to the prediction $y$. But if you are speaking about convexity with respect to the NN weights, the resulting function generally will be non-convex.

I think you are asking about Fully Input Convex Neural Networks as proposed in [1].
ReLU is in fact a convex function, and the sum of convex functions can only produce convex functions. However, unlike you said, composition of convex functions can produce non-convex functions, unless they are non-decreasing.
With FICNNs you can only learn convex functions. For that all weights W must be non-negative for the activation function g
From [1] the interesting part is:

The function f is convex in y provided that
all $W_i(z)_{1:k}$ are non-negative, and all functions $g_i$ are convex and non-decreasing.
The proof is simple and follows from the fact that non-negative sums of convex functions are also convex and that
the composition of a convex and convex non-decreasing
function is also convex (see e.g. Boyd & Vandenberghe
(2004, 3.2.4)).

[1] Amos, Brandon, Lei Xu, and J. Zico Kolter. "Input convex neural networks." International Conference on Machine Learning. PMLR, 2017.

I'd recommend using LaTeX to format your equations and symbols for example $W_i$ and $g_i$


text:
Notation used in paper on Continuous Time Reinforcement Learning
I am working on implementing the learning shown in this paper:
https://homes.cs.washington.edu/~todorov/courses/amath579/reading/Continuous.pdf
In the paper, the authors devise a continuous time learning extension of the actor critic method in reinforcement learning. My question is simply about the dimensions of some objects in equations shown in the paper.
I will summarize the relevant equations for my problem. Let the state of the system be described by a state vector $\mathbf{x}(t)\in\mathbb{R}^n$ and the actions undertaken by an agent interacting with the system be described by $\mathbf{u}(t)\in\mathbb{R}^m$. Then the natural generalisation of the evolution of the system to continuous time is:
$$\dot{\mathbf{x}}(t)=\mathbf{f}(\mathbf{x}(t),\mathbf{u}(t)) \tag{1} $$
A quick note is that I am simply copying the paper's notation (which is part of the problem) and I am also numbering the equations according to where they appear in the paper. From this we also see that $f\in\mathbb{R}^n$. I mention these equations just to set the scene and dimensions of the various objects. They then do some derivations, and end up saying that they will describe the actor as:
$$\mathbf{u}(t) = s\left( \mathbf{A}(\mathbf{x}(t);\mathbf{w}^A) + \sigma \mathbf{n}(t)\right) \tag{19}$$
Here, $s()$ is an activation function, $\mathbf{w}^A = \{ \omega_i^A \}$ is a parameter vector, $\mathbf{n}(t)\in\mathbb{R}^m$ and $\mathbf{A}(\mathbf{x}(t);\mathbf{w}^A)\in\mathbb{R}^m$. The last two make sense since $\mathbf{u}(t)\in\mathbb{R}^m$ and I would guess then that the function $s()$ is applied elementwise. We have that $\mathbf{n}(t)$ is some noise to encourage exploration. Now $ \mathbf{A}(\mathbf{x}(t);\mathbf{w}^A)$ is defined by further refining the above expression in the appendix to
$$\mathbf{u}(t) = \mathbf{u}^{\text{max}}s\left( \sum_k \omega_k^{A} b_k(\mathbf{x}(t))+ \sigma \mathbf{n}(t)\right) \tag{33}$$
Here is where the problem comes in. We would expect the the summation above outputs an $m$ dimensional vector as it is added to the presumably $m$ dimensionl $\mathbf{n}(t)$, then $s()$ (a sigmoid function) is applied pointwise so that each of the $m$ elements is limited to a max value of 1, and then $\mathbf{u}^{\text{max}}$ pointwise multiplies each of those elements to choose its new maximum possible value. However, the $b_k(\mathbf{x}(t))$ above apparently form a Normalized Gaussian Network and are defined as follows:
$$b_k(\mathbf{x}) = \frac{a_k(\mathbf{x})}{\sum_{l=1}^K a_l(\mathbf{x})}$$
with
$$a_k(\mathbf{x}) = e^{-||\mathbf{s}_k^T(\mathbf{x}-\mathbf{c}_k)||^2} \tag{32}$$.
So each $\mathbf{c}_k$ is the center in state space of the $k$'th basis vector from which the current position in state space is subtracted. This is dotted with $\mathbf{s}_k$ to account for units and the whole thing is put through a norm and exponentiated. So I believe the function $a_k(\mathbf{x})$ in some sense measures the distance of the current state $\mathbf{x}$ to $k$'th basis vector. Then $b_k(\mathbf{x})$ does the same thing but normalized over all distances.
But here is my question: surely $b_k(\mathbf{x})$ is a scalar due to its form as a ratio of $a_k$ functions (and the form of $a_k$)? In that case, it does not output an $m$ dimensional vector.
In my code (which does seem to work) I have $m=1$ so I haven't encountered this issue so far but I need to understand to build more complicated models.
An answer to how the above makes sense may be all I need, but if you think the solution is merely that we replace the scalar with an $m$ dimensional vector whose every entry is the scalar value, then we get another issue later on. When we define a Normalized Gaussian Network $\hat{\mathbf{f}}$ to try to learn $\dot{\mathbf{x}}(t)$ as follows:
$$\dot{\mathbf{x}}(t)\sim \hat{\mathbf{f}}(\mathbf{x}(t),\mathbf{u}(t)) = \sum_k \omega_k^M b_k(\mathbf{x}(t),\mathbf{u}(t)) \tag{35}$$
They say very very little about this, but I would assume they mean that the function $b_k$ takes as input the vectors $\mathbf{x}(t)$ and $\mathbf{u}(t)$ "stacked" in some sense, and gives you the distance to basis vectors in the larger space of states of the system and actions taken by the agent. The issue is then with the way the $\omega_k^M$ are updated, which obeys the following equation:
$$\dot{\omega_k^M}(t)=\eta^M\left(\dot{\mathbf{x}}(t)-\hat{\mathbf{f}}(\mathbf{x}(t),\mathbf{u}(t))\right)b_k\left(\mathbf{x}(t),\mathbf{u}(t)\right)$$
Now you will note that the weights $\omega_k^M$ are vectors, since the difference $\dot{\mathbf{x}}(t)-\hat{\mathbf{f}}(\mathbf{x}(t),\mathbf{u}(t))$ is a difference of elements of $\mathbb{R}^n$ and we are assuming the $b_k$ are scalars. I do not know if this is correct and there is very little explanation given in this part of the paper.
Any help understanding the dimensions of these objects would be appreciated.
Edit 1: For some additional information that may be helpful for the deduction of the dimensions involved, the weights $\omega_k^A$ in equation (33) are updated by the following equation:
$$\dot{\omega}_i^A = \eta^A \delta(t) \mathbf{n}(t) \frac{\partial \mathbf{A}(\mathbf{x}(t);\mathbf{w}^A)}{\partial \omega_i^A}\tag{20}$$
where $\eta^A$ and $\delta(t)$ are definitely scalars.
As a further note, I pointed out above (and they specifically state in the paper) that $\mathbf{w}^A$ is a "parameter vector" so I would be surprised if $\omega_k^A$ in (33) is a vector too. However, they say almost nothing about (35), so there I'm less sure.

Overall your understanding of dimensions here is correct. The function $a_k(\mathbf{x})$ measures the exponential of the negative squared Euclidean distance between the input state $x$ and the center $c_k$, and then weighted by $s_k$ of the $k$-th basis function. Then the normalized scalar $b_k(\mathbf{x})$ is a value between $0$ and $1$, representing the contribution or activation of the $k$-th basis function in a normalized way.
This type of function is commonly used in radial basis function (RBF) networks, where Gaussian-like functions centered at specific points are combined to form a basis for approximating a target function. The $b_k(\mathbf{x})$ functions are often interpreted as activation functions for the input state or stacked with action as joint input representation, determining how much influence each basis function has at a given input representation.
Further clarification based on your comment question, indeed here's a little messy and the authors were not clear enough assuming readers already have much background in HJB equations of classical physics and optimal control theory, you can further review the linked reference of HJB equation. As clearly defined in (32) here they're using linear function approximation thus the summations in both (33) & (35) express the usual inner product of the parameter vector and the basis vector $[b_1,...,b_k]$. But notice value function is scalar thus the parameters $ω_k$ in (32) is interpreted as usual which has no upper right superscript if you take minute notice, however, the parameters in (33) & (35) have superscript $A,M$, respectively. So as you rightly concerned to make dimension match with their LHS, their parameters should be construed as matrix multiplication with their respective basis vector $[b_1(x),...,b_k(x)],[b_1(x,u),...,b_k(x,u)]$ instead of simply inner product producing a scaler.

Thank you @cinch for your reply and clarification! I am still confused however, on how the scalar value in (33) could then be added to the vector valued $\mathbf{n}(t)$? If the answer is that $\omega_k^A$ is a vector, then I have included an edit at the bottom where I would like to then ask what that expression would mean? If that is not the answer then don't worry about that. Thank you!

As clearly defined in (32) here's linear function approximation thus the summations in both (33) & (35) express the usual inner product of the parameter vector and the basis vector $[b_1,...,b_k]$. But notice value function is scalar thus the parameters $\omega_k$ in (32) has no upper right superscript interpreted as usual. But the parameters in (33) & (35) have superscript $A,M$, respectively. So as you rightly concerned to make dimension match with LHS, their parameters should be treated as matrix multiplication with their respective basis vector $[b_1(x),...,b_k(x)],[b_1(x,u),...,b_k(x,u)]$


text:
model.fit fails using keras sequential (slice index ### of dimension 0 out of bounds)
this is the most simple model I can think of for my data yet I can't use the fit function, it gives an error.
the desired procedure is to make a simple autoencoder : from 576 nodes to 64 then back to 576
why this doesn't work ?
model = Sequential(name = 'sth')
model.add(Input(shape=(576,)))
model.add(Dense(576, activation='relu')
model.add(Dense(64, activation='relu') #bottleneck
model.add(Dense(576, activation='sigmoid')

model summary gives :
┌─────────────────────────────────┬────────────────────────┬───────────────┐
│ Layer (type)                    │ Output Shape           │       Param # │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ input (Dense)                   │ (None, 576)            │       332,352 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 64)             │        36,928 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ output (Dense)                  │ (None, 576)            │        37,440 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 406,720 (1.55 MB)
 Trainable params: 406,720 (1.55 MB)
 Non-trainable params: 0 (0.00 B)

then compile and then fit :
model.compile(optimizer='adam', loss='mse', metrics=['accuracy'])
model.fit(x=Xdata, y=Xdata, validation_data=(X_val, X_val),
          batch_size= 32, epochs= 50)

the Xdata and X_val are numpy arrays of shape (900, 576) and (100, 576) respectively
gives this error :
ValueError: slice index 576 of dimension 0 out of bounds. for '{{node strided_slice_576}} = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](ReadVariableOp_576, strided_slice_576/stack, strided_slice_576/stack_1, strided_slice_576/stack_2)' with input shapes: [576,64], [1], [1], [1] and with computed input tensors: input[1] = <576>, input[2] = <577>, input[3] = <1>.

This stack exchange is for the discussion of AI.  This is likely a programming issue and should probably be asked on stack overflow.

Dense layer expects 1-D inputs. Your data seems to be 2-D.

there was an untraceable mistake in one of my Dense layers, and that was I creating the layer and passed the argument kernel_regularizer=HeNormal(), instead of kernel_initializer=HeNormal()
wasted 3 days of tweaking everything, while every time I was reading that line of code I just read it as initializer automatically.