text:
Is hyperoperator a generalization of neural network?
We know that artificial neural network (ANN) components are:

$x$: input
$y$: output
$w$: weight
$b$: bias

For example, a single layer of ANN with a linear activation function is basically a linear regression: $$ y=xw+b $$
Using hyperoperator, we can construct the equivalent formula.
$$ y=(x[2]w)[1]b$$
Therefore, a single-unit neuron based on a hyperoperator can be defined as: $$ y=x[k]w $$
Where $k$ as an operator selector, $x,w,y,k$ having the range of $[0,2^n-1]$ where $n$ is the number of bits, and they are all positive integer. $k$ and $w$ are parameters that can be tuned by the optimizer.
It seems the model has linear activation. However, my intuition said it's possible a single neuron has non-linear activation since $k$ can be a parameter that is tuned by the optimizer, especially for $k\geq3$, which is an hyper exponential operator. Thus, it has similarity with KAN, in which the activation is the tunable part
Since they are all positive integers, any continuous floating point can be transformed into integer space. E.g., a normalized $P$ like:
$$
P:[0,1], P \in \mathbb{R}
$$
is analogous to:
$$
P:[0,2^n-1], P \in \mathbb{N_0}
$$
Therefore, it's just scaling and normalization issue when the inputs, outputs, and the weights in real domain.
Actually, it's up to the architect how to define a single layer, like these examples, note we can replace $k$ with indexed $w$ as they are tunnable parameters:

$y=w_0[w_1]w_2$
$y=w_0[w_1]x$
$y=w_0[x]w_1$
$y=w_0[x]x$
$y=x[w_0]w_1$
$y=x[w_0]x$
$y=x[x]w_0$
$y=x[x]x$ (deterministic output)

Hence, to construct a multi-layer neural network, just perform recursivity like these examples:

$y=x[k](...)$
$y=(...)[k]x$
$y=x[...](...)$
$y=x[w_0[k]w_1]x$
$y=x[x[k]x]x$
$y=x[x](...)$
$y=(...)[...](...)$ and so on...

So, is hyperoperator a generalization of neural network?

Perhaps to qualify as an appropriate generalization of the popular and powerful MLPs, such new types of neural networks need to be able to do stable first order gradient descent optimization which is scalable with big training dataset.
However, since your proposed operator selector $k$ is a discrete parameter, optimizing over $k$ introduces challenges in training, for example, gradient-based methods struggle with discrete optimization. In addition, the interpretability of such networks becomes harder as $k$ grows, and large $k$ will obviously introduce numerical instability which impacts convergence of your proposed neural network's training.
Finally hyperoperators sequence allowed Goodstein to rigorously express and study numerical growth beyond exponentiation (similar to the Ackermann function), and Goodstein's theorem ultimately highlighted the incompleteness limitation of Peano Arithmetic (PA) which cannot prove that every Goodstein sequence eventually terminates at 0, but the same can be proved in the stronger second order arithmetic or ZFC. Therefore the hyperoperators dealing with large number growth are better suited for theoretical edge case studies, but not for the everyday applications target by common ML models. Comparatively speaking, your referenced KANs may have better application since KANs have much less neurons to achieve similar performance of MLPs due to the famous Kolmogorov-Arnold representation theorem linking multivariate/univariate real-valued continuous functions.

The optimizer is not only about gradient descent. It can also be possible to solve it by quantum computing for the optimization problem. Put $k$ and $w$ in superposition so that it explores possible values, and let the system evolve, finding the optimized parameter based on input-output pairs.

I don't see quantum computing is in OP's scope at all, which is yet to become applicable or relevant for everyday ML apps in the near future. Also bear in mind quantum computing as a new and potentially extremely efficient computational technique in some cases don't change computability or incompleteness edge case.

Same for the gradient descent, which implies the question has an agnostic optimizer method.


text:
Is it better to train neural network with ideal data before resuming with non-ideal data?
Speaking generally, AFAIK Neural network relies on initial state at their weights. Mostly it started with random weights.
Does that mean it will be better to train neural network with ideal data first? Then after the model's loss convergence, it will resume training with non-ideal data such as noisy data or something that related with real-world scenerio.

If you have the full dataset (comprising both "ideal"/cleaned and "noisy"/real examples), then no, it is not necessarily better to pretrain it with clean data just to initialise the weights and then update it with real examples. I don't see why it would be better to make the model overfit on clean data first and then correct it by using more noisy data, instead of just training it with all the data from the beginning.
On the other hand, if you  have only clean data to start with (for example let us say you have some synthetic or human annotated data) then it could be worthwhile to pretrain the model with that to initialise the weights and when you get the real data update the model (so it could benefit the head start).

How about I have different sets? First one is ideal how the data is with timesteps 512, while the second one is more complex that the timestep is 3600. Specifically the data is an ECG signal where I'm using LSTM on it. In the medical domain, the 512 is a beat which represent foundation of entire signal. While 3600 timesteps is the real world scenerio, though the timesteps is actually has dynamic size,

I mean, we can't train the data with different shape or different dimension?

I don't think it will help you to train first on the "easy" 512 timesteps examples and then train on the "real" longer dynamic sized examples (the first para of my answer applies here). And dynamic length of the sequence in sequence models like LSTMs isn't considered a chance in shape or dimension. Sequence models are meant to handle this. Although that is probably a separate topic since this question is about the noise in the examples.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why time based neuron still needed when the time-series data can be converted to time-freq domain (image) and use CNN for that?
LSTM, BI-LSTM, GRU, RNN are time-step based neuron. Why it still needed specifically for time-series data?
I mean, we can just transform the time-series data into spectrogram and use CNN for that.
For example, given time series data (vector) and its sample rate (scalar). We can preprocess data with Short-time Fourier Transform (STFT). Hence, our tensor will be transformed from 1D (vector) to 2D (matrix).
Basically, transforming 1D signal to 2D signal (image). Where 1D signal has 2 axis (x-axis is time and y-axis is amplitude). While 2D signal has 3 axis (x is time, y is frequency, and the amount of pixel intensity is amplitude).
So, its dimension for training will be transformed from (batch_size, amplitude) to (batch_size, width, height, 1).
With this, we don't need time dependencies anymore with just using CNN, Therefore, true parallel processing can be achieved with AI accelerator. As far as I know AI accelerators have some problem for parallel processing for time based neuron such as LSTM.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Which model would recognize the rotated version of its input without explicit training during inference?
Training an MNIST classifier with a regular ANN will make the model recognize its unrotated version.
But is there such a model where I train the unrotated version as usual, but it also recognizes its rotated version, e.g., the 90-degree version, a.k.a. the transposed version that I never fed the rotated version during training before?

this https://arxiv.org/abs/2211.11812

If you only care about whether an image contains a a certain digit after arbitrary rotation, not where the digit is located, rotation invariance suffices and the latest Rotation-Invariant Coordinate-CNNs as recommended in the comment could be a simple straightforward approach without rotated data augmentations during training in theory, simply based on the input space's polar coordinate system, where after filter convolutions in polar space aggregating over the angular axis makes the representation invariant to the specific angle of rotation.
However, in deep learning equivariance is a more important and useful property than invariance as Cohen & Welling emphasized in their 2016 paper on the G-CNNs to address similar rotation issue albeit in a group-theoretic way.

the network or layer $Φ$ that maps one representation to another should be structure preserving. For G-spaces this means that $Φ$ has to be equivariant: $Φ(T_g x)=T′_gΦ(x)$ (1). That is, transforming an input $x$ by a transformation $g$ (forming $T_g x$)... Equivariance can be realized in many ways, and in particular the operators $T$ and $T′$ need not be the same... From equation (1) we see that the familiar concept of invariance is a special kind of equivariance where $T′_g$ is the identity transformation for all $g$. In deep learning, general equivariance is more useful than invariance because it is impossible to determine if features are in the right spatial configuration if they are invariant.

Therefore in many real world CNN deep learning facial-recognition and object-detection tasks rotation-equivariance is required instead of mere invariance for classification tasks. For example, for self-driving cars knowing the relative location of the pedestrian is more important than knowing a pedestrian exists. If you rotate and shift the captured image, the feature maps in a CNN should also rotate and shift correspondingly preserving the structure of features. In such cases Cohen & Welling's 2016 steerable CNNs could be kept in mind for many types of equivariance preservation including rotation, translation and reflection. Here's another blog dedicated to such topic.


text:
Why softmax/sigmoid use base e instead of 2?
Performing -ln(ε) in NumPy returns relatively small values like this:
print(-np.log(np.finfo(np.float32).eps))
print(-np.log(np.finfo(np.float64).eps))

Output:
15.942385
36.04365338911715

Compared to -log2(ε) which has a greater range compared to base e:
print(-np.log2(np.finfo(np.float16).eps))
print(-np.log2(np.finfo(np.float32).eps))
print(-np.log2(np.finfo(np.float64).eps))

Output:
10.0
23.0
52.0

So, why don't softmax/sigmoid functions use 2 instead of e as the base for the exponential function? Using base 2 for the logarithm when converting back to logits would have more precision compared to base e in IEEE 754 representation.

IEEE 754 provides sufficient precision for most ML tasks even with base $e$, and the choice in softmax and sigmoid functions is rooted in mathematical and practical considerations.
Notice the function $e^x$ has a unique property that its derivative is itself, therefore this simplifies gradient calculations during backpropagation of the common CE loss. Changing the base $2$ doesn't gain much precision but complicates existing math formulations in machine learning field and optimization computations potentially across many places. Therefore both softmax and sigmoid functions often used in probability and information theory typically use natural logarithm and exponential function based on $e$ in practice, such as the CE loss implemented in PyTorch. Often in information-theoretic discussions the base $2$ for the unit of bits is preferred such as Shanon's original paper about entropy.

The base of the logarithm change the range, but it not necessarily affect the precision.
There exists the following formula:
$\log_b(x) = \log(x) \cdot \frac{1}{\log(b)}$
In which b is the base.
Then, the selection of the base affect the range. Larger bases generate smaller ranges.
Also, computing gradients with log with base e is more easy that when using other bases. Gradients are needed for backpropagation.

$\log_2$ results in larger numbers than $\ln$, but that doesn't make them more precise. Floating point arithmetic has the same relative precision for numbers of any size.


text:
global minimum loss always best metric?
Suppose the hardware constraint is not a problem anymore, so that the quantum computer is everywhere.
If we define a neural network model that has many params, traditionally (using gradient descent) its params (weights) is adjusted based on direction (gradient) of next layer reference (from output layer).
In quantum annealing (tunneling), it doesn't relly on weights initialization as it replaced by superposition state which all possible states of weights has explored simultenously.
So, it's possible that in future, what deep learning engineer tune is only about the layers used for the hyperparameters and the dataset preprocessing, but still need to monitor the metric even though the loss is always in global minimum.
Learning rate (optimizer hyperparams) would be not used anymore as it replaced by quantum superposition.
Speaking generally, regardless of metric being used (classification like accuracy and regression like rmse), is global minimum loss always correlation and causation to the metric?
Like, global minimum = best metric?
Or, is it possible that the global minimum which not achieving best metric is due to misconfigured in defining the architecture or miss annotated the dataset?

Achieving the global minimum of a loss function doesn't guarantee it's the best metric if there are problems with the architecture or training dataset.
For example, if the training dataset contains mislabeled examples or very noisy data, the model might overfit to these flawed data at the global minimum of the right loss function even with quantum computing, leading to poor metrics like low accuracy or high RMSE due to large variance.
Or if your model architecture is misconfigured to be underparameterized, such a simple model might minimize the loss but lack the capacity to capture complex patterns in the data, leading to suboptimal metrics due to large bias.


text:
What is the different between feeding transformer with continous data and with bins data (token) for timeseries?
For timeseries that consists of amplitude, and time domain.
For amplitude, the data is encoded into bins for every ±0.1 increment.
E.g. Amplitude value range 0.1 until 0.2 is Token with index 1, 0.0 until 0.1 is token with index 0.
So, it's similar like NLP
But, what is the different if I directly feed with direct continous data into transformer?

In NLP, words are tokenized to unique integer tokens. These indices are subject to an embedding table, which basically "converts" a token into a (learnable) high-dimensional, continuous, dense vector.
These dense vectors (or embeddings) are the representation learned by the transformer model to represent the different words. Now, if you want to use continuous input features you need to drop the embedding table and feed them directly to the transformer blocks.
Moreover, the continuous input can be problematic depending on you task (e.g., classification, regression, or auto-regressive generation). In particular, in NPL the task is often to predict the next word (i.e., auto-regressive generation) which entails a classification head and learning objective over the discrete tokens (classes). Therefore, you may need to figure out a suitable output head for your model in order to accept continuous features.


text:
What's make transformer encoder difference from its decoder part?
Encoders are commonly used for understanding sequences (like classification, NER, etc.), while decoders are used for generative tasks in an autoregressive manner (e.g., GPT-style text generation).
Decoders are autoregressive because they generate one token at a time, conditioning on previously generated tokens.
In contrast, encoders process the entire input sequence in a single pass and return contextualized representations of the tokens.
If I train the encoder part of a Transformer using a sliding window approach, where the model uses the output of the previous inference as input for the next step (similar to a feedback loop), wouldn't it behave in a way that's similar to a decoder's autoregressive behavior?
In this setup, I'm essentially giving the model the last predicted output as the next input, so wouldn't this iterative process have similar properties to autoregressive generation in a decoder?
For example Xt, yt here to train "transformer encoder only model" for every timestep:
x1: [I, <NEXT>, <PAD>, <PAD>, <PAD>]
y1: [I, LOVE, <NEXT>, <PAD>, <PAD>]

x2: [I, LOVE, <NEXT>, <PAD>, <PAD>]
y2: [I, LOVE, KANADE, <NEXT>, <PAD>]

x3: [I, LOVE, KANADE, <NEXT>, <PAD>]
y3: [I, LOVE, KANADE, <EOT>, <PAD>]

x4: [I, LOVE, KANADE, <EOT>, <PAD>]
y4: [I, LOVE, KANADE, <EOT>, <PAD>]

x5 and so on: same as x4
y5 and so on: same as x4

Both encoders and decoders use self-attention mechanisms. Given that self-attention is responsible for capturing dependencies between tokens, what are the key bounds or differences between the two?
Is the primary difference just that the decoder generates tokens one by one, while the encoder processes the whole sequence at once?
I find this situation reminiscent of the traditional autoencoder structure, where the encoder maps the input to a latent space, and the decoder maps this latent space back into a human-readable representation. But, transformer encoder-decoder model is unintuitive for me.

You’re right that encoder-decoder transformer aligns with the traditional autoencoder (AE) structure except AE’s encoder output is usually a compressed latent representation while transformer’s encoder output is an enriched contextualized representation.
While your sliding window approach makes an encoder behave similarly to a decoder, it lacks causal constraints in the sense that your encoder processes input tokens in parallel, not sequentially. This can introduce dependencies that violate autoregressive constraints, for instance, in your above window 2 the encode can attend to <NEXT> token to predict the next token. Also transformer decoders are optimized for token-by-token autoregressive generation, while your sliding windows require reprocessing overlapping inputs which can be computationally expensive.

So, the transformer decoder is strictly be used for autoregressive generation only? When my problem didn't have to include generation, I shouldn't use decoder?

Not necessarily, though most decoder transformers such as GPT/Mistral/LLaMA are designed specifically for token-by-token causal generation, they sometimes can also be used in non-generative contexts if causal constraints are appropriate for your task, such as Masked Language Modeling (MLM) or some ML interpretability problems involving understanding how earlier steps influence later ones. For classification problems or NER, certainly you don't use causal decoder-only transformer but encoder to have better bidirectional contextualization and more efficient parallelism. Hope this clarifies.

For MLM, isn't encoder only enough?

If you apply causal mask correctly in encoder then it's enough, but you can also leverage decoder's inherent causal autoregressive feature for MLM, at least in principle without considering parallelism efficiency.

So, you said decoder is not efficient in computation compared to encoder? Is it during inference or training?

While the decoder processes sequences in parallel during training by using teacher forcing where the target sequence is fed as input, this still involves extra complexity due to masking compared to the fully parallel encoder. And during inference, decoder is significantly less efficient compared to the encoder since the token-by-token generation cannot be parallelized which is a bottleneck for tasks like text generation or machine translation. Hope now clarifies.


text:
What is it called when model asking for validation?
As we know, classification problems are just a bunch of probabilities, commonly it comes from logits or softmax output. Performing $argmax$ to get the most favorable class by model, discarding us from some information. Such as the distribution of the probability distribution.
For deep lerning model that doesn't have reasoning capabilities, So I formulate new term based on model favorable and model asking for validation so I have four quadrants. It's based on purely softmax output only.
For example, here 8 softmax output with 3 classes:
Out0 [0.99  0.005 0.005]: Confidence: 0.98, Uncertainty: 0.00
Out1 [0.8 0.1 0.1]: Confidence: 0.70, Uncertainty: -0.00
Out2 [0.5  0.25 0.25]: Confidence: 0.25, Uncertainty: 0.00
Out3 [0.33 0.33 0.33]: Confidence: -0.00, Uncertainty: 0.01
Out4 [0.99  0.009 0.001]: Confidence: 0.98, Uncertainty: 0.80
Out5 [0.8  0.19 0.01]: Confidence: 0.70, Uncertainty: 0.90
Out6 [0.5  0.49 0.01]: Confidence: 0.25, Uncertainty: 0.96
Out7 [0.1  0.45 0.45]: Confidence: 0.18, Uncertainty: 0.64

Rescaling that range from $[0,1]$ to $[-1, 1]$ for visualization, draws:

Then I tried with real data of MNIST handwritten digit of 10 classes. Focusing on label number 1, subjevtively model should be easy for him to classify it, but what if I ask him for something unusual that never seen before, it's 90° degree rotate version of number one.

As I expected, model should be wrong to classify something unusual which is rotated version of number one, and therefore leading to low confidence and high needed assistance.
If I visualize with the four quadrants, it's clear that the blue (which is rotated sample) is seeking for the validation or assistance compared to red (the original).
Therefore this is useful for critical inference such as medical diagnoses that doesn't need ground truth for explainable AI.

The question is, what is the better description to represents x-axis and y-axis?
As you see I have inconsistent to use the terms for y-axis:

Uncertainty (I think it doesn't fit with my definition)
Affirmation (Maybe there is another better)
Assertion
Assistance
Vagueness

So, what is the best name for this? (no, this is not opinion question, it's objective question that intersect with philosophy)

"Such as the distribution of the probability distribution", ah? The distribution of the probability distribution? What do you mean?

Can you explain how each row like Out0 [0.99  0.005 0.005]: Confidence: 0.98, Uncertainty: 0.00 is computed? How is the confidence and uncertainty computed here?

"this is useful for critical inference such as medical diagnoses that doesn't need ground truth for explainable AI." - You don't need ground-truth for medical diagnosis? Really?

I said, model doesn't need ground-truth to know whether he is asking for validation of his inference itself. So, when confidence high, but the uncertainty is also high, it's indicator for model which mean he is asking validation by the domain specific expert in this case the doctor specialist.

The distribution of the probability distribution is local distribution and relative from its the maximum value. With excepting its maximum value, we have distribution of another class (which the model says it's not supposed to be the answer).

How I compute Confidence is simply taking the maximum value of a class and then normalized it relatively with its mean.
While Uncertainty simply comparing the most minimum with relative uniform value (excepting value that has the most maximum). Therefore, uncertainty is representing entropy and how assert is the model sure he doesn't know its inference itself based on another classes consideration.

The quadrant of Don't Trust My Inference is related with Socrates's philosophy. I know that I know nothing, therefore Don't Trust My Inference!.

The terms you choose for the x-axis and y-axis should ideally align with the concepts of most people's intuitive understanding who will view these results. "Confidence Score" of your probability classifier as the predicted maximum class probability itself is a well-understood term in ML and statistics, and accurately describes what your x-axis captures, so no change is needed here. "Assistance Needed" could be the best term comparatively from your candidate list since it essentially measures the spread of all the predicted class probabilities like entropy. Therefore in ML a good term for your y-axis could be "Predictive Entropy" or "Confidence Calibration".

Calibration can mean procedures in statistical classification to determine class membership probabilities which assess the uncertainty of a given new observation belonging to each of the already established classes.

But they are two words. Not a single word.

I don't see in OP it specifies to be a single word only. In the meanwhile in applied areas like AI single word terminology might be hard to get a patent since most fundamental fields may have already used them up. So 2-word term is common in AI/ML to make them pick out more definite and distinct descriptions. If you have to coin a single word for your y-axis, I'd suggest just 'entropy' to account for your softmax spread info. Hope this completely clarifies your lingering concern for this question.

Okay I will use entropy.


text:
Approximate the chance of a number is prime using neural network
How to create a model that can give an output with a range of 0 to 1 with a sigmoid activation function where the value closer to 0 means the lesser chance that the input number is not prime and the closer to 1 means the greter chance that the input number is prime.
This means that the output layer will contain one neuron with a sigmoid activation function.
But I don't know what the input looks like.
So, from this problem, I would like to ask, what does the dataset look like?
Is it a one-dimensional list of prime numbers? like: [2,3,5,...,9997]
Or is it actually an unsupervised learning problem?
What do the input, hidden, and output layers look like?
So in the end, I was expecting this kind of input-output:
#1
Input: 77772
Output: 0.01

#2
Input: 27777
Output: 0.69

@RAG The dataset is the question too, whether it will be unsupervised learning or supervised learning.

One idea that comes to mind is having the features (inputs) be the digits of the number, rather than the number itself. You might also try converting to binary before hand and then using those digits as features, or use some kind of one hot encoding for the decimal digits. The rationale being that finding primes is a discrete task dealing with integers, you don't want your model interpreting the input as a continuous value.
You might get better results if you include the modulo of the input with some primes (or just random integers) as features as well.
As for the hidden layers pick your favorite architecture. Standard linear layers followed by ReLU activations are acceptable. As you said, the last layer would be a sigmoid activation (preceded by a linear layer, not a ReLU). You can experiment with things like batch norm if you want although I wouldn't imagine they would help much.
With this setup your training data would have features as the binary digits of a positive integer, and the target would be whether or not that integer is prime (0 or 1). Your loss would be binary cross entropy. Train with stochastic gradient descent or friend until convergence. Note that your training data is very imbalanced as most numbers are not prime.


text:
Is sparse categorical crossentropy support arbitrary custom label encoding?
As we know that in AI tools like tensorflow has loss named sparse_categorical_crossentropy which bassically categorical_crossentropy. The different is, categorical_crossentropy for one-hot encoding while sparse_categorical_crossentropy is for label encoding or integer encoding. In a nutshell, sparse_categorical_crossentropy will convert the label encoding to one-hot encoding at the end CMIIW.
Encoding label encoding commonly use incremental integer such as [0, 1, 0, 2, 0] that has unique value (0, 1, 2) therefore it is 3 classes.
So I want to confirm if sparse_categorical_crossentropy support custom label encoding such as [0, 27, 27, 0, 0, 24, 24, 0, 1, 1,] that has unique value (0, 1, 24, 27) or 4 classes; in any tensor processing tools.


text:
Is Computer Vision always related to Machine Learning?
So I have AI project about motion detection with image subtraction.
Regardless what are the object used, if there are change between two frames according threshold value, then it will categorized as motion.
So in my project I only use OpenCV library in python.
My program take two input. Where first frame or background frame will assumed/labeled as no motion frame for a refference. Second frame is any frame that captured currently.
So, with just using image processing like
resizing -> grayscaling -> blurring -> substracting (absdiff) -> thresholding

Basically my program/project is just comparing between two images if there are changes in its pixel.
Beside my project is related to computer vision obviously, is my project related to machine learning too? Especifically supervised learning because I labelled what is no motion image looks like to the machine.
But in other hand, I don't feel any statistically method where machine learning usually use it. My mathematical operation was using substracting method only.

No - not all computer vision is machine learning.
With machine learning, the computer designs its own algorithm (often by gradient descent) based on a "blank slate" version of the algorithm.
Since you have just told the computer an algorithm, it's not machine learning.

Also, is my project related to CNN since CNN was subset of Deep Learning and it was subset of ML too?

@MuhammadIkhwanPerwira A CNN is a type of neural network. You're not using any neural network.

Actually grayscaling and blurring are convolutional operations, and thresholding can be seen as an "activation function" (think of a sigmoid with a high gain). And resizing can be implemented by an average pooling layer. But since you have hard-coded these parameters (the blur radius and threshold), there is no ML involved.
Then again it could be a fun exercise to apply a gradient descend to those layers. To run the training, you'd need to supplement the network with training data. In this case it would be a "binary" image where you have defined for each pixel whether it belongs to the background or foreground. Since there are so few parameters to tune, I expect that you wouldn't need that many training examples.

if there are change between two frames according threshold value, then it will categorized as motion.

Ah now that I read you question more carefully, your training data could be just yes/no label for the whole picture. You aren't looking for object segmentation.

You said I'm using convolutional operations, but CNN is type of neural network in Deep Learning, and Deep Learning is subset of Machine Learning. So does it mean I'm currently using CNN? If true, but you also said there is no ML involved.

Well this is a bit of a gray area, but at least I'm confident that there is no supervised learning involved since you don't run any training against a labeled dataset. And there isn't unsupervised learning either, no any optimization steps or gradients so this isn't ML. You just use the same mathematical operations as CNN networks would use to solve the same problem.

@MuhammadIkhwanPerwira CNN is Convolutional Neural Network. You are using convolution, but you are not using a neural network.

@user253751 it's clear now. One thing I still confused, is convolution operation is part of image processing or computer vision? I actually can't differentiate both term of them. I feel they are same.

@MuhammadIkhwanPerwira why not both? There is no rule that says it can only be one or the other. I would say that image processing includes all the stuff you find in photoshop (like blur/unblur/zoom/swirl), while computer vision is trying to make computers see things by themselves (where is the face? how many cars are there?)

@user253751 okay so we can conclude that Computer Vision is always related to Image Processing.

@MuhammadIkhwanPerwira Yes. It's impossible make computers see without processing images. You can also draw things in photoshop, and that is image processing, but not computer vision.


text:
Reinforcement Learning vs Supervised Learning
I have never tried reinforcement learning in my life.
I'm planning to apply it in robotics.
I have some experiences using supervised learning mainly deep learning. So, that's mean I will use neural network in the projects.
We know that supervised learning works on forward propagation to infer and backward propagation to learn.
I have read some articles about reinforcement learning concept . As far as I know, I feel reinforcement learning is doing same as supervised learning, except the robot (the agent) training process in real-time where we (the environment) give reward (for true positive/negative) and punishment (for false positive/negative) from its action (infer).
Unlike supervised deep learning where the splitted dataset for training, validating, and testing have been defined. I mean the X_train, y_train, X_test, y_test have been defined. The reinforcement learning works similiar like that, but the X data wouldn't be defined until the agent gives action and y data wouldn't be defined until the environment gives reward or punishment.
So basically, the agents doing forward propagation, the environment (such as we are as human) giving backward propagation to them whether it's a reward or a punishment based on its action (infer).
We can conclude that, reinforcement learning is basically supervised learning with binary classification problem (reward/punishment) where the X data and y data will be defined in real-time.
The question is, please confirm that my statements were true. So were my statements true? Correct me if I'm wrong.

No, this is completely wrong, just one difference, in RL there is an environment, in Supervised Learning, there is no environment.

@Dr.Snoopy In some ofthe broad ideas, it is not completely wrong. Most RL can be split conceptually into the rules for calculating values to associate with states and actions, and rules for learning from those values, and the latter is essentially supervised learning (with some constraints - mainly that the algorithm must be capable of online learning i.e. forgetting older incorrect data)

No, I say completely wrong because the RL setup is wrong , in RL you do not do supervised learning of actions, and there are different RL algorithms from supervised learning (Q Learning, Policy Gradients, Value Iteration, etc). Some of these algorithms do not do supervised learning at all.

This seems to be kind of a duplicate of Can supervised learning be recast as reinforcement learning problem?

Your statements are mostly incorrect, there are very large differences between reinforcement learning (RL) and supervised learning (SL).
In SL, you have labels that should be the correct answer that a model predicts, in RL you have rewards (which are continuous, not binary), and these rewards do not tell you the right answer.
In RL, you predict actions in an environments based on learning actions on an accumulated reward, in SL there are no actions and there is no environment at all.
While there are many RL algorithms that also use SL, there are RL algorithms that do not use SL at all, like value and policy iteration. This last statement should show you the actual differences between SL and RL.


text:
Deep reinforcement learning:DQN can't converge

i am work on a project.i use the dqn to maximize return. this picture are some env states.
i found that dqn did learn a bit, but after a while it stopped improving and even started to decline.
this my current Hyperparameters:lr 0.01,lr_decay 9e-7,mineps 0.1, maxeps 1,epsdecay 1/25000,memory buffer 200000,discount 0.999,Weight_update_frequency 20step,batch_size 16.
i try a lot.i use the lr:0.001,0.01,0.0001,used eps:0.05,0,02. the lr_decay:1e-7,1.5e-7,1.2e-7,eps_decay:1/10000,1/20000,1/25000,1/40000,1/50000.Weight_update_frequency:4epoch,20step.batch_size:16,32,64.the model size:64,64,128,32,32,64.
Its trend is almost the same, and it has learned something, but then the shaking is very serious, or even declines.

this is current training figure, anyway,it is better than most of my other training figure.
ADD:
source of my dqn is from a book AI-Crash-Course,this is the github site https://github.com/PacktPublishing/AI-Crash-Course/tree/master/Chapter%2011.
the code is in chapter 11.i made Some changes  on the basis of these codes.
about environment:my project is divided in two parts,the schedule algorithm and the DQN part.env will produce many tasks.the schedule algorithm will schedule tasks to a device for processing.The dqn is used to give the best device id.
When a ready task is solved,the environment will change,then the schedule algorithm send the state、reward to DQN.DQN send the device id to schedule algorithm.
action is device id.Range from 0 to 5.

What implementation do you use for DQN ? Is it something you wrote yourself, Tensorflow, something else ? People could probably help if you gave some link toward the detail of those parameters, some documentation.

Also, could you give some information about the markov decision process you're working on ? What's the environment, what kind of actions the agent can take ? All of these might help to find the right parameters.

I don't personnally have enough knowledge of DQN yet to go through this code and check for errors, but my suggestion would be to try what you did on another implementation of DQN such as the Tensorflow one. That way you'll know if's it's a parameter issue or an implementation issue.


text:
What data can I obtain from CNN model (H5 file)?
I created a CNN model and it is saved in h5 format. I used the Netron app, where I obtained the model architecture, however batchnormalization was not present.
CNN model:
model = Sequential()
model.add(Conv2D(16, (3, 3), input_shape=(height, width, 3), kernel_regularizer=l2(0.001)))
model.add(Activation('relu'))
# BatchNormalization()
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2)))

BatchNormalization()

model.add(Conv2D(32, (3, 3), kernel_regularizer=l2(0.001)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2)))

BatchNormalization()
model.add(Dropout(.5))

model.add(Conv2D(32, (3, 3), kernel_regularizer=l2(0.001)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2)))

BatchNormalization()
model.add(Dropout(.3))

model.add(Conv2D(64, (3, 3), kernel_regularizer=l2(0.001)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2)))

BatchNormalization()
model.add(Dropout(.3))

model.add(Flatten())

model.add(Dense(256))
model.add(Activation('relu'))

model.add(Dropout(.5))

model.add(Dense(4,activation='softmax'))

The output of Netron app:

I have saved this model in h5 format. Is there a method to determine whether this h5 model has a batchnormalization layer or not. Again, is it possible to know the learning rate?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

We can't see the BatchNorm layer in Netron, so NO. It doesn't have BatchNorm but for good reason. In testing, you don't need BatchNorm (that is the possible reason why it has been done).
Second, for learning rate, you can't get that because .h5 stores the network and its weights. You don't need learning rate to test or run a model.

Your main answer seems to be correct, but it seems misleading to state that batchnorm is "not needed" during testing. It is certainly true one often does not want to update the batchnorm parameters during testing, but it does not follow that one should remove it - the normalization can still be crucial for the overall network to work if it was trained with batchnorm.

Could the batchnorm weight and bias be "adopted" to the subsequent layer's parameters? I think it is mathematically possible, but I have no idea whether the framework does that. I guess you could save and reload the model, and check. Actually dropout isn't needed either, don't you just multiply all outputs by a factor when running interference?


text:
In the policy gradient method, state dependent baseline does not affect gradient of the objective function. Then how this is better approach?
In policy gradient theory subtracting state dependent baseline from Q(s,a) does not affect gradient of the objective function. I understand the proof shown below. One things is, if it's not affecting the gradient, then how this approach is better?

Short answer: variance reduction. Some details may be found, e.g., here.

@mikkola thank you for your time end effort sir. I will check this out.


text:
Training a neural network simultaneously with two different loss functions rather than considering the weighted sum
This is a follow up on the already asked question: Is the neural network 100% accurate on training data if epoch loss is minimized to 0?
I want to train a neural network that works as an approximator for a function that satisfies a few constraints.
For example, I want the network $\mathcal{N}(x)$ to satisfy $\mathcal{N}(x) \leq \eta, \ \forall x \in D_1$, and $\mathcal{N}(x) \geq \eta, \ \forall x \in D_2$, where $D_1 \cup D_2$ is the training data set.
So I define a custom loss function as:
$$L_1 := \cup_{x \in D_1} \mathrm{ReLU}( \mathcal{N}(x) - \eta) + \cup_{x \in D_2} \mathrm{ReLU}(-\mathcal{N}(x) + \eta) $$
such that I can generate an output that minimizes the loss and let the training converge when $L_1=0$ so that all the conditions are satisfied on the training set (Yes I need this, even at the cost of overfitting?).
Now, I also want the trained neural network to have some Lipschitz bounds, so I consider the log-det barrier method used in the following paper:
https://arxiv.org/abs/2201.00632
Correspondingly, I add an extra loss term $L_2$ and try to minimize $L=\lambda_1L_1+\lambda_2 L_2$.
However, my question is that if I do this, I may not able to achieve $L_1=0$ anymore since $L_2$ can be highly negative even if $L_1$ is positive can result in minimization of the overall loss function. Is there any way that I can simultaneously train the neural network for $L_1$ and $L_2$ minimization without using the weighted sum? Or to recover from the final training loss $L_1$ and separately use this value for convergence?
Any help in this area is appreciated, I'm not so well-versed with neural networks. Another concern for me is if enforcing $L_1=0$ is impossible since $L_2$ is kind of like a regularization term that will prevent NN from overfitting, and $L_1$ and $L_2$ will work against each other.


text:
Why is the output of my graph neural network not permutation equivariant?
I am using Pytorch to train a graph neural network on a 4x4 graph. Each node has one feature, and the output has one feature. Essentially, the architecture of my GNN looks like this (I'm training the networks using the pre-defined modules in https://github.com/unstable-zeros/grnn-comms-codesign):
 LocalGNN_t(
  (GFL): Sequential(
    (0): GraphFilter_t(in_features=1, out_features=1, filter_taps=2, edge_features=1, bias=True, GSO stored)
    (1): ReLU()
  )
  (Readout): Sequential(
    (0): Linear(in_features=1, out_features=10, bias=True)
    (1): ReLU()
    (2): Linear(in_features=10, out_features=10, bias=True)
    (3): ReLU()
    (4): Linear(in_features=10, out_features=1, bias=True)
    (5): Hardtanh(min_val=-1e+16, max_val=1e+16)

So I have one graph filter layer that takes information from the immediate neighbors for every node, and applies ReLU nonlinearity. The output of this is given to an MLP, node-wise.
I have read that graph neural networks are permutation equivariant, so if the input is permuted, then the output must be accordingly permuted.
Now, I am working with the following graph adjacency matrix:
S= [1,1,0,1 ; 1,1,1,0 ; 0,1,1,1 ; 1,0,1,1]

This means that every node is connected to two nodes that are adjacent to it in a circular fashion.
After training the neural network on PyTorch, I find that the output is not permuted according to the graph structure. I would assume that for the input
x=torch.tensor([1,2,3,4])
if I get the following output:
tensor([20.7212, 20.7212, 20.5522, 20.2472])
for
x=torch.tensor([2,1,3,4])
I should get:
tensor([20.7212, 20.7212, 20.2472, 20.5522])
But this is the output I actually get:
tensor([20.7212, 20.7212, 20.5715, 20.2280])
I do not understand this behavior, or whether I'm missing something or making a mistake. Any help will be appreciated.

It seems that the graph filter layer in your GNN takes information from the immediate neighbors for every node and applies a ReLU nonlinearity. However, the architecture you've shown does not explicitly consider the graph structure or utilize any permutation equivariance properties.
Permutation equivariance in graph neural networks refers to the property that the output of the network should remain unchanged when the input nodes are permuted while keeping the graph structure intact. In your case, it appears that the network does not explicitly enforce permutation equivariance.
To achieve permutation equivariance, you can incorporate graph convolutional layers or graph attention layers into your network architecture. These layers are designed to capture the graph structure and propagate node information accordingly. By using these layers, the network can learn to output results that are invariant to the ordering of the input nodes.
If your goal is to enforce permutation equivariance, you might need to modify the network architecture or consider different graph neural network models that explicitly account for the graph structure and permutation equivariance.

You're right, my graph filter for any given node collects information from itself and its neighboring nodes, and applies a graph convolution operator. It seems that they way it is defined, it is shift invariant over the input features, but not permutation equivariance. But permutation equivariance would be preserved if the graph structure is also permuted accordingly.


text:
Training loss decreases very fast after few epochs
I am implementing an ANN whose training loss is in Figure:

As you can see training loss decreases very fast and it is approximately 3.2 at epochs 2, 3, ..., 8, ... 10, and so on. (batch learning)
The diagram is similar even though I increases MAX_EPOCHS to 2000.
Is it correct?
Note that my training set is a toy:
X = np.asmatrix([
        [2, 5, 3.2, 7.2],
        [1, 0.8, 1, 8.5],
        [2, 7, 3, 1.4]
    ])

targets = np.asarray([0,2,1])
targets = np.asmatrix(np.eye(np.max(targets) + 1)[targets])  # 1-hot

And I am using Gradient Descent with momentum

What is the architecture  of your ANN? The dataset seems extremely small (with only 3 samples) so it is difficult to explain the behaviour of the ANN. It is possible that with the given architecture and this dataset, the loss cannot be reduced beyond 3.2.

You're loss is converging fast simply cause you're totally overfitting on you're toy input samples, which is good, at least it proves that your training script works.

Thank you. I was using the toy samples for that purposes, indeed.


text:
How is MNIST only providing the training and the test sets? What about the validation?
I was taught that, usually, a dataset has to be divided into three parts:

Training set - for learning purposes
Validation set - for picking the model which minimize the loss on this set
Test test - for testing the performance of the model picked using metrics such as accuracy score

How is MNIST only providing the training and the test sets? What about the validation?

The test set should never be seen and ran once at the end of training.
The validation set is used to help you select hyperparameters and it would be cheating to tune your model on the test set because you would be giving your model information about the test set. This would give your model an unfair advantage and skew the results; that simply means that if you are essentially using the test set for training data model, the model overfits to your test set, and will not generalize well to new, unseen data.
For this reasons, the validation set must be a portion of the training data which is selected out and evaluated on during training so that you can do this. It's not necessary if you're not doing model selection.

Helpful answer. Can you edit it for explaining why "it would be cheating to tune your model to the test set"?

It would be cheating to tune your model to the test set because you would be giving your model information about the test set that it would not have during actual prediction. This would give your model an unfair advantage and skew the results.

that simply means that if you are essentially using the test set for training data model, the model overfits to your test set, and will not generalize well to new, unseen data.

that's correct!

How is MNIST only providing the training and the test sets? What about the validation?
In order to perform cross-study comparison of model performance it makes a lot of sense to have a single test set for benchmarking.  This way, different investigators can compare their respective models in an "apples to apples" manner.  The test set is tested only once, as already mentioned, and is in a practical sense the final arbiter of model performance.  As an aside, the MNIST not only provides an important source of data for testing individual hypotheses, but it also serves to provide "standards".  After all, NIST stands for National Institute of Standards and Technology.
The reason why a validation set is not parsed out (like the way the test set is).
Different investigators may want to perform different types of validations (hold out, n-fold cross-validation, leave-one-out cross-validation).  Thus, MNIST does not limit investigators from doing this by separating out a validation set.  MNIST leaves it up to the investigator to parse their "training set" into yes, a training set and validation set they way they prefer.  Unlike the test set, there is really no need to standardize the validation.  In contrast, test set does need to serve as a standard for benchmarking models.
NOTE:  To be clear, this does not mean that you should not perform a validation.  You absolutely must.  The only thing is that you must "carve out" the validation set in a manner of your choosing.

Good question, it probably should, but I guess since it's the "hello world" of deep learning, they may have wanted to simplify it.
But in your day to day use, you should split the training into train/validation, but keep test the way it is.

There are a few reasons why MNIST only provides training and test sets, and not a validation set:

Since, the MNIST dataset is intended to be a simple and straightforward benchmark for machine learning models, It is important to have a standard test set that can be used to compare different models.

Secondly, the MNIST dataset is well-known and well-studied.

Thirdly splitting it into three sets (training, validation, and test) would reduce the size of each set too much & there is less need to have one.

The validation set is used to assess the performance of the model on unseen data. In this case, the validation set is not needed because the performance of the model can be assessed on the test set.
Another common method is to use cross-validation, which is where the data is split into multiple sets and each set is used to train and test the model. This allows for a more accurate assessment of the model's performance.


text:
Training and validation loss are almost the same (perfect fit?)
I am developing an ANN from scratch which classifies MNIST digits.
These are the curves I get using only one hidden layer composed of 100 neurons activated by ReLU function. The output's neurons are activated by the softmax function:

Is it correct that training and validation loss are almost identical? Does it mean that my model perfectly fit the data?

Usually it means you did something wrong. Double check everything.

@lpounng What If I do not spot errors?

If you have implemented from scratch, I would verify that your validation data is being held out during training.  It can happen that the two track quite tightly (as these do) but in my experience it is somewhat unusual for them to reflect the same hills and valleys throughout.

@DavidHoelzer No bugs spotted...

@tail try get someone else to look at it, or post more details here so we can help.

MNIST is a very easy dataset so I'd not be that surprised to have a good validation score. Try with Fashion MNIST which is a bit more challenging to see if the correlation remains.

If your accuracy is 50% in train and val, would you say that the model fit the data perfectly?
One guess is that your val and train data are pretty similar. Have you split the data before training the model?
The loss seems quite high. What does your accuracy look like? Have you tried to train for more epochs?

Accuracy on both data are around 95% at the end of the learning. Yes I have split the data. Yes I have tried to train for more epochs and the losses still seem to be identical.

50% accuracy is relative.  On a 10 class problem, 50% is much better than on a 2 class problem.  Also, loss is entirely relative and mostly meaningless without another loss for the same problem (and same loss function) to compare it to.  Here we have only the validation loss for comparison.

@tail
Have you tried kfold split? https://www.askpython.com/python/examples/k-fold-cross-validation

@DavidHoelzer that's the point. It's relative. It does highly depend on the data chosen for training and validation. I can have easily 100% accuracy on a multiclass classification with arbitrary number of classes when choosing the validation data-set accordingly.

@DavidHoelzer
"50% accuracy is relative. On a 10 class problem, 50% is much better than on a 2 class problem."
That not generally true. It depends on whether your data is balanced or not. For balanced data, it would be true.


text:
Why diffusion model always use U-Net?
I want to know why diffusion models always use U-Net.
In my opinion, they use U-Net because you can see features of different resolutions and skip connection is good to add detail of images. But I am not sure if it is the main reason why they use U-Net.
Are there other reasons they choose U-Net rather than other architectures?

I don't have a definitive answer but I'd state my intuitions anyways:
Diffusion models are highly related to the idea of stacked denoising autoencoders [Kumar et al. (2014)]. Additionally, U-Net-like architectures are a very common architecture for autoencoders on images.
Here, I would agree with your intuition that the bottleneck and the skip-connections help with denoising because they provide representations of different granularity.
Another thought is that U-Nets are empirically well-suited for image segmentation problems. Despite the fact that pixels are classified individually, you want the segmentation output to have consistent areas on top of objects in the image. So you kind of turn a very noisy segmentation mask (original image input) into a mask with much less noise (segmented output).
I think the latter is debatable, I'd be happy to hear your thoughts.

Hi, thanks for reply. The "bottleneck" you mean is 1x1 conv or the process of lowering resolution? If it is the former, I saw an article that says diffusion model cares about output more than latent representations which I think is true. Maybe this is also one of the reasons why they remove bottleneck layer from architecture?  As for image segmentation, because I am not familiar with it, I am sorry that I cannot see the connection between them XD.

With bottleneck I mean the decrease in layer size. This forces the encoding to discard some information which should improve denoising properties. The noise cannot be encoded lower dimensions.

Regarding the image segmentation part: Just look at this image and you know what image segmentation means. What you see there is the model output (colors) superimposed on the model input (a regular image). The goal of the model is to classify every pixel based on which object it belongs to (colors denote object classes). Thus, you could think of this task as the transformation of a very noisy version of this segmentation map (which is the regular image) to a very clean segmentation map. I hope this made it somewhat clearer?

Yes, it is clearer, thanks for explanation! If that is the case, I think I prefer the former idea. Maybe it is really just a simple reason like this that supports them to use it :o.

I think stable diffusion uses UNet because diffusion probabilistic models used UNet, DPM used UNet because PixelCNN++ used UNet, PixelCNN++ used it because it was 2017, that's what they had and it was good enough. Deep learning research has always been about cargo cult empirical research with weak (if any) justification. That said, the other choice is stuff like MaskRCNN and RPNs don't make any sense on latent representations

I totally get your point and agree to some extend. I still think it is very fruitful to try to interpret a little bit more into why certain architectures are used in a given framework. This has led to great advances, e.g. the whole area of geometric deep learning is kind of an artifact of identifying the benefits of specific models and casting them into more general versions.

They do not.
Even "the" seminal paper does not (it was put on axiv before "U-Net" was a thing :).

Generally, U-Nets are the default choice when dealing with functions where inputs and outputs have the same size/shape (such as in image segmentation) due to the lack of a real information bottleneck. Thus, those are a natural good choice when dealing with diffusion models, where the network must predict the residual noise.


text:
Deep Reinforcement Learning - How to Avoid a Naive Solution?
I'm implementing a connect-4 agent: using DQN, training by playing vs previous versions of the network.
However, many times the network learns that it's best to simply put 4 pieces in the same column, which is usually able to beat the previous (also naive) versions.
What could be the reason? how can it be dealt with?
Thanks!

There are many potential reasons why your AI might be learning that it's best to put $4$ pieces in the same column. One possibility is that AI is simply finding this strategy to be the most effective way to win the game. Another may be that the AI is overfitting to the training data and is not generalizing well to new situations i.e not exploring enough during training, which would prevent it from finding new and better strategies
To avoid a sub-optimal(naive solution), it is important to use a state representation that is informative enough to allow the agent to learn a good strategy. if state representation is not informative enough, then the agent will be not able to learn a good strategy from the data it has.
To combat this issue try to use feature engineering to make the state representation more informative, such as adding a binary feature for each column indicating whether or not that column is full. This would encourage the agent to try to spread its pieces out over the board more, rather than just dumping them all in the same column.
Another possibility could be that the AI is not exploring enough and is getting stuck in a local optimum so the approach would be to use a different reinforcement learning algorithm that is less likely to get stuck in local optima, such as Q-learning or SARSA. You could also try increasing the exploration rate during training, which would also help the agent escape from local optima

ok I found out that I was overriding the model after it finished training lol.
now it works!
However I appreciate your answer! It clarified some terminology and have given some interesting general ideas. Thanks!


text:
Transformers - how do the decoder attention input matrices look like, in terms of future tokens?

I have a question regarding the original transformer implementation (as in "Attention is all you need").
Assuming I want to translate English to German.

In the Decoder part, in the self-attention layers, the input is only the German tokens the model already translated, but never the future tokens.
So in the input matrices X=K=Q=V, what are the values in those rows? some placeholder null token?
If so, what happens in the (not self) attention afterwards? How does its Q matrix look like as a result from the previous self attention? What bothers me is that its Q matrix would have null/random-valued rows, which would affect the attention result.

Much appreciated.

In the implementation of transformers, there are specific methods employed to address this issue. The attention mechanism initially establishes a context length, which refers to the number of tokens each attention head processes. The issue you're describing seems to relate to what happens when the number of tokens generated by the decoder is less than the product of the context length and the number of attention heads. This situation is typically resolved by inserting a placeholder token named '[PAD]' for padding purposes. The network, therefore, functions as intended, and the attention calculations include the embeddings of this special token that is added in the vocabulary for that specific purpose. Consequently, the Q matrix will not contain null or random-valued rows, allowing the process to continue smoothly.
As a reference, when programming the tokenizer for the transformer with Hugginface's Transformers library, you can specify whether to use padding in the tokenizer class. This will add the '[PAD]' token to the Lookup table dictionary. This is customizable for different applications; for example, you can choose whether to add the paddings to the right or left side.

In addition to the answer by Cesar Ruiz:
In the Transformer model, padding tokens are used to make all sequences the same length. These padding tokens are not actual data and are assigned a logit of -inf so that they have no effect on the model's computations. When the softmax function is applied, the output probability associated with a logit of -inf will be effectively 0. This ensures that the padding tokens do not influence the model's predictions.
For a more detailed explanation, read this.


text:
Model Suggestions for Real Life local Hospital Data
I'm doing a machine learning project and was looking for suggestions. It's meant to get the date, household, age, sex, doctor, date of the medical appointment, and type of medical appointment of a patient in order to predict whether he will actually attend the appointment or not (many simply don't show up, without any further notice). The data set consists of around 26000 people.
I'm new to ML, and I was wondering if it makes sense to build an AI out of this situation. If the answer is yes, what model should I use? I'm using sklearn in Google Colab. I have tried DecisionTreeClassifier and MLPClassifier and both gave horrible accuracy, although that may be due to other reasons I'm not aware of.
Thanks in advance.

The model you're using is not telling the entire story. Lets briefly describe some approaches you should consider:
The Data itself
In most cases, working on the data itself may be more important than choosing the right model to train.

how descriptive are the features? can you use dummy variables on some of your textual features?
Are you working with balanced data? or perhaps $90\%$ of it is instances of "patient did now show up". If so, can you add more samples? can you remove samples? is augmented data relevant to your case?
do you have features in the form of free text? perhaps you can convert those to meaningful features using (for example) Word2Vec

Evaluating your model
In terms of the algorithmic approach, the model is not the only parameter

how do you validate the success of your model? what metrics are important to you? for example, the accuracy might not be as relevant if the data is unbalanced.
Do you care about Type 1 or Type 2 errors? If so, avoid a basic misclassification loss
Can you identify overfitting when comparing the training loss to the test loss? If so, try adding a meaningful regularization. If not - your model may not be complex enough.
To what baseline are you comparing your results? How does a simple linear regression perform?

Meta-algorithms

classical ML approaches usually benefit when using ensemble methods
How are you tweaking the hyper-parameters of your model? are you working with a validation set? have you tried using k-Fold cross validation?

I think that answering those questions may help you with reaching your goal. I would also not advise considering big guns like deep neural networks unless you have exhausted more approachable methods


text:
Similarities between 2d-vectors. (to flatten or to not)
I have this scenario where I need to measure the similarity between a 2d tensor t1: (100,8) and 61 tensors of the same shape(100,8). 100 represent time-steps and 8 is the no. of options.
I first tried flattening all tensors so that I can use the cosine similarity measure, but it slows down performance significantly.
I used pairwise comparisons without flattening like the approach below, but it made executing mathematical operations more complicated (e.g., diagonal operations):
How to calculate a meaningful distance between multidimensional tensors.
What should I do to speed-up ops on flattened tensors/or are there alternatives to flattening?

I suggest you to flatten and use FAISS. I had the same problem but using faiss it was made simple it is creating indexing which is faster to compare. Using faiss indexing i am able to compare 1 vector with 21 million vector in 6 milliseconds.

Thank you for your comment. I tried FAISS based on you suggestion, but it does not seem to support incremental addition of training samples. Based on your experience, is there a possible workaround for such a scenario or should I look for something different?


text:
Can the Jacobian of a Neural Network be Full Column Rank?
Let $\mathcal{X}$ be the input data space and $\mathcal{Y}$ be the output data space.
$f: \mathcal{X} \to \mathcal{Y}$ is a function represented by some Neural Network.
Is it possible to to check if the Jacobian $J_{f}:=\left(\frac{\partial f_{i}}{\partial x_j}\right)$ is full Column rank? or are there some conditions (like specific activation function) we can assume on the Neural network to make it full Column rank?
Also can it be safely applied to CNNs and not only ANN? (because I think CNN can not be treated as in vector space)


text:
Pointers to (deep) latent variable models that admit analytical approximations
I am aware that there is a plethora of deep generative models out there (e.g. variational autoencoders (VAE), GANs) that can model high-dimensional data as the images of latent variables under a non-linear mapping (typically neural network).
In more traditional methods such as probabilistic PCA, the latent variables can be marginalised analytically. In Bayesian PCA (BPCA), we can additionally integrate out the linear mapping, from the latent space to the observation space, by adopting the variational lower bound that leads to closed form updates of the parameters. The Gaussian Process Latent Variable (GPLVM) model adopts a non-linear probabilistic mapping (a Gaussian process) that can be marginalised. These two models enjoy to a certain degree analytical solutions concerning the inference of the latent variables and the mapping.
I have been wondering whether there is any research into more "complex" models (perhaps I should call them deep) that are capable of modelling more complex data distributions than the GPVLM and BPCA, but retain analytical solutions when inferring the posterior of the latent variables (like BPCA) or the mapping (like GPLVM)?
What I like about the GPLVM and BPCA is that they possess an objective function that can be analytically optimised, as opposed to the intractable objective of VAEs that necessitates Monte-Carlo averages and stochastic gradient. Could somebody please point me to such  examples of more complex generative models that admit analytical  inference for working out the posterior of the latent variables or the mapping?

Also posted on reddit


text:
What are the fundamental differences between VAE and GAN for image generation?
Starting from my own understanding, and scoped to the purpose of image generation, I'm well aware of the major architectural differences:

A GAN's generator samples from a relatively low dimensional random variable and produces an image. Then the discriminator takes that image and predicts whether the image belongs to a target distribution or not. Once trained, it can generate a variety of images just by sampling the initial random variable and forwarding through the generator.

A VAE's encoder takes an image from a target distribution and compresses it into a low dimensional latent space. Then the decoder's job is to take that latent space representation and reproduce the original image. Once the network is trained, it can generate latent space representations of various images, and interpolate between these before forwarding through the decoder which produces new images.

What I'm more interested is the consequences of said architectural differences. Why would I choose one approach over the other? And why? (for example, if GANs typically produce better quality images, any ideas why that is so? is it true in all cases or just some?)

You're already describing both approaches, and, as far as I know, these descriptions are accurate. So, are you asking that someone tries to tell you differences that you are not able to capture from your description? Is this your first question? I suppose, of course, you're also interested in knowing if you missed something in terms of the high-level description of both the VAE and GAN.

The other question "when would you use one over the other" is actually very interesting and useful, I would say, given that they are both generative models, and it's not clear when one would perform better than the other, so when to choose one over the other.

@nbro yes I mean the more interesting question. I hoped using "fundamental" in the title would hint that I don't care about the obvious architectural differences. I'll edit to try and make it clearer.

GANs generally produce better photo-realistic images but can be difficult to work with.  Conversely, VAEs are easier to train but don’t usually give the best results.
I recommend picking VAEs if you don’t have a lot of time to experiment with GANs and photorealism isn’t paramount.
There are exceptions such as Google’s VQ-VAE 2 which can compete with GANs for image quality and realism.  There is also VAE-GAN and VQ-VAE-GAN.
As a note, GANs and VAEs are not specifically for images and can be used for other data types/structures.

Thanks Brian. I think this answers the question, although I will also change my question a bit as I realised this is not exactly what I'm looking for. Looking to get something a little more fundamental.


text:
What is an appropriate size for a latent space of (variational) autoencoders and how it varies with the features of the images?
I am training an autoencoder and a variational autoencoder using satellite and streetview images. I have tested my program on standard datasets such as MNIST and CelebA. It seems that the latent space dimension needed for those applications are fairly small. For example, MNIST is 28x28x1 and CelebA is 64x64x3 and for both a latent space bottleneck of 50 would be sufficient to observe reasonably reconstructed image.
But for the autoencoder I am constructing, I needed a dimension of ~20000 in order to see features. The variational autoencoder is not working, and I only see a few blobs of fuzzy color.
For those who have experience with training the autoencoders with your own images, what could be the problem? Are certain features easier to compress than others? or does it look like a sample size problem? (i have 20000 images in training) Is there any rule of thumb for the the factor of compression? Thanks!
Here is a best example of what I have got with my VAE. I am using a ResNeXt architecture and the image dimension is 64x64x3, the latent space dimension is very large (18432).

You are asking about several things here and while related, solving one, will not necessarily "solve" your problem. Let's look at them separately:

Optimal dimension of the latent space.
Blurry reconstructions.
Optimal sample size.

Optimal dimension of the latent space.
I'm unaware of a one-fit-all way to find the optimal dimensionality of $z$ but an easy way is to try with different values and look at the likelihood on the test-set $log(p)$ - pick the lowest dimensionality that maximises it. This is a solution in tune with Deep Learning spirit :)
Second solution, maybe a little more grounded, is to decompose your training data with SVD and look at the spectrum of singular values. The number of non trivial (=above some small threshold) values will give you a rough idea of the number of latent dimensions you are going to need.
Finally, you could allow for a lot of z-dimensions but augment your loss function in such way, that the encoder will be forced to only use what it needs. This is sometimes called Sparsity promoting, L1 or Lasso-type regularisation and is also something that can help with overfitting. Take a look at arXiv:1812.07238.
Blurry reconstructions
This is a notorious problem with VAE's and while there are a lot of theories on why this happens, my take is that the reason is two fold.
First, the loss function. With typical Cross-Entropy or MSE losses, we have this blunt bottom of the function where the minimum is residing allowing for a lot of similar "good solutions". See arXiv:1511.05440 and especially https://openreview.net/forum?id=rkglvsC9Ym for an easy fix that seems to improve the quality/sharpness of the reconstructions.
Second, the blurriness comes from the Variational formulation itself. Handwavy explained, we are trying to model some very-very complex data (images in your case), with a "simple" isotropic Gaussian. Not surprisingly, the result will be something the best the model can do given this constraint. Recall that the loss function in VAE's is called ELBO - Evidence Lower Bound - which basically tells us that we are trying to model a Lower Bound as best as we can and not the "actual data" distribution. Typically, introduction of a KL-multiplier, $\beta$, which relaxes the influence of the Gaussian prior, will give you better reconstructions (see $\beta$-VAE's).
Finally, if you are feeling especially adventurous, take a look at discrete VAE's (VQ-VAE's), which seem to have reconstructions on pair with GAN's. Sampling from them is not trivial, however.
Optimal sample size
As for optimal sample size, just choose an architecture that will not overfit. Decrease the number of neurons/layers, check your $\log(p)$ on the test-set, introduce Dropout, all the usual stuff.

Thank you for summarizing and troubleshooting my problem! I was indeed asking about quite a few things because I did not know what is causing the problem. Currently trying your suggestions!

From my experience (on MNIST digits), even when using a latent space of only $10$ nodes, the decoded reconstruction was pretty much ok. perhaps the architecture itself lacks the capabilities of encoding/decoding properly. Check out this summary and see if you can improve your results using a similar approach.


text:
Is my neural network working?
I recently just finished programming a neural network in c#, and it seems like it's working. My question is if I'm doing it right. It's a very confusing process so I will explain.
Basically every neuron in the network has a bias (except the first layer) and fully connected weights to the next. every neuron also has a double[] desiredBias which holds all the desired changes to the neuron for every iteration in a given mini batch. The same thing for the weights. So I start from the last layer, and from there I calculate the "sensitivity" you might say between the cost of a given neuron on that last layer and the bias, and i take note of it (put it into changes). Then for every weight connected to the layer behind, I also calculate the sensitivity for that one. And I keep doing that until I get to the second to first layer. And to take note of the desiredChanges for neurons not in the last layer, I use this array called "changes". For the first layer it's basically just a list of all the derivatives of the activations (z) with respect to the actual activations (sigmoid(z)) respectively. After I shift to the layer behind, and behind, every time I do that, I create a new changes array, which is the size of the layer behind, and for each item in the previous changes array, I take the sum of all of them multiplied by the partial derivative of that same z-activation with respect to the previous layer's z activation. I do this for every neuron in the previous layer.
after all of that I end up with for every neuron a list of desired changes to it's bias, and a list of lists which contain desired changes to the weights of each neuron. Then for each neuron (except for the first of course) I average together all of the biases and SUBTRACT it from the neuron's bias. I do the same thing for the weights.
Is this the correct process to go through for exactly one batch of training data?
here's my code:
using System;

namespace NeuralNetwork {

    public class Functions {

        // Activation Functions

        public static double sigmoid(double x) {
            return 1 / (1 + Math.Pow(Math.E, -x));
        }
        public static double ReLU(double x) {
            return Math.Max(0, x);
        }
        public static double Tanh(double x) {
            return Math.Tanh(x);
        }
        public static double Linear(double x) {
            return x;
        }

        // Derivatives

        public static double sigmoidDerivative(double x) {
            return sigmoid(x) * (1 - sigmoid(x));
        }
        public static double ReLUDerivative(double x) {
            return (x > 0) ? 1 : 0;
        }
        public static double TanhDerivative(double x) {
            return 1 - (Math.Pow(Tanh(x) * Tanh(x), 2));
        }
        public static double LinearDerivative(double x) {
            return 1;
        }

        public enum ActFunction {
            Sigmoid,
            ReLU,
            Tanh,
            Linear
        }

        // Loss Functions

        public static double MSE(double[] outputs, double[] desired) {
            double sum = 0;
            for (int i = 0; i < outputs.Length; i++) {
                sum += Math.Pow((outputs[i] - desired[i]), 2);
            }
            return sum / outputs.Length;
        }

        // Derivatives

        public static double[] MSEDerivative(double[] outputs, double[] desired) {
            double[] derivatives = new double[outputs.Length];
            for (int i = 0; i < outputs.Length; i++) {
                derivatives[i] = 2 * (outputs[i] - desired[i]);
            }
            return derivatives;
        }

        public enum LossFunction {
            MSE
        }

    }

    public class Neuron {

        public double activation = 0;
        public Functions.ActFunction activationFunction = Functions.ActFunction.Linear;

        public double bias;
        public double[] weights = new double[] {};
        public int x;
        public int y;
        public Neuron[][] network;

        public double[] desiredBias;
        public double[][] desiredWeights;

        public void fire() {
            for (int i = 0; i < network[x + 1].Length; i++) {
                Neuron next = network[x + 1][i];
                next.activation += activation * weights[i];
            }
        }

        public void activate() {
            double n_activation = activation + bias;

            switch(activationFunction) {
                case Functions.ActFunction.Sigmoid:
                    activation = Functions.sigmoid(n_activation);
                    break;
                case Functions.ActFunction.ReLU:
                    activation = Functions.ReLU(n_activation);
                    break;
                case Functions.ActFunction.Tanh:
                    activation = Functions.Tanh(n_activation);
                    break;
                case Functions.ActFunction.Linear:
                    activation = Functions.Linear(n_activation);
                    break;
            }
        }

    }

    public class Network {

        public Neuron[][] network;
        public double learningRate;
        public int iterations;
        public Functions.LossFunction lossFunction;
        public int trainingLength;

        public Network(int[] size, double _learningRate, int _iterations, int _trainingLength, Functions.LossFunction _lossFunction) {
            learningRate = _learningRate;
            iterations = _iterations;
            trainingLength = _trainingLength;
            lossFunction = _lossFunction;
            network = new Neuron[size.Length][];
            for (int i = 0; i < size.Length; i++) {
                network[i] = new Neuron[size[i]];
            }
            for (int i = 0; i < size.Length; i++) {
                for (int j = 0; j < size[i]; j++) {
                    network[i][j] = new Neuron();
                }
            }
            for (int i = 0; i < network.Length - 1; i++) {
                for (int j = 0; j < network[i].Length; j++) {
                    Neuron neuron = network[i][j];
                    neuron.weights = new double[network[i + 1].Length];
                    neuron.desiredWeights = new double[neuron.weights.Length][]; // i would have put iterations right here but the compiler thinks otherwise
                    for (int k = 0; k < neuron.desiredWeights.Length; k++) {
                        neuron.desiredWeights[k] = new double[trainingLength/iterations];
                    }
                }
            }
            for (int i = 1; i < network.Length; i++) {
                for (int j = 0; j < network[i].Length; j++) {
                    Neuron neuron = network[i][j];
                    neuron.desiredBias = new double[trainingLength/iterations];
                }
            }
        }

        public void init() {
            Random rand = new Random();
            for (int i = 0; i < network.Length; i++) {
                for (int j = 0; j < network[i].Length; j++) {
                    Neuron neuron = network[i][j];
                    neuron.x = i;
                    neuron.y = j;
                    neuron.network = network;

                    if (i != 0) {
                        neuron.bias = (rand.NextDouble() * 2) - 1;
                    }

                    if (i != network.Length - 1) {
                        for (int k = 0; k < network[neuron.x + 1].Length; k++) {
                            neuron.weights[k] = (rand.NextDouble() * 20) - 1;
                        }
                    }
                }
            }
        }

        public void info() {
            Console.WriteLine("Here are the statistics for the network:\n");

            for (int i = 0; i < network.Length; i++) {
                for (int j = 0; j < network[i].Length; j++) {
                    Neuron neuron = network[i][j];
                    Console.WriteLine("Neuron " + j + " of layer " + i);
                    //Console.WriteLine("Activation: " + neuron.activation);
                    Console.WriteLine("Bias: " + neuron.bias);
                    Console.WriteLine("X and Y: " + neuron.x + ", " + neuron.y);
                    string weights = "Weights: ";
                    foreach (double weight in neuron.weights) {
                        weights += weight + " ";
                    }
                    Console.WriteLine(weights);
                    Console.WriteLine();
                }
            }

            Console.WriteLine();
        }

        public double[] run(double[] inputs) {
            // initialize inputs
            for (int i = 0; i < network[0].Length; i++) {
                network[0][i].activation = inputs[i];
            }

            for (int i = 0; i < network.Length - 1; i++) {
                for (int j = 0; j < network[i].Length; j++) {
                    Neuron neuron = network[i][j];
                    neuron.fire();
                }
                for (int j = 0; j < network[i + 1].Length; j++) {
                    Neuron neuron = network[i + 1][j];
                    neuron.activate();
                }
            }

            // get outputs

            Neuron[] outputs = network[network.Length - 1];
            double[] activations = new double[outputs.Length];

            for (int i = 0; i < outputs.Length; i++) {
                activations[i] = outputs[i].activation;
            }

            return activations;
        }

        public double iteration() {
            Random rand = new Random();

            int iterationLength = trainingLength/iterations;
            double[] costs = new double[iterationLength];

            // training

            for (int iter = 0; iter < iterationLength; iter++) {
                double[] inputs = { (rand.NextDouble() ), ( rand.NextDouble() )};
                double[] answers = { inputs[0] + inputs[1] };
                double[] outputs = run(inputs);

                for (int i = 0; i < network.Length; i++) {
                    for (int j = 0; j < network[i].Length; j++) {
                        Neuron neuron = network[i][j];
                        neuron.desiredBias = new double[iterationLength];
                        if (i == network.Length - 1) continue;
                        neuron.desiredWeights = new double[network[i + 1].Length][];
                        for (int k = 0; k < neuron.desiredWeights.Length; k++) {
                            neuron.desiredWeights[k] = new double[iterationLength];
                        }
                    }
                }

                double cost = 0;
                switch(lossFunction) {
                    case Functions.LossFunction.MSE:
                        cost = Functions.MSE(outputs, answers);
                        break;
                }

                costs[iter] = cost;

                double[] changes = new double[network[network.Length - 1].Length];
                switch(lossFunction) {
                    case Functions.LossFunction.MSE:
                        changes = Functions.MSEDerivative(outputs, answers);
                        break;
                }

                for (int i = 0; i < changes.Length; i++) {
                    Neuron neuron = network[network.Length - 1][i];
                    double act = neuron.activation;
                    switch(neuron.activationFunction) {
                        case Functions.ActFunction.Sigmoid:
                            changes[i] *= Functions.sigmoidDerivative(Math.Log(act / (1 - act)));
                            break;
                        case Functions.ActFunction.Linear:
                            changes[i] *= Functions.LinearDerivative(act);
                            break;
                    }
                }

                for (int i = network.Length - 1; i > 0; i--) {
                    for (int j = 0; j < network[i].Length; j++) {
                        Neuron bNeuron = network[i][j];
                        bNeuron.desiredBias[iter] += changes[j]; // bias partial derivative is always one
                        for (int k = 0; k < network[i - 1].Length; k++) {
                            Neuron aNeuron = network[i - 1][k];
                            aNeuron.desiredWeights[bNeuron.y][iter] += aNeuron.weights[bNeuron.y] * changes[j]; // chain rule
                        }
                    }

                    double[] holderChanges = changes;
                    changes = new double[network[i - 1].Length];
                    for (int j = 0; j < changes.Length; j++) {
                        for (int k = 0; k < holderChanges.Length; k++) {
                            Neuron aNeuron = network[i - 1][j];
                            Neuron bNeuron = network[i][k];
                            switch(aNeuron.activationFunction) {
                                case Functions.ActFunction.Sigmoid:
                                    double activation = aNeuron.activation;
                                    changes[j] += Functions.sigmoidDerivative(Math.Log(activation / 1 - activation)) * aNeuron.weights[bNeuron.y] * holderChanges[k];
                                    break;
                                case Functions.ActFunction.Linear:
                                    activation = aNeuron.activation;
                                    changes[j] += Functions.LinearDerivative(activation) * aNeuron.weights[bNeuron.y] * holderChanges[k];
                                    break;
                            }
                        }
                    }
                }

                // reset activations

                for (int i = 0; i < network.Length; i++) {
                    for (int j = 0; j < network[i].Length; j++) {
                        Neuron neuron = network[i][j];
                        neuron.activation = 0;
                    }
                }
            }

            // after everything we now actually add the desired changes

            // the weights
            for (int i = 0; i < network.Length - 1; i++) {
                for (int j = 0; j < network[i].Length; j++) {
                    Neuron neuron = network[i][j];
                    for (int k = 0; k < neuron.desiredWeights.Length; k++) {
                        double weightSum = 0;
                        foreach (double weight in neuron.desiredWeights[k]) {
                            weightSum += weight;
                        }
                        weightSum /= iterationLength;
                        neuron.weights[k] -= learningRate * weightSum;
                    }
                }
            }

            // the biases
            for (int i = 1; i < network.Length; i++) {
                for (int j = 0; j < network[i].Length; j++) {
                    Neuron neuron = network[i][j];
                    double biasSum = 0;
                    foreach (double bias in neuron.desiredBias) {
                        biasSum += bias;
                    }
                    biasSum /= iterationLength;
                    neuron.bias -= learningRate * biasSum;
                }
            }

            double netcost = 0;
            foreach (double cost in costs) {
                netcost += cost;
            }

            return netcost/costs.Length;
        }

        public void epoch(bool showInfo) {
            Console.WriteLine("Starting training...\n");

            for (int iter = 0; iter < iterations; iter++) {
                double cost = iteration();
                if (showInfo) info();

                Console.WriteLine("Cost for iteration " + iter + ": " + cost + "\n");

                // reset desired

                for (int k = 0; k < network.Length - 1; k++) {
                    for (int l = 0; l < network[k].Length; l++) {
                        Neuron neuron = network[k][l];
                        for (int i = 0; i < neuron.desiredWeights.Length; i++) {
                            for (int j = 0; j < neuron.desiredWeights[i].Length; j++) {
                                neuron.desiredWeights[i][j] = 0;
                            }
                        }
                    }
                }
                for (int k = 1; k < network.Length; k++) {
                    for (int l = 0; l < network[k].Length; l++) {
                        Neuron neuron = network[k][l];
                        for (int i = 0; i < neuron.desiredBias.Length; i++) {
                            neuron.desiredBias[i] = 0;
                        }
                    }
                }
            }

            Console.WriteLine("Training finished!\n");
        }

        public void prompt() {
            while(true) {

                Console.WriteLine("What would you like to do?");
                Console.WriteLine("Train, info, run or end?");

                string answer = Console.ReadLine();

                if (answer.ToLower() == "end") {
                    Console.WriteLine("...");
                    break;
                }

                switch(answer.ToLower()) {
                    case "info":
                        Console.WriteLine();
                        info();
                        Console.WriteLine();
                        break;
                    case "run":
                        double[] runanswer = new double[network[0].Length];
                        while (true) {
                            try {
                                for (int i = 0; i < network[0].Length; i++) {
                                    Console.WriteLine("Enter input " + i);
                                    runanswer[i] = Convert.ToDouble(Console.ReadLine());
                                }

                                Console.WriteLine("Running...\n");

                                double[] outputs = run(runanswer);
                                string _string = "";

                                foreach (double output in outputs) {
                                    _string += output;
                                }

                                Console.WriteLine("Here were the outputs:");
                                Console.WriteLine(_string + "\n");
                                break;
                            } catch (FormatException exception) {
                                Console.WriteLine("Only numbers!");
                                continue;
                            }
                        }
                        break;
                    case "train":
                        int trainanswer = 0;
                        int trainanswer2 = 0;
                        double trainanswer3 = 0;
                        while (true) {
                            try {
                                Console.WriteLine("Enter epoch length:");
                                trainanswer = Convert.ToInt32(Console.ReadLine());
                                trainingLength = trainanswer;

                                Console.WriteLine("\nEnter batch size:");
                                trainanswer2 = Convert.ToInt32(Console.ReadLine());
                                iterations = trainanswer2;

                                Console.WriteLine("\nEnter learning rate:");
                                trainanswer3 = Convert.ToDouble(Console.ReadLine());
                                learningRate = trainanswer3;

                                for (int i = 0; i < trainanswer2; i++) {
                                    double cost = iteration();
                                    Console.WriteLine("The cost was: " + cost + "\n");
                                }
                                break;
                            } catch (FormatException exception) {
                                Console.WriteLine("Only numbers!");
                                continue;
                            }
                        }
                        break;
                    default:
                        Console.WriteLine("Invalid command!\n");
                        break;
                }
            }
        }

    }

    public class Run {
        public static void Main(string[] args) {

            Network brain = new Network(new int[] { 2, 1 }, .01, 50, 50, Functions.LossFunction.MSE);
            brain.init();

            int i = 7;
            while (i > 0) {
                brain.epoch(false);
                i--;
            }

            brain.prompt();

        }
    }

}

*** to clarify if you don't understand my question, I'm just asking if i'm implementing backpropagation correctly

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

Unfortunately I highly doubt anyone is going to check your code or exact implementation as that is a very time consuming process. Luckily for you though, you don't need someone to check, you can check it yourself with gradient checking! The basic process is to manually calculate the exact gradient for every single parameter for an input by varying every parameter one at a time, then seeing how each changes your output. With a proper implementation of backpropagation, the gradients will match.
Here's another question describing this process on stack overflow
Note though that with ReLU the function is undefined at 0, and you will quickly see how this is a problem when you start doing grad checking. The gradients wont match if you try and check a parameter using the output of a ReLU function when the input to said ReLU is 0. I did some research on this a while ago when I had this problem, and I believe the consensus was to just ignore it

You're right. I don't know why I thought anyone would. I will try this method, thank you.


text:
If GPT-3 is trained on predicting the next token, how is it able to take commands?
From my understanding, GPT-3 is trained on predicting the next token from a sequence of tokens. Given this, how is it able to take commands? For instance, in this example input, wouldn't the statistically most likely prediction be to insert a period and end the sentence?
Input: write me a beautiful sentence

Output: I cannot put into words how much I love you, so I'll just say it's infinite.

No, it's actually a very precise guess determined by relatively complex statistical equations.
GPT-3 is able to take commands because it has been trained on a dataset of commands. This means that it is able to recognize patterns in sequences of tokens that indicate a command. In the example input, the most likely prediction is to insert a period and end the sentence because this is what typically happens at the end of a sentence. However, GPT-3 is able to take commands because it is able to recognize patterns that indicate a different kind of sequence, such as a command.

The dataset has been curated in such a way as that it believes the next most likely set of tokens are answers.
i.e. Its been shown that after a command surrounded by special tokens the next tokens are always responses to the commands.