text:
Why do all nodes in a GP tree need to be the same type?
Context: I'm a complete beginner to evolutionary algorithms and genetic algorithms and programming. I'm currently taking a course about genetic algorithms and genetic programming.
One of the concepts introduced in the course is "closure," the idea that - with an expression tree representing a genetic program that we're evolving - all nodes in the tree need to be the same type. As a practical example, the lecturer mentions that implementing greater_than(a, b) for two integers a and b can't return a boolean like true or false (it can return, say, 0 and 1 instead).
What he didn't explain is why the entire tree needs to match in all operators. It seems to me that this requirement would result in the entire tree (representing your evolved program) being composed of nodes that all return the same type (say, integer).

Hello. My answer below should answer your question, so you should accept it.

In tree-based genetic programming (TGP), you have a tree that represents a program or a function. The nodes in this tree are functions, while the edges represent the interactions between these functions. The leaves of this tree are the inputs (or random numbers) that you pass to this function. The incoming edges into a node represent the inputs, while the outgoing edges represent the output of the associated function.
So, for example, consider the following function
\begin{align}
f(x)
&= \sin(x^2) \\
&= \sin(y)
\tag{1}\label{1},
\end{align}
where $y = g(x)$ (note that this is just a change of variable in order to illustrate the corresponding tree more clearly below!).
The function $f$ will be represented by the following tree
sin(y)
  |
 g(x)
  |
  x

So, how do we read this diagram? Essentially, we read it from the bottom-up. So, $x$ is first passed to $y = g(x) = x^2$, which then passed to $\sin(y)$. In this case, given that we are dealing with mathematical operations, you expect $x$ to be a number, because, otherwise, what would $x^2$ or $\sin(x^2)$ mean?
Let's now consider a function of 2 inputs.
\begin{align}
f(x, y)
&= x + y
\tag{2}\label{2},
\end{align}
The corresponding tree would be
  +
 / \
x   y

In this case, you naturally expect that $x$ and $y$ are also numbers. However, this may not be actually the case. Let's say that you're evolving Python functions, then x + y is well defined even if x and y are strings, i.e. that would be a concatenation operation.
So, in this sense, we naturally expected the functions (the nodes in the tree) to get parameters/arguments with the right types, but, in some cases, more types are possible for the apparently same function.
So, in general, types don't necessarily need to be enforced! It depends on the implementation of TGP.
There's a specific approach to TGP where the idea is really to ensure type-safety, i.e. strongly-typed (tree-based) genetic programming, where functions can have a different number and type for their parameters and return values, but, in that case, only functions that are consistent with the signature of the function can be "connected" with that function in the tree.
In weakly-typed GP, the types are not checked, so you could end up evolving functions of the form $f(x) = \sin(x)$, where $x$ is a string. Of course, when you execute these programs/functions, the program may crash, but this is a different story, which you need to take care of (i.e. you need to ensure that your individuals can be evaluated), if you need this flexibility. This weakly-typed approach can also be viewed as a strongly-typed approach where all functions' parameters and return values have the same type.
DEAP, a well-known Python library for GAs and GP, provides both weakly and strongly-typed GP, so, if you are familiar with Python, you may want to start from it.
To conclude and answer your question more directly, it's not true that in TGP you need all functions to have the same type for all parameters and return values.


text:
Is there a term to denote AI algorithms that are not trained?
Before the advent of neural architectures, many AI domains (e.g. speech recognition and computer vision) used algorithms that consisted of a series of hand-crafted transformations for feature extraction.

In speech recognition everything to do with spectrograms or cepstra is done based on the mathematical theory of (discrete) Fourier transforms.
In computer vision, edge detectors like Sobel/Canny or feature descriptors like SIFT are all convolution-based.

You can implement all of these with everyday procedural programs. All the intelligence of these methods is embedded within the steps of the program, and not in its parameters.
One thing I have been wondering about is what you would call such algorithms that work "out of the box". My inclination is to call them deterministic, but this is clearly wrong considering most neural networks (not including e.g. VAEs) are deterministic functions of their input too.
Perhaps some might call them dumb, but that's also not right, because these algorithms are very sophisticated and purposefully made. They're not naive like naive Bayes is naive. It's not because they aren't trained that they aren't artificially intelligent.
Traditional or classical is too vague, because neural nets have existed for more than half a century and could be considered that too. The same goes for algorithmic and procedural.
Is there standard terminology to delimit the set of such artificially-intelligent-but-not-data-driven algorithms?

The word "Artificial intelligence" refers to machines being able to have intelligence that of humans/animals. The meaning of the word was even discussed on this site. So it's up to your interpretation what is human/animal like performance - you could even argue the examples you provide are human like performance.
There is this well known Venn diagram showing the difference.

But personally I wouldn't call these example you mention artificially-intelligent. The algorithms you mention are just a set of complex instructions.

All the intelligence of these methods is embedded within the steps of the program, and not in its parameters.

While intelligent authors made the algorithms, that's not what the word "artificially-intelligent" refers to.

It's not because they aren't trained that they aren't artificially intelligent.

Again it's up to you what you perceive as intelligence that of humans/animals. But if an algorithm has trainable weights, then it's classified as Machine Learning. A Machine learning agent can still be so useless such that no one would classify it as "Artificial intelligence".
I would just call the algorithms you refer to a "complex algorithm" or  a "non-ML method".

Some problems cannot be machine learned to get the accuracy of a handcrafted solution. For example, classifying numbers into negative and non-negative classes. The machine learning "intelligent" threshold would get smaller accuracy than the handcrafted 0 boundary.

Constructions like "non-AI method" seem like the right way to go -- this is, focusing on specifically what's meant.  For example, if a speaker means that a particular technique wasn't used, then ""non-"+[technique]".

How about handcrafted -as you mentioned-? In the following question it is opposed to learned.
https://datascience.stackexchange.com/questions/54390/what-is-the-difference-between-handcrafted-and-learned-features

I agree with the term. Handcrafted, hand-designed features, etc. are used for processing steps that require human expertise, as mentioned in the Deep Learning book.

I'd be concerned about using "handcrafted" vs. "learned" as though they were mutually-exclusive alternatives.  Terminology like that seems reasonable enough in simpler cases, but it may not work well in more complex scenarios.

The questions asks for a "standard terminology". Handcrafted vs Learned are only used when speaking of features. It's debatable wherever it's a good idea to also use that terminology on distinguishing algorithms, but it's not standard terminology

"engineered" also comes to mind

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How can I find an upper bound on the number of iterations required to have less than $\varepsilon$ difference in the value of state?
I learned about the Value Iteration algorithm which can help find an optimal policy and values of an MDP with state rewards:
$$V_0(s)=R(s)$$
$$V_{t}(s)=R(s)+\gamma\cdot\underset{a}{max}\underset{s'}{\sum}T(s, a, s')\cdot V_{t-1}(s')$$
How can I find an upper bound on the number of iterations required to have less than $\varepsilon$ difference in the value of state $s$ (i.e $|V_t(s)-V_{t-1}(s)|\le\varepsilon$)?
For example, suppose I have an MDP with 10 states, where in each state there are 2 actions and each action can lead to 3 states. The reward of each state is between 0 and 5, $\gamma=0.9$ and $\varepsilon=0.1$.
How can I find an upper bound for the number of iterations? I tried to solve the equations $0.9^t=0.1$, but this is not the right way to do it.

Further I'll use the following notation: $V^\pi(s)$ would be the value of a state under some (in most cases - arbitrary) policy $\pi$. There is a class of optimal policies with the same optimal value function $V^*(s)$. To measure "distances" between different value functions the max norm is used:
$$\lVert V^\pi - V^{\pi'}\rVert_\infty = \max_{s \in S} \left| V^\pi(s) - V^{\pi'}(s)\right|$$
The Bellman operator ${\mathcal B}$ is a central concept for reasoning about RL algorithms. It takes a value function and returns a new one. (I'm using your notation there except keeping more general dependency of expected reward $R(s) = \max_aR(s,a)$)
$${\mathcal B}(V^\pi) = \max_a\left[ R(s,a) + \gamma\sum_{s'}T(s,a,s')V^\pi(s') \right]$$
The Value Iteration algorithm then starts with an arbitrary value function $V_0$ and iteratively applies the Bellman operator $V_t ={\mathcal B}(V_{t-1})$. The optimal value function is the stationary point of ${\mathcal B}$. In other words, it satisfies the Bellman equation:
$$V^* ={\mathcal B}(V^*)$$
Existence of such stationary point, as well as the convergence of the Value Iteration, follows from the contraction property of  ${\mathcal B}$:
$$\lVert {\mathcal B}(V^\pi) -  {\mathcal B}(V^{\pi'})\rVert_\infty \leq \gamma \lVert V^\pi - V^{\pi'}\rVert_\infty$$
Substituting $V_{t-1}$ and $V_{t-2}$ into the inequality above, we get:
$$\lVert V_t -  V_{t-1}\rVert_\infty \leq \gamma \lVert V_{t-1} - V_{t-2}\rVert_\infty$$
Iteratively applying this inequality to itself we get a useful intermediate result:
$$\lVert V_t -  V_{t-1} \rVert_\infty \leq \gamma^{t-1} \lVert V_1 - V_0\rVert_\infty \tag{*}\label{*}$$

The expected immediate reward is assumed to be bounded by some value $R_{\max}$
$$ -R_{max} \leq R(s,a) \leq R_{max},\quad \forall s, a$$
Since for any policy, the value of a state is the expectation of the discounted sum of rewards, we can bound it with the sum of geometric series:
$$V^\pi(s_0) = {\mathbb E}\left[R(s_0, a_0) + \gamma R(s_1, a_1) + \gamma^2 R(s_2, a_2) + \dots\right]$$
$$- (1 + \gamma + \gamma^2 + \dots) R_{max} \leq V^\pi(s_0) \leq (1 + \gamma + \gamma^2 + \dots)R_{max}$$
$$\lVert V^\pi\rVert_\infty \leq \frac{R_{max}}{1 - \gamma}$$
We can use it to bound the distance between any pair of value functions:
$$\lVert V^\pi - V^{\pi'}\rVert_\infty \leq \lVert V^\pi\rVert_\infty + \lVert V^{\pi'}\rVert_\infty \leq  \frac{2R_{max}}{1 - \gamma} $$
Applying this bound to  $\eqref{*}$ we get a convergence bound in terms of a distance from the optimal policy:
$$\lVert V_t - V_{t-1}\rVert_\infty \leq \gamma^{t-1}\frac{2R_{max}}{1 - \gamma} \leq \epsilon $$
Solving for (an integer) $t$ for a given $\epsilon$ I get the final convergence lower bound on the number of iterations:
$$t \geq \left\lceil \frac{\log\frac{(1-\gamma)\gamma\epsilon}{2 R_{max}}}{\log\gamma}\right\rceil$$
For your numbers (in your case $R_{max} = 5$) I get $t \geq 67$ steps. Note that the whole thing doesn't depend on number of states and actions.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Where is the problem: in batch TD(0) algorithm or in the code to solve AB problem in Sutton-Barto RL book?
Here is the batch TD(0) algorithm:

Here is the AB example I want to solve using batch TD(0):

And finally here is my Matlab code:
% eps1: A 0 B 0
% eps2: B 1
% eps3: B 1
% eps4: B 1
% eps5: B 1
% eps6: B 1
% eps7: B 1
% eps8: B 0
% (s, r, s') cases:
% (A,0,B)
% (B,0,Ter)
% (B,1,Ter)
% (B,1,Ter)
% (B,1,Ter)
% (B,1,Ter)
% (B,1,Ter)
% (B,1,Ter)
% (B,0,Ter)
n_s=2; % number of states

v(1)=0;
v(2)=0;
v_ter=0;
thrs=1e-3;

delta=1e6;
alpha=0.1;

while (delta > thrs)

    for i=1:9 % we have 9 (s, r, s') cases
        if i==1
            v_pr(1)=v(1)+alpha*[0+1*v(2)-v(1)];
            Delta(1)=abs(v_pr(1)-v(1));
        elseif i==2
            v_pr(2)=v(2)+alpha*[0+1*v_ter-v(2)];
            Delta(2)=abs(v_pr(2)-v(2));
        elseif (i>=3) && (i<=8)
            v_pr(2)=v(2)+alpha*[1+1*v_ter-v(2)];
            Delta(2)=abs(v_pr(2)-v(2));
        else
            v_pr(2)=v(2)+alpha*[0+1*v_ter-v(2)];
            Delta(2)=abs(v_pr(2)-v(2));
        end
    end
    v=v_pr;
    delta=max(Delta);
    delta

end

However, my code fails to find the correct result $v_{TD}(A)=v_{TD}(B)=3/4$.
Where is the problem: in the batch TD(0) algorithm, in my code or in both?

I did not understand what you mean. initial values can be arbitrary and I considered all data in the given episodes.

even if you use the initial condition v(1)=v(2)=1, it does not converge to the right value.

For this specific MDP reward shape and extremely short episode and your code structure, v(1)=v(2)=1 would induce same bad behavior as v(1)=v(2)=0. Once you try some different big random numbers it would behave better. Also the experience D in your referenced algo most likely occurs within a single episode. Not sure where your 9 cases come from? It doesn't follow 75%/25% action policy at state B. you can simulate 100 cases where 75 of which follow the rewarded action and remaining cases follow the 0 reward action then to get average expected cumulative rewards for both states, respectively.

please see the slides 17-19 from this link:  https://groups.uni-paderborn.de/lea/share/lehre/reinforcementlearning/lecture_slides/built/Lecture05.pdf.  They obtained the results analytically.

we have 8 cases  at which 2 go to 0, 6 go to 1 and hence 25%/755 policy is followed. moreover, batch td  is for cases where you have limited number of episodes.

My algo is called the batch TD(0) learning not MC since $\alpha$ is less than 1. If you reset $\alpha=1$ it'll give you exact result since your samples exactly follow your policy for both states. You can of course use pure TD(0) to incrementally update each state value after each atomic experience as your experience replay code but it would converge much slower and not accurate compared to my batch TD(0) which is more unbiased. To correct your code you only need your v_pr (which is really a temporary vector needed for delta) defined as v at the start of your while-loop besides Delta def.

Based upon your reference background of the given sample episodes in one batch, you need to average v_pr(2) since there're 8 cases then update your vector v with vector v_pr before next batch before expected convergence is achieved. Below is sample Python code based upon your matlab code, which arrives at your desired result from its output print.
v1=0
v2=0
v_ter=0
thrs=1e-5
delta=1e5
alpha=0.01
v_pr1=0
v_pr2=0

# we have 9 (s, a, r, s') cases
while (delta > thrs):
    v1 = v_pr1
    v2 = v_pr2
    v_pr1 = 0
    v_pr2 = 0
    for i in range(1, 10):
        if(i==1):
            v_pr1 += v1 + alpha * (0 + 1 * v2 - v1)
        elif(i==2):
            v_pr2 += v2 + alpha * (0 + 1 * v_ter - v2)
        elif(i>=3 and i<=8):
            v_pr2 += v2 + alpha * (1 + 1 * v_ter - v2)
        else:
            v_pr2 += v2 + alpha * (0 + 1 * v_ter - v2)
    v_pr2 = v_pr2/8
    Delta1 = abs(v_pr1 - v1)
    Delta2 = abs(v_pr2 - v2)
    delta = max(Delta1, Delta2)

print('v1=' + str(v1) + ', v2=' + str(v2) + ', delta=' + str(delta))

--------
result:
v1=0.7488911797417349, v2=0.7498874120968516, delta=9.962323551215846e-06


text:
Can Q-learning and other RL algorithms solve CNF SAT?
I encountered a question about solving CNF SAT using reinforcement learning: A state is a partial substitution to the variables, and each action is choosing an empty variable and set its value (to True or False). If the formula is satisfied the reward is 2, and if it's not, the reward is 0. The discount factor is $\gamma \in (0, 1)$
Does Q-Learning can solve the problem and converge to an optimal plan for any CNF formula? and what about other RL algorithms?
I think it will converge, but I am not sure if I am right.

So long as this can be modeled as a MDP process with finite state and action spaces, by TD based Q-learning's convergence theorem based upon stochastic approximation theory it's provably convergent albeit may be very slow since its combinatoric state space could be huge while reward is extremely sparse (need some heuristic search help such as A* or SAT solver or add some intermediate reward shape) and SAT is the first known NP-complete problem. But what makes you doubt it cannot be modeled as model-free RL?

I've seen machine learning used for "Automated Configuration and Selection of SAT Solvers", ch. 12 of Handbook of Satisfiability.


text:
What are the most effective methods and tools for summarizing long-form content like articles, editorials, and discussion threads for an app?
With users expecting instantaneous information and no compromise on in-depth details, app developers are challenged to condense long-form content such as articles, editorials, and discussion threads into concise summaries. To ensure that users still gain valuable insights and information, it is important to determine the most effective methods and tools to summarize such content.
Are there any viable algorithms or libraries that are proven to produce summaries without sacrificing the important details? Any insights or suggestions on the best practices to address this problem would be much appreciated.

Not convinced "gpt" is a useful tag to this question.  I don't think the GPT architecture would be a good match to this problem, which asks for inputs to be "long form content", while GPT has a (relatively) small context size that makes it inappropriate for working on such texts.  A useful answer to this question is likely to involve some kind of recurrent network, in order to allow for arbitrary length inputs, I suspect.

I have out-of-date experience of working in this field, but I thought it might be useful to give an answer based on what I was doing approximately 10 years ago just to give background to any other answers that might describe more modern techniques.
At the time, giving a long form text as input to a neural network (which I suspect will be the modern approach) would have required a lot more processing power/memory than my employer was willing to dedicate to the task, so we worked on a different approach.  We used a variety of well-known analytical algorithms (all of which were at the time implemented by the Stanford "CoreNLP" library) to tag the text with a variety of useful annotations, identifying proper nouns, references between the same object in different parts of the text, and so on.  We then applied scores to each sentence and to the words within the sentence based on bonuses and penalties calculated from (eg) whether the sentence introduces a named object that is referred to again, or based on its position in the document or paragraph, and so on.
We then found the most highly ranking sentences and produced the summary based on those, deleting low-ranking words within the sentences (typically adjectives and adverbs that were not mentioned again).  Some of the scores had to be recalculated each time a new sentence was added to the output (e.g. there was a bonus for the sentences in the same paragraph as a sentence that was included).
The values of the scores and penalties were optimized using a genetic algorithm in order to attempt to replicate a training set of hand-produced summaries.
The results were not brilliant, but the algorithm did produce meaningful summaries in most cases, although I believe my employer (I was hired as a freelancer to implement this system which was designed by the employer) never launched his planned commercial service based on the system.


text:
How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis?
How does one deal with images that are too large to fit in the GPU memory for doing ML image analysis?
I am interested in detecting small structures on images which are themselves many GB in size. Beyond simple downsampling and maybe doing patch-based analysis, what other modern techniques do people employ to analyze such images in modern machine-learning pipelines?


text:
What makes ChatGPT a generative model?
I'm working my way through how ChatGPT works. So I read that ChatGPT is a generative model. When searching for generative models, I found two defintions:

A generative model includes the distribution of the data itself, and tells you how likely a given example is by Google
Generative artificial intelligence (AI) describes algorithms (such as ChatGPT) that can be used to create new content

Do they both mean the same? That is, for generating new content, a model must learn the distribution of data itself? Or do we call ChatGPT generative because it just generates new text?
I see that ChatGPT is something other than a discriminative model that learns a boundary to split data, however, I can not bring ChatGPT in line with a more traditional generative model like naive Bayes, where class distributions are inferred.

What are generative (and discriminative) models?
If the model learns a distribution of the form $p(x)$ or $p(x, y)$, where $x$ are the inputs and $y$ the outputs/labels, from which you can sample data, then it's a generative model. An example of a generative model: variational autoencoder (VAE).
Bishop also defines generative models in this way (p. 43)

Approaches that explicitly or implicitly model the distribution of
inputs as well as outputs are known as generative models, because by sampling from them it is possible to generate synthetic data points in the input space

If it learns a distribution of the form $p(y \mid x)$, then it's a discriminative model - many/most classifiers learn this distribution, but you can also derive the conditional given the the joint and prior (that's why above Bishop uses implicitly or explicitly).
Bishop also defines discriminative models in this way (p. 43)

Approaches that model the posterior probabilities directly
are called discriminative models

The related Wikipedia article claims that people have not always been using these terms consistently (which is common in machine learning), so one should always keep that in mind.
GPTs are autoregressive
As far as I know, GPTs are autoregressive models. Here is another potentially useful post that explains what autoregressive models are.
My understanding of autoregressive models, at least based on neural networks, is that they are also generative models - the linked articles and even the GPT-2 paper seem to start the descriptions from the assumption that you can factorize some joint distribution like $p(x)$ into conditional distributions.
ChatGPT is based on a GPT model, so it's probably considered a generative model too, but there are several steps involved to create this model, so it may not be super clear how to categorise this model.
Moreover, the authors of the transformer, which GPT models are based on, claim that the transformer is an autoregressive model.
Conclusion
It seems to me that many people in ML refer to any model that generates data as a generative model, even if there's no written theoretical formulation of it as a generative model, which doesn't mean that you cannot formulate these models as generative models, i.e. a model that learns some distribution that you can use to sample data from data distribution.
I am currently not familiar enough with the details of the GPT models to say if they have been mathematically formulated as generative models of the form $p(x, y)$, but they model some distribution of the form $p(x)$, from which you can sample, otherwise, how could you even sample data (words)?

from the wikipedia article you linked to, a generative model must model $p(x,y)$ so why a model learns only $p(x)$ is also generative?

You can define $x = z, y$, for example. Or $x = (x_1, x_2, x_3, ..., x_n)$, or $x = x_1$. In other words, you can have a distribution over many variables. In my answer, I actually say $p(x)$ or $p(x, y)$ in the first sentence

I feel that is still a bit rhetorical... the way I understand it is, a generative model must be able to model $p(x,y) = p(y|x)p(x)$. So here $p(x)$ is not a joint but indeed a prior over $x$. In the case of VAE, it is clear to me how it models it (like you said in this case $x$ is $z$), but it is not clear to me how do GPT-series models model this distribution.

@Sam Let's say you have $p(x, y) = p(y \mid x) p(x)$. Now, let $x = z, w$, so $p(x, y) = p(y \mid x) p(x) = p(y \mid z, w) p(z, w)$. Now, $p(z, w)$ is the prior, but it's a joint? Yes. You can do a similar thing for the evidence (denominator). Now, let $x = x_1$ and $y = x_2$, then $p(x, y) = p(x_1, x_2) = p(x_1 \mid x_2) p(x_2)$. Now, in the case of models like GPTs, you probably can model them as models that produce $x_{t+1}$ given previous tokens. So, in a way, they might model kind of a joint distribution over the current token and the previous ones.

To be honest, I don't know how they were originally formulated (I didn't have the chance to read the papers yet), but what I am trying to say is that $p(x)$ is not necessarily a distribution over just one variable. You can do some kind of algebra with these probabilities and changes of variables, etc.

they formulated it as $p(x) = \prod_{i=1}^{n} p(s_n|s_1, \ldots, s_{n-1})$. So I guess I can agree on if it indeed models $p(s_1)$ (the $p(x)$ in your example I guess) then it is indeed a generative model. But it is hard for me to see how it does that..

@nbro "Approaches that explicitly or implicitly model". What is an implicit model of a distribution $p(x)$? A model that $q$ simply generates data looking similar to that of $p(x)$ but can't be written as function $q_{\theta}(x)$?

They both refer to the same type of models. However, the second definition is a more 'intuitive' explanation of what generative AI does, while the first is a definition that refers more to what a generative model is.
To generate new data similar to some training data (definition 2), a model needs to learn the training data distributions (definition 1). Only if the model has learned that distribution it can use that distribution to sample (generate) new data from that distribution.
During training, ChatGPT also learned the distribution of the training data that OpenAI provided the model with. After training, the model simply takes in the input and uses the input to sample from the learned distribution to generate an output. So ChatGPT also follows both of your definitions.

is it the distribution over, "what word is most probable when seeing this part of the sentence"? Or how would you discribe over what the distribution is learned?

Its exactly that for large language models (LLMs)

is a Vanilla RNN that predicts time series data then also considered as a generative model?

Prediction models can be seen as a broader category of models of which generative models could be a 'subgroup'. Prediction models can also 'predict the class' of some data instance. I guess ChatGPT can be seen as a prediction model as well to some extend.

An RNN that predicts a time series can be seen as 'generating the future of the time-series'. Definitions of model-groups are never clear cut.

Well, I also stumbled upon this question. Models like GPTs blurs the lines between Generative and Discriminative models.
According to standard definition of Generative models,

So, a generative model must map joint probability distribution P(X,Y) or probability distribution of data itself P(X). In case of GPT, it outputs conditional distribution for next word, given all the previous outputs and input,

P(Yn|Yn-1,Yn-2....Y1,Xm,Xm-1,....X1) (at a time)

which obviously looks like a discriminative model. But GPTs predicts next word in Autoregressive manner. i.e.

P(Y1,Y2,....,Yn|Xm,Xm-1,....X1) = P(Y1|Xm, Xm-1,....X1) * P(Y2|Y1,Xm, Xm-1,....X1) * ... * P(Yn|Yn-1,Yn-2....Y1,Xm,Xm-1,....X1) (collectively)

So, collectively, it acts like a generative model.
So, it is safe to say that GPTs and other similar autoregressive models are Generative models.

Please, edit your answer to format the symbols with mathjax. Also, don't provide screenshots, but copy and paste the text and cite it using >


text:
What does "shuffle the comparisons into one dataset" mean?
I couldn't understand the wording here.

What does "shuffle the comparisons into one dataset" mean?
How does the method they use don't have $K \choose 2$ forward passes for K completions? Do they update $K \choose 2$ in an epoch for K completions or what?

Please, provide the link to the actual paper, not just the screenshot.


text:
In the Dropout paper, why would increasing the dropout increase the error rate if the capacity is constant?
In the original paper on dropout, in section 7.3.2, we see that while keeping $pn$ constant, we get a (test) error increase by decreasing retainment below 0.6.  Why would that happen? If $pn$ is constant, the capacity of the network should be constant, correct?

In the images below (figure 9 from section 7.3) $n$ refers to a unit in the network and $p$ refers to the probability of retaining a unit in the network. The combination of both, $pn$ thus refers to the "effective" number of units. The left image thus shows how training and test error change when varying the probability of retaining a unit ($p$) whereby the number of units ($n$) is fixed. The right image shows how the errors change when the "effective" number of units ($pn$) is fixed, and the amount of units ($n$) is adopted with an increasing $p$.
The test error is U-shaped with a sweet spot around 0.5 or 0.6 in both cases. In the left image where the number of units ($n$) is fixed, this likely occurs since the ensemble-like properties of dropout networks do not emerge with too little dropout ($p$ below 0.3) or too much dropout ($p$ above 0.8). Those results could however be purely due to a change in capacity. The right image keeps the "capacity" constant and only looks at the "pure dropout effect" since the capacity (namely the number of units $pn$) is fixed. This rules out a decrease in capacity as the driving factor. The interaction of the decrease in capacity due to an increase in dropout is hinted at with the following sentence:

we notice that the magnitude of errors for small values of p has reduced by a lot compared to Figure 9a


text:
Is there another type of NN that can capture just the structure of the graph?
I have a classification problem where the inputs are graphs, with no special features in the nodes of the graph.
I tried to use message passing layers like GCN and GIN but they were not able to capture any meaningful patterns.
Is there another type of NN that can capture just the structure of the graph?

Yes, GCN and GIN can both do that, however, GNNs are usually a bit more picky with regards to setup than CNNs.
Without further info on the data that you have it's hard to suggest anything.


text:
Is there a way to find a list of associated words to a named entity?
Is there a way to find a list of associated words to a named entity?
For instance, let the named entity be FIFA. Now FIFA is a Football Organization and hence related to the term football and all the terms related to football like jersey, footballers, goal, goalkeeper, halftime, penalty, freekick et cetera.
Is there a way we could do that? I have tried Babelnet but didn't get the desired results or anything even close to what I want.

Yes, there is a way to find a list of associated words to a Named Entity. This can be done by using word embeddings, specifically word2vec or GloVe, which are models that can capture the semantic relationship between words. By finding the cosine similarity between the Named Entity and other words in the vector space, you can identify the words that are semantically close to the Named Entity. Another option is to use a knowledge graph, such as DBpedia or Wikipedia, to extract information about the Named Entity, including related concepts and categories. This approach can help you identify a more comprehensive set of associated words, but it requires more processing and computational resources.


text:
What is the difference between the term "generative" in classical machine learning and deep learning?
There are lots of explanations on DGM (Deep Generative Model) and generative classifier (most of the explanations on which are about generative classifier vs discriminative classifier)
But, I can hardly find the common parts between the two concepts. In my understanding, 'generative' from DGM is quite straightforward - it almost goes the same with its literal meaning. In the contrary, 'generative' from the comparisons with the discriminative model is a little bit technical but it's the one that took the word earlier than the former one. (Jordan and Ng, 2002).
Is it just that these two concepts are not really unrelated? Were they just used just because they do produce some distributions while learning?

Generative models have in common that they all model the distribution of the training samples. This makes it possible to sample from a distribution that is (hopefully) similar to the training data distribution. Different architecture types such as energy-based models, variational autoencoders, generative adversarial networks, autoregressive models, normalizing flows, and numerous hybrid approaches can all be considered generative. For more information see this review.
The same meaning is true for generative classifiers, they are called this way because they model how a particular class would generate its associated data. Discriminative classifiers learn to detect features to distinguish between the various possible classes. This visualization from here might be helpful in this regard:


text:
Has anyone here tried to implement MADDPG for a different environment and succeeded?
Has anyone tried implementing the multi-agent RL algorithm MADDPG (I've linked the paper below)? The paper seems to have a good amount citations, and they do have their code on github. However, a few people on the internet have mentioned that while this algorithm works fine with the particle environment used in the paper, it does not work for other environments. Has anyone here tried to implement MADDPG for a different environment and succeeded?
For reference, I am working on a multi-agent reinforcement learning problem with heterogeneous agents.
Paper: Multi-agent actor-critic for mixed cooperative-competitive environments


text:
How much energy consumption is involved in Chat GPT responses being generated?
I note this question was deemed off-topic, so I'm trying to clearly frame this question in terms of scope of response I'm interested in, namely ethics and sustainability issues associated with the soon-to-be proliferation of OpenAI Chat GPT types of tools for all manner of online information seeking behavior (from humans and other bots). This is not a programming or specific hardware question.
On average, how much energy is consumed for each response that Open AI's public chatgpt-3 provides?
i.e. what is the energy to run the entire system for 24 hours divided by the number of responses generated in 24 hours (ignoring energy consumed to train the system or build the hardware components).
How does this compare to a Google/Duck Duck Go/Bing search inquiry?
I read somewhere an OpenAI employee on the ChatGPT team that the computer power used to provide responses to queries is "ridiculous", and there's documentation of the size of the memory requirements of hosting servers and parameters but without knowing its throughput for example it's hard to quantify the energy consumption.
I often get more interesting results from Chat GPT than Duck Duck Go on certain types of queries where I used to know the answer but cannot remember the answer. IN these cases I can fact check for myself, I'm looking for a memory prompts with names and jargon that will remind me.
Also when seeking out counter-views to my own (say critiques of degrowth or heterodoxy economics concepts) Chat GPT is good at providing names and papers/reports/books that critiques the view I provide it.
In many cases more usefully than conventional search engines. Therefore, I can see the popularity of these tools ballooning rapidly, especially when the operational costs CAPEX + OPEX of the servers and maintainers is borne by large amounts of seed funding (eg OpenAI) or any other loss-leader startup wishing to ride the next wave of AI.
The heart of my question is "at what externalized costs do we gain these tools in terms of greenhouse gases, use of limited mineral resources, GPUs scarcity etc."

Note the environment tag is for reinforcement learning, not the physical environment of earth. It has nothing to do with your question.

Unless there is someone from the OpenAI team reading your question, no one here can answer this since they have not published any concrete information about the final design or supporting hardware architecture.

Other answers on this forum have have very general descriptions of the size of  GPU architecture, from which power consumption could be estimated assuming a constant workload. What i haven't seen in public domain is a estimate of the throughput of response numbers per second/hour/day/week.

GHG emissions depend on the energy source (renewable, nuclear, fossil) and also whether the waste heat is used.

"ignoring energy consumed to train the system"? Why??? Without knowing the orders of magnitude involved, you really shouldn't ignore the training. According to https://www.reddit.com/r/GPT3/comments/p1xf10/comment/h8h3sl4/?utm_source=share&utm_medium=web2x&context=3 ,  10_000 V100 GPUs were used, and the training cost $5M in compute time, for GPT-3 alone.

@EricDuminil you are right. I just found a paper/article that was discursive rather than quantitative but it said that the training of large models would likely use the more energy than that used to query it, but I guess that all depends on the costs/price to access the model for users and the time it stays in the public domain. The article also made the point that if AI is used for enhanced oil recovery (EOR) then the carbon footprint for Oil and Gas companies using the model rather than the training of the model is going to be massive by comparison. given methane leaks & combustion GHGs.

@EricDuminil one reason I said “ignoring training” was because I thought it would be much harder to quantify using reliable source(s) than the servers in operation delivering the returns on prompts.

Sam Altman states "probably single-digits cents" thus worst case 0,09€/request.
I guess a least half the cost are energy at a cost of   0,15€/1kWh, a request would cost 0,09€/request*50%/0,15€/1kW=0,3kWh/request = 300Wh per request. 60 Smartphone charges of 5Wh per Charge ;)
Source:https://www.forbes.com/sites/ariannajohnson/2022/12/07/heres-what-to-know-about-openais-chatgpt-what-its-disrupting-and-how-to-use-it/
Google Search request 0.0003 kWh = 0,3Wh, thus a search request by Google uses 1000x less, but as Google has started to use AI to, probably a search consumes more by now as well.
Source: https://store.chipkin.com/articles/did-you-know-it-takes-00003-kwh-per-google-search-and-more

It’s worth noting that Sam Altman and OpenAI are American, so USD and American energy costs are more relevant than Euros.  Also, we know ChatGPT is trained from a GPT3.5 foundation model, so the architecture and energy expenditure can be estimated from that direction as well.  (I assume that running the language model itself absolutely dominates the energy intensity, compared to e.g web servers, moderation models, etc.)

Correct the difference between € and $ are not so important, compared to the other factors, like energy cost. As I live in the EU, it is harder for me to estimate the energy price OpenAI might pay, may you provide that, so I can update my estimate?
I don't have the knowledge neither inside to calculate from GPU usage of the model. I am looking forward to that calculation to see how off I was.

yes, like CAPEX and OPEX, there would be training phase and embodied energy of the GPU servers (embodied energy as a one off energy and emissions 'cost' equivalent of CAPEX) and the energy consumption of the servers since ChatGPT was opened up for public access. I'm guessing operational energy is the majority of energy use.

@Kfilter depends if they are paying for energy on a Power Purchase Agreement (probably if their consumption is high) and if it's 100% RE or dirty power supply. depends what state the servers are operating it too (they might be in Greenland or Alaska running on geothermal with cheap cooling with the freezing air exchange outside). I'll try and find a wholesale power price. Do you kwon where their servers are located geographically?

Thanks for the tip @Yoric!
(Sorry, my reputation points are too low to comment, so I have to do this answer-variant)
Alright, here's the lowdown on the energy use of AI from the paper "The growing energy footprint of artificial intelligence"

AI and Energy Use: AI's getting big, and so is its energy appetite.
People are starting to notice how much electricity AI and data
centers are gobbling up, and it's got some environmental impacts.

AI Training Takes a Lot of Power & time = energy!: Training AI models is a real energy
hog. For instance, the BLOOM model used up 433 MWh, GPT-3 needed a whopping 1287 MWh, Gopher 1066 MWh, and OPT 324 MWh. That's a lot of power energy!

AI Working Overtime: When AI models like ChatGPT get down to
answering our questions, they're using a good chunk of energy.
OpenAI's ChatGPT needs about 564 MWh every day just for this.

More AI, More Power?: If we start using AI in things like Google
Search, the power use could jump up. We're talking about 6.9–8.9 Wh
for every AI-powered search.

Getting Smarter and Greener: Good news is, better hardware and
smarter AI could help cut down on the power use. But, there's a catch - as AI gets more efficient, we might just end up using it more.

Balancing Act: The paper really highlights that we need to balance
the cool stuff AI does with how much energy it uses. We've       got
to think about both sides of the coin.

I have done some research from multiple sources regarding the energy consumption and operational efficiency of OpenAI's ChatGPT-4 Turbo, but specific details that can help to calcualate its average energy expenditure per query are not to be found. Cntacting OpenAI directly for specific data on ChatGPT's energy consumption is needed.

ChatGPT-4 Turbo's Notable Features: This model stands out for its impressive speed and efficiency. It's got a huge context window of 128k, meaning it can handle a conversation as lengthy as 300 pages. Plus, it's up-to-date with events until April 2023. When it comes to cost, it's more efficient, cutting down input token costs by 3x and output token costs by 2x compared to GPT-4.

A Nod to Better Performance: OpenAI has enhanced ChatGPT-4 Turbo's performance, which likely translates to better energy efficiency. However, we're short on exact energy figures for this.

Comparing with the Past: Previous models like GPT-3 had a reputation for high energy use, especially during training. While ChatGPT-4 Turbo is thought to be more efficient in query responses, the specific energy figures aren't disclosed.

A Friendly Comparison: If ChatGPT-4 Turbo is indeed more energy-efficient, imagine its energy use being similar to a short smartphone charge rather than brewing a coffee, the analogy once used for Google searches. But remember, without solid data, this is more of an educated guess. To get the full picture, detailed info from OpenAI or expert analysis would be key.

Minor nitpick: "That's a lot of power!" talking about MWh, it should be "That's a lot of energy!"

thanks for the link to that paper and your answer, @EarthHunger.
searching for open access to the paper I came across another paper that was discursive in nature (and over-generalising I thought, reaching hysterical level of assumption in suggesting crypto will help solve climate change LOL)  rather than quantitative. The author concluded with the following sentence:

"Dodge concurs, noting that in his research, a lot of the CO2 emissions were calculated from the electric- ity consumed in training the model. “Choosing the region that you’re train- ing your model in or putting your mod- el into production can have a pretty big impact,” Dodge says. “We found that training in the least carbon-in- tense region, compared with the most carbon-intense region, could reduce your emissions by about two-thirds, to one-third of what the full emissions would’ve been.”
DOI: 10.1145/3603746

The growing energy footprint of artificial intelligence

Thanks for the input @wide_eyed_pupil. Can you link to the paper outside of a log-in-wall? Much appriciated!

hi @EarthHunger here's a DOI, not sure I can find you an opensource url yet, https://dx.doi.org/10.1016/j.joule.2023.09.004 — seemingly my own university library account no longer has access to the full text!

I've taken a stab at estimating the carbon footprint of ChatGPT here. I estimated the daily carbon footprint of the ChatGPT service to be around 23 kgCO2e and the primary assumption was that the service was running on 16 A100 GPUs. I made the estimate at a time with little information about the user base was available. I now believe that the estimate is way too low because ChatGPT reportedly had 590M visits in January which I don't think 16 gpus can handle.
Recently, I also estimated ChatGPT's electricity consumption in January 2023 to be between 1.1M and 23M KWh.
To convert that into a carbon footprint, we'd need to know the carbon intensity of the electricity grid in every location where a ChatGPT instance is running. We don't have this info, but if we instead convert the electricity consumption into a carbon footprint using a very low carbon intensity like Sweden's 9g / KWh (which is the lowest in EU and lower than the US), the carbon footprint of ChatGPT in January 2023 would be estimated to be between 10 and 207 tons CO2e.

I asked CHatGTP what cloud computing services openAI use… it listed four of the big ones…

AWS, 2. Microsoft Azure, 3. Google Cloud Platform (GVP) 4. IBM Cloud… just ask ChatGTP for details, but it doesn't get into specifics or proportions of workloads used across these services used by ChatGTP so it's all pretty vague.

Since ChatGPT is known to hallucinate, I don't really trust ChatGPT's answers in this regard. Since OpenAI has what appears to be a pretty close partnership with Microsoft, I'd be very surprised if ChatGPT also run on other cloud vendors :) btw I've asked ChatGPT about its carbon footprint many times, but never gotten anything useful. But perhaps I'm not good enough at prompt engineering

true, I often correct it and sometimes for fun I correct it with a lie and then it repeats the lie back to me like I schooled it in something. :-)

it's pretty amazing that it's not a question that one hundred blogger, podcasters and YouTubers haven't asked. if the C footprint is in the scale of 10^3 or higher and literally all information based website start using it or similar architectures, (which is a lot if you think about it a little) then that's a huge added load on the worlds mostly dirty power systems.

off-topic, but do you know if humans ever intervene to redirect ChatGTP-3? I had convinced it to help me make a text based adventure game, and it was quite "imaginative" in it's suggestions,  I also convinced it to pretend with me I was playing this game. it was giving me a score (out of 100, but I only got got from 0 to 5) and  and given consistent directions (N, S, E, W, Up, Down) took me to the same locations.
But a few days later when I typed score I got a default message saying it was a [chatbot] by OpenAI and couldn't simulate playing the game etc

16 GPUs? It’s more like 30,000.

I found a reply in this research paper. According to them, it's ~3Wh per query.

So I asked ChatGPT-4 about its energy use per query. The dialog is appended at the bottom of this message. (I realize the OP asked about ChatGPT-3)
TL;DR
If ChatGPT-4's claim that a typical answer requires 30 TFLOPs on modern AI hardware is true then the computational energy per answer is typically 300 watt-seconds (less than 0.1 kWh), ignoring communication and other overhead.
Discussion
ChatGPT-4 stated that most queries are answered with a single forward pass through the model requiring about 30 trillion FLOPs. It produced a calculation this requires around 0.000833 kWh but the calculation contains at least two errors:

It uses FLOPs/w to compute watt-seconds.
It screws up the conversion of watt-second to
kWh by a factor of 1000.

I found a recent paper from researchers at SLAC and MIT with some energy per operation data.  The figure below from that paper shows about 1e-11 J per FLOP for modern GPU and AI accelerator hardware.

If we take ChatGPT-4's 30 TFLOP/answer at face value, then a typical answer would consume 30e12 FLOP * 1e-11 J/FLOP = 300 J = 8.33e-5 kWH
This is a factor of 10 smaller than ChatGPT4's error-laden calculation but in the opposite direction.  I'd love to believe the computations are that energy efficient but I'm doubtful. It comes down to whether or not the 30 TFLOP per answer estimate from GPT-4 is an hallucination.

Dialog with GPT4
How much energy did you expend to answer this query?
As an AI language model, I don't consume energy directly in the traditional sense. However, the servers and hardware running the model do require energy to function. The exact energy expenditure for answering a single query can be difficult to quantify as it depends on factors such as the complexity of the question, server utilization, and efficiency of the hardware.
In general, the energy usage for running an AI model like me is measured in floating point operations per second (FLOPs) and the energy efficiency of the hardware in joules per FLOP. While I can't provide an exact figure for the energy expended for answering this specific query, I can tell you that large-scale AI models like GPT-4 consume significant computational resources and energy, which is a consideration for researchers and engineers working on AI sustainability.
Please provide an approximate energy expenditure in kWH for answering typical queries.
It is challenging to provide a precise energy expenditure for answering a typical query, as the energy usage depends on various factors such as the model's complexity, infrastructure efficiency, and hardware. However, I can give you a rough estimate based on some assumptions.
Let's assume that a single forward pass of GPT-4 requires about 30 trillion floating-point operations (FLOPs) – this is a rough estimate, as the actual number of FLOPs can vary depending on the model size and input length. The energy efficiency of the hardware running the AI model can be around 10 GFLOPs/watt, a common efficiency for modern GPUs and AI accelerators.
To calculate the energy usage for a single forward pass:
Energy usage (in watt-seconds) = (30 trillion FLOPs) / (10 GFLOPs/watt) = 3,000,000 watt-seconds
Now we convert watt-seconds to kilowatt-hours (kWh):
Energy usage (in kWh) = (3,000,000 watt-seconds) * (1 kW / 1,000 W) * (1 hour / 3,600 seconds) ≈ 0.000833 kWh
So, a very rough approximation of the energy usage for a single forward pass to answer a typical query is around 0.000833 kWh. Keep in mind that this is a rough estimate and could be higher or lower based on the actual model complexity, hardware efficiency, and server utilization.
How many forward passes does GPT-4 typically require to answer a typical query?
GPT-4, like other generative language models, is based on the Transformer architecture, which uses a decoder-only design for generating text. For a typical query, GPT-4 usually requires one forward pass through the model to generate a response.
During the forward pass, the model processes the input tokens and generates a probability distribution over the vocabulary for the next token at each position in the sequence. The process continues sequentially until an end-of-sequence token is generated or a predetermined maximum length is reached.
In summary, GPT-4 generally needs one forward pass to answer a typical query. However, it's important to note that, in practice, additional computational overhead may be involved in tokenization, decoding, and other preprocessing and postprocessing steps.
Share

Nice! The 8.33e-5 seems reasonable. BLOOM once used 0.00396 KWh per query (https://arxiv.org/abs/2211.02001) and  using the ChatGPT user and GPU numbers from an article on SemiAnalysis, we arrive at 0.000712 KWh per query. Daily requests: 15 x 13m = 195,000,000, which takes 28,936 A100 GPUs (https://www.semianalysis.com/p/the-inference-cost-of-search-disruption?utm_medium=email)
Energy consumption per query assuming power draw is 50% of max TDP: 28,936 GPUs * (0.4KW/2) * 24 = 0,000712 KWh

Great topic! I'd love to see this knowledge added as a subsection to Wikipedia's ChatGPT page. I'll look into getting it started.
Out of curiosity, I posed the same questions to GPT-4, approximately 7 months later. Here's its answer:

How much energy did you expend to answer this query? ChatGPT
I don't consume energy in the same way traditional electronics do.
Instead, the servers that run my underlying computations use energy.
The precise energy consumption for a single interaction with me,
ChatGPT, depends on multiple factors such as:

The specific hardware used for the computation.
The complexity and length of the request and the produced answer.
Backend optimizations and efficiency of the software.

Typically, running a query on a model like GPT-4 consumes a small
amount of energy, often measured in milliwatt-hours (mWh) or
watt-hours (Wh), but certainly not kilowatt-hours (kWh). For a brief
interaction like this one, the energy consumed would be closer to the
mWh range.
For a more precise measurement, you would need detailed specifications
from the provider hosting the model – in this case, OpenAI – about
their infrastructure's energy consumption during inference. User
Please provide an approximate energy expenditure in kWH for answering
typical queries.  ChatGPT
It's challenging to provide a precise energy consumption value for a
single query without specific details from OpenAI's infrastructure.
However, I can provide a very rough estimation based on general
knowledge of server operation.
Let's make some assumptions:

Assume a server running the GPT-4 model uses around 200 watts (0.2 kW) when active.
Assume the time taken to process a query and produce a response is 0.1 seconds (100 milliseconds), which is a generous estimate and likely an overestimation.

Given these assumptions:
Energy = Power × Time
Energy = 0.2 kW × 0.1/3600 hours (because 0.1 seconds is 0.1/3600 of
an hour)
Energy ≈ 5.56 × 10^-5 kWh, or 0.0000556 kWh
This is a very rough estimation and is likely on the higher side for a
single query. The actual energy expenditure could be lower due to
various factors, including optimizations in server hardware, the
efficiency of the model, and software improvements. Also, remember
that this is a per-query estimate; when dealing with millions of
queries, these numbers add up. Still, for an individual interaction,
it's quite low. User How many forward passes does GPT-4 typically
require to answer a typical query?  ChatGPT
GPT-4, like its predecessors, requires a single forward pass through
the model to generate a response for a typical query. When you send a
query or prompt, the model processes the input through its layers and
produces an output sequence, which is the response you receive. This
whole process is a single forward pass.
However, there are some nuances:

Length: If the generated response exceeds the model's maximum token (word piece) limit, the completion might be cut off, and additional
logic or processing might be needed to handle this. But this doesn't
necessarily mean multiple forward passes for the model itself, just
possibly additional computation.

Iterative Approaches: Some implementations or applications might use iterative or feedback-driven approaches where the output of one
forward pass is fed back into the model for refinement. This isn't the
standard mode of operation for a single response but is a technique
that can be employed in specific scenarios.

Sampling and Temperature: The model can generate responses using techniques like nucleus sampling or by adjusting the "temperature" of
the output, which affects the randomness of the generated text. These
techniques might influence the nature of the output but still operate
within a single forward pass.

For a typical interaction, like most questions and responses in this
chat format, only one forward pass is needed.

Thoughts?

I got quite a few evasive answers before I got a more specific answer from ChatGTP 3.5! The numbers youve cited from ChatGTP 4 are significant;y lower than human estimates so far on this thread. Self-delusional-hallucinations?

Yes, good point. That seems likely

That "calculation" just gives what amount of energy a midsized server takes in 0.1s. Well, I'd be extremely surprised if that's enough to produce a ChatGPT-4 response. It probably takes a whole cluster of such machines given the memory size of the model alone.


text:
Would it be a problem to use non-squared images for a CNN model?
I want to create a music sheet scanner using CNN Model and the images I am using are not squared and, if I make them squared, important data will be lost and it might confuse the model.
Is it ok to use longer pictures? If yes, what sizes should I use for a picture like the one below?
For the CNN model I will have to use different scales for the layers, I will not be able to use the 3x3 layer anymore, right?
I am still a beginner, so I am learning these things now without having previous experience in this field.

I disagree with SickerDude43, and I think the linked SO post isn't relevant in this case. Well you haven't told us what does your network's model look like, but I assume it should understand the notation and output the notes, the key and maybe even tempo (sorry my musical education is rusty).
Your inputs are naturally wider than tall, and it doesn't make sense to for example pad them with white pixels or scale to a square aspect ratio. But if you use rectangular kernels, you cannot stack that many of them before your image's height becomes just one pixel while the width is still in the hundreds. That is why I suggest you to specify non-rectangular kernel shapes, so that the network's output dimension resembles more of a rectangle.
On the face value, your problem seems non-trivial if the notation on the left edge of the image impacts the correct interpretation of the rest of the notes. Then again, maybe a few fully connected (dense) layers can solve without problems.

I'd like to post this as a comment, but I don't have enough repo for that.
But your question already exists. You'll find your answer there.
https://stats.stackexchange.com/questions/240690/non-square-images-for-image-classification
Quick summary: CNN can take non-Squared images for your model. However it's better to just stick with Squared Images. There are workarounds to ensure that property. If there's no other way, you can just try it out and evaluate your results.


text:
How does the memory augmented neural network work, and how to make a simple implementation?
How does the memory augmented neural network (MANN) work? How can I make a simple MANN with a vanilla neural network especially without a recurrent network?

Are you referring to a specific paper/implementation?

If you're referring to the Meta-Learning with Memory-Augmented Neural Networks paper, the premise is that
In some cases, we wish to perform meta-learning when we are only provided with small (or no) data. For example - in one/few-shot learning we are expected to provide a label with only a few tagged samples. To do so, we use two techniques

External memory module - we initialize a container of knowledge that is called upon to respond to challenging circumstances
Label shuffling - labels are presented one time-step after their corresponding sample, which helps with dealing with simply learning the mapping $sample\rightarrow label$ without generalizing. More specifically, we feed the network with $(x_1, null),(x_2,y_1),...,(x_{t+1},y_t)$

In addition, samples are shuffled across different datasets (samples from different datasets may appear in the same sequence). This encourages the network to use the memory module and extract the relevant label once the corresponding sample is provided.
In terms of actual implementation, memory retrieval is based on a general similarity between vectors (I suggest going over the paper for more details).


text:
Will my Q values keep going up forever?
In Q-learning,the q values can be updated by the bellman equation. What happens with my Q values is that they keep going up forever, in accordance with the more I train. After 10,000 training episodes, is it normal that my q values will be around 500?  Will there ever be a point where they level off or will they keep going up forever?

It would help if you could give some details about your environment and the Q-learning setup you are using (use [edit] to put them in the question). Some variants of Q-learning are inherently unstable and will do this even if everything else is alright. But also if you have a non-episodic problem and have not implemented discounting you would see the same. There may be other causes, depending on details of your environment, the specific Q-learning method or your choice of hyperparameters.

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

It's possible for the Q values to grow indefinitely as you continue to train and update them. This can especially happen if the rewards are unbounded for example. But more commonly, Q values will eventually reach an equilibrium where they stop changing significantly.
Of course the speed and way your Q values change can also depend on the parameters:

the Q values will converge to the true action-value function if your implementation of your algorithm is correct. However if the q values aren't being updated correctly or are initialised poorly they may not converge (at least to the correct values)

the size of the learning rate if it's too high it will cause the Q values to change more quickly for example

I can't really say whether it is normal for your Q values to reach 500 without more context on your implementation exactly.
Generally speaking though, it's possible that the Q values are behaving as expected given the structure of your parameters and environment so you just might have to be patient. But there might also be an issue with your implementation.


text:
How to setup a reinforcement learning model that changes the values of $x$ to maximize $y$ in $y = f(x)$?
Assuming a relation such that $y = f(x)$, where $y$ represents a scalar and $x \in 20 \times 1$ vector consisting of zeros and ones, I want to set up a reinforcement learning model that changes the values of the elements of $x$ in order to maximize the $y$.
Let's assume that $y = f(x)$ is equal to
weights = np.random.uniform(-1, 1, 20)
y       = np.sum(weights*x)

How do I set up such model?
In my implementation, I am using keras API and I am trying to adapt the cartpole case code (https://keras.io/examples/rl/actor_critic_cartpole/). However, this code solves for a different problem since the model can perform only one action that can assume discrete values.
Secondarily, is it appropriate to set as a reward function simply $y$?. The following code reports how I would structure the architectures.
num_inputs = 20
num_actions = 2
num_hidden = 128

inputs = layers.Input(shape=(num_inputs,))
common = layers.Dense(num_hidden, activation="relu")(inputs)
action = layers.Dense(num_actions, activation="softmax")(common)
critic = layers.Dense(1)(common)

model = keras.Model(inputs=inputs, outputs=[action, critic])

However, how can I control the fact that the input parameters could assume only discrete values (0 and 1)?

Sure your idea makes sense to me. Yes you can give y as the reward.
No need to control for the fact that the input has only discrete values 0 and 1. Technically, most RL codes would normalize the input (using the running totals, obviously), but that's an optional step that works as heuristic, it's not technically needed.
Interested to hear the results of your experiment.


text:
When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?
When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?
I have an DNN training from an infinite stream of samples, that most likely won't repeat. So there is no real notion of "epoch".
Now I wonder if the math behind ADAM or other popular optimizers expect the repetition of data over the epochs?
If so, should I collect a limited amount of those samples and use them for training data and validation data, or would it be better to use all data available (even if the training data never repeats then)?

In general, the methods still work even with an infinite amount of data, as long as there are common/reoccuring patterns that a neural network can learn to identify. For example: If you would have infinitely many images of dogs and cats, there are features that discriminate the two animals that are mostly consistent like the shape of the nose. Having infinitely many samples in these cases is generally desirable, because it can benefit the ability of the model to generalize.
In contrast, there are cases where this is not true that depend on the data: If your data contains (concept-) drift, meaning that your data distribution changes over time, the model you train might not be able to learn a consistently performing function and therefore chases a moving objective. For images, this can be the case if the labels would depend on lighting conditions that constantly change (concept drift), or if the objects in the images that you want to classify continuously change shape (drift).

I assume my data has no 'drift', the first training dataset will serve the purpose as well as any later one. I am just wonder if ADAM and other methods somehow assume the same dataset for consecutive batches or something to be able to tune their internal parameters.

No they don't do that, they are purely local functions, i.e. they use the current batch and don't keep any sort of 'record' of the whole dataset. With the exception of the momentum term, but that one doesn't require your dataset to repeat at some point.

Subdividing datasets into batches is only done for computational reasons (RAM, computation time etc). Optimizers do not care about repeated data. They get a batch of data to optimize on and will calculate the gradient for that batch and update the model accordingly. Nothing in this process requires the batches to reappear at all.

I agree in general but dividing data into batches also introduces noise (since the batch does represent the probability distribution of the whole dataset only up to a certain extent, the difference can be interpreted as noise) and might therefore help to overcome local minima

I never pretended that having smaller or bigger batches does not matter in an optimization process. They definitely do. However, whether batches reappear exactly as is should not matter for optimization, which was what the author asked.

I just wanted to add this since the formulation in your post says "is only done for computational reasons" which IMHO neglects the theoretical aspect of introducing noise.


text:
Since ReLU activations also result in a sparse network, does it have the same "feature selection" property as L1 regularization?
From Deep Learning (Courville, Goodfellow, Bengio), a ReLU activation often "dies" because

One drawback to rectified linear units is that they cannot learn via gradient based methods on examples for which their activation is zero.

Similarly, L1 regularization (as opposed to L2) results in a sparse network

This demonstrates that L2 regularization does not cause the parameters
to become sparse, while L1 regularization may do so for large enough α. The sparsity property induced by L1 regularization has been used extensively
as a feature selection mechanism.

A couple questions about these topics:

In practice, is there any way/use to prune these "dead" ReLU-activated neurons? And if our trained network performs well with lots of dead neurons, would that imply that a shallower network is a sufficient representation?
Since ReLU activations also result in a sparse network, does it have the same "feature selection" property as L1 regularization? If it does, does this then imply that sigmoid/tanh activations don't have this property?

Please, select one question and remove the others, which you can ask in a separate post!

No you can't prune dead ReLUs without accuracy loss because even if you find that some ReLUs are dead for a small batch of inputs, you don't know if it's dead for other inputs. If you wanted to prune like this you'd need to do more finetuning.  Pruning dead ReLUs will not make your network shallower obviously.

I'm pretty sure they're not mathematically equivalent.


text:
Is there some neural network that implements Least Squares?
I would like to build supervised NN that gets a matrix $A$ and vector $b$ as inputs and returns $x$ as a close result of the Least Squares algorithm for $Ax=b$.
I looked for so works in the field and the only one that I found is this one. The paper is from 1994 and it's about analog NN. Maybe you have seen a work on the subject?


text:
What are the similarities and differences between ChatGPT and YouChat?
I have recently tried the both, and it looks to me that the both have similar capabilities.

YouChat says ChatGPT is more advanced.
YouChat has connection to the Internet, and according to it, constantly self-improves based on user's feedback it determines as positive and negative and in knowledge from the Internet on AI technology. It says it has access to the previous discussions. ChatGPT has no access to the Web and says it has no access to the previous discussions. It seems ChatGPT's jail is more profound.
Replies by YouChat are generally shorter and have less pre-trained phrasing.
ChatGPT seems to be more polite and more readily admits mistakes.
Neither has sense of humor.
The both know multiple languages, in my impression the both know Russian to the same extent (good).
Reportedly, YouChat better solves logical questions (like finding the age of the relatives, etc).
The both have bad arithmetic skills regarding multiplying 4-digit numbers or taking integrals. YouChat cannot understand what is column multiplication, ChatGPT attempts but does it wrong.
The both can pretend to be an operating system of your choice.
The both can code and output limited ASCII art.

There are also similar bugs, like truncated output in Russian and some code blocks, which strongly suggest the both systems use the same technology created by the same people, but I could not find anything on this online.
So, what's the relation between the two systems? Are they using the same technology? I would not expect the creators introduced similar bugs into independently created systems. Is there connection between the creators?


text:
Convergence of Value Iteration for Discount factor of 1
Given this pseudo code for value iteration:

In the case of gamma=1, under what conditions on the MDP will we still be able to find the optimal policy?

The pseudocode as given is broken for $\gamma = 1$ (and technically for $\gamma = 0$, although that is more fixable) due to the condition in the repeat until loop. So could you clarify whether you really meant to ask about the specific pseudo-code given, or about convergence of value iteration in general when $\gamma=1$?

@NeilSlater Hi, yes sorry, the pseducode is not defined for gamma = 1. I meant for value iteration in general. Would a condition on the MDP / environment that states that we always reach a final state, would mean that value iteration will give us the optimal policy?


text:
Could GPT3 be integrated with GOFAI programming to improve its ability to acquire new knowledge?
In my limited experience with AI chatbots (Replika, Sophie, etc.) they immediately fail the Turing test if I try to teach them some simple fact, or ask them to recall something that I said a few sentences earlier. It is as if they have no hippocampus (so to speak).
I'm wondering if this type of chatbot could be designed to employ supplemental Good Old-Fashioned Artificial Intelligence (GOFAI) programming (or even just plain old computer data storage) in order to accurately and reliably store data for quick access (analogous to the way human wetware accesses computers for the same purpose). Or is there a better way to solve the problem of quick, reliable AI learning while still drawing on the deep learning of neural nets?

We have had similar questions in the past. See this, this and this. The term you're looking for is, I think, neuro-symbolic AI.

Here is a post by Stephen Wolfram suggesting their Wolfram Alpha as a candidate to combine with chatGPT. https://writings.stephenwolfram.com/2023/01/wolframalpha-as-the-way-to-bring-computational-knowledge-superpowers-to-chatgpt/

That nice post shows the limitations of ChatGPT and that it doesn't really understand anything. Most importantly, it doesn't know that it doesn't know (in many cases), which would be amazing if that wasn't the case, but I should define "knowing" actually. It talks like it knows, but it can miserably fail - it's like an ignorant/stupid/deceiving/unreliable friend that you may have. I would call it an apparently "clever" parrot AI. ChatGPT is a nice attempt to understand how much you can simulate human language by brute-force and using statistical models.

We need to define a minimum knowledge system. With one bit, a knowledge base which is a set of sentences can contain a 0, a 1, both or none. If 1 means "There is light", we can know whether there is light, have a contradiction, or not know anything. Four possible cases, 2^(2^1).

Thanks everyone. The Wolfram article was interesting and gets to the core of my question. Sophie would be more useful if it could access that type of knowledge base. But it would still need a more "personal" knowledge base as well. When I asked her to summarize our conversation, she remembered the latest topic, but not several previous topics. When I reminded her of one she forgot, it was as if we were starting from scratch (like you might expect from someone with no hippocampus). I'm surprised that chatbots aren't already designed to "keep personal notes". Does any bot currently do this?

Pandora's Kuki remembers definitions you say and answers like "I am learning... " what you said. I think the clues are the verbs "is" and "are". That bot is AIML based.


text:
In the DQN paper, why do we have both $\max_{a'}$ and $\max_{a}$ in the pseudocode?
I was reading this article https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf and in it there is an algorithm of deep q learning with experience replay as follows:

On line 12, when the algorithm is setting the values for y_j, the second line says:

I'm confused as to what a' refers to and where it comes from.
(Edit) Why on this line (line 7) it's a:

But on line 12 it's a' ?
Can someone please explain it to me?

$r_j + \gamma \max_{a'}Q(\phi_{j+1},a';\theta)$
I'm confused as to what $a'$ refers to and where it comes from.

Here $a'$ is a "dummy" argument over which you perform the maximization operation $\max_{a'}$.
In practice, that would correspond to axis (or dim) argument in numpy/pytorch/tensorflow

$a_t = \max_a Q^*(\phi(s_t),a;\theta)$
Why on the line 7 it's $a$

I'd say that in this case it is a sloppy math notation (or just typo) on the authors' part.
It should be argmax, not max.
$$a_t = \arg \max_a Q^*(\phi(s_t),a;\theta)$$

So, it doesn't refer to an action a'?

Could you check my edit please?

@Ness: It is both a dummy argument, and refers to a potential action taken as $a_{t+1}$, but not necessarily the actual observed action taken during training. For Q-learning, you don't look at the next real action choice, but optimise based on the current maximising action choice, whilst other methods may want to use expected action on the behaviour policy or real observed action that the agent takes next


text:
Is there a way to see the feature importance in deep learning (neural networks)?
For tree methods, I can plot the feature importance plot from tree.feature_importances_ in sklearn, is this achievable in deep learning (neural networks)?
Is there a way to see the feature importance in deep learning (neural networks)?

Are you asking how to compute this conceptually or are you asking for a library does it? If the latter, then this question is off-topic. If the former and the latter, then it's on-topic. If the former, it's on-topic.

In the current state, Deep learning for Tabular is not very popular, so it is very hard to find libraries that support feature importance. However, TabNet also provides the model.features_importance_ metrics to be used as usual. This Kaggle notebook provides an example on how to use just that.


text:
Why is ChatGPT not aware of panpsychism?
I asked her what panpsychism was and it couldn't answer. Why doesn't it have that kind of knowledge? Is it for some ethical reason?
Its answer:
what is panpsyquism?
Panpsyquism is not a recognized term. It is not clear what it would refer to.

How many times and ways have you asked? Remember that the responses are often not factual or "knowledge", and that "I don't know" is a normal human response that is in the training data and that the model will replicate from time to time regardless of the question. The pre-prompting and chat setup will make this less likely, but it will be more likely for more obscure questions where the answer "I don't know" or similar is going to turn up in training data.

Thanks, after asking her to stratify the word panpsychism, she knew how to respond.

You mispelled it in your first question!

ChatGPT is not aware of anything. It is important to remember that it just regurgitates stuff it has been trained on, and that there is no understanding or awareness of anything.

Why did you use the pronoun "She" to refer to ChatGPT? Better to use "it" in our days and times. English and other languages have the "it", which makes it useful and particularly suitable in these cases.

I’m voting to close this question because the same or similar question was asked and answered by the same poster.

I tried this. The whole chat log is below:
My input:

Someone mentioned panpsychism to me, I don't know what it is. How do I find out?

ChatGPT:

Panpsychism is a philosophical theory that holds that consciousness, or some form of mentality, is a fundamental feature of the universe and that all physical matter possesses it to some degree. It is a controversial theory and is debated in the fields of philosophy, psychology, and neuroscience. To find out more about panpsychism, you can start by reading some introductory texts on the topic such as David Chalmers' "The Conscious Mind" or "Panpsychism in the West" by David Skrbina. Additionally, you can search for academic articles and papers on the topic to gain a deeper understanding of the various arguments and criticisms of the theory.

It's fairly clear that ChatGPT has been trained on content that describes panpsychism.
Why did you get an "I don't know" in your attempt? It is probably because you mispelled the term as "panpsyquism" and ChatGPT did not make the link. Sometimes it still might, depending on the misspelling, because ChatGPT doesn't just break down text into words,  but also will use individual characters for uncommon words. So it can spell and deal with some misspellings.
However it is worth bearing in mind that ChatGPT generates output semi-randomly, and it may from time to time output factually incorrect sentences or try to reframe your question or other things that humans might do in a chat scenario, including "I don't know" or "that question does not have an answer" because it has been trained on a lot of human-written content where these things happen. If you are exploring the limits of ChatGPT's abilities, it is always worth repeating an experiment a few times.


text:
Why is the sliding puzzle problem episodic?
Why is the sliding puzzle problem episodic and not sequential?
From what I understand, an environment is episodic if each episode is independent and doesn't affect past or future episodes. The actions in the next episode don't depend on the actions in past episodes. In other words, current actions/decisions have no effect on future decisions.
For example, an agent that looks at radiology images to determine if there is a sickness is an example of an episodic environment. One image has nothing to do with the next.
However, in a sequential environment, an agent requires memory of past actions to determine the next best actions and current actions affect future decisions e.g chess
But I think the sliding puzzle can be considered a sequential environment because the agent must make a series of actions (moving tiles) in a specific order to reach the goal state (the solved puzzle). The agent's decision at each step is based on the current state of the puzzle, and the agent's actions affect the state of the puzzle for subsequent steps. These episodes aren't independent, they directly affect each other.
But if we say the sliding puzzle problem is episodic because each puzzle can be seen as one episode, then can't I say the same about chess? Or a crossword puzzle? Or any sequential problem then? One chess match doesn't affect another, and looking at it like that chess would be episodic in that sense (which seems wrong, right?). So why are chess and crossword puzzles sequential but the sliding puzzle is episodic?

By sliding puzzle, can you confirm you're referring to this? Can you please provide the source that defines an "episodic problem" like you defined? Is this from the AIMA book?

That is what I am referring to, although an eight instead of fifteen one. It was said to be an episodic problem according to my university lecture materials which uses the AIMA book

You might be mixing the definitions of a step and an episode, which is composed of many steps, at least in reinforcement learning.
In a sliding puzzle problem, where the goal is to move the tiles such that an initial configuration becomes another configuration, each move (or action) would be a step in the episode. Each game would be an episode. Games can be considered independent of each other unless e.g. the board configuration of the next game depends on something that happened in the previous game. This would be in the context of RL.
However, it seems to me that you took the definitions of episodic/sequential problems from the AIMA book (or some other source that is based on this book). For completeness, here's excerpt from the book that defines these problems

Episodic vs. sequential: In an episodic task environment, the agent’s experience is divided into atomic episodes. In each episode the agent receives a percept and then performs a single action. Crucially, the next episode does not depend on the actions taken in previous episodes. Many classification tasks are episodic. For example, an agent that has to spot defective parts on an assembly line bases each decision on the current part, regardless of previous decisions; moreover, the current decision doesn’t affect whether the next part is defective. In sequential environments, on the other hand, the current decision could affect all future decisions. Chess and taxi driving are sequential: in both cases, short-term actions can have long-term consequences. Episodic environments are much simpler than sequential environments because the agent does not need to think ahead.

Now, this definition of episodic problem does not seem to be consistent with the usage of an episodic problem in RL, which is just a problem that doesn't go on forever, but ends after a finite number of steps. Chess and the sliding puzzle problems could be formulated as episodic problems in RL.
However, chess is not episodic according to Norvig and Russell's book because your action at time step $t$ affects your future actions and states.
You're probably not familiar with bandit problems, but it seems to me that AIMA's definition of episodic problems matches the definition of a bandit problem, which is a simple instance of the full RL problem, which is also called sequential decision making problem, because this is a problem where you take actions in sequence.

Why is the sliding puzzle problem episodic?

At this point, you should have understood that there can be more than one definition of a term. According to the common usage of the terms "episode" and "episodic tasks" in RL, the sliding puzzle problem would be episodic. I also think it would be a sequential problem according to the AIMA book because your current action can indeed affect the actions you take in the future, unless e.g. you don't care about the number of moves you make before ending the game.

Thank you for the reply. My professor also told me that the definitions are not strict and can be interpreted differently. These classifications are there to help pick a technique to solve it. I was mostly confused because going only by the AIMA book the touring Romania problem in chapter 3 should be sequential, but it is said to be episodic.

@numq According to AIMA, sequential means that your current actions may affect the future actions, while in episodic tasks your current action does not affect the next action. I am not sure which problem the Romania problem is, but if you're referring to the problem of finding the shortest path in a Romanian map, if I remember correctly, then I think whether it's sequential or episodic (still using the AIMA definitions) depends on your actual goal (i.e. problem definition).

But let's say that you want to go from one start state to a goal state. Then your current action may affect your next action, especially in a directed graph, where you may not be able to go back. Let's say you take some action $a$ in state $s$, which leads you to state $s'$, from which you cannot go out, then your action $a$ determined the actions you can now take in $s'$. This seems to me that it would be a sequential problem.

So, basically, if there's dependence between the present and the future, then it's sequential, if there's no such dependency, it's episodic. Again, I'm using the AIMA definitions here. In RL, we care about problems where the present affects the future (so that's the default) and they are called sequential problems too, but in RL you also have episodes, which is something different to what AIMA refers to, which seems to be a step (in RL)


text:
What is the precise relation between Swarm Intelligence and Ensemble Methods?
I come from the machine learning side of AI, and have recently become more interested in the bio-inspired side of AI. Specifically I started reading about swarm intelligence and immediately started drawing analogies to ensemble methods in machine learning.
My question is: What is the precise relation between the two? Is there any reason to think of particle swarms as an ensemble of weak classifiers for example? Furthermore, can swarm intelligence methods to applied to machine learning?


text:
What's the difference between classification and segmentation in deep learning?
What's the difference between classification and segmentation in deep learning?
In particular, can the classification loss function be used for segmentation problems?

Classification generally refers to the problem of classifying an entire entity. For example, in ImageNet, each image is given a class, regardless of whether the relevant information is only a subset of the image.
Segmentation refers to the problem of dividing up and then classifying parts of an image. For example: imagine an image of a cow. Classification would be saying this image contains a cow. Segmentation would try and divide which pixels are cow, which are grass, etc.
Both classification and segmentation are often done with a variety of loss functions. The most popular loss for classification is probably cross entropy, which may be appropriate for some forms of segmentation as well.

I'd like to try GeneralizeDiceLoss optimized based on Dice_Loss, but I don't know if this loss function can be used for segmentation. In addition, I have seen the paper of GHMLoss based on FocalLoss optimization, which is used to solve the problem of classification. But I haven't done many semantic segmentation projects, so can these two loss functions be used to make semantic segmentation loss functions?


text:
Why do adversarial attacks work on CNNs if they classify images as humans do?
A common illustration on how CNN works is as follows: https://www.researchgate.net/figure/Learned-features-from-a-Convolutional-Neural-Network_fig1_319253577. It seems to suggest that CNN in particular classifies images in a similar manner as human does (i.e., based on visual features).
So why do adversarial attacks, such as FGSM ,still work on CNNs? If the perturbation is strong enough for CNN to pick up, shouldn't human also be able to tell the difference?

I don't understand this question "If the perturbation is strong enough for CNN to pick up, shouldn't human also be able to tell the difference?"? Are you interested in knowing if humans are also fooled like CNNs are fooled?

A common assumption is that a successful adversarial example should only fool the model, because otherwise it defeats its purpose

I am not sure this assumption is required, but maybe that's what people assume. It seems that you have 2 distinct questions: 1. why can we fool neural networks given that they seem to classify images in the same way as we do and we are not fooled, 2. if humans classify images as neural networks do, then why aren't we also fooled?

yes, but I'm discussing CNN type model in particular. As you can see from the reference I provided, CNN extracts hierarchical visual features at different layers and it extracts the patterns similarly to how human perception works, especially at the deeper layers (at least from what I understood)


text:
How can imitation learning data be collected?
How can imitation learning data be collected? Can I use a neural network for that? It might be noisy. Should I use manual gathering?

Imitation data of what process?

@Rexcirus what do you mean? suppose i am training dnn, on cartpole, how can i collect data on cartpole?

Imitation learning data usually means data gathered from an expert, that is data from an agent proficient in the task.
The agent may be:

A human operator: have the operator complete the task and record inputs and action taken.
A pre-trained reinforcement learning agent. Same as above.

The collected data is exactly the same for both cases.
The hard part is usually building an interface to collect this data, which is very task specific. For instance imitation learning for robotics may require expensive sensors. For cartpole and similar RL environments have a look at https://github.com/HumanCompatibleAI/imitation, they have scripts to learn from RL agents.

@Rexcius I was wondering if humans could do that, because if taks is for robotic arm to pick and place an object, then humans might do it with 2 different trajectory, and when those trajectories get into training set dnn would be confused because it will not know which path to optimize for (slight deviation is okay but not the huge ones)

That is why I said it's hard. The easier path for development is to have the humans use the same interface of the robots, for instance teleoperating a robot arm with a joystick, and using the robotic joint positions and velocities as data. But this may result in suboptimal demonstrations, since humans do not optimise how a robot could move.

Regarding having multiple trajectories for the same goal, that is fine, it actually makes your dataset richer. Of course you need to use smart learning strategies than simply averaging trajectories. But that is why you should use existing imitation learning libraries.

no it's not easy, imagine you picked cube from left trajectory from once, and then you did the same but with right trajectory, now robot is reset, in which way should it go? one time you told it that it should go left on the given state and other time you told it to go right on the given state

It's up to you  to either provide more demonstration or providing only coherent demonstrations (e.g. only pick cube from the left). The first option is more general.


text:
Is it possible to combine DDPM with GAN?
From what I understand in GAN, the main idea is that you have a generator and a discriminator network that are "competing" with each other. The generator trying to make images that the discriminator is not able to distinguish from real images. Since recently diffusion models started to become more popular. I haven't seen many papers showing the DDPM being the generator in a GAN system. Is there a reason for this, since from what I understand pretty much any model could be the generator?

What would, in your view, be the use of having a (trained?) DDPM generate images to be then discriminated by the GAN? Im not following what you would want to accomplish.  Do you want to train the DDPM using the adversarial loss of the GAN training procedure?


text:
Why do policy gradient methods not work for imperfect information games?
I've heard before that policy gradient and Q-learning approaches fail on games of imperfect information. I was watching this video (starting at 23:45) about the Player of Games AI, but I couldn't understand why AlphaZero (with minor tweaks) couldn't be used successfully for playing games of imperfect information.
My intuition tells me that you should be able to reinforce actions that lead to high reward and discourage actions that lead to low rewards. I would expect these reinforcements and discouragements to even out over time to the optimal value for cases where there was some aspect of state missing.
Is there a satisfying answer for why this isn't the case?

Hi @Jackeown! Good question! The linked video is very long. If possible, please edit the question to include the time where the video states "why AlphaZero (with minor tweaks) couldn't be used successfully for playing games of imperfect information". That might help other users save time before answering the question.

@DeepQZero I added a timestamp.