text:
What does the approximate posterior on latent variables, $q_\phi(z|x)$, tend to when optimising VAE's
The ELBO objective is described as follows
$$ ELBO(\phi,\theta) = E_{q_\phi(z|x)}[log p_\theta (x|z)] - KL[q_\phi (z|x)||p(z)] $$
This form of ELBO includes a regularisation term  in the form of the KL divergence which drives $q_\phi(z|x) \rightarrow p(z)$ when optimising ELBO.
However we also have the overall expression for the loglikelihood which is defined as follows (proof provided here)
$$ p_\theta(x) = ELBO(\phi,\theta) + KL[q_\phi(z|x)||p_\theta(z|x)] $$
Rearranging the above equation as follows
$$  \max\limits_\phi ELBO(\phi,\theta) = \max\limits_\phi p_\theta(x) - KL[q_\phi(z|x)||p_\theta(z|x)]  $$
We can see that maximising ELBO w.r.t $\phi$ in this form causes $q_\phi(z|x) \rightarrow p_\theta(z|x)$
These two ways of describing how VAEs learn conflicts my understanding of what happens to the approximate distribution during training.
Is it simply just trying to match both the prior $p(z)$ and the posterior $p_\theta(z|x)$ or am I missing something

Practically, when optimizing VAE, you assume that prior $p(z)\sim N(0,1)$; i.e. the unit Gaussian distribution. However, in testime you sample z from $p(z|x)$; the encoder model. Why is that?
Let's go back to the start. We have a model $p_{\theta}(x)$ and the data $\{x_1, ..., x_N\}$. Solving the maximum log-likelihood problem, we have
\begin{equation}
\begin{split}
\theta &= argmax_{\theta}\frac{1}{N}\sum_i log p_{\theta}(x_i) \\
&= argmax_{\theta}\frac{1}{N}\sum_i \left( \int p_{\theta}(x_i|z)p(z)dz \right)
\end{split}
\end{equation}
which is intractable to calculate. So what to do now?
Here it comes the Variational Inference: "Use the expected log-likelihood instead."
\begin{equation}
\begin{split}
\theta &= argmax_{\theta}\sum_i E_{z \sim p_{\theta}(z|x)}\left[ logp_{\theta}(x,z) \right]
\end{split}
\end{equation}
We approximate $q(z) \approx p_{\theta}(z|x)$. Thus, we unfold $logp(x)$:
\begin{equation}
\begin{split}
logp(x) &= log \int p_{\theta}(x_i|z)p(z)dz \\
&= log \int p_{\theta}(x_i|z)p(z) \frac{q(z)}{q(z)}dz \\
&= log E_{z \sim q(z)}\left[ \frac{p_{\theta}(x|z)p(z)}{q(z)} \right] \\
&\geq E_{z \sim q(z)}\left[ log \frac{p_{\theta}(x|z)p(z)}{q(z)} \right]  \\
&= E_{z \sim q(z)}\left[ logp_{\theta}(x|z) + logp(z) \right] - E_{z \sim q(z)}\left[ logq(z)\right] \\
&= E_{z \sim q(z)}\left[ logp_{\theta}(x|z) + logp(z) \right] + H(q) \\
&= ELBO(p,q)
\end{split}
\end{equation}
where $H(q)$ is the entropy of q. As you highlighted we can also write
$$ ELBO(p,q) = logp(x) - D_{KL}\left( q(z) || p(z|x) \right) $$
This means that (a) maximizing $ELBO(p,q)$ w.r.t. to $q$ then KL-Divergence is minimized and (b) maximizing $ELBO(p,q)$ w.r.t to $p$ then the model is improved as the log-likelihood is improved. So this point of yours is true.
However, how can we actually train this model?
The answer is by using Amortized Variational Inference! Practically, we use 2 Neural Networks, $\phi$ and $\theta$, so that we have 2 models: $q_{\phi}(z|x)$ (encoder) and $p_{\theta}(x|z)$ (decoder). Thus, we replace $q(z)$ with $q_{\phi}(z|x)$ and $ELBO(p,q)$ with $ELBO(\theta,\phi)$ .
To fully answer your question, I should reach your first ELBO formula. I will unfold my first equation about ELBO:
\begin{equation}
\begin{split}
ELBO(\theta,\phi) &= E_{z \sim q_{\phi}(z)}\left[ logp_{\theta}(x|z)\right] + E_{z \sim q_{\phi}(z)}\left[ logp(z) \right] + H\left(q_{\phi}(z|x)\right) \\
&= E_{z \sim q_{\phi}(z)}\left[ logp_{\theta}(x|z)\right] - D_{KL}\left( q_{\phi}(z|x) || p_{\theta}(z) \right)
\end{split}
\end{equation}
Therefore, using the Amortized Variational Inference, we maximize the above objective (which is the same as the one above).

Amortized VI doesnt change the objective you use, they both hold, ultimately the aim of ELBO is to act as a lower bound for for the log likelihood, the proof for the first objective you show still holds if we use q(z|x) instead of q(z) meaning both objectives apply in amortized VI.

Of course. I clarified it on text above. Thanks for the feedback. I hope I answered your question above.

My question is what does the distribution q(z|x) tend to when training VAEs, i'm not sure how this answer addresses this.


text:
What does all the formula and pictures mean?
https://www.nature.com/articles/s41467-020-17419-7
I am a medical school graduate and I really want to learn AI/ML for computer-aided diagnosis.
I was building a symptom checker and I found the material. It clarifies the drawbacks of associative models which are performing differential diagnosis. And it suggests counterfactual(causal) approach to improve accuracy.
The thing is I couldn't understand what the formulas mean in the article, e.g.:
$$P(D \mid \mathcal{E}; \theta )$$
I really want to know what | and ; are doing here, what do they mean, etc.
I would really happy if someone can directly answer or just provide me some references to get general idea quickly.
Here comes the most tricy part...

This is a notation of a conditional probability, which means: The probability of $D$ given $\mathcal{E}$ with some parameters $\theta$. In other words, the likelihood of the event $D$ if event $\mathcal{E}$ happens. $\theta$ denotes parameters for the model (e.g., for neural network).

Thanks, @ArayKarjauv, I really want to know what the tau, sigma, and the division meant.

According to the provided article,
$$
\begin{equation} \tag{1}
P(D| {\mathcal{E}};\ \theta )
\end{equation}
$$
is a probability of disease $D$ given findings $\mathcal{E}$, and a model $\theta$ that is used to estimate this probability.

$D$ represents a disease or diseases, and findings $\mathcal{E}$ can
include symptoms, tests outcomes and relevant medical history.

The Sheffer stroke symbol (vertical bar) | in the conditional probability notation is read "given that", whereas the semicolon symbol ; tells that we use a model (or parameters for the model) to calculate this probability. For instance, we can define this probability as $P(D| {\mathcal{E}};\ \theta ) = M_{\theta}(\mathcal{E})$, where $M$ can be a nueral network with parameters $\theta$ that takes $\mathcal{E}$ as input and returns the probability of $D$.
As we read the article further, we see that Equation 1 is nothing more than the posterior probability from Bayes' theorem:
$$
\begin{equation} \tag{2}
P(D| {\mathcal{E}};\ \theta )=\frac{P({\mathcal{E}}| D;\ \theta )P(D;\ \theta )}{P({\mathcal{E}};\ \theta )}.
\end{equation}
$$
where $P({\mathcal{E}}| D;\ \theta )$ is a likelihood of findings $\mathcal{E}$ given that we have the disease $D$, $P(D;\ \theta )$ is the prior probability of the disease $D$, and $P({\mathcal{E}};\ \theta )$ is a likelihood of findings $\mathcal{E}$.
As the article suggests, Theorem 2 is related to the Noisy-OR model, which itself is a large area of research. I encourage you to read the references provided in the article to learn more about approaches used by authors. If you have further questions regarding this theorem, I suggest you to open another question.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Are PDDL and classical planners (e.g. FF) used in industry or consumer products?
I am learning classical planning techniques, and I know many examples of robots are in books. However, I am more curious about the real-world usage of these techniques.
Are PDDL and classical planners (e.g. FF) directly used in industry or consumer products? Or does every company or organization have its own representation of problems rather than directly using PDDL as representation language?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
The model learns well, but the validation decreases over time
I have trained a model for four days. I noticed a behaviour quite strange/unnatural.
During the training, the score and loss look like this:

However, when I see the validation score, I got:

It seems the model learning by heart at the beginning and not generalise well afterwards. Is this a natural behavior? Maybe it's really not normal and there must be some errors in the code or algorithm? I don't know what to think anymore. Can you help me? What is a good solution?

This post was closed as "needs more details or clarity", and I think that's correct. You're just showing us some plots, but you forgot to explain what problem you are trying to solve, which model you're training, how you're training, which learning algorithm you're using, what's your dataset, etc. You may want to edit your post to include all these details.

This is simply overfitting. The model is performing well on train data but bad on test (unseen) data; you can measure it also by noticing a huge difference between the training accuracy and the validation accuracy. Of course this is not a natural behavior, to solve this you need to apply some data or network modifications in order to avoid overfitting. Some techniques that might help you:

Reduce network complexity by removing some of the layers if it is complex
Use Dropout() inside your network
Apply Regularization
Check your classes distribution and make a train - dev/test split


text:
Is there a limitation to the amount of data that a genAI model could be trained upon?
My friend says that genAI would become more human like, and perhaps even smarter than humans if it were simply trained on more and more data. I say that this would overtrain the models, and we would perhaps lose the generative aspect.
I looked up on the internet, and it seems overtraining is actually a problem... but could it be said that there is a limitation to the amount of data we could feed to a model which would kill it's generativeness off completely?

theoretically speaking, your friend is correct, with theoretically meaning that you have a powerful enough universal function approximators, but for neural networks, it's been observed a phenomenon called double descent, where increasing the dataset size in certain cases, actually hurts the overall performance  https://arxiv.org/pdf/2303.14151.pdf

also, overtraining on the internet usually means training for more epochs, not on more data, leading to overfitting

I'll try to deconstruct your question and give you the most informative answer:

Is there a limitation to the amount of data that a genAI model could be trained upon?

In the way that this question is framed, the answer is clearly no. You can continue training on data indefinitely, given that you have the data and monetary resources to do so.
However, I do not think that this is the answer you are looking for. Let me attempt to rephrase the question to align with what you actually want to know:

Is there a threshold/point after which training a genAI model on more data is no longer beneficial towards the quality of the output?

This is a much more intricate question.

There is the very well-known problem of overfitting models on training data. Small models are generally trained on a training set, which is iterated over many times during training. If the training loss decreases, but the model does not generalize well (the validation loss increases), then we typically (not always) say that your model is overfitting. Training more would, generally, not be beneficiary for your model.
Large LLMs are, however, not trained by iterating over a dataset multiple times. As far as I know, the training sets are so enormous that each sample is only seen once. It is highly unlikely that training on more data will cause the model to overfit.
The post OpenAI made on GPT 4]1 state certain benchmarks over time. They show that these models do seem to slowly converge to a state where the benchmarks do not improve anymore. This can allude to two things, namely (1) the benchmark is completed optimally or (2) the model no longer learns. The plots seem to suggest the former, further reinforcing the second point I made.
However, many people have suggested that ChatGPT is getting worse, while being further trained. Whether this is true, I will not judge. Nevertheless, these models can get worse over time if the data on which it is trained is not of high quality. High-quality data has been shown to significantly matter in the rate of convergence of these models. If you train the model on lower quality data than it has been previously trained upon, your model can get worse.
What is the 'final state/optimal state' of these models? Will they be 'smarter than human' (whatever that may mean)? These models are restricted to the data that we feed it. They are able to construe sentences together based on the data is has been trained on. Whether that means they can be smarter than humans according to your definitions of intelligence is up to philosophers to debate.

Regarding "You can continue training on data indefinitely, given that you have the monetary resources to do so" - this is only true if the data exists, or you have truly large enough resources to create it. With the largest LLMs we may soon reach a "peak usable data" limit for text resources, where there simply is not enough text available  to scale in the same way naively - https://www.lesswrong.com/posts/Couhhp4pPHbbhJ2Mg/will-we-run-out-of-ml-data-evidence-from-projecting-dataset

Adapted my answer to reflect your comment. I was thinking more along the lines of 'you can simply iterate over your existing dataset' which would still constitute 'training'. But the framing of the question is ambiguous, depending on how you interpret 'limitation on the amount of data'. Is that new data, or just data in general. I interpreted it as just data in general, which you can keep training on given monetary resources. However, I can see how you can interpret it as new data, given the specific mention of 'amount'.

There is a proposed optimal data-to-paramaters ratio (20:1) for LLMs. Continuing to train on repeated data may not be effective. Separately to this question there is also the issue of "poisoning the well" in that the LLM owners may soon not be able to collect data knowing it was generated by humans. There is research showing LLMs re-training on their own output is not a good idea, either.


text:
Are "prompt engineering" and "prompt design" used as synonymous?
Are "prompt engineering" and "prompt design" used as synonymous / equivalent terms on the day to day communications (not research papers) in Artificial Intelligence community ? Do you simply say "prompt"?

I'm "following" questions about ChatGPT. I think that there are too many questions and online content in general that are not making the most appropriate use of terms like

ChatGPT
prompt engineering
prompt design

This makes hard to find helpful content.
ChatGPT
ChatGPT has being used as a common name like calling a code library "chatgpt" on questions about the OpenAI API, mostly of them specifically about the text completions end-point, some related to packages or libraries that use this end-point. Regarding this term, IMHO, it's clear that it's too early to consider that ChatGPT is a common name and people should be encouraged to avoid to use it this way in order to make their post clear about what they are talking about.
Prompt Engineering
So far I have read stuff that I was able to find without investing too much time like

Wikipedia article prompt engineering that presents prompt engineering as something very broad.

Best practices for prompt engineering with OpenAI API that focus on writing prompts.

As an engineer, I like what I understood about what is prompt engineering from the Wikipedia article, not the second as it looks to me that trivializes the term.
Some questions, including here, apparently use prompt engineering to refer to the writing prompts. While might not be wrong to use this term, it looks to me to be way too broad for such specific task.
Related:
The following answers use the term "prompt engineering" but they look to focus on writing prompts:

Answer to What causes ChatGPT to generate responses that refer to itself as a bot or LM? (from this site)
Answer to How to avoid word limit in ChatGPT in R? (from Stack Overflow)

Prompt design
As well that I have done in relation to prompt engineering, so far I have read stuff  like the OpenAI API documentation, i.e. Prompt design that focus on writing prompts.

From my experience from a research standpoint, we describe "prompt engineering" as "the process of exploiting the LLMs knowledge by providing domain-specific keywords in the input text, in order to achieve better performance". It is the same as using Google search: let's say when I search for How to do multiplications in Python, I have have (possibly) better answers by using How to do multiplications in Python site:stackoverflow.com, which only searches answers on SO.


text:
OpenAI ChatGPT gives a network error at long responds. How can I fix it?
When OpenAI's ChatGPT replies with a very long answer, it will return a network error. When you check the network console, the POST request will fail with a ERR_HTTP2_PROTOCOL_ERROR:

The time of the "crash" is around 1 minute.

Getting the same thing. Very frustrating. It even removes the already relevant first part of the answer. Pressing retry doesn't redo the same answer, but gives you a new variation - which if equally long, also terminates in a network error. It helped in some instances to restart the session and ask the same question again.

Unfortunately I can't write an answer because the question is closed, but what worked for me, when the response is stuck and returned half way, is just typing in "Show me the rest" and the bot just continues where it left off

@eja There was an update on Dec 15, 2022 that somehow makee it possible your  "solution". Considered the answers posted before the question were closed I think that it would not work before that update.

Ah, that makes sense then, thanks! In either case for reference and future readers - "Show me the rest" now works :)

I resolved it by asking it to show me the first 10-20 lines, then the next 10 and so on until the end. In my case those were lines of code so it worked for me.

can you share how you managed to made her split the response code, i tried multiples thing can't do it properly always network error. Thanks,

A workaround to this issue is quite simple;
When asking a question that will be lengthy, simply tell the AI to present it to you in waves, for example:
Write me 1000 words about World War 2, but only show me the first 250 words.
(250 seems to be a safe limit without the AI maxing out,) and it will do exactly as asked. Since the AI is trained to reaccess previous questions, simply ask "show me the next 250 words."

Danilo's answer works well, here is an example how to do it:

can you please show me a code example of webpack which compiles React,
CSS and Fonts and break your answer up every 10 lines

UPDATE: The following seems to work as well:

can you please show me a code example of webpack which compiles React
and CSS. if a 'network error' happens, please revert and resume your
answer

I've tried to resolve this with its own information and it seems right now we're stuck with this severe limitation.
"When I ask you a question; if the answer exceeds 20 lines, could you send me a separate message per 20 lines?"

I'm sorry, but I'm not able to separate my responses into multiple messages or lines. I'm a large language model trained by OpenAI to assist with a variety of tasks, such as answering questions, providing information, and offering suggestions. I don't have the ability to separate my responses into multiple messages or lines. I'm designed to provide responses in a continuous stream of text, and I do my best to provide accurate and helpful answers to the questions I'm asked. If you have any specific questions, I'll do my best to help. Just let me know.

"Can I restrict the amount of information you give in your answer to keep your answers short?"

I'm sorry, but I'm not able to display error messages or restrict the amount of information in my responses. I'm a large language model trained by OpenAI to assist with a variety of tasks, such as answering questions, providing information, and offering suggestions. I don't have the ability to display error messages or restrict the amount of information in my responses. My responses are based on the information and knowledge I've been trained on, and I do my best to provide accurate and helpful answers to the questions I'm asked. If you have any specific questions, I'll do my best to help. Just let me know, and I'll provide as much or as little information as you need.


text:
Advantage Actor Critic model implementation with Tensorflowjs
I am trying to implement an Actor Critic method that controls an RC car. For this I have implemented a simulated environment and actor critic tensorflowjs models.
My intention is to train a model to navigate an environment without colliding with various obstacles.
For this I have the following:
State(continuous):

the sensors distance(left, middle, right): [0..1,0..1,0..1]

Action(discrete):

4 possible actions(move forward, move back, turn left, turn right)

Reward(cumulative):

moving forward is encouraged
being close to an obstacle is penalized
colliding with an obstacle is penalized

The structure of the models:
buildActor() {
      const model = tf.sequential();
      model.add(tf.layers.inputLayer({inputShape: [this.stateSize],}));

      model.add(tf.layers.dense({
        units: parseInt(this.config.hiddenUnits),
        activation: 'relu',
        kernelInitializer: 'glorotUniform',
      }));

      model.add(tf.layers.dense({
        units: parseInt(this.config.hiddenUnits/2),
        activation: 'relu',
        kernelInitializer: 'glorotUniform',
      }));

      model.add(tf.layers.dense({
        units: this.actionSize,
        activation: 'softmax',
        kernelInitializer: 'glorotUniform',
      }));

      this.compile(model, this.actorLearningRate);

      return model;
    }

buildCritic() {
      const model = tf.sequential();

      model.add(tf.layers.inputLayer({inputShape: [this.stateSize],}));

      model.add(tf.layers.dense({
        units: parseInt(this.config.hiddenUnits),
        activation: 'relu',
        kernelInitializer: 'glorotUniform',
      }));

      model.add(tf.layers.dense({
        units: parseInt(this.config.hiddenUnits/2),
        activation: 'relu',
        kernelInitializer: 'glorotUniform',
      }));

      model.add(tf.layers.dense({
        units: this.valueSize,
        activation: 'linear',
        kernelInitializer: 'glorotUniform',
      }));

      this.compile(model, this.criticLearningRate);

      return model;
    }

The models are compiled with an adam optimized and huber loss:
compile(model, learningRate) {
      model.compile({
        optimizer: tf.train.adam(learningRate),
        loss: tf.losses.huberLoss,
      });
    }

Training:
trainModel(state, action, reward, nextState) {
      let advantages = new Array(this.actionSize).fill(0);

      let normalizedState = normalizer.normalizeFeatures(state);
      let tfState = tf.tensor2d(normalizedState, [1, state.length]);
      let normalizedNextState = normalizer.normalizeFeatures(nextState);
      let tfNextState = tf.tensor2d(normalizedNextState, [1, nextState.length]);

      let predictedCurrentStateValue = this.critic.predict(tfState).dataSync();
      let predictedNextStateValue = this.critic.predict(tfNextState).dataSync();

      let target = reward + this.discountFactor * predictedNextStateValue;
      let advantage = target - predictedCurrentStateValue;
      advantages[action] = advantage;
      // console.log(normalizedState, normalizedNextState, action, target, advantages);

      this.actor.fit(tfState, tf.tensor([advantages]), {
        epochs: 1,
      }).then(info => {
          this.latestActorLoss = info.history.loss[0];
          this.actorLosses.push(this.latestActorLoss);
        }
      );

      this.critic.fit(tfState, tf.tensor([target]), {
        epochs: 1,
      }).then(info => {
          this.latestCriticLoss = info.history.loss[0];
          this.criticLosses.push(this.latestCriticLoss);
        }
      );

      this.advantages.push(advantage);
      pushToEvolutionChart(this.epoch, this.latestActorLoss, this.latestCriticLoss, advantage);
      this.epoch++;
    }

You ca give the simulation a spin on https://sergiuionescu.github.io/esp32-auto-car/sym/sym.html .
I found that some behaviors are being picked up - the model learns to prioritize moving forward after a few episodes, but then hits the wall and it reprioritizes spinning - but seems to completely 'forget' that moving forward was ever prioritized.
I've been trying to follow https://keras.io/examples/rl/actor_critic_cartpole/ to a certain extent, but have not found an equivalent of the way back-propagation is handled there - GradientTape.
Is it possible to perform training similar to the Keras example in Tensorflowjs?
The theory i've went through on Actor Critic mentions that the Critic should estimate the reward yet to be obtain until the rest of the episode, but i am training the critic with:
reward + this.discountFactor * predictedNextStateValue where reward is the cumulative reward until the current step.
Should i keep track of a maximum total reward in previous episodes and subtract my reward from that instead?
When i am training the actor i am generating a zero filled advantages tensor:
let advantages = new Array(this.actionSize).fill(0);
let target = reward + this.discountFactor * predictedNextStateValue;
let advantage = target - predictedCurrentStateValue;
advantages[action] = advantage;

All other actions than the taken one will receive a 0 advantage. Could this discourage any previous actions the were proven beneficial?
Should i average out the advantages per state and action?
Thanks for having the patience to go trough all of this.

I've been trying to follow https://greentec.github.io/reinforcement-learning-fourth-en/ which has an in depth explanation for the A2C method.
It seems that the formula for calculating the advantage is in line with:
A(s,a)≃R+γV(s′)−V(s)
or advantage = reward + this.discountFactor * predictedNextStateValue - predictedCurrentStateValue

After tinkering a bit more with my experiment, i got it to consistently manifest the intended behavior after around 200 episodes.
Changes to the model itself were minimal: i replaced the loss function on the actor to tf.losses.softmaxCrossEntropy.
Some changes to the training environment seemed to have a significant impact and improved training:

Ending the episode after the reward reaches a minimum threshold - this prevents the models being polluted when the car was stock in a corner or flat against a wall.
Making sure that the reward used in training was inline with the action that produced that reward - the model training in my case is asynchronous to the physics of the environment - i am sampling the simulation and providing inputs, but the simulation is not interrupted by model related processing.


text:
Why is the margin attained with $\Phi=\left[2 x, 2 x^{2}\right]^{T}$ greater than the margin attained with $\Phi=\left[x, x^{2}\right]^{T}$?
I am trying to understand the solution to part 4 of problem 3 from the midterm exam 6.867 Machine learning: Mid-term exam (October 15, 2003).
For reproducibility, here is problem 3.

We consider here linear and non-linear support vector machines (SVM) of the form:
$$
\begin{equation}
\begin{aligned}
\min w_{1}^{2} / 2 & \text { subject to } y_{i}\left(w_{1} x_{i}+w_{0}\right)-1 \geq 0, \quad i=1, \ldots, n, \text { or } \\
\min \mathbf{w}^{T} \mathbf{w} / 2 & \text { subject to } y_{i}\left(\mathbf{w}^{T} \Phi_{i}+w_{0}\right)-1 \geq 0, \quad i=1, \ldots, n
\end{aligned}
\end{equation}
$$
where $\Phi_{i}$ is a feature vector constructed from the corresponding real-valued input $x_{i}$. We wish to compare the simple linear SVM classifier $\left(w_{1} x+w_{0}\right)$ and the non-linear classifier
$\left(\mathbf{w}^{T} \Phi+w_{0}\right)$, where $\Phi=\left[x, x^{2}\right]^{T}$.

Here is part 4 of the same problem.

In general, is the margin we would attain using scaled feature vectors
$\Phi=\left[2 x, 2 x^{2}\right]^{T}$

greater
equal
smaller
any of the above

The correct answer is the first (greater). Why is that the case?

@DuttaA would you clarify a bit more? I didn't get the point.


text:
Does there necessarily exist "dominated actions" in a MDP?
In a Markov Decision Process, is it possible that there exists no "dominated action"?
I define a dominated action the following way:
we say that $(s,a)$ is a dominated action, if $\forall \pi, a \notin \text{argmax}\ q^{\pi}(s,.)$, where $\pi$ are policies.
For now, I am only considering the cases where all q-values are distinct and therefore the max is always unique.
I also only consider the case of deterministic policies (mappings from state space to action space).
We can consider MDP in which each state has at least 2 actions available to get rid of the corner cases where there is only one possible policy.
I am struggling to find a counter-example or a proof.

I say that an action is "dominated" if, for all policies, it is not the argmax of the q-value of that policy. An action would be "dominated" if it is  never the best action you can perform, regardless of the policy considered.

Ok, now I think I better understand your question. So, an action can be the argmax for some policy but not be the argmax for another policy. In this second case, it would be a dominated action for that second policy, but not for all policies. But you define a dominated action one that is not the argmax but for all policies. Is this interpretation correct?

Yes, i define "domination" to be for all policies. A dominated action would therefore be a very bad choice, which will never occur as an improvement in an algorithm such as Policy Iteration!


text:
Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?
There are proofs for the universal approximation theorem with just 1 hidden layer.
The proof goes like this:

Create a "bump" function using 2 neurons.

Create (infinitely) many of these step functions with different angles in order to create a tower-like shape.

Decrease the step/radius to a very small value in order to approximate a cylinder. This is what I'm not convinced of

Using these cylinders one can approximate any shape. (At this point it's basically just a packing problem like this.

In this video, minute 42, the lecturer says

In the limit that's going to be a perfect cylinder. If the cylinder is small enough. It's gonna be a perfect cylinder. Right ? I have control over the radius.

Here are the slides.

Here is a pdf version from another university, so you do not have to watch the video.
Why am I not convinced?
I created a program to plot this, and even if I decrease the radius by orders of magnitude it still has the same shape.
Let's start with a simple tower of radius 0.1:

Now let's decrease the radius to 0.01:

Now, you might think that it gets close to a cylinder, but it just looks like it is approximating a perfect cylinder, because of the zoomed out effect.
Let's zoom in:

Let's decrease the radius to 0.0000001.

Still not a perfect cylinder. In fact, the "quality" of the cylinder is the same.
Python code to reproduce (requires NumPy and matplotlib): https://pastebin.com/CMXFXvNj.
So my questions are:
Q1 Is it true that we can get a perfect cylinder solely by decreasing the radius of the tower to 0 ?
Q2 If this true, why is there no difference when I plot it with different radii(0.1, vs 1e-7) ?
Both towers have the same shape
Clarification: What do I mean with: same shape ? Let's say we calculate the volume of an actual cylinder(Vc) with the same raius and height as our tower and divide it by the volume of the tower(Vt) .
Vc = Volume Cylinder
Vt = Volume Tower
ratio(r) = Vc/Vt
What this documents/lectures claim that is the ratio of these 2 volumes depends on the radius but in my view it's just constant.
So what they are saying is that: lim r -> 0 for ratio(r) = 1
But my experiments show that: lim r -> 0 for ratio(r) = const and don't depend on the radius at all.
Q3 Preface
An objection i got multiple times once from Dutta and once from D.W is that just decreasing the radious and plotting it isn't mathematical rigorous.
So let's assume in the limit of r=0 it's really a perfect cylinder.
One possible explanation for this would be that the limit is a special case and one can't approximate towards it
But if that is true this would imply that there is no use for it since it's impossible to have a radius of exactly zero. It would only be useful if we could get gradually closer to a perfect cylinder by decreasing the radius.
Q3 So why should we even care about this then ?
Further Clarifications
The original universal approximation theorem proof for single hidden layer neural networks was done by G. Cybenko. Then I think people tried to make some visual explations for it. I am NOT questioning the paper ! But i am questioning the visual explanation given in the linked lecutre/pdf (made by other people)

What's your question?  I don't see a question here.  A question usually ends with a "?".  You can't verify or disprove a statement about what happens "in the limit" by looking at finite instances.

There is a question mark in the title. Well if I decrease the width to a very small value the shape of the tower should change and get closer to a cylinder imo. If there doesn't happen any transformation of the shape there is no use for the universal approximation theorem. It's only useful if the tower gets closer to a cylinder when the step width decreases. If it's only a perfect cylinder when the step width is exactly zero, but a completly different shape when it's 0.0000000000001 above zero there is no use for it.

Let me try to put it into mathematical terms as good as i can. Let;s define ts as the shape of the tower. ts depends on the number of neurons used, and on the step width(w). Let's ignore the neurons for now. So we have ts(w) . So you are saying that ts(w=0) is a perfect cylinder ? But ts(w=1) = ts(w=0.1) = ts(w=0.01) = ts(w=0.001) are all the same shape. So ts would be a discontinous function. Well, if this is really true, then what's the point of the Universal Approximation for a single hidden layer ?

Could you [edit] the question to include a self-contained statement of the theorem, and then tell us what kind of answer you are looking for?  I don't really want to watch an external video to understand what you are asking; and we discourage questions that rely on us to click external links to understand what is being asked.  Slides and informal statements like "nearly perfect cylinder" are not a substitute for an actual mathematical exposition with a precise statement of the theorem.

It seems like the fact that you have to zoom in farther and farther to see the inaccuracy in the approximation is an indication that there is a sense in which it is becoming a closer approximation to a perfect cylinder.  No, it will never be a perfect cylinder for any finite value, but that is not a contradiction.  I am not sure what kind of answer you are looking for, though, or exactly what the question is.

When I zoom I can see that the shape of the tower is exactly the same. So it doesn't get any closer to a cylinder.  It's the same shape just smaller. There is no difference in the quality of the shape.

So it's not becoming a closer approximation of a cylinder. I'm aware that it will never become a perfect cylinder BUT i expect the shape to change. BUT it does NOT change it's exactly the same shape just smaller.

I'll update the question later to include some clarifications.

I suspect you've misunderstood the claim.  I'll look forward to the revision.

It's really unfortunate that you started a bounty. I didn't yet follow the discussions above, but it looks like you now have many different questions in this post. It's not a good thing to have multiple questions in the same post. You should focus on the simplest problem that you have that you need to solve before moving to the next and ask a question about that one. Everything else should be moved to the next post. Please, read this.

So, what I suggest that you do, although you already started a bounty, is to simplify as much as possible this post and ask the first simplest question that you need an answer to, while providing all the necessary context for us to be able to answer such a question. If you have further questions, you should ask them in separate posts. In any case, I understand that when someone is confused about something, often, we don't really know what the best first question to ask is, and that's why often we end up asking multiple questions (this also happened to me).

Hey @KoKia! Did this question ever get resolved? I realise that I have the exact same question. It seems to be me that the method of stacking lots of bumps doesn't yield a "circular tower", not matter how small you make the radius. Have a look at my question, which is the same as yours: https://datascience.stackexchange.com/questions/106905/constructing-circular-towers-to-show-that-single-hidden-layer-feedforward-neural

@Just_a_fool Hey I did a lot of work related to this topic. I haven't had yet time to fully write up everything. But i am planning to do that in the future. I am going to have a look at your question soon.

@KoKlA Amazing. Thank you for your willingness to share. I am looking forward to see what you have found out whenever its done!

The more I think about it the more convinced I am that the visual explanation from the linked lecture is wrong. But the good news is there are still some ways to get close to the cylinder but not before the activation of the last neuron but instead afterwards. I haven't done it with simgoid. But I tried with ReLu instead for now.
We can cut the tower at the very top (thanks to ReLu and a bias). The closer to the top we cut it the more it will be like a cylinder.
We can control the height of the tower with the weights.
First in 2d:

Unfortunately the closer we put this towers together the more they will start two influence each other.

But we can counter that with a negative tower between them.

Now in 3d:

This Answer is work in progress I will update it when i find out something new.

I think you misunderstood that part of the proof: you first need a limit on the number of neurons to get closer and closer to a cylinder. You are keeping them constant at 1000, thus indeed not getting any closer to the cylinder and exponential vanishing behavior.
Once you have the "epsilon-perfect" circles/cylinders, then you make them smaller and smaller, thus needing more and more copies of the "1-circle" setup.
My understanding is that this proof has those two numbers going to infinity: neurons-to-approximate-cylinder, and number-of-cylinders. You took into account the latter, but not the former.


text:
How to extract parameters from a text using AI/NLP
lets say I have three texts:

"make a heading that says hello word"
"make a heading of hello world"
"create heading consist of hello world"

How can I fetch those groups of words using AI which is referring to heading i.e hello world in this case. Which AI frameworks or libraries can do that?
in all examples heading is pointing to hello world (which i am referring as group of words). so basically i want those words which will be a part of heading or in other word there is a relationship between them. another example i can give is "I am watching Breaking bad" so there is a relationship between watching and breaking bad and i want to extract what are you watching.
What's the best approach? Do I have to train a model for that or there are some other techniques that can get it done?

Please explain more clearly.  What does 'fetch those groups of words' mean?  You already have words.  Why do you want more words?  What is it you are hoping AI is going to do for you?

in all examples heading is pointing to hello world (which i am referring as group of words). so basically i want those words which will be a part of heading or in other word there is a relationship between them. another example i can give is "I am watching Breaking bad" so there is a relationship between watching and breaking bad and i want to extract what are you watching. I hope I have made myself more clear. thanks for taking your time out.

Hello @ukasha, could you update your question to include the details that you just provided - this may help you to get more answers

Spacy is a machine learning library to (among others) analyze linguistic dependency and offers solutions for Named Entity Recognition (i.e. finding names or titles like "Breaking Bad" which refer to some entity). You can either use pretrained models or train your own models. So, this library might offer what you are looking for.

I think U are looking for PLSA
for PLSA either U find out those topics(catogeries) with EM or NNMF
Personally I recommend NNMF
or u can use LDA which is Bayesian version of PLSA
here is code for PLSA:
https://github.com/Man-ash/Probabilistic-Latent-semantic-analysis
which use NNMF
for EM method I code it by myself but i am not sure if it is right
https://stackoverflow.com/questions/65783880/problem-about-m-step-in-probabilistic-latent-semantic-analysis-with-code

You can also pose your problem as co-reference resolution. Try Huggingface's neuralcoref library.

I've noticed that you have provided many shorts answers. Sometimes, an answer doesn't have to be longer than 1-2 lines, but that's rarely the case. Note that I'm not saying that long answers are better. In fact, very long answers are also not good. I've seen so many answers that were bad only because they were so long and didn't focus on answering the question. So, please, before providing another answer, take a look at https://ai.stackexchange.com/help/how-to-answer. So, in this case, you could at least provide a link to what you're suggesting the OP to look into.


text:
Keras 1D CNN always predicts the same result even if accuracy is high on training set
The validation accuracy of my 1D CNN is stuck on 0.5 and that's because I'm always getting the same prediction out of a balanced data set. At the same time my training accuracy keeps increasing and the loss decreasing as intended.
Strangely, if I do model.evaluate() on my training set (that has close to 1 accuracy in the last epoch), the accuracy will also be 0.5. How can the accuracy here differ so much from the training accuracy of the last epoch? I've also tried with a batch size of 1 for both training and evaluating and the problem persists.
Well, I've been searching for different solutions for quite some time but still no luck. Possible problems I've already looked into:

My data set is properly balanced and shuffled;
My labels are correct;
Tried adding fully connected layers;
Tried adding/removing dropout from the fully connected layers;
Tried the same architecture, but with the last layer with 1 neuron and sigmoid activation;
Tried changing the learning rates (went down to 0.0001 but still the same problem).

Here's my code:
import pathlib
import numpy as np
import ipynb.fs.defs.preprocessDataset as preprocessDataset
import pickle
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras import Input
from tensorflow.keras.layers import Conv1D, BatchNormalization, Activation, MaxPooling1D, Flatten, Dropout, Dense
from tensorflow.keras.optimizers import SGD

main_folder = pathlib.Path.cwd().parent
datasetsFolder=f'{main_folder}\\datasets'
trainDataset = preprocessDataset.loadDataset('DatasetTime_Sg12p5_Ov75_Train',datasetsFolder)
testDataset = preprocessDataset.loadDataset('DatasetTime_Sg12p5_Ov75_Test',datasetsFolder)

X_train,Y_train,Names_train=trainDataset[0],trainDataset[1],trainDataset[2]
X_test,Y_test,Names_test=testDataset[0],testDataset[1],testDataset[2]

model = Sequential()

model.add(Input(shape=X_train.shape[1:]))

model.add(Conv1D(16, 61, strides=1, padding="same"))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling1D(2, strides=2, padding="valid"))

model.add(Conv1D(32, 3, strides=1, padding="same"))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling1D(2, strides=2, padding="valid"))

model.add(Conv1D(64, 3, strides=1, padding="same"))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling1D(2, strides=2, padding="valid"))

model.add(Conv1D(64, 3, strides=1, padding="same"))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling1D(2, strides=2, padding="valid"))

model.add(Conv1D(64, 3, strides=1, padding="same"))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dropout(0.5))

model.add(Dense(200))
model.add(Activation('relu'))

model.add(Dense(2))
model.add(Activation('softmax'))

opt = SGD(learning_rate=0.01)

model.compile(loss='binary_crossentropy',optimizer=opt,metrics=['accuracy'])

model.summary()

model.fit(X_train,Y_train,epochs=10,shuffle=False,validation_data=(X_test, Y_test))

model.evaluate(X_train,Y_train)

Here's model.fit():
model.fit(X_train,Y_train,epochs=10,shuffle=False,validation_data=(X_test, Y_test))

Epoch 1/10
914/914 [==============================] - 277s 300ms/step - loss: 0.6405 - accuracy: 0.6543 - val_loss: 7.9835 - val_accuracy: 0.5000
Epoch 2/10
914/914 [==============================] - 270s 295ms/step - loss: 0.3997 - accuracy: 0.8204 - val_loss: 19.8981 - val_accuracy: 0.5000
Epoch 3/10
914/914 [==============================] - 273s 298ms/step - loss: 0.2976 - accuracy: 0.8730 - val_loss: 1.9558 - val_accuracy: 0.5002
Epoch 4/10
914/914 [==============================] - 278s 304ms/step - loss: 0.2897 - accuracy: 0.8776 - val_loss: 20.2678 - val_accuracy: 0.5000
Epoch 5/10
914/914 [==============================] - 277s 303ms/step - loss: 0.2459 - accuracy: 0.8991 - val_loss: 5.4945 - val_accuracy: 0.5000
Epoch 6/10
914/914 [==============================] - 268s 294ms/step - loss: 0.2008 - accuracy: 0.9181 - val_loss: 32.4579 - val_accuracy: 0.5000
Epoch 7/10
914/914 [==============================] - 271s 297ms/step - loss: 0.1695 - accuracy: 0.9317 - val_loss: 14.9538 - val_accuracy: 0.5000
Epoch 8/10
914/914 [==============================] - 276s 302ms/step - loss: 0.1423 - accuracy: 0.9452 - val_loss: 1.4420 - val_accuracy: 0.4988
Epoch 9/10
914/914 [==============================] - 266s 291ms/step - loss: 0.1261 - accuracy: 0.9497 - val_loss: 4.3830 - val_accuracy: 0.5005
Epoch 10/10
914/914 [==============================] - 272s 297ms/step - loss: 0.1142 - accuracy: 0.9548 - val_loss: 1.6054 - val_accuracy: 0.5009

Here's model.evaluate():
model.evaluate(X_train,Y_train)

914/914 [==============================] - 35s 37ms/step - loss: 1.7588 - accuracy: 0.5009

Here's model.summary():
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv1d (Conv1D)              (None, 4096, 16)          992
_________________________________________________________________
batch_normalization (BatchNo (None, 4096, 16)          64
_________________________________________________________________
activation (Activation)      (None, 4096, 16)          0
_________________________________________________________________
max_pooling1d (MaxPooling1D) (None, 2048, 16)          0
_________________________________________________________________
conv1d_1 (Conv1D)            (None, 2048, 32)          1568
_________________________________________________________________
batch_normalization_1 (Batch (None, 2048, 32)          128
_________________________________________________________________
activation_1 (Activation)    (None, 2048, 32)          0
_________________________________________________________________
max_pooling1d_1 (MaxPooling1 (None, 1024, 32)          0
_________________________________________________________________
conv1d_2 (Conv1D)            (None, 1024, 64)          6208
_________________________________________________________________
batch_normalization_2 (Batch (None, 1024, 64)          256
_________________________________________________________________
activation_2 (Activation)    (None, 1024, 64)          0
_________________________________________________________________
max_pooling1d_2 (MaxPooling1 (None, 512, 64)           0
_________________________________________________________________
conv1d_3 (Conv1D)            (None, 512, 64)           12352
_________________________________________________________________
batch_normalization_3 (Batch (None, 512, 64)           256
_________________________________________________________________
activation_3 (Activation)    (None, 512, 64)           0
_________________________________________________________________
max_pooling1d_3 (MaxPooling1 (None, 256, 64)           0
_________________________________________________________________
conv1d_4 (Conv1D)            (None, 256, 64)           12352
_________________________________________________________________
batch_normalization_4 (Batch (None, 256, 64)           256
_________________________________________________________________
activation_4 (Activation)    (None, 256, 64)           0
_________________________________________________________________
flatten (Flatten)            (None, 16384)             0
_________________________________________________________________
dropout (Dropout)            (None, 16384)             0
_________________________________________________________________
dense (Dense)                (None, 200)               3277000
_________________________________________________________________
activation_5 (Activation)    (None, 200)               0
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 402
_________________________________________________________________
activation_6 (Activation)    (None, 2)                 0
=================================================================
Total params: 3,311,834
Trainable params: 3,311,354
Non-trainable params: 480
_________________________________________________________________

The solution for my problem was implementing Batch Renormalization: BatchNormalization(renorm=True). In addition normalizing the inputs helped a lot improving the overall performance of the neural network.


text:
How can I apply naive Bayes classifier for three classes (Positive, Negative and Neutral) in text data?
I found a naive Bayes classifier for positive sentiment or a negative sentiment Citius: A Naive-Bayes Strategy for Sentiment Analysis on English Tweets. But with most available datasets online, sentiments are classified into 3 types: positive, negative, and neutral.
How does the naive Bayes formula change for such cases? Or does it remain the same, and we only consider the positive and negative to calculate the log likelihoods-likelihoods?

post a sample of the data your trying to classify.   Will it have one output label or multiple?   Are you using CountVectorizer or TfIDF to tokenize your words?

naive bayes is a classifer combined in a pipeline with countvectorize in the preprocessing part of the pipeline.

Use TextBlob to get the text sentiment polarity and subjectivity. polarity is negative or positive sentiment and subjectivity is how factual the sentiment is.
my_valance=TextBlob(sentence)
df.loc[key,'sentiment_polarity']=my_valance.sentiment.polarity
df.loc[key,'sentiment_subjectivity']=my_valance.sentiment.subjectivity

you want to reverse engineer the nlp equation for sentiment? why? just use the existing api TextBlog.   you could get spacy nlp and then find the sentiment portion of the open source code and reverse engineer the calcualtions.   certain words are weight negative and certain words are weighted positive and certain words are facts.  Proximity to each other have a distance and a weight

The question is out of context then and should be removed.

How can I apply naive Bayes classifier for three classes (Positive, Negative and Neutral) in text data?   The Sentence implies a working solution which I provided using the TextBlog api  .   naive Bayes classifier is a classifier in the pipeline.   You must use textblob to get the sentiment and polarity of the sentence then countvectorize the text into tokens for vectorization to be used by the classifier.

the question about log-likelihood implies an misunderstanding of where sentiment is determined.   Sentiment is not part of the bayes classifer.  I clarified the content.


text:
Alpha Zero does not converge for Connect 6, a game with huge branching factor - why?
I have a problem with applying alpha zero self-play to a game (Connect 6) with a huge branching factor (30,000 on average).
I have implemented the MCTS as described but I found that during the MCTS simulations for the first move, because P(s, a) is so much smaller than Q(s, a), the MCTS search tree is extremely narrow (in fact, it only had 1 branch per level, and later I have added dirichlet noise and it changed the tree to have 2 child branches instead of 1, per level).
With some debugging I figured that after the first simulation, the last visited child would have a Q value which is on average 0.5, and all the rest of the children would have 0 for their Q values (because they haven't been explored) while their U are way smaller, about 1/60000 on average. So comparing all the Q + U on the 2nd and subsequent simulations would result in selecting the same node over and over again, building this narrow tree.
To make matter worse, because the first simulation built a very narrow tree, and we don't clear the statistics on the subsequent simulations for the next move, the end result is the simulations for the first move dictated the whole self-play for the next X moves or so (where X is number of simulations per move) - this is because the N values on this path is accumulated from the previous simulations from the prior moves. Imagine I run 800 simulations per move but the N value on the only child inherited from previous simulations is > 800 when I started the simulation for the 3rd move in the game.
This is a basically related to question 2 raised in this thread:
AlphaGo Zero: does $Q(s_t, a)$ dominate $U(s_t, a)$ in difficult game states?
I don't think the answer addressed the problem I am having here. Surely we are not comparing Q and U but when Q dominates U then we ended up building a narrow and deep tree and the accumulated N values are set on this single path, preventing the self-play to explore meaningful moves.
At the end of the game these N values on the root nodes of moves played ended up training the policy network, reinforcing these even more on the next episode of training.
What am I doing wrong here? Or is AlphaZero algorithm not well suited for games with branching factor in this order?
Note: as an experiment I have tried to use a variable C PUCT value, which is proportional to the number of legal moves on each game state. However this doesn't seem to help either.

Hello. Welcome to AI SE. I want to let you know that you can use latex/mathjax on this site, so I suggest that you edit your post to format the math symbols properly with them.

I don't know if my experience can be of any help to anyone, but I got this exact scenario because I unintentionally initialize v (or q?) != 0 for newly encountered state when building the Monte-Carlo tree. Basically then the search never considers certain states/branches ever again because the agent thinks it can get better results in the other branches.

Yes @Nuclear03020704 I believe this is what I ran into. In my case I was initialising Q as 0 but I need it set as 0.5 rather, because my value head has an activation function that returns [0, 1] than [-1, 1]. So it depends on what your representation is for a draw game evaluation :)

There are a few things you can do to address this
(1) Setting Q correctly on unexplored nodes

With some debugging I figured that after the first simulation, the last visited child would have a Q value which is on average 0.5, and all the rest of the children would have 0 for their Q values

This already points out a bug, as the rest of the children should start with the "average Q-value". Remember that Q is just our guess for the EV of an action. Your comment says that you set it to "0.5", but this still isn't quite right. You'll want to modify this by initializing it to the "average value" of a "random move" in that position.
For Chess and Go, this is quite close to 0 from most positions, since most valid moves are terrible. For your game, it's possible that this value is higher, if most moves are "reasonable". But, it should still be set to the "Average value of a random move in that particular position", not simply 0.5.
When your network is very weak, its choice will have a value of approximately "the average value of a random move", so by setting the initial Q correctly, it will cause the difference in Q between two different moves to be very small, letting the "U" term dominate initial exploration. To be precise, the difference in Q will tell you How much better you current path is, over just picking some other random move, which is exactly what you want the difference in Q to be.
See Lc0's "FPU / First Play Urgency" configuration, which sets unvisited node's initial value to be 0.44 less than the parents' evaluation.

Theoretically, if you have high confidence in your NN, you can lower FPU for increased efficiency.

(2) Increase $C_{puct}$ with respect to the number of paths down that edge.
In the face of extreme depth down a particular path, $C_{puct}$ should begin to increase to at least "double-check" the moves we've been ignoring thusfar. AlphaZero already does this:

With their numbers, $C_{puct}$ starts at 2.5, but rises to 2.8 when Depth = 20k, and 2.97 when Depth = 40k. ($2.5 + \log(2)$, and $2.5 + \log(3)$, respectively). You'll have to adjust this manually, often in relation to your simulations-per-move.
(3) Increase Dirichlet Noise
I'm not sure what $\alpha$ you used, but based on AlphaZero, $\alpha = 10/60000 =$ 1.66e-4 is what you should be aiming for, if you want to match AlphaZero's configuration of 10/AverageValidMoves.

Thanks for the detailed answer. Regarding this statement: "For Chess and Go, this is quite close to 0 from most positions, since most valid moves are terrible." - In this does the 0 represent a loss or draw? Because in many sources the range of Q is [-1, 1]  than [0, 1]. It just happens that in my implementation I used [0, 1] from the value head. If I initialise Q to 0 or close to 0 it is almost saying an unexplored move is estimated to be a loss, and this will give significant weightings to the explored moves during selection when the network is still very weak.

@javaPhobic It would represent a loss. The LeelaZero team generally found that your odds of winning drops about 44% when you make a random move. So 50/50 game goes to 6%/94%. And a 94% chance of winning drops to a 50% chance of winning.

It is important that for LeelaChess, FPU is still > 50%, when you start off with a 98% chance of winning. You don't want to blindly set it, you want it to be relative to the parent EV. For Chess/Go, it's still harsh though. But for 60,000 moves, it's likely the case that a lot of them are reasonable, if your NN has to pick the "one magic move" out of 60k, it'll take a lot of Dirichlet noise to figure that out lol. This is definitely a "hyperparameter" you'll want to tune a lot, lower FPU values will more aggressively use the NN.

For Go, this was set to "0" for all positions, since in Go, even if you have a 90% chance of winning, most random moves are terrible. Only a small set of the dozens upon dozens of valid moves are reasonable.


text:
PPO2: Intuition behind Gumbel Softmax and Exploration?
I'm trying to understand the logic behind the magic of using the gumbel distribution for action sampling inside the PPO2 algorithm.
This code snippet implements the action sampling, taken from here:
def sample(self):
    u = tf.random_uniform(tf.shape(self.logits), dtype=self.logits.dtype)
    return tf.argmax(self.logits - tf.log(-tf.log(u)), axis=-1)

I've understood that is a mathematical trick to be able to backprop over the action sampling in case of categorical variables.

But why can't I just put a softmax layer on top of the logits and sample according to the given probabilities? Why do we need u?

Tere is still the argmax which is not differential. How can backprob work?

Does u allows exploration? Imagine that at the beginning of the learning process, Pi holds small similar values (nothing is learned so far). In this case the action sampling does not always choose the maximum value in Pi because of logits-tf.log(-tf.log(u)).
In the further course of the training, larger values arise in Pi, so that the maximum value is also taken more often in the action sampling? But doesn't this mean that the whole process of action sampling is extremely dependent on the value range of the current policy?


text:
Reinforcement learning and Graph Neural Networks: Entropy drops to zero
I am currently working on an experiment to link reinforcement learning with graph neural networks.
This is my architecture:
Feature Extraction with GCN:

there is a fully meshed topology with 23 nodes. Therefore there are 23*22=506 edges.
the original feature vector comprises 43 features that range from about -1 to 1.
First, a neuronal network f takes calculates a vector per edge, given the source node and target node features.
After we have calculated 506 edge vectors, function u aggregates the results from f per node (aggregation over 22 edges)
A function g takes the original target feature vector and concatenates the aggregated results from u. Finally, the output dimension of g determines the new feature vector size for each node.
At last, the function agg decides which information is returned from the feature extraction, e.g. just flatten the 23xg_output_dim feature vectors or building the average

After that:

The output of the feature extractor is passed to the OpenAi Baseline PPO2 Implementation. The frameworks adds a flatten layer to the output and maps it to 19 action values and 1 value-function value.

I have made some observations in the experiments and do not manage to explain them. Hyperparameter are: an output dimension for f and g of 512. U=sum, aggr=flatten. A tanh activation is applied on the outputs of f  and g. For PPO2: lr=0.000343, stepsize=512.
This gives me  the following weight matrices:
<tf.Variable 'ppo2_model/pi/f_w_0:0' shape=(86, 512) dtype=float32_ref>
<tf.Variable 'ppo2_model/pi/g_w:0' shape=(555, 512) dtype=float32_ref>
<tf.Variable 'ppo2_model/pi/w:0' shape=(11776, 19) dtype=float32_ref>
<tf.Variable 'ppo2_model/vf/w:0' shape=(11776, 1) dtype=float32_ref>

The following problem occurs. Normally you wait for the entropy in the PPO2 to decrease during the training, because the algorithm learns which actions lead to more reward.
With the described hyperparameters, the entropy drops abruptly to 0 within 100 update steps and stays zero even after >15.000 updates (=150M steps in the game). This means that the same action is always selected.
What I found out: the problem is that by making the sum over 22 edges, very large values are created (maximum 22*1 and 22*-1). The values are then given to the function g and thus ends up in the saturation region of the tanh. As a result, the new features of the 23 nodes contain many 1's and -1's. Because we flatten, the weighted sum of 11776 input neurons flows into each of the 19 action neurons, resulting in very large values in the policy. An action is then calculated from the policy with the following formula:
u = tf.random_uniform(tf.shape(logits), dtype=logits.dtype)
action = tf.argmax(logits - tf.log(-tf.log(u)), axis=-1),

Most of the time tf.log(-tf.log(u) gives sommething between 2 and -2 (in my opinion). This means that as soon as a very large value appears in the policy, the corresponding action is always selected and not the second or third most probable one, which might lead to more exploration.
What I don't understand 1): As soon as negative reward occurs, shouldn't the likelihood decrease again, so that in the end I choose other actions again?
I did some experiments with relu and elu activations:
These are the value histogram of the output of g after the using relu, tanh and elu:

These are the value histograms of the policy, when using relu, tanh and elu:

Histogram over resulting actions:
What I don't understand 2:Using Relu you see that in the first steps in the policy were large values, but then the model learns to reduce the range, which is why in this example also the entropy does not drop. Why does this not work when using tanh or elu?

We have found out 3 things with which the problem does not occur or is delayed. These are my assumptions. Are the correct in your opinion?:

using smaller output dimension of f and g, like 6 or using aggr=mean -> For each of the 19 action neurons, less input neurons are averaged -> smaller values in the policy --> more exploration
Using u=mean and not sum, averages the outputs of f, therefore the aggregated values are not only 1 and -1
Smaller learning rate -> Making the weights too big, increases the chance of the 19 action values to be  big. If there is no negativ reward, there is no need for the algorithm to make the weights smaller.

I know this is a lot of information, so I would be grateful for any small tip.!

Hello. Could you please put your main specific question in the title? "... Issue with entropy" is not really a question but an announcement of some very general problem.

Sorry for nitpicking a little, but that title can still be improved. It would be better if you asked the question in the form "Given situation X, why is the entropy dropping to zero"? You don't need to have "RL and GNNs" in the title, because they are already part of the tags.

Please, share what is goal in your neural network and what is your input data. It will be helpful if you put structure of neural network as well.


text:
How should we regularize an LSTM model?
There are five parameters from an LSTM layer for regularization if I am correct.
To deal with overfitting, I would start with

reducing the layers
reducing the hidden units
Applying dropout or regularizers.

There are kernel_regularizer, recurrent_regularizer, bias_regularizer, activity_regularizer, dropout and recurrent_dropout.
They have their definitions on the Keras's website, but can anyone share more experiences on how to reduce overfitting?
And how are these five parameters used? For example, which parameters are most frequently used and what kind of value should be input? ?

This pre-print Tikhonov Regularization for Long Short-Term Memory Networks could be useful: you may be already able to implement this in Keras. This paper Recurrent Neural Network Regularization says that dropout does not work well in LSTMs and they suggest how to apply dropout to LSTMs so that it is effective. There's also this paper Noisin: Unbiased Regularization for Recurrent Neural Networks.

In addition to that, you have here a very similar post and this article could be useful too. Here is apparently the paper that introduced "recurrent dropout". There's also this paper that could be useful.

You can probably find more related/relevant papers on the web. If I had more time, I would go through them and provide a formal answer, but not now. Feel free to read them and provide a formal answer to your own question!

One LSTM layer should be enough unless you have lots of data. The same thing goes for the number of nodes in the layer. Start small first so 5 to 10 nodes and increment it until the performance is reasonable.
Once you have a model working you can apply regularization if you think it will improve performance by reducing overfitting of the training data. You can check this by looking at the learning curves or compring the error on the validation and test sets.
In my experiments I've used the L1 and L2 regularizers along with dropout. These can all be mixed together in fact using both L1 and L2 at the same time is called the ElasticNet.
I tend to apply the regularizers on the kernel_regularizer because this affects the weights for the inputs. Basically feature selection.
The value for the L1 and L2 can start with the default (for tensorflow) of 0.01 and change it as you see fit or read what other research papers have done.
Dropout can start at 0.1 then increment it until there is no performance gain. This is basically a percentage so 0.1 would remove about 10% of your nodes.
Finding the best regularizer is the same as any other hyperparameter optimization which is mostly trial and error.

If what is mentioned above, that is probably in the context of lstm networks. I would suggest using the keras tuner bayesian optimizer and making the l1 or l2 number a parameter of the kernel space. This way you find the optimal values, and its a great way to hypertune. Just keep in mind, the greater the range of parameters, or kernel if i am not wrong, the higher computer power you need.
from tensorflow import keras
import keras_tuner as kt

def model1(hp):
  model=Sequential()
  model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20),
                              dropout=hp.Float('droput',min_value=0.15, max_value=0.99, step=0.05),
                              recurrent_dropout=hp.Float('redroput',min_value=0.05, max_value=0.99, step=0.05),
                              activation='relu',
                              return_sequences=True,
                              input_shape=(30,1)))
  Attention()
  model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20),
                              dropout=hp.Float('droput',min_value=0.15, max_value=0.99, step=0.05),
                              activation='relu',return_sequences=True))
  Attention()
  model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20), activation='relu'))
  model.add(keras.layers.Dense(1))

  model.compile(loss='mean_squared_error',optimizer=tf.keras.optimizers.Adam(hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4, 1e-7, 1e-10])))
  return model

bayesian_opt_tuner = kt.BayesianOptimization(
    model1,
    objective='val_loss',
    max_trials=200,
    executions_per_trial=1,
    project_name='timeseries_bayes_opt_POC',
    overwrite=True,)

xval=X_test
bayesian_opt_tuner.search(x=X_train ,y=X_train,
             epochs=300,
             #validation_data=(xval ,xval),
             validation_split=0.95,
             validation_steps=30,
             steps_per_epoch=30,
             callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_loss',
                              patience=4,
                              verbose=1,
                              restore_best_weights=True),
                        tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss',
                                   factor=0.1,
                                   patience= 2,
                                   verbose=1,
                                   min_delta=1e-5,
                                   mode='min')]
             )
This is where the magic happens. Something I composed myself. If interested holla

Regularization is trying to discourage complex information from being learned so we want to eliminate the model from actually learning to memorize the training data. We don't want to learn like very specific pinpoints of the training data that don't generalize well to test data.
Dropout, the idea of drop out is that during training we randomly set some of the activations of the hidden neurons to zero with some probability say 0.5. This idea is extremely powerful because it allows the network to lower its capacity, it also makes it such that the network can't build these memorization channels through the network where it tries to just remember the data because on every iteration 50% of that data is going to be wiped out so it's going to be forced to not only generalize better but it's going to be forced to have multiple channels through the network and build a more robust representation of its prediction. We repeat this on every iteration so on the first iteration we dropped out one 50% of the nodes on the next iteration we can drop out a different randomly sampled 50% which may include some of the previously sampled nodes as well and this will allow the network to generalize better to new test data.
Early stopping, when the network is improving its performance during training there comes a point where the training data starts to diverge from the testing data, at some point the network is going to start to do better on its training data than its testing data, what this means is basically that the network is starting to memorize some of the training data and that's what you don't want so what we can do is we can identify this inflection point where the test data starts to increase and diverge from the training data so we can stop the network early and make sure that our test accuracy is as minimum as possible.

The question was not 1. "What is regularization?" or 2. "How does dropout work" Instead, the questions were 1. "And how are these five parameters used?", 2. "For example, which parameters are most frequently used and what kind of value should be input?" (I believe this second question was the main question). Of course, the other question was assumed to be in the context of LSTMs and not in general. Your question only addresses "how to usually reduce over-fitting in general", but that wasn't really the question. As far as I understand, the question is clearly in the context of LSTMs.

How to apply these methods in LSTM?


text:
Why don't those developing AI Deepfake detectors use two detectors at once so as to catch deepfakes in one or the other?
Why don't those developing AI Deepfake detectors use two differently trained detectors at once that way if the Deepfake was trained to fool one of the detectors the other would catch it and vice-versa?
To be clear this is really a question of can deepfakes be made to fool multiple high-accuracy detectors at the same time. And if so then how many can they fool before they become human detectable from noticeable noise?
I've heard of papers where they injected a certain noise into their deepfake videos which allows them to fool a given detector (https://arxiv.org/abs/2009.09213, https://delaat.net/rp/2019-2020/p74/report.pdf), so I thought well if they simply used two high-accuracy detectors then any pattern of noise used to fool one detector would interfere with the pattern of noise used to fool the other detector.

Because it is possible to fool many different models at once.
See table 2 in this paper, for an example using adversarial perturbations: https://arxiv.org/pdf/1610.08401.pdf
That being said, there is no reason to think that using two detectors at once will not increase chance to detect deepfakes. It will just not resolve the problem completely.


text:
Are there any approaches to AGI that will definitely not work?
Is there empirical evidence that some approaches to achieving AGI will definitely not work? For the purposes of the question the system should at least be able to learn and solve novel problems.
Some possible approaches:

A Prolog program
A program in a traditional procedural language such as C++ that doesn't directly modify its own code
A program that evolves genetically in response to selection pressures in some constructed artificial environment
An artificial neural net
A program that stores its internal knowledge only in the form of a natural human language such as English, French, etc (which might give it desirable properties for introspection)
A program that stores its internal knowledge only in the form of a symbolic language which can be processed unambiguously by logical rules

To answer your second question, you could look at this and this (in particular, the computational theory of mind). So, I would clarify that your main question is "Are there any approaches to AGI that will definitely not work?" and actually focus on that interesting question (and remove the last one, which is different, and partially answered in the linked posts), which can also be improved, such as by asking "Is there any empirical evidence that a certain approach cannot be used to implement an AGI?".

Thanks for the reference to the CTOM, I will read further.

i would vote for ANN to do AGI, it's real biology

datdinhquoc, I would agree intuitively that ANN is the most likely approach though I don't have any strong evidence for thinking so.

@persiflage Of course, most people nowadays would say "neural networks" (not to your question, but to the question "Which technique is the most promising to create AGI?"). However, yesterday, I was watching a recent video where 1 AI researcher and 1 cognitive scientist were discussing similar topics, and one of them said that he would be surprised if we could create an AGI only out of neural networks. He pointed out that we are mainly symbolic. We think in terms of symbols and concepts ("cat", "house", etc.). Neural networks are sub-symbolic (they do non-linear regression).

The cognitive scientist, on the other hand, pointed out that we really need the ability to "learn how to learn" and we really need the neural network (or whatever) to be able to solve new problems without having really been programmed/trained to do so explicitly. Nowadays, most neural networks are actually programmed/trained to solve a specific task, and they would completely fail in all other tasks. This is absolutely no good. Unfortunately, neural networks trained with back-propagation actually may inherently suffer from this issue.

That's why I also think that neural networks alone are probably not the path to AGI.

Very interesting question. Assuming that the programming languages used are powerful enough (say Turing Complete), all of the above actually should lead to an AGI. The difference is in how efficiently they can do it, both in term of number of computations required and in the length of the program.
So the question could be rephrased as: which approach cannot lead to an AGI using less than X resources and being shorter than Y characters?
The second question is basically asking the Kolmogorov complexity of the AGI in that language, which is uncomputable. Since we cannot find the shortest program, I don't think we can make conclusions about the maximum program efficiency either.
In summary I don't see a way to rule out any of those approaches (but I would be very happy to be proven wrong).


text:
Is my proof of equation 0.6 in the book "Reinforcement Learning: Theory and Algorithms" correct?
In Sham Kakade's Reinforcement Learning: Theory and Algorithms, this equation (page 17) is used preceding the proof of performance difference lemma.

I am attempting to prove equation 0.6. Here is my current attempt:
\begin{align*}
        \mathbb{E}_{\tau \sim \rho^\pi}\left[\sum\limits_{t=0}^\infty \gamma^t f(s_t,a_t)\right] &= \sum\limits_{t=0}^\infty \gamma^t \mathbb{E}_{\tau \sim \rho^\pi} [f(s_t,a_t)]\\
        &= \sum\limits_{t=0}^\infty \gamma^t \mathbb{E}_{s_t, a_t} [f(s_t,a_t)]\\
        &= \sum\limits_{t=0}^\infty \gamma^t \sum\limits_{s, a} \mathbb{P}(s_t = s, a_t = a) f(s,a)\\
        &= \sum\limits_{t=0}^\infty \gamma^t \sum\limits_{s} \mathbb{P}(s_t = s) \sum\limits_{a}\pi(a_t = a|s_t = s) f(s,a)\\
        &= \frac{1 - \gamma}{1 - \gamma}\sum\limits_{t=0}^\infty \gamma^t \sum\limits_{s} \mathbb{P}(s_t = s) \mathbb{E}_{a \sim \pi(s)} [f(s,a)]\\
        &= \frac{1}{1 - \gamma} \sum\limits_{s} (1-\gamma) \sum\limits_{t=0}^\infty \gamma^t \mathbb{P}(s_t = s) \mathbb{E}_{a \sim \pi(s)} [f(s,a)]\\
        &=\frac{1}{(1-\gamma)} \mathbb{E}_{s \sim d^\pi}\left[\mathbb{E}_{a \sim \pi(s)}\left[f(s,a)\right]\right] \\
\end{align*}
Is the swapping of expectation and summation in this way allowed (given that the series converges)?
Note that this is not the proof of the performance difference lemma, but just an attempt to show equation 0.6, which is used but not proved in the book.

Hi, late question, but why is the seond equalitys valid in your proof? Specifically why we can switch from trajectory to state-acton notation

The expectation of a sum is equal to the sum of the expectation this just follows from the linearity property of expectations
$$ \begin{aligned} E[\sum_{t} f(s_t,a_t)] &= \sum_{\tau} p(\tau)\left(\sum_t f(s_t,a_t)\right)
 \\ &= \sum_\tau\sum_{t}p(\tau)f(s_t,a_t)
 \\ &= \sum_t\sum_\tau p(\tau)f(s_{t,\tau},a_{t,\tau})
 \\ &= \sum_tE[f(s_t,a_t)]
\end{aligned} $$
note that in the penultimate line I swapped the sums around which can be understood as looking at all trajectories for a single time-step instead of looking at time-steps within  a single trajectory (the second expression). I append subscript for variables in the penultimate line for clarity since they also depend on which trajectory they were drawn from.

As mentioned by nboro in the comments, the linearity of property only holds for infinite sums if
$$\sum_{i=0}^\infty E[|X|]< \infty  \quad or \quad E\left[\sum_{i=0}^{\infty}|X|\right]. <\infty$$
More details can be found here. Using this theory alongside knowledge that since we know that value functions in RL are always bounded (in continuing tasks discounting factors are introduced) then we can say there exists an $M$ such that
$$ |f(s_{t,\tau},a_{t,\tau})| \leq M \qquad \forall t,\tau $$
Then
$$ \begin{aligned}\sum_t\sum_\tau \lambda^t p(\tau)|f(s_{t,\tau},a_{t,\tau})| &\leq \sum_t\sum_\tau \lambda^t p(\tau)M
\\ &=M\sum_t\lambda^t\sum p(\tau)
\\ &=M\sum\lambda^t
\\ &< \infty
\end{aligned}$$
As for the rest of the proof is there anything else you need clarity on? it looks good to me. From that step to the next step where you change distribution from $p(\tau)$ to $p(s,a)$ is just because of marginalisation but I think you got that.

This may explain the first step of the derivation, but what about the other steps? Moreover, maybe you should clarify if or when the linearity of the expectation also applies when you have an infinite sum, as it is the case. Maybe take a look at https://math.stackexchange.com/q/575974/168764.


text:
Reasoning behind performance improvement with hopfield networks
In the paper Hopfield networks is all you need, the authors mention that their modern Hopfield network layers are a good replacement for pooling, GRU, LSTM, and attention layers, and tend to outperform them in various tasks.
I understand that they show that the layers can store an exponential amount of vectors, but that should still be worse than attention layers that can focus parts of an arbitrary length input sequence.
Also, in their paper, they briefly allude to Neural Turing Machine and related memory augmentation architectures, but do not comment on the comparison between them.
Has someone studied how these layers help improve the performance over pooling and attention layers, and is there any comparison between replacing layers with Hopfield layers vs augmenting networks with external memory like Neural Turing Machines?
Edit 29 Jan 2020
I believe my intuition that attention mechanism should outperform hopfield layers was wrong, as I was comparing the hopfield layer that uses an input vector for query $R (\approx Q)$ and stored patterns $Y$ for both Key $K$ and Values $V$. In this case my assumption was that hopfield layer would be limited by its storage capacity while attention mechanism does not have such constraints.
However the authors do mention that the input $Y$ may be modified to ingest two extra input vectors for Key and Value. I believe in this case it would perform hopfield network mapping instead of attention and I do not know how the 2 compare.

The paper that you're mentioning "Hopfield networks is all you need" is from 2020 (so very recent) and is apparently only a pre-print (i.e. it's not yet been published in any journal or conference proceedings). There's this blog post and the code. Have you already looked at them? Also, why are you claiming this "but that should still be worse than attention layers that can focus parts of an arbitrary length input sequence."? Can you clarify that? Please, edit your post to clarify that.

Moreover, please, put your main specific question in the title, otherwise, this post looks like you're asking us to review a 95-papes paper, which is of course off-topic here, as it's too broad.

@nbro I understand that the question is broader than anyone (and I) would like it to be, but since my question is if someone knows the reasoning behind the performance claims of the network, how would you re-phrase it?
I am updating my question regarding my intuition of attention vs hopfield as I realized an error in my assumption from previous reading.

I suggest that you re-write your post as if there was no edit because I don't think that anyone is already writing an answer to your question(s). While reviewing your post, try to be more specific and now that you have a clearer idea of what you want to ask, ask it explicitly. Moreover, if they are showing the results of the experiments that show that their model outperforms other models, what else do you want to get as an answer?

For example, when you ask "Has someone studied how these layers help improve the performance over pooling and attention layers", but, in the paper, aren't they showing, with experiments, that this is the case? Or are they just claiming that? It may be a good idea to quote the parts of the paper where they claim (without proving) what you want to be proved.

Will try to formulate my understanding of the ideas in this paper, mention my own concerns that I see are relevant to your question, and see if we can identify any confusions along the way that might clarify the issue
On eq(6) of the relevant blog post, they identify the weight matrix of a discrete, binary Hopfield network as
$$ \boldsymbol{W} = \sum_i^N x_i x_i^T $$
with N raw stored entries, that are retrieved by iterating the initial guess $\xi$ with the following update rule
$$ \xi_{t+1} = \text{sgn}( \boldsymbol{W} \xi_{t} - b ) $$
Now to the paper in question, the update rule for the generalization they propose for continuous states that would be used is (eq 22 in OP):
$$ \xi_{t+1} = \boldsymbol{X} \text{softmax}( \beta \boldsymbol{X}^T \xi_{t} ) $$
Where $\boldsymbol{X} = (x_0, x_1, \dots , x_N ) $
The first substantial difference I see is that, while on the case of binary entries, all the weights of the network are encoded in the matrix $\boldsymbol{W}$, hence the storage is constant regardless of how many actual patterns are stored in it. In contrast, in this continuous case generalized rule, the $\boldsymbol{X}$ matrix seems to grow linearly with the size of entries, in fact it keeps all the stored entries directly. By their update rule, it doesn't seem to be a way around storing and keeping around the entire entries, and the update rule seems to only find a "best fit" among the entries, using the scalar dot product of the attention mechanism. I still think I might be missing something important here


text:
UCB-like algorithms: how do you compute the exploration bonus?
My question concerns Stochastic Combinatorial Multiarmed Bandits. More specifically, the algorithm called CombUCB1 presented in this paper. It is a UCB-like algorithm.
Essentially, in each round of the sequential game the learner chooses a super-arm to play. A super-arm is a $d$-dimensional vector $a \in \mathcal{A}$ where $\mathcal{A} \subset \{0,1\}^d$. In each super-arm $a$, when the $i$-element equals to $1$ ( $i \in \{0, \dots, d\}, a(i)=1$ ), that means that the basic action $i$ is active. Basically, in each round the learner plays the basic actions that are active in the chosen super-arm. The rewards of the basic actions are stochastics and a super-arm receives as a reward the sum of the rewards of the basic active actions.
The algorithm mentioned above presents a UCB-like algorithm, where with each basic action is associated a UCB-index and in each round the learner plays the super-arm that maximises that index.
My question concerns the confidence interval around the mean of the rewards of the basic actions, presented in equation $2$ of the mentioned paper. Here, the exploration bonus is
$c_{t,s} = \sqrt{\frac{1.5 \log t}{s}}$
I don't understand where that $1.5$ is coming from.
I've always known that one needs to use Chernoff-Hoeffding inequality to derive the exploration bonus in a UCB-algorithm. Am I wrong and it needs to be computed in other ways? I've always seen the same coefficient but with $2$ instead of $1.5$ (reference).
Could someone explain me where does $1.5$ come from, please?
I know there is a similar question here, but I cannot really understand how that works here.
Thank you in advance in case you have time to read and answer my questions.

Do you mean the $6$ from eq. 6 and $48$ from Theorem 2 in (Kveton et al., 2015)? So, you're saying that if I substitute those constants with variables $k_1, k_2, k_3$ and optimise, I should be able to get back to that 1.5. Have I understood correctly? Thank you for your answer, I really appreciate it!


text:
classification of unseen classes of image in open set classification
I have a scanned image, and they need to be classified in one of the pre-defined image classes, so that it can be sorted. However, the problem is the open nature of the classes. At testing time, new classes of scanned images can be added and the model should not only classify them as unseen (open set image recognition), but it should be able to tell in which new class it should belong (not able to figure out the implementation for this.)
So, I am thinking that the below option can work for the classification of unseen classes

Zero-shot learning: Once the image is classified as unseen, we can then apply zero-shot learning to find its respective class for sorting.

Template matching: Match the test image of unseen classes with all available class images, and, once we have a match, we can do sorting of images.

Meta learning-based approach: I am not sure how to implement this, suggestions are much appreciated.

Note: I already tried the classical computer vision approach, but it's not working out. So, more open for neural net-based approach.
Is my approach to solving the problem correct? If possible, suggest some alternative to find the corresponding match/classification of the unseen class image. As I could think of these 2 alternative solutions only.

I have one possible solution for you inspired by real-time face recognition system. It is a similar case to you.

Create embeddings for each class i.e. person or in your case, a class. (Using Siamese network with ArcFace loss)
When a new image comes, take L1 or cosine distance with the embeddings. You will need a threshold above which you create a embedding for the new guy and give it a new class.

One issue you will think at this point of time is how do you do that when have billions of data points. The search for the matching embeddings can then be done using Hierarchical Navigable Small World graphs (see https://github.com/nmslib/hnswlib).
This solution is loosely connected to the plausible techniques you have mentioned here. And if you still  want something in that regard, I shall add the keyword Active Learning.

Thanks for suggesting facial recognition based approach. I have thought the same but input images are very noisy, say lot of extra things are written on the image(noise) before scan, so I was thinking that Face recog based approach will have hard time finding similarity. ON top of that I have many classes in which image can go, so timing is also a problem with face recog based approach. I can compromise with time issue but not sure if 1st issue(noise) can be taken care by Siamese based network.

Try cleaning your images with a GAN or VAE based approach and then move on with this one. Apart from that, it will be a good idea to give this a try since facial recognition system really get a lot of different kind of noises still go on.

Many thanks for VAN /GAN based cleaning approach suggestion. Will try first FR based and then will go for GAN/VAN based approach.

Matching network and prototypical network few-shot learning can be a better option than Siamese network, There are many implementation available online. Also, these methods are fast in computation or classification.


text:
What error should I use for RNN?
I'm relatively new to machine learning, and I don't know what error I should use for an RNN.
I want to use a simple Elman RNN to predict the cases of Covid-19 there will be in a hospital for the next 15 days. I modeled this as a regression problem, treating the input like a bunch of dots in a graph to predict the tendency that the data is going to take (only show if there will be more cases or less).
With that bunch of dots I in fact refer to this:

Then I would treat this problem as a regression.
I actually don't have anything programmed yet. Firstly I want to write it all on a paper and then get down to work.  I am also considering focusing the problem to predict the actual plot of the time-series input, but right now I want to try the regression.
I've come to the conclusion that I can use these four different errors:

MSE
RMSE
Entropy
Cross-entropy

What are the different characteristics of these errors? Which to use? Where and when to use them?

Can you clarify this part "treating the input like a bunch of dots in a graph"? Why is the data like that? What is the exact input you're using?

What does the y and x axes on your graph/plot represent?

The x represents the time (in days), and the y is the number of cases in that day.

To provide a good answer would fill several pages.  To keep it very simple try many different loss functions on your model.  Your goal is to have the highest performance based on some desired prediction metric (e.g., RMSE, MAE, MAPE, etc.).  You almost always have plenty of time to try many loss functions so you don't need to have a full understanding, and few people do, to start your project.
I recommend you read the following to learn more:

5 Regression Loss Functions All Machine Learners Should Know by Prince Grover
Loss functions: Why, what, where or when? by Phuc Truong
Loss Functions in Deep Learning: An Overview

Thank you, the second page was really useful and clear. Now I understand better this topic

@mariogarcia79 You should still clarify what I asked under your post, if you want to get a more specific answer. At this point, your question has nothing to do with RNNs, but you're just asking when to use MSE as opposed to CE.

@nbro This is what I was refering to.


text:
Improving Mask RCNN by arbitrary scaling head input
Currently, I am looking at how Mask R-CNN works. It has a backbone, RPN, heads, etc. The backbone is used for creating the feature maps, which are then passed to the RPN to create proposals. Those proposals would then be aligned with feature maps and rescaled to some $n \times n$ pixels before entering box head or mask head or keypoint head.
Since conv2D is not scale-invariant, I think this scaling to $n \times n$  would introduce scale-invariant characteristics.
For an object that is occluded or truncated, I think scaling to $n \times n$ is not really appropriate.
Is it possible if I predict the visibility of the object inside the box head (outputting not only xyxy [bounding box output], but also xyxy+x_size y_size [bounding box output  + width height scale of object]). This x_size and y_size value would then be used to rescale $n \times n$  input.
So, if only half of the object is seen (occluded or truncated), inputs inside the keypoint head or mask head would be 0.5x by 0.5x.
Is this a good approach to counter occlusion and truncation?

Can you please clarify this part "outputting not only xyxy, but also xyxy x_size y_size"? What xyxy and xyxy x_size y_size? What does that mean? Note that I changed from x by x to $n \times n$ because of latex.

Editted for clarifitcation xyxy means xmin, ymin, xmax, ymax (i believe this is bounding box output) or xywh (xmin, ymin, width, height)