text:
Why are the Bellman operators contractions?
In these slides, it is written
\begin{align}
\left\|T^{\pi} V-T^{\pi} U\right\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \tag{9} \label{9} \\
\|T V-T U\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \tag{10} \label{10}
\end{align}
where

$F$ is the space of functions on domain $\mathbb{S}$.
$T^{\pi}: \mathbb{F} \mapsto \mathbb{F}$ is the Bellman
policy operator
$T: \mathbb{F} \mapsto \mathbb{F}$ is the Bellman
optimality operator

In slide 19, they say that equality $9$ follows from
\begin{align}
{\scriptsize
\left\|
T^{\pi} V-T^{\pi} U
\right\|_{\infty}
=
\max_{s} \gamma \sum_{s^{\prime}} \operatorname{Pr}
\left(
s^{\prime} \mid s, \pi(s)
\right)
\left|
V\left(s^{\prime}\right) - U
\left(s^{\prime}\right)
\right| \\
\leq \gamma \left(\sum \operatorname{Pr} \left(s^{\prime} \mid s, \pi(s)\right)\right) \max _{s^{\prime}}\left|V\left(s^{\prime}\right)-U\left(s^{\prime}\right)\right| \\
\leq \gamma\|U-V\|_{\infty}
}
\end{align}
Why is that? Can someone explain to me this derivation?
They also write that inequality \ref{10} follows from
\begin{align}
{\scriptsize
\|T V-T U\|_{\infty}
= \max_{s}
\left|
\max_{a}
\left\{
R(s, a) + \gamma \sum_{s^{\prime}} \operatorname{Pr}
\left(
s^{\prime} \mid s, a
\right) V
\left(
s^{\prime}
\right)
\right\}
-\max_{a} \left\{R(s, a)+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, a\right) U\left(s^{\prime}\right)\right\} \right| \\
\leq \max _{s, a}\left|R(s, a)+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, a\right) V\left(s^{\prime}\right)
-R(s, a)-\gamma \sum \operatorname{Pr}\left(s^{\prime} \mid s, a\right) V\left(s^{\prime}\right) \right| \\
=
\gamma \max _{s, a}\left|\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, a\right)\left(V\left(s^{\prime}\right)-U\left(s^{\prime}\right)\right)\right| \\
\leq \gamma\left(\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, a\right)\right) \max _{s^{\prime}}\left|\left(V\left(s^{\prime}\right)-U\left(s^{\prime}\right)\right)\right| \\
\leq
\gamma\|V-U\|_{\infty}
}
\end{align}
Can someone explain to me also this derivation?

Which particular bits do you not understand? It would also be good if you introduced some of the notation, e.g. what is $T$ here? I assume it is some kind of bellman backup operator but that is just a guess. Also, what is $U$ here? it looks like it'll be a value function but without specifics it is hard to comment.

@DavidIreland the first inequality in (10) can be understood using |maxA - maxB| <= max| A - B|   However, I am not sure about the two inequalities at the very last line of the proof

The inequality
\begin{align}
\left\|T^{\pi} V-T^{\pi} U\right\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \label{1}\tag{1},
\end{align}
where $U$ and $V$ are two value functions, follows from the definition of Bellman policy operator (at slide  16)
\begin{align}
T^{\pi} V(s)
&\triangleq
R(s, a)+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, a\right) V\left(s^{\prime}\right) \\
&=R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s)\right) V\left(s^{\prime}\right), \; \forall s \in S
\tag{2}\label{2},
\end{align}
where $\triangleq$ means "defined as". Note the $\pi$ in the definition, hence the name Bellman policy operator (BPO), and note that the BPO holds for all $s$.
To prove (\ref{1}), first recall that
\begin{align}
\left\|\mathbf {x} \right\|_{\infty } \triangleq \max _{i}\left|x_{i}\right| \label{3}\tag{3}.
\end{align}
In the case of value functions $V$ and $U$, we have
\begin{align}
\left\|V - U \right\|_{\infty } \triangleq \max_{s \in S}\left|V(s) - U(s) \right|.
\label{4}\tag{4}
\end{align}
Note also that $Pr$ is always non-negative (specifically, between $0$ and $1$).
Successively, we expand the left-hand side of (\ref{1}) by applying the definition (\ref{2}) and using the properties just mentioned
\begin{align}
&\left\|T^{\pi} V-T^{\pi} U\right\|_{\infty}
=
\\
&\left\|
\left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) \right) -
\\
\left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right) \right)
\right\|_{\infty}
=\\
&\max_{s \in S}
\left|
\left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) \right) -
\\
\left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right) \right)
\right|
=
\\
& \max_{s \in S}
\left|
\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) - \gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right)
\right|
=
\\
& \gamma
\max_{s \in S}
\left|
\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) - \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right)
\right|
=
\\
& \gamma \max_{s \in S} \left|
\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s)\right) \left ( V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right)
\right|
=
\\
& \gamma \max_{s \in S}
\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s)\right) \left| V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right| \\
& \leq
\gamma \max_{s \in S}
\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s)\right) \max_{x \in S }\left| V\left(x\right) - U\left(x\right) \right|
\label{5}\tag{5}
\\
& \leq
\gamma \max _{x \in \mathcal{S}}\left|V\left(x\right)-U\left(x\right)\right|
\label{6}\tag{6}
\\
&=
\gamma \| V - U \|_{_{\infty}}
\label{7}\tag{7}
\end{align}
Here are a few notes to help you understand this derivation

Equation \ref{7} is just the direct application of the definition of the $\infty$-norm in equation \ref{4}

The inequalities \ref{5} and \ref{6} come from the fact that $\mathbb{E}[f(x)] \leq \max_x f(x)$. When we take $\max_s$, we choose among all conditional distributions $p$ (which are conditioned on $s$), but the differences $\left| V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right|$ don't change in that process. So, no matter which $p$ we choose, i.e. no matter which distribution of the function $\left| V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right|$ we choose, we know that $\mathbb{E} \left[ \left| V\left(s^{\prime}\right) - U\left(s^{\prime}\right)  \right| \right] \leq \max _{x \in \mathcal{S}}\left|V\left(x\right)-U\left(x\right)\right|$

why from 5 to 6 there is an inequality sign? shouldn't the sum of Pr(s'|s,pi(s)) be 1?


text:
What happens if our target network overestimates the value?
When we use DDQN, we often use the target network in case our online network overestimates a value, but this doesn't make sense to me, because

What happens if our target network is the one that overestimated a value, then we’d keep using that overestimated value

Why can't we use both target network for selection and evaluation

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the memory complexity of the memory-efficient attention in Reformer?
When I read the paper, Reformer: The Efficient Transformer, I cannot get the same complexity of the memory-efficient method in Table 1 (p. 5), which summarizes time/memory complexity of scaled dot-product, memory efficient, and LSH attention.
The memory complexity of the memory-efficient method is as follow:
$$\max \left(b n_{h} l d_{k}, b n_{h} l^{2}\right)$$
$b$: batch size
$l$: sequence length
$n_h$: the number of attention head
$d_k$: the dimension of query or key
To the best of my knowledge, the memory-efficient method will do a loop for each query, therefore, the whole attention matrix will not show up.
So, shouldn't the memory complexity be $\max(b n_h l d_k, b n_h l)=(b n_h l d_k)$ instead of $\max(b n_h l d_k,b n_h l^2)$?

authors: حاجیه مهلقا اخوند باز زاده

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
When to use NLP, NLG and NLU in conversation agents?
I had read some blogs (like 1, 2 or 3) about what the difference between all three of them is. I am trying to build an open domain conversation agent using natural language AI. That agent can do casual conversation, like a friend. So, for that, I want to know what is the importance of NLP, NLG, and NLU, so that I can learn that part first.

They're all important. NLP is an umbrella term that includes the other two; NLG is only concerned with generating language, ie transforming some internal data structure into human language. NLU is about processing information contained in language, and putting it into relation with a knowledge base etc.
If you don't know anything about any of these fields, then I suggest your aim is far too optimistic. I work for a company that provides a conversational AI platform for businesses to develop their own agents, and it is a complex area.
If you want to have a quick go and pick up some experience, I suggest you start with ELIZA. Even though this is 'ancient', many modern chatbots still work on the same principles. There are many implementations in a number of programming languages, so you should be able to find one that suits you and you can tinker around with it.


text:
What are the differences between a knowledge base and a knowledge graph?
During my readings, I have seen many authors using the two terms interchangeably, i.e. as if they refer to the same thing. However, we all know about Google's first quotation of "knowledge graph" to refer to their new way of making use of their knowledge base. Afterward, other companies are claiming to use knowledge graphs.
What are the technical differences between the two? Concrete examples will be very useful to understand better the nuances.

What is a knowledge graph?
Appendix A.3 "Knowledge Graphs": 2012 Onwards of the survey Knowledge Graphs (which is probably the most extensive survey on KGs) states that knowledge graphs have been defined in different ways in recent years. Each of these definitions raises questions about the relationship between KGs and other related concepts, like graph databases, knowledge bases, and ontologies.
One definition of a KG is

a graph where nodes represent entities, and edges represent relationships between those entities. Often a directed edge labelled graph is assumed (or analogously, a set of binary relations, or a set of triples)

The question here is: what's the difference between KGs and graph databases (like Neo4j)? Graph databases have been used to build KGs, but is there any actual difference between these 2 terms?
Another definition of a KG is

a knowledge graph is a graph-structured knowledge base

So, according to this definition, a KG would be a type of knowledge base (KB).
What is a knowledge base?
In the same appendix, the authors write

The phrase "knowledge base" was popularised in the 70's (possibly earlier) in the context of rule-based expert systems [72], and later were used in the context of ontologies and other logical formalisms [68]

They conclude that a KB has also been defined in ambiguous ways in the past.
Norvig and Russell, in chapter 7 (p. 235) of their AIMA book (3rd edition), define a KB as a set of sentences/facts, for example, expressed in propositional logic. You then use inference techniques to derive new knowledge from this knowledge base. The programming language PROLOG is based on this definition of a KB.
What is the difference between a KG and KB?
So, there is not a single answer to your question because knowledge graphs (KGs) and knowledge bases (KBs) have been defined in multiple (often ambiguous) ways in the past. Some people say that KGs are different from KBs, while other people use the term KG as a synonym for KB or define it as a type of KB.
One possible answer
However, if we define a KG as a graph with nodes that represent entities (like city and country) and edges that represent the relations between those entities (which is a common definition of a KG), we can view a KG as a visual representation of a KB, defined as a set of sentences/facts (for example, expressed in propositional logic). To see why this is the case, consider the following simple KG.

Let's denote this KG by $K = \{E, R\}$, where $E = \{ \text{Santiago}, \text{Chile}, \text{Perú}\} = \{ S, C, P\}$ is the set of entities (with a property) and $R = \{ \text{capital}, \text{borders}\} = \{ c, b \}$ is the set of relations. In $K$, we have the following relations

Santiago is the capital of Chile: $(S, c, C)$
Chile borders Perú: $(C, b, P)$
Perú borders Chile: $(P, b, C)$

which can be put into a set of facts $F = \{(S, c, C), (C, b, P), (P, b, C) \}$, which is the KB associated with this specific KG.
In the context of KGs, we also have a task/problem similar to the inference (or knowledge reasoning) one in the context of KBs, which is known graph completion, which can be divided into other subtasks, like entity prediction, relation prediction and triple classification, which use knowledge graph embeddings.
Further reading
You should really read Appendix A and successive appendices of the mentioned survey to get more information about the issues concerning the history and definition of knowledge graphs and knowledge bases, and other related concepts, like schemas. You can also read chapter 7 (p. 235) of the AIMA book, 3rd edition to know more about knowledge bases.

Based on the related Wikipedia, a knowledge base (KB) is:

a technology used to store complex structured and unstructured information used by a computer system. The initial use of the term was in connection with expert systems which were the first knowledge-based systems.

As there are different representation model for a KB, we can find different terminology in different domains. For example, in some AI articles, it's called ontology.
Knowledeg graph (KG) is another object model to KB realization which is introduced by Google for its search engine (as you have mentioned). Hence, KG is a specification of KB. You can find more information in the paper Knowledge Graphs, such as more history about the KG or a formal definition of that:

knowledge graph is a graph of data intended to accumulate and convey knowledge of the real world, whose nodes represent entities of interest and whose edges represent relations between these entities.

Moreover, you can find some articles about contextual KG (CKG) in the paper Learning Contextual Embeddings for Knowledge Graph Completion and KG$^2$: Learning to Reason Science Exam Questions with Contextual Knowledge Graph Embeddings.

I am not sure if an ontology is a synonym for KB. For example, read this article What’s the Difference Between an Ontology and a Knowledge Graph?, where the author says that a knowledge graph is an "instantiation" of an ontology. See also Ontologies and knowledge bases: towards a terminological clarification (1995) by Nicola Guarino et al. The term ontology has been defined differently in many cases. There's also this related blog post.

@nbro You're right. They are not a synonym. It's also a realization of a KB. However, the term "ontology" is mostly used in AI as a reference to a KB

I have to support OmG here. I have always seen the word 'ontology' used in AI. Whether that is 'correct' or not is a different question, sadly.

In a book coauthored by Xin Luna Dong, one of the creators of the Google Knowledge Vault, it reads that:

Over the last decade, large-scale knowledge bases, also known as knowledge graphs, have been automatically constructed from web contents and text sources, and have become a key asset for search engines.

Then I thought knowledge graphs are just large-scale knowledge bases, and this statement can be proved by this snippet in this article on Emily Moxley, the product manager of the 2012 knowledge graph project:

Then, a year ago, when the search giant bought a company called Metaweb and its collaborative knowledge base called Freebase, made up of metadata collected from multiple sources, Moxley took on the responsibility to help integrate Freebase into Google’s search UI.
“At Metaweb, they were collecting factual information about real world things, like baseball teams and tourist destinations and celebrities. Once they came to Google, it was an amazing match, because we have all this data, including Maps, Business Photos, Google Books – gigantic data sets that let us identify real world things.”
Fast forward to the present, and from the 12 million entities Metaweb brought with it to Google, the Knowledge Graph has ballooned to include 500 million items and 3.5 billion facts. As for the related search results that are algorithmically displayed, Google recognizes patterns in search data no competitor can see due to its massive scale.


text:
Can we achieve what a CNN can do with just a normal neural network?
When I was learning about neural networks, I saw that a complex neural network can understand the MNIST dataset and a simple convolution network can also understand the same. So I would like to know if we can achieve a CNN's functionality with just using a simple neural network without the convolution layer and if we can then how to convert a CNN into an ANN.

The convolutional aspect of a CNN comes purely from the connections between layers. Instead of a fully-connected network, which can be difficult to train and tends to overfit more, the convolutional network utilizes hierarchical patterns in the data to limit the number of connections - a local edge detection feature in an image analysis network, for example, only needs input from a small number of local pixels, not the entire image. But in principle, you could assign weights to a fully-connected network to perfectly mimic a convolutional one - you just set the weights of the unneeded connections to zero. Because a general ANN has all the connections present in a CNN plus more, it can do anything a CNN can do plus more, although the training can be more difficult.

Might be worth showing with an example, just how crazy the numbers get if you try to do this with a typical small image size and multi-layer CNN. It really demonstrates how efficient CNN approach is

Yeah I agree with @NeilSlater, I didn't quite get the "utilizing of hierarchical patterns" part of the answer. Can you please explain that part in brief with an example?

It can be argued that CNN will outperform a fully connected network if they have the same structure (number of neurons).
Normal neural networks can probably learn to detect things like CNNs, but the task would be a lot more computationally expensive. In a CNN, all neurons in a feature maps share the same parameters, so if CNN learns to recognize a pattern in one location, it can detect the pattern in any other location. Furthermore, CNNs take into account the fact that pixels that are closer in proximity with each other are more heavily related than the pixels that are further apart, this information is lost in a Normal neural network.
Read More here.


text:
Why can't neural networks be applied to preference learning problems?
In section 6.1 of the paper Neural Networks in Economics, the authors say

this leads to the problem, that no risk can be formulated which shall be minimized by a Neural Network learning algorithm.

So, why can't neural networks be applied to preference learning problems?
See sections 6.0 of the same paper for a definition of preference learning.

Please, next time, do not assume that the readers of your post are familiar with what you are writing, so provide more context in the post.


text:
What are the differences between constraint satisfaction problems and linear programming?
I have taken an algorithms course where we talked about LP significantly, and also many reductions to LPs. As I recall, normal LP is not NP-Hard. Integer LP is NP-Hard. I am currently taking an introduction to AI course, and I was wondering if CSP is the same as LP.
There seems an awful lot of overlap, and I haven't been able to find anything that confirms or denies my suspicions.
If they are not the same (or one cannot be reduced to the other), what are the core differences in their concepts?

LP is a mathematical problem to optimize a linear function subject to linear (in)equality constraints of the sort:
$$
min_x  w^t x
$$
$$
Ax<b
$$
where $x$ is a continuous variable (vector). If the problem is feasible, then a unique global optimal solution exists for $x$
A constraint satisfaction problem on the other hand is just the constraints part of the above LP. So we are not interested in the optimal solution, just a set of values that will satisfy the constraints. If the domain of your variables is continuous, then it can be recast as a dummy LP with a fake objective e.g. $min_x 0, s.t. Ax<b$, and then solved with an LP solver (e.g. simplex algorithm)
However many CSPs have discrete variables so they are not LP. Also CSP problems may have too many variables for plain LP algorithms to handle, so there are problem specific shortcut algorithms. We are often happy to find a feasible solution and forget about "optimality"

@PrabhDhali If think this answer is useful, you should upvote it by clicking on the arrow pointing upwards. And, if you think this answer answers your question, you should accept it by clicking on the check mark.


text:
Is there a tutorial for beginners on capsule neural networks?
I am interested in capsule neural networks. I have already read the paper Dynamic Routing Between Capsules, but it is a little bit difficult to follow. Is there a tutorial for beginners on capsule neural networks?


text:
Is the generator distribution in GAN's continuous or discrete?
I have some trouble with the probability densities described in the original paper.  My question is based on Goodfellow's paper and tutorial, respectively: Generative Adversarial Networks and NIPS 2016 Tutorial: Generative Adversarial Networks.
When Goodfellow et al. talk about probability distributions/densities in their paper, are they talking about discrete or continuous probability distributions? I don't think it's made clear.
In the continuous case, it would imply, for instance, that both $p_{data}$ and $p_g$  must be differentiable since the optimal discriminator (see Prop. 1) is essentially a function of their ratio and is assumed to be differentiable. Also, the existence of a continuous $p_g$ is non-trivial. One sufficient condition would be that $G$ is a diffeomorphism (see normalising flows), but this is rarely the case. So it seems that much stronger assumptions are needed.
In the case that the answer is discrete distributions: the differentiability of $G$ implies continuous outputs of the generator. How can this work together with a discrete distribution of its outputs? Does the answer have something to do with the fact that we can only represent a finite set of numbers with computers anyway?


text:
How to choose a suitable threshold value for the Shi-Tomasi corner detection algorithm?
While implementing the Shi-Tomasi corner detection algorithm, I got stuck in deciding a suitable threshold for corner detection.
In the Shi-Tomasi algorithm, all those points that qualify $\min( \lambda_1, \lambda_2) > \text{threshold}$ are considered as corner points. (where $\lambda_1, \lambda_2$ are eigenvalues).
My question is: what is a suitable criterion to decide that threshold?

If the answer below answers your question, you should upvote and accept it.

The threshold is typically chosen empirically, so there is no exact answer.
It's dependent on how many corners you wish to select, and how strict you want the detection, which could depend on the use case, the dataset, and the block size of the algorithm. If you’re not sure what to choose for the threshold, I would suggest using a scheme relative to your images to deduce the best threshold, based on what performs best for your use case.
For example, let the criteria for selection, or scoring function, be $R$, which in the Shi-Tomasi case is $R = min(\lambda_1, \lambda_2)$. We could choose some value $q \in (0,1)$, so that the threshold becomes
$t = q\max_p{R}$,
where the max $R$ is calculated over all points $p$, an approach similar to OpenCV Good Features to Track.
You could examine the corners detected and adjust $q$ based on how stringently you wanted to select corners. If you don't want to waste the time manually fine-tuning the threshold, you could check out some automated approaches in the literature, such as Automated thresholding for low-complexity corner detection


text:
What are some good loss functions used to minimize extreme errors in regression and time series forecasting?
I'm working on a time series forecasting task, and, in some specific cases, I don't need perfect accuracy, but the network cannot by any means miss by a lot. So, in detriment of a smaller mean error, I want to have fewer big mistakes.
Any suggestions of loss functions or other methods to solve this issue?

Off the top of my head, you could have a loss function with a penalty term that penalizes higher individual losses. I don't know exactly how this could be implemented in your case, because I also don't understand how you're currently implementing your loss function and training your neural network.


text:
Under what conditions can one find the optimal critic in WGAN?
The Kantorovich-Rubinstein duality for the optimal transport problem implies that the Wasserstein distance between two distributions $\mu_1$ and $\mu_2$ can be computed as (equation 2 in section 3 in the WGAN paper)
$$W(\mu_1,\mu_2)=\underset{f\in \text{1-Lip.}}{\sup}\left(\mathbb{E}_{x\sim \mu_1}\left[f\left(x\right)\right]-\mathbb{E}_{x \sim \mu_2}\left[f\left(x\right)\right]\right).$$
Under what conditions can one find the optimal $f$ that achieves the maximum? Is it possible to have an analytical expression for $f$ that achieves the maximum in such scenarios?
Any help is deeply appreciated.


text:
Are there any good tutorials about training RL agent from raw pixels using PyTorch?
Is there any good tutorials about training reinforcement learning agent from raw pixels using PyTorch?
I don't understand the official PyTorch tutorial. I want to train the agent on the atari breakout environment. Unfortunately, I failed to train the agent on the RAM version. Now, I am looking for a way to train the agent from raw pixels.

I always found that towards data science articles can be a good source for code for these types of problems. They usually have a git repo with everything you need and walk through the less obvious steps in the article. This article may be of interest to you.

I am using Jetson reinforcement for my quadcopter reinforcement in simulation. Maybe it will help you. because you can create AI agents that can learn from the interactive environment, gather experience, and system of reward with deep RL. You can also use end-to-end neural networks that translate raw pixels into action as per your need and use that RL-trained agent to complete complex tasks.
The best thing is you can easily transfer RL-agent which simulated in the simulator to real-world robots. and We are using multiple Nano's easily to perform the complex tasks of navigation and co-ordination of Our Quad-copter.
Webinar of Deep RL on Jetson
Google group to get help for Deep RL on jetson nano
Hope it helps you.

The question was about tutorials, so you should at least explain why your suggestion could help to solve the issue.


text:
How can the target rely on untrained parameters?
I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function
$$ L_i(\theta_i) = [(y_i - Q(s,a;\theta_i))^2] $$
where
$$ y_i = r + \gamma * max_{a'}Q(s',a';\theta_{i-1}) $$
is the target value in the loss function.
From my understanding, experiences are pulled from the replay buffer, then the DQN is used to estimate the future sum of the discounted rewards for the next state (assuming it plays optimally) and adds this onto the current rewards $r$ to create the target value. Then the DQN is used again to estimate $Q$ value for the current state. Then the loss function is just the difference between the target and the estimated $Q$ value for the current state. Afterward, you optimize the loss function.
But, if the parameters of the DQN start off randomly, then surely the target value will be completely wrong since the parameters that define that target function are random. So, if the target function is wrong, then it will minimize the difference between the target value and the estimated value, but it will be learning to predict incorrect values, since the target values are wrong?
I don't understand why the target value works if the parameters of the DQN needed to create that target are completely random.
What obvious mistake am I making here?

Normally when calculating the target $y_i$, we should use parameters/weights $\theta^{-}$ of the target network, which are a periodically-updated copy of the main network parameters $\theta$ (See here). So, just checking if this is a typo in your notation, and if even when fixed, your doubts still exist.

Note that, in the case of tabular Q-learning (Q-learning without function approximation), the initial targets are also noisy, but hopefully they will converge to the correct target, as you update the state-action value function and explore (and exploit) the environment.

Thank you for the replies. I think I have a better idea about what is happening.


text:
Are the final states not being updated in this $n$-step Q-Learning algorithm?
I am reading this paper and in algorithm 3 they describe an $n$-step Q-Learning algorithm. Below is the pseudo-code.

From this pseudo-code, it looks as though the final tuples that they would visit in don't get added to the memory buffer $M$. They define a sample size $T$, but also say in the paper that an episode terminates when $|S| = b$.
This leaves me two questions:

Have I understood the episode termination correctly? It seems from the pseudocode they are just running an episode for $T$ time steps but also in the paper they have a definition for when an episode terminates, so I'm not sure why they would want to truncate the episode size.
As I mentioned, it seems as though the final state $S_T$ that you would be in won't get added to the experience buffer as we only append the $(T-n)$th tuple. Why would you want to exclude the information you get from the final tuples you visit?


text:
How does one prove comprehension in machines?
Say we have a machine and we give it a task to do (vision task, language task, game, etc.), how can one prove that a machine actually know's what's going on/happening in that specific task?
To narrow it down, some examples:
Conversation - How would one prove that a machine actually knows what it's talking about or comprehending what is being said? The Turing test is a good start, but never actually addressed actual comprehension.
Vision: How could someone prove or test that a machine actually knows what it's seeing? Object detection is a start, but I'd say it's very inconclusive that a machine understands at any level what it is actually seeing.
How do we prove comprehension in machines?

Comments are not for extended discussion; this conversation has been moved to chat.

This is one of the most important issues in the philosophy of artificial intelligence.
The most famous philosophical argument that attempts to address this issue is the Chinese Room argument published by the philosopher John Searle in 1980.
The argument is quite simple. Suppose that you are inside a room and you need to communicate (in a written form) with people outside the room in a certain language that you do not understand (in the particular example given by Searle, Chinese), but you are given the rules to manipulate the characters of this language (for a given input, you have the rules to produce the correct output). If you follow these rules, to the people outside the room, it will seem as if you understand this language, but you don't.
To be more concrete, when I say "apple", you understand that it refers to a specific fruit because you have eaten apples and you have a model of the world. That's understanding, according to Searle.
The most famous mathematical model of computers, the Turing machine, is essentially a system that manipulates symbols, so the Chinese Room argument directly applies to computers.
Many replies or counterarguments to the CR argument have been discussed, such as

the system reply (the symbol manipulator is only a part of the larger system).
the robot reply (the symbol manipulator does not understand the meaning of the symbols because it has not experienced the associated real-world objects, so it suggests that understanding requires a body with sensors and controllers)
the brain simulator reply (the symbol manipulator can actually simulate the activity in the brain of a person that understands the unknown language)

So, can we prove that machines really understand? Even before Searle, Turing had already asked the question "Can machines think?". To prove this, you need a rigorous definition of understanding and thinking that people agree on. However, many people do not want to agree on a definition of intelligence and understanding (hence the many counterarguments to the CR argument). So, if you want to prove that machines understand, you need to provide a proof with respect to a specific definition of understanding. For example, if you think that understanding is just a side effect of symbol manipulation, you can easily prove that machines understand many concepts (it just follows from the definition of a Turing machine). However, even if understanding was just a side effect (what does a side effect actually mean in this case?) of symbol manipulation, would a machine be able to understand the same concepts and in the same way that humans understand? It's harder to answer this question because we really do not know if humans only manipulate symbols in our brains.

Comments are not for extended discussion; this conversation has been moved to chat.

I recently came across a neat definition of understanding in Roger Schank's Dynamic Memory:
Basically, you store everything you experience in your memory, but you need to index it in order to be able to use it for processing. Obviously, all experiences are slightly different, eg going to a restaurant is broadly the same, but the details vary. So you need to abstract away the details and store those only if necessary (eg if the food or service was particularly good or bad). Otherwise you just store a general template (or 'script') of the event.
In your memory (note: this is modeled, not neurologically correct) you thus have a whole set of event scripts that you can retrieve. So currently I would be accessing my reply-to-stack-exchange-question script to guide me how to best write this answer without getting downvoted for ludicrous claims etc.
Understanding, then, would be to receive (through sensory input, or language) an event, and putting it into the right area in your memory. So if I told you I just went to Burger King, you would understand it when this activates your fast-food-restaurant memory set. If I then told you I went there to wipe the floor, it should instead activate cleaning-job, rather than fast-food-restaurant. So you understand the sequence "I went to Burger King to clean the floor" by linking it to the correct memory region. If a computer then responded with "What did you eat?" it would clearly not have understood the input. But a response of "Do you get free food for working there?" would indicate some level of comprehension/understanding, as it might recognise that people working in food outlets might get free food as a work-related benefit.
If you experience something completely new, you recognise it as a new experience, and start a new cluster of experiences. For example, if you have been to restaurants before, but never to fast food ones. The first time it will be strange and different, but you remember it as differences to the existing restaurant script. Over time it becomes strong enough (assuming to go to more fast-food restaurants), and it will become its own area, still linked to restaurants, but also not quite the same.
What I like about this is that it is a generic mechanism, rather than an explicit processing of content. It is based on learning and experience, which I believe are key aspects of intelligent behaviour.
UPDATE: This answer is more concerned with trying to find a workable definition of what it means to comprehend something, rather than trying to operationalise it in a dialogue system. You can probably pass the Turing test with some clever tricks, without any comprehension at all. But the point is, what does it mean to understand something? And in the current definition it means to classify related events together, and to recognise similarities and differences between similar experiences. The reaction (ie a response) is not the understanding itself, but only a reflection of the internal state that would demonstrate understanding.
The difference to a neural network is, I would guess, that it can cope with a broad range of experiences, where a NN would need vast amounts of training data (as it doesn't comprehend). Comprehension involves compression of information through abstraction and evaluating differences. This is still a hard problem, and I'd think difficult to achieve just with automated machine learning.
UPDATE 2: With regards to the Turing Test, in a way it goes back to deep philosophical points about empiricism. How do you know the world around you exists? You can see it. But how do you know your eyes tell you the true picture? You can quickly descend into a Matrix-like scenario where you don't know anything for certain.
The Turing Test is a proxy for showing understanding. You don't know the computer understands what you say, so you observe its responses and interpret them accordingly. Just like at school: the teachers asks a question, and from the pupils' answers infers whether they show understanding. If you simply regurgitate a memorised answer, that's not understanding. If you paraphrase in different words, that shows some sort of comprehension. If you draw analogies to similar issues and analyse why and how they are distinct, now there you show that you really get it.
Because we cannot inspect the internal state of a pupil, we cannot measure objectively whether they understood something. We only have communication as an interface between our mind and theirs, and so far chatbots have focused on getting that right. But I think what we really need is to work on memory and memory processing to get further towards comprehension or understanding. And I say this as a computational linguist who specialises in the language parts...

Comments are not for extended discussion; this conversation has been moved to chat.


text:
What are the keys and values of the attention model for the encoder and decoder in the "Attention Is All You Need" paper?
I have recently encountered the paper on NLP. It is very new to me and I am still unable to see how that works. I have used all the resources over there from the original paper to Youtube videos and the very famous "Illustrated Transformer".
Suppose I have a training example of "I am a student" and I have the respective French as "Je suis etudient".
I want to know how these 3 words are converted to 4 words. What are the query, keys, values?
This is my understanding of the topic so far.
The encoder part is:

Query: a single word embedded in a vector form. such as "I" expressed as a vector of length 5 as $[.2, 0.1, 0.4, 0.9, 0.44]$.
Keys: the matrix of all the vectors or in simple words, a matrix that has all the words from a sentence in the form of embeddings.
Values = Keys

For decoder:

Query: the input word in the form of a vector (which is output given by the decoder from the previous pass).
Keys = values = outputs from the encoder's layers.

BUT there are 2 different attention layers and one of which do not use the encoder's output at all. So, what are the keys and values now? (I think they are just like encoder, but just the generated until that pass)?

BUT there are 2 different attention layers and one of which do not use the encoder’s output at all. So, what are the keys and values
now?

The first attention layer in the decoder is the "Masked Multi-Head Attention" layer and is the self-attention layer, calculating how much each word is related to each word in the same sentence. However, our aim in the decoder is to generate the next French word and so for any given output French word we can use all the English words but only the French words previously seen in the sentence. We, therefore, "mask" the words that appear later in the French sentence by transforming these to 0 so the attention network cannot use them.

How these 3 words are converted to 4 words

The second attention block in the decoder is where the English to French word mapping happens. We have a query for every output position in the French sentence and a key/value for every English input word. We calculate relevance scores from the dot product of the query and key and then obtain output scores for each predicted word from multiplying the relevance and value. The following diagram is useful to visualise how, for each predicted word, we can have relevance scores that can predict that one English word can be translated to multiple, or no French word.

In summary, the encoder discovers interesting things about the English sentence whilst the decoder predicts the next French word in the translation. It should be noted they use "Multi-Head Attention", meaning that a number (8 in the original paper) of attention vectors are calculated to learn attention mechanisms to pay attention to different things, for example, grammar, vocabulary, tense, gender, and the output is a weighted average of these.


text:
How can I predict the label given a partial feature vector?
Most of the traditional machine learning algorithms need a feature vector of a constant dimension to predict the label.
Which algorithms can be used to predict a class label with a shorter or partial feature vector?
For example, consider a search engine. In search engines, when the user types a few letters, the search engine predicts the context of the query and suggests more queries to user.
Similarly, how can I predict a class label with an incomplete feature vector? I know one way is to pad the sequence, but I want a better solution.

Something that works at character level? search engines are more complex though

can you please introduce me some paper to study and get the idea?

I am not sure about any specific paper, but now I realize you want something like partial embedding? Make do with 20 dimension out of 50? But for the search engine example, you can predict the whole word or query based on characters already input (there are other algorithms/DS like 'tries' for word completion but I don't think that's what you are asking for). It would help if you give a more proper example or a description for a more specific thing.

Thanks dear @SajanGohil. You are right. One solution is to predict the query based on the typed characters and then give it to model. in my question, imagine i have a tabular data in which number of features is constant (e.g. n). I need to use a model which can process samples with fewer features without padding the shorter sample.


text:
Can we force the initial state of a neural network to produce an "unknown" class?
Has anyone investigated ways to initialize a network so that everything is considered "unknown" at the start?
When you consider the ways humans learn, if something doesn't fit a class well enough, it falls into an "unknown category".
I would argue we all ultimately do some type of correlation matching internally with a certain threshold existing for recognition.
Deep networks don't currently have this ability, everything falls into a class. What I'm curious about is how might we force a network to initially classify things as "unknown" as the default state.


text:
How many training runs are needed to obtain a credible value for performance?
I'm trying to optimize a neural network. For that, I'm changing parameters like the batch size, learning rate, weight initialization, etc.
A neural network is not a deterministic algorithm, so, in each training set, I train the neural network from scratch and I stop it when it's full converged.
After training is complete, I calculate the performance of the neural network in a test dataset. The problem is, I trained the neural network from scratch 2 times with the same parameters, but the difference in performance was almost 5%, which is a BIG DIFFERENCE.
So, what's the reasonable number of training runs to obtain a credible performance number of a neural network?

Are you shuffling the data before passing it to the net? Shuffling the data can cause these "issues" because that operation introduces randomness.

No, I'm not shuffling. My neural network is a CNN and in each mini-batch i pad all images to have same size. So, all images in same mini batch has the size of the biggest image in that mini batch.
But if i'm not shuffling, should not be the same in the 2 runs?

Are you using the same test set for both experiments? Also, have you tried to initialize the weights with the same values (not with the same strategy)?

Yes, the trainning,validation and test sets are always the same. I'm initialize the weights with Glorot Uniform with default parameters

Initializing the weights with Glorot uniform may create a different set of initial weights. So, you may want to try initializing both neural networks with the exact same weights, to see if that solves the issue.

And if I want to test the use of Glorot Uniform initialization in my neural network? How many runs It's reasonable to do?

Is it normal doing more than 1 run for gain credibility? I know in genetic algorithms, it's usual to run at least 30 runs, but I never saw nothing about this is Neural Networks

Nitpick: "A neural network is not a deterministic algorithm". Stochasticity comes from the samples and weight initialization. Everything else is deterministic. Plus neural network is not an algorithm but I guess I know what you mean.

@JoãoCastilho If you remove all sources of randomness, the outputs of the networks and their parameters should be exactly the same, if they are trained in the same way and for the exact same number of iterations (epochs and steps). If you introduce randomness, then, yes, you may want to train different networks to gain a sense of confidence about the network and its output.

Please add a seed value in your train/test split code and weight Initialization. Though 5% is too much to be attributed to randomness.


text:
How to implement RAM versions of Atari games
I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in short, I used double deep Q-learning, and trained it like it was CartPole or lunar-lander environment. In CartPole, the observation was a vector of 4 components. In that case, my double deep Q-learning agent solved the environment, but in the breakout-ram version whose observation was a vector of 128 elements, it was not even close.
Did I miss something?

It does sound like something's off.  Have you already used aggregates of multiple frames so that the movement of the ball and paddle are captured?  My own solutions to this a few years back easily achieved perfect/infinite games after a few hours.

To my knowledge from reading about model-based and model-free reinforcement learning,
DQN and Double DQN are model-free reinforcement learning methods. (Why am I mentioning this, see below):
https://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-7320-6_674-1
You should see under the heading "Definition" (you may need to scroll down in the web-page) in the web-page (can be accessed by the above link), it states that: "model-free techniques require extensive experience."
Now extensive experience, depending of how fast you can go through states, can take a few days even weeks...model-free methods require a lot of samples to learn.
Also there are a lot of states: there are ($256^{128}$) states. That is a really big number (I'm just emphasizing that training may take a long while)
Due implementation specifics not being supplied in question, I am assuming your implementation is correct, although...you are using RAM. DQN used image data, I very quickly skimmed the Double DQN paper: https://arxiv.org/pdf/1509.06461.pdf . I am assuming they used image data as well because Double DQN was compared to DQN.


text:
What does it mean when the discriminator's loss gets a constant value while the generator's loss keeps on changing?
While training a GAN-based model, every time the discriminator's loss gets a constant value of nearly 0.63 while the generator's loss keeps on changing from 0.5 to 1.5, so I am not able to understand if this thing is happening either due to the generator being successful in fooling the discriminator or some instability in training. I have been stuck in this confusion for so many days.


text:
What kind of problems cannot be solved using machine learning techniques?
For the problems that can be solved algorithmically.
We have very good formal literature for which problems can be solved in polynomial, exponential time and which cannot. P/NP/NP-hard
But do we know some problems in machine learning paradigm for which no model can be trained? (With/without infinite computation capacity)

At least you should be aware of two points:

P/NP/NP-hard (and all other class of complexities) are thoroughly valid for the machine learning area as well. Because these concepts are related to the fundamental of computations (theory of computation), and machine learning is not an exception here.
One of the useful concepts in the complexity of the learning problem is the VC dimension, PAC learnability, and their related concepts (such as sample complexity). Although these concepts can't be enough to measure the time complexity, they are useful for finding the learner model's capacity.

Unsupervised disentanglement learning with arbitrary generative models is impossible without inductive biases [1].
In fact, in general, any kind of learning is impossible without inductive biases.
[1]: Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations

Maybe you could explain what UDL is and why UDL with arbitrary generative models is impossible without inductive bias.


text:
Will adding memory to a supervised learning system makes it into a Bayesian learning system?
Seung et.al recently published GameGAN paper, GameGAN learned and stored the whole Pacman game and was able to reproduce it without a game engine. The uniqueness of GameGAN is that it had added memory to its discriminator/generator which helped it to store the game states.

In Bayesian interpretation, a supervised learning system learns by optimizing weights which maximizes a likelihood function.
$$\hat{\boldsymbol{\theta}} = \mathop{\mathrm{argmax}} _ {\boldsymbol{\theta}} P(X \mid \boldsymbol{\theta})$$
Will adding memory which can store prior information makes GameGAN a Bayesian learning system?
Can GameGAN or similar neural network with memory can be considered as a bayesian learning system. If yes, then which of these two equations(or something other) correctly explains this system (considering prior as memory)?

$$\mathop{\mathrm{argmax}} \frac{P(X \mid \boldsymbol{\theta})P(\boldsymbol{\theta})}{P(X)}$$

or

$$\mathop{\mathrm{argmax}} \frac{P(X_t \mid \boldsymbol{X^{t+1}})P(\boldsymbol{X^{t+1}})}{P(X_t)}$$

PS: I understand GAN's are unsupervised learning systems, but we can assume discriminator and generator models separately trying to find weights that maximize their individual likelihood function.

If you're maximizing the likelihood (maximum likelihood estimation), that doesn't make the system Bayesian, in the sense that you just get a point estimate rather than a distribution and you are not specifying any prior. In Bayesian learning, you need to be able to specify a prior on the parameters and you often get a posterior distribution over them. So, your sentence "In Bayesian interpretation, a supervised learning system learns by optimizing weights which maximizes a likelihood function" is wrong, in the sense that's just a probabilistic interpretation (rather than a Bayesian one).

You might also be interested in this post.


text:
Overcome caveats on using Deep Learning for faster inference on limited performance availability
I am working in the field of Machine Vision, where accuracy and performance both play a major factor in deciding the approach towards a problem. Traditional rule based approaches work quite well in such cases.
I am gradually migrating towards deep learning, due to its umpteen advantages, where the results seem promising albeit with two huge caveats:

Lack of Training data in this field. To be precise, the lack of erroneous data.
Performance issues on inference. Accuracy and speed are required in equal proportion, and cannot be compromised.

In industrial settings, Point 1 plays a strong factor. I have been dabbling with Transfer learning techniques and using pre-trained models to overcome this situation. For simpler applications such as classification, this suits and gives good results. In other cases such as detection and localization, I have tried using MaskRCNN, which gives really good results but poor inference speed, means it is not production-ready.
The worrying factor in both cases is how slow detection and inference is, compared to traditional vision algorithms. A solution would be to buy Machine Vision software specifically from companies such as Cognex, HALCON, etc, who sell deep learning bundles. They are quite expensive and are to be used out- of- box with minimal modifications, which does not suit me currently.
Point 2, is highly necessary in production lines, where each iteration/image may take less than 500ms for execution.
Deep Learning gives a lot of opportunities in getting state of the art results with very less data in most of the situations, but in general without inference optimization in using apps such as TensorRT, the "time" metric does not give good results.
Is there an approach in using open source that can solve both point 1 and point 2? Creating a CNN from scratch is out of the question.
This post is to discuss ideas if possible, I know a concrete solution is not really possible in the scope of this question. I am the only person working on this problem at my company, thus any discussion ideas would be highly appreciated!

What do you mean by "erroneous data" exactly? Anyway, have you tried to augment your dataset? That has proven to be beneficial in some cases (e.g. see the u-net paper).

Hi, erroneous data - dataset of errors.  Yes data augmentation is an option, but when the error dataset has like 10 images , its very hard to increase the dataset. In general practice, this causes severe overfitting. Have also tried going the anomaly detection route.


text:
Why am I getting better performance with Thompson sampling than with UCB or $\epsilon$-greedy in a multi-armed bandit problem?
I ran a test using 3 strategies for multi-armed bandit: UCB, $\epsilon$-greedy, and Thompson sampling.
The results for the rewards I got are as follows:

Thompson sampling had the highest average reward
UCB was second
$\epsilon$-greedy was third, but relatively close to UCB

Can someone explain to me why the results are like this?

Hi. Just to clarify, in the $\epsilon$-greedy, you used a constant value for $\epsilon$ (if so what was this value?) or did you decay it?

for epsilon greedy, epsilon is decayed at each iteration: 1/i, where i is the current iteration

What was the problem domain? In general these techniques will sort in effectiveness depending on the problem domain. If it is a test or toy problem, please characterise it (how reward functions were set up). Also, can you give more details of your experiment and clarify what you mean precisely by "highest average reward" - there is a large difference in terms of approach when considering reward from start of training, or results after training. The number of training episodes is also a big factor.

Please, address Neil's comment by editing your post and adding more details about your problem domain, etc., otherwise, I will not re-open this post.

Hi @nbro, I see you closed the post. In fact, I agree with Neil's comment, but I've only seen it after finishing typing my answer. Should I delete my answer accordingly?

@user5093249 Hi. I didn't fully read your answer, but it looks good. I wouldn't delete it for now. Let's wait a little bit more until the user clarifies his/her question. Maybe your answer will still be valid, as it is. Please, next time, if you find an post that is unclear or needs more details to be answered, you should flag it to be closed. This is a very important work that the community members should do and that rarely do here (on this site). I shouldn't need to close questions. The community should do it. Anyway, thanks for caring about the post and to make the site a better place!

The first thing to note here is that your results seem aligned with the results commonly found in the bandit literature.
Second thing to note would be that the performance of bandit algorithms is usually measured in terms of regret. This is the difference between (i) the amount of rewards accumulated by an oracle policy having prior knowledge about the true rewards of the bandit arms and (ii) the amount of rewards accumulated by the bandit strategy you are evaluating. In other words, this measures the loss your incur due to not knowing what the optimal arm is.
Ideally, the regret should be logarithmic. In theory, all three algorithms achieve logarithmic regret.
See [1] for the regret of decaying $\epsilon$-greedy and UCB and [2] for the regret of Thompson sampling.
Despite their similar regret bounds, these algorithms may behave differently in practice.
For example, according to experiments in [3]:

It appears that Thompson
sampling is more robust than UCB when the delay is long. Thompson sampling alleviates the
influence of delayed feedback$^*$ by randomizing over actions; on the other hand, UCB is deterministic
and suffers a larger regret in case of a sub-optimal choice.

For decaying $\epsilon$-greedy, the fact that it obtained the lowest rewards may be due to the fact that the action selected for exploration is chosen uniformly at random (even if it has a bad reward estimate). In addition, it will explore actions even if they have already been selected many times, unlike UCB which considers the number of action selections to refine the confidence bound.
That said, besides these aspects, there may be other factors that could influence the results obtained by a given bandit algorithm. These may include:

The reward distributions you chose to model your actions, and their
characteristics.
The number of actions you have, and how far each
action’s reward is from the optimal action’s reward.

$^*$ Delayed feedback occurs when the outcome of an action selection is not revealed immediately but after some time delay.


text:
What is the equation of the learning rate decay in the Adam optimiser?
Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, each weight will get updated differently based on the accumulated squared gradients in their respective dimensions, even though $\alpha$ might be constant. There are other StackOverflow posts that have said that Adam has a built-in learning rate decay. In the original paper also, the authors of adam paper says that the learning rate at time step $t$ decays based on the equation $$\alpha_t = \alpha \cdot \frac{\sqrt{1-\beta_2^t}}{{1-\beta_1^t}}$$
Is the second equation the learning rate decay that has been built into the Adam algorithm?


text:
Designing a reward function for my reinforcement learning problem
I'm working on a project lately and I'm trying to solve a problem with reinforcement learning and I have serious issues with shaping the reward function.
The problem is designing a device with maximum efficiency. So we simulated the problem as follows. There is a 4x4 grid (we defined a 4x4 matrix) and the elements of this matrix can be either 0 or 1 (value 0 means "air" and 1 means a certain material in reality), so there is a 2^16 possible configurations for this matrix. Our agent starts from the top left corner of this matrix and has 5 possible actions: move up, down, left, right and flip (which means flipping a 0 to 1 or vice versa). Based on flipping action, we get a new configuration and each configuration has an efficiency (which is calculated by maxwell equations in the background).
Our goal is to find the best configuration so that the efficiency of the device is maximum.
So far we have tried many reward functions and non of them seemed to work at all! I will mention some of them:

reward = current_efficiency - previous_efficiency (the efficiency is being calculated in each time step)

 if current_efficiency > previous_efficiency:
    reward = current_efficiency
    previous_efficiency = current_efficiency

 diff = current_efficiency - previous_efficiency
 if diff > 0:
    reward = 1
 else:
    reward = -2

and some other variations. Nothing is working for our problem and the agent doesn't learn at all! So far, we have used different approaches to DQN and also A2C method and so far no positive feedback. We tried different definitions of states as well, but we don't think that is the problem.
So, can somebody maybe help me with this? It would be a huge help!

how expensive is it to calculate the efficiency? 2^16 doesn't seem like a huge number, could you not just brute force it? other than that, what are you defining as a state, which position in the grid the agent is in?

To find the best configuration seems more like a search rather than a reinforcement learning one ?

@DavidIreland The problem is that this is a test case. My actual Problem has 10 ^120 possible states. My state consist of the configurations of the grid ( a matirx of 0s and 1s) and another matrix which has all the values = 0 except where currently the agent is. There the value =1. Then these two are flatend and added together.

@calveeen yes, but we want to find the best policy with reinforcement learning. Thats the whole idea about the project.


text:
Can the addition of unnoticeable noise to images be used to create subliminals?
I was reading this report: https://www.theverge.com/2017/4/12/15271874/ai-adversarial-images-fooling-attacks-artificial-intelligence
Researchers used noise to trick machine learning algorithms to misidentify or misclassify an image of a fish as a cat. I was wondering if something like that can be used to create subliminals.
What I mean by subliminals: United Nations has defined subliminal messages as perceiving messages without being aware of them, it is unconscious perception, or perception without awareness. Like you may be aware of a message but cannot consciously perceive that message in the form of text, etc.
All the reports about the noise trick said the noise was so transparent that humans couldn't detect it. This can be changed to make it noticeable unconsciously but unnoticeable at a conscious level so a human can register the subliminal but not be aware of it.
Is it possible to take an output from a hidden layer to construct such subliminal for humans, with trial and error one can find right combination? Can it be possible to come up with a pixel pattern or noise with ML which allows one to impose subliminals?

Basically,  subliminal images are a set of images that we can perceive unconsciously but not consciously. There are some speculations whether certain companies have used such images in their advertising campaigns. To my best of understanding, OP is asking whether ML is every involved or can be involved, to create images that exhibit such property.

I would say no, at least not now. To achieve that you need to have a model that represents the structure of a human brain at a very detailed level, so that inherits the same flaws.

I'm wondering if a knowledge of how neurons process and function in computer vision can be enough to create something like that, like when scrolling up and down, or seeing static or moving images. So you are basically saying someone with enough computation power like Google, Microsoft, CIA would be able to do it because they use neurologists everyday.

I do not think that we understand how human brain operates, so only a lucky guess can occur.


text:
Which machine learning approach can be used to predict a univariate value?
I have a stream of data coming in like below (random numbers 0-9)

7, 7, 0, 0, 8, 9, 2, 7, 3, 8, 2, 8, 5, 7, 0, 8, 7, 8, 5, 3, 2, 6, 1, 9, 5, 7, 5, 3, 4, 9, 1, 3, 5, 5, 0, 7, 7, 5, 2, 8, 8, 7, 5, 5, 5, 2, 9, 7, 2, 1, 0, 0, 5, 7, 1, 4, 2, 7, 8, 8, 5, 2, 7, 5, 7, 1, 7, 2, 0, 5, 7, 5, 2, 6, 3, 6, 3, 6, 1, 9, 1, 9, 7, 2, 3, 9, 8, 8, 4, 9, 8, 2, 5, 3, 4, 0, 3, 1, 0, 7, 2, 3, 8, 7, 5, 7, 3, 6, 0, 3, 3, 3, 6, 3, 1, 3, 0, 6, 9, 8, 0, 1, 4, 4, 9, 9, 3, 7, 4, 1, 0, 5, 0, 6, 8, 8, 8, 1, 7, 6

Ask: is to Predict the next numbers(at least 3-10).
Which approach would be helpful in getting through this problem?

Assuming your data is not purely random (otherwise it will be difficult to make useful predictions), you can try the following:

Hidden Markov Models: Treating your discrete numbers as states, you can use HMMs to predict the next state (next number) by looking at the last or the $n$ last states.
Recurrent Neural Networks: Assuming you have a lot of number sequences, you can treat your numbers as time series. Train a model and predict the next number(s). Here is a nice tutorial which uses LSTM cells.
Autoregressive models: AR models are quite powerful as well and have already been successfully used in finance and signal processing before the rise of deep learning.

Further this post (Problem in discrete valued time series forecasting) on Cross Validated might answer your question as well.


text:
How to autocorelate multiple variants of same text into one?
I want to improve quality of translations for open-source projects in Ukrainian language. We have multiple translations from different authors. We can also translate messages using machine translations. Sometimes machine translation is even better than human translation.
Given multiple variants of translation of the same original text, I want to create AI which will be able to "translate" from Ukrainian to Ukrainian, using these multiple variants in parallel as the source, to produce one variant of higher quality.
So, in general, given multiple similar input sequences, the neural network needs to "understand" them, and produce a single output sequence.
$$S_1, S_2, \dots \rightarrow S$$
For a simple example, we may want to train a NN to recognize a sequence of natural numbers: $1,2,3,4, \dots$. We give two sequences to NN: $23,4,24,6,8$ and $3,65,5,6,23$, then trained the NN is expected to produce $3,4,5,6,7$.
How to modify an existing neural network to achieve that? Is it possible at all?


text:
How do I know that the DQN has learnt an appropriate Q function?
Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it normal that Q values for the non-terminal states have higher values than those of the terminal states?
The reason why I want to know whether Q values learnt are appropriate is because I want to apply the doubly robust estimator for off-policy value evaluation. Using doubly robust requires a good Q value estimate to be learnt for each state.

To answer the question "is it normal that Q values for the non terminal states have higher values than those of the terminal states ?", you should ask yourself what the value function represents. What is the definition of the value function? After that, you should be able to answer your question ;)

I believe the answer is yes the Q values for non terminal states can be higher, an example would be possible if the rewards from the start state to the terminal state is positive. However, the Q values are insanely high compared to the rewards. For the tabular learning it is easier to visualise and understand the Q values. For the continuous state space however, i am not sure how to wrap my head around that.

My rewards are of magnitude 10^-2 - 10^-1 range. but my Q values are of magnitude 10^1 - 10^2. Hence I believe something is off with my training..

Well, maybe you should tell us about the discount factor, learning rate and how many steps and episodes, etc. That could help people to understand better if there's something wrong.

I will try to look more into the problem first..

DQN is famous for doing over-approximation on Q function. However, having over approximated Q does not imply that it does not perform well in the environment. (unless it looks ridiculously high) From my experience, high learning rate usually cause over approximated Q, or mistakes made in the code. Best way to check is to see plot of Q function when running on environment to see if it makes any sense. (e.g. It should go lower for bad states and vice versa)

What does "over-approximation on Q function" mean? Can you provide at least one source (i.e. a research paper) that supports that claim?

Double Q-learning (Hado van Hasselt)

Please, just edit your answer to provide the link to the paper.

You’re welcome ;)


text:
Can a machine learning approach solve this constrained optimisation problem?
I had done with different classification, regression and clustering approaches for predictions of values, etc. I was wondering if there is a machine learning approach for distribution of a whole based on some features (I do not know if there is an approach for that I just could not find one with my research).
An easy example might be lets consider we have height and weight data of many children and we have to distribute a given number of pizza slices amongst them so that skinny children get more pizza as compared to obese ones because pizza is more beneficial for skinny as compared to obese. So might have to find out the optimum number of slices for each child out of the total number of slices so that each child gets maximum possible nutrients. A more complex version could incorporate more features like age, overall health, blood sugar content, physical activity index, daily calorie consumption, and others.
A similar example might be to find out the optimal value of fuel to be allocated to each vehicle if we have a total of 100 gallons. Features might be distance they have to travel, mpg, driver competency, engine horsepower, etc., so that all of them might travel the maximum distance possible.
So, can we achieve a task like this with machine learning/deep learning approaches? If not what are the hurdles achieving this?

Hi. What do you mean by "for distribution of a whole"? I don't understand why you're using this expression. Your problem is about allocating resources given some constraints .

Hi @nbro! I used that because I could not think of any term more explaining than this . Thanks for the edit though it makes it more clear!

Sounds more like a optimization problem than a deep learning / machine learning problem to me.
For machine learning you would have the features of every child / vehicle and the optimal amount of pizza / fuel already given, but you don't know how exactly the optimal amount is computed. So the target is to find a function which maps features to target.
However in your case you don't know the optimal value, you just have a number of constraints. So my suggestion would be to use linear programming.
Here is an simple example: drive the maximum distance with a given amount of fuel
$$\text{max }\ m_1 \cdot x_1 + ... + m_n \cdot x_n$$
$$\text{s.t. }\ x_1 + ... + x_n \leq F$$
$$m_i \cdot x_i \geq T_i\  \forall\  1 \leq i \leq n$$
$x_i$ is the amount of fuel (in gallons) given to car $i$, $m_i$'s are miles-per-gallon for car $i$. $F$ is the total amount of fuel available, this is our first constraint, we cannot distribute more fuel than we have. The second constraint say's that we like to drive at least $T_i$ miles with car $i$. This ensures we do not just give all the fuel to the most efficient car.
You can come up with more constraints for drivers competency, engine horse power etc. In order so solve your problem with linear programming, you just have to make sure it is still linear and convex. Problems of this kind can be solved using the Simplex algorithm (scipy.optimize.linprog).
If your objective or your constraints are more complex you can use numerical methods for non-linear constrained optimization problems (see scipy.optimize for algorithms).

So @Tinu it means this type of problems cannot be solved by machine learning approaches even though total amount is given?


text:
Why is it useful to track loss while model is being trained?
Why is it useful to track loss while the model is being trained?
Options are:

Loss is only useful as a final metric. It should not be evaluated while the model is being trained.
Loss dictates how effective the model is.
Loss can help understand how much the model is changing per iteration. When it converges, that's an indicator that further training will have little benefit.
None of the above

Hi and welcome to AI SE! Do you know something about machine learning or neural networks? This question should be easy to answer if you know something about neural networks and you have trained at least one once in your life.

This looks like coursework? If it is, could you please add some of your own thoughts/analysis (use [edit]). Also, you may want to consider whether you have agreed to any code of conduct on such a course - e.g. if your correct answer leads to gaining a certificate - because posting a question from course material directly in order to get someone else to answer for you may lead to your results being nullified.

The loss function (aka cost function) measures the correctness of the predictions of the model.
For example, a simple cost function could be $|y - f(x)|$, where

$\hat{y} = f(x)$ is the prediction of the model $f$ when the input is $x$,
$y$ is the ground-truth label for input $x$ (i.e. what the model is supposed to output when the input is $x$), and
$|\cdot|$ is the absolute value

The loss function is also used to understand whether a model is overfitting (and underfitting) or not. More specifically, if the loss of the model evaluated on the training data (the training loss) keeps decreasing while the loss of the model evaluated on the validation data (validation loss) starts to increase, that's a good sign that the model is overfitting (i.e. just memorizing the training data, but it will likely perform poorly on non-training data). If the training loss does not decrease, that's a good sign of underfitting (i.e. the model is not capable of learning the patterns in the training data), so you may need to use a model with a bigger capacity, change the loss, or do something else.
The loss is typically not directly used as a measure of the performance of the model because the loss doesn't directly represent the performance of the model. In fact, the loss can be relatively big (with respect to another model's loss) and the model still performs well, but typically a lower loss corresponds to higher performance. The performance of the model is calculated differently depending on the task, model, and your goals. The most common performance measure is the accuracy (the number of correct predictions over the total number of predictions), but there are many other performance measures, such as the precision, recall, f1 score, AUC, etc., that emphasize different behavior that you expect from your model.


text:
After a GAN is trained, which parts of it are used to generate new outputs from data?
After a GAN is trained, which parts of it are used to generate new outputs from data?
Options are:

Neither
Discriminator
Generator
Both Generator and Discriminator

Is this being asked as part of coursework? If so, could you add any insight or opinion that you already have? Use [edit] to add details.

I think so you are new to GANs.
As the name itself implies, the generator task is to generate new images to match the distribution of the model and the input data, hence it is the generator that contributes in the generation of the data, while the discriminator just tries to differentiate between the generated images by the generator and real data.


text:
Is my 57% sports betting accuracy correct?
I have been creating sports betting algorithms for many years using Microsoft access and I am transitioning to the ML world and trying to get a grasp on determining the success of my algorithms. I have exported my algorithms as CSV files dating back to the 2013-14 NBA season and imported them into python via pandas.
The purpose of importing these CSV files is to determine the future accuracy of these algorithms using ML. Here are the algorithm records based on the Microsoft access query:

A System: 471-344 (58%) +92.60

B System: 317-239 (57%) +54.10

C System: 347-262 (57%) +58.80

I have a total of 8,814 records in my database, however, the above systems are based on situational stats, e.g., Team A fits an algorithm if they have better Field Goal %, Played Last Game Home/Away, More Points Per Game, etc...

Here is some of the code that I wrote using Jupyter to determine the accuracy:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

clf = LinearSVC(C=1.0, penalty="l2", dual=False)

clf.fit(X_train, y_train)

pred_clf = clf.predict(X_test)

scores = cross_val_score(clf, X, y, cv=10)

rfe_selector = RFE(clf, 10)

rfe_selector = rfe_selector.fit(X, y)

rfe_values = rfe_selector.get_support()

train = accuracy_score(y_train, clf.predict(X_train))

test = accuracy_score(y_test, pred_clf)

print("Train Accuracy:", accuracy_score(y_train, clf.predict(X_train)))

print("Test Accuracy:", accuracy_score(y_test, pred_clf))

print(classification_report(y_test, pred_clf, zero_division=1))

print(confusion_matrix(y_test, pred_clf))

print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

Here are the results from the code above by system:
A System:

Train Accuracy: 0.6211656441717791
Test Accuracy: 0.5153374233128835
F1 Score: 0.52
CONFUSION MATRIX: [[16 50] [29 68]]
Accuracy: 0.55 (+/- 0.10)

B System:

Train Accuracy: 0.6306306306306306
Test Accuracy: 0.5178571428571429
F1 Score: 0.52
CONFUSION MATRIX: [[49 23] [31 9]]
Accuracy: 0.55 (+/- 0.08)

C System:

Train Accuracy: 0.675564681724846
Test Accuracy: 0.5409836065573771
F1 Score: 0.54
CONFUSION MATRIX: [[15 29] [27 51]]
Accuracy: 0.57 (+/- 0.16)

In order to have a profitable system, the accuracy only needs to be 52.5%. If I base my systems off of the Test Accuracy, only the C System is profitable. However, all are profitable if based on Accuracy (mean & standard deviation).
My question is, can I rely on my Accuracy (mean & standard deviation) for future games even though my Testing Accuracy is lower than 52.5%?
If not, any suggestions are greatly appreciated on how I can gauge the future results on these systems.

My question is, can I rely on my Accuracy (mean & standard deviation) for future games even though my Testing Accuracy is lower than 52.5%?

If by Accuracy you mean training accuracy, then absolutely you should not trust those values. For almost all machine learning algorithms there is a problem with overfitting to training data, which will result in reporting over-estimates for metrics against the training data. This is why you should always have a test data set of unseen data, because the performance on the training data is not what you truly care about - it is the performance on new unseen data, or how the model generalises, that matters when you want to use it to predict new values.
The test data set gives you a measure of how well your model will generalise, because it simulates the performance of the model against unseen data.
However, the test data set is not perfect for all uses. Random chance will cause your test measurements to vary. If you use your test results to select a model from a list of models based on performance, then:

The model you selected does have the highest likelihood of being the best one, as you want. It is not guaranteed to be the best though.

The test result is likely to be an over-estimate of general performance. That is because you cannot separate the random fluctuations in test results from real performance improvements, you only see the combination.

The more tests you run, the more likely you will have an inflated view of the performance of your best model.
The usual fix for this is to use cross validation. Cross validation uses yet another data set to help you with the first step of choosing the best model. After you have chosen your best performing model using cross validation, then you can use a test set that you have kept in reserve to measure the performance. Because you have not used that test set to select a model, then it will give you an unbiased measure of performance. You should still bear in mind that this measure still comes with implied error bars (and with other caveats, such as any inherent sampling bias).
When predicting future results from past data, you do also need to be concerned about population drift and non-stationary problems. This is a common issue in any data set that includes complex behaviour that can evolve over time. This is very likely to affect results from sports teams where many conditions affecting performance evolve over the same timescales that you are trying to predict. In practice this means you will want to feed in new data ad re-train your models constantly, and despite this your models will tend to lag behind reality. It is unlikely you will achieve even the test result accuracy in the long term.
You can add one more thing to your testing routines to help assess the impact of non-stationarity - when reserving data for the test or cross-validation sets, don't do it at random, instead reserve all the latest results (e.g. last four weeks of data) for test only. You are likely to see a reduction in the metrics when you do this for a problem domain like sport, but that should give you a more realistic assessment of the model you are building.

Hi Neil,
Thanks for responding! I am not referring to my Training Accuracy. Taking a looking at my A System Results:
Train Accuracy: 0.6211656441717791
Test Accuracy: 0.5153374233128835
F1 Score: 0.52
CONFUSION MATRIX: [[16 50] [29 68]]
Cross Validation Accuracy: 0.55 (+/- 0.10)
Should I trust the Test Accuracy over the Cross Validation Accuracy?

@Sports_Stats: Sorry I didn't spot you were already using CV. In general CV accuracy is  tainted by using it to select the best agent, so it will be an over-estimate. So yes you should use test accuracy as a guage for how well that model will do "in production". You also need to keep selecting your best agent according to CV, don't select by test accuracy (otherwise you cannot trust it is unbiased). The worst thing you could do here is look through the models, compare test accuracy and pick the best.

@Sports_Stats: Actually your question does not mention CV and states "Train Accuracy"? There is an addional "Accuracy" value at the end - is that the CV accuracy? If so, the CV results are also pushing you to pick model C, so you are in the clear to simply trust what the test accuracy is there and not otherthink things :-)

Sorry for the confusion but to clear things up:
The Accuracy label under the CONFUSION MATRIX label is == Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2)
and
scores = cross_val_score(clf, X, y, cv=10)
Also, when I change the random_state from 42 to 0, my new results are much better:
Train Accuracy: 0.6306306306306306
Test Accuracy: 0.5803571428571429
Cross Validation Accuracy: 0.55 (+/- 0.08)

@Sports_Stats: Given your SD on the estimates is +- 0.08 I would not get too excited about small improvements you see by tweaking params - they could just be luck and not represent true values. You could maybe look into k-fold cross validation to try and get that variance in the estimate down

Hey Neil,
The Cross Validation Accuracy for C System is 57% with a Standard Deviation of (+/- 0.16). Doesn't that indicate that 95% of future predictions will be between 57-0.16= 56.84% and 57 + 0.16 = 57.16% ?

@Sports_Stats: The +-0.16 is standard error on the CV Accuracy, which is not reported as a percentage but a probability, so it's a lot worse than that. It means you are 95% confident that the "true" accuracy is somewhere between 41% or 73%. This is complicated by you then using the value of 0.57 to select this as "the best", so there is also an unmeasured (and difficult to assess) bias in there. That range is very large, probably because you don't have much data to spare for cross validation, which is why I suggest k-fold validation.

Much Appreciated Neil! I truly thank you for all your responses and helpfulness. I'll go ahead and implement the K-Fold Validation and see what results I can come up with. Also, if I can get this algorithm to prove that it can be profitable on future predictions, and if you're interested, I'll send you my future picks :)

I hope you do well. In all likelihood you are about to discover a deeper rabbit hole as the kinds of predictions you are looking at are hard to get right, and model accuracy can drift a lot over time as factors you are not including change. I think FiveThirtyEight might make some good reading for you - https://fivethirtyeight.com/tag/sports-betting/ Thanks for the offer of sharing results, but I am not really a betting person so I would not use them. If they ever get good, keep them to yourself :-)

That unfortunately looks like the case as I just received a KFold Validation of -0.251765602234286 with a 57% Testing Accuracy. Which I guess I subtract the KFold Validation from the Testing Accuracy and receive a 32% Accuracy?

@Sports_Stats: That doesn't seem right. If you run k fold, you should simply get multiple Accuracy values, and be told a mean Accuracy and (reduced) standard error. For instance if you ran 10-fold cross validation on your data set, I would expect you see a mean accuracy somewhere around 0.50 (50%) +- 0.03 (+- 3%) - it's the reduction of the standard error that you care about, it means you can trust the CV values a bit more.

Ok.......I think I figured out where I went wrong and in addition to running K-Fold, I also ran Stratified K-Fold both with n_splits=10 and random_state=0, here are the results (including my original cross_val_score: Cross Validation Accuracy: 0.55 (+/- 0.18)   /    K-Fold Accuracy: 0.49 (+/- 0.16)    /   Stratified K-Fold Accuracy: 0.52 (+/- 0.12).

Hmm, I would expect those standard errors to be lower (around 0.05 if a single measure was around 0.16). I'm not sure what's up there, I don't know your library and how it reports things.

Oops....I imported a different csv....here are the correct results for A System: Cross Validation Accuracy: 0.54 (+/- 0.10)    /    K-Fold Accuracy: 0.50 (+/- 0.12)    /    Stratified K-Fold Accuracy: 0.48 (+/- 0.11)

The hardest part to fathom is the fact that after 815 results from A System, which had a 58% Accuracy when only needing to be 52.5% accurate, is that those results can't be trusted to be better than 52.5% over the next 800 results?


text:
What are the main differences between a language model and a machine translation model?
What are the main differences between a language model and a machine translation model?

cf. Deep Learning "12.4 Natural Language Processing", "12.4.2 Neural Language Models", and passim.

The simple language model will give you the probability of a sequence of tokens(sentence) for that language. So lets say if you have trained a model for English language your model can give you the probability for any random english sentence.
Consider some sentence
$X$ $=$ "the quick brown fox jumps over the lazy dog" $=$ $x_1 \ x_2 \ x_3 \ ... \ x_n$
model will give you $P(X)$
Moreover if the model has been trained properly, in the following scenario, where
$X$ $=$ "the quick brown fox jumps over the lazy dog" $=$ $x_1 \ x_2 \ x_3 \ ... \ x_n$
$Y$ $=$ "dog brown quick fox over the jumps lazy the" $=$ $y_1 \ y_2 \ y_3 \ ... \ y_m$
model will always give $P(X) > P(Y)$, as it has learnt the structure of language.
On the other hand Machine translation model gives you the conditional probability of the next token given your source sentence and partial-target sentence.
So if
$X = $ "I am a student" and $Y = $ "je suis" $=y_1, \ y_2$
model will give you $P(y_3 | X,y1_,y_2)$ .
where $X$ is a source sentence and $Y= y_1, y_2$ is a partial target sentence. The probability of word/token "étudiant" would be maximum among all words of vocabulary.


text:
Is the self-attention matrix softmax output (layer 1) symmetric?
Let's assume that we embedded a vector of length 49 into a matrix using 512-d embeddings. If we then multiply the matrix by its transposed version, we receive a matrix of 49 by 49, which is symmetric. Let's also assume we do not add the positional encoding and we only have only one attention head in the first layer of the transformer architecture.
What would the result of the softmax on this 49 by 49 matrix look like? Is it still symmetric, or is the softmax correctly applied for each line of the matrix, resulting in a non-symmetric matrix? My guess would be that the matrix should not be symmetric anymore. But I'm unsure about that.
I ask this to verify if my implementation is wrong or not, and what the output should look like. I have seen so many sophisticated and different implementations of the transformer architecture with different frameworks, that I can't answer this question for myself right now (confusion). I still try to understand the basic building blocks of the transformer architecture.

I compared my results visually to a second implementation known to be working - "The annotated transformer".  I compared the pytorch calculation results of the attention-method to my implementation results.
The answe is - the softmax is applied row by row. Therefore the resulting matrix p-attn is not equal to its transposed version.


text:
Is Reinforcement Learning what I need for this image to image translation problem?
I have a paired dataset of binary images A and B: A1 paired with B1, A2-B2, etc., with simple shapes (rectangles, squares).
The external software receives both images A and B and it returns a number that represents the error.
I need a model that, given images A and B, can modify A into A' by adding or removing squares, so that the error from the software is minimized. I don't have access to the source code of the software so I don't know how it works.
I tried to make a NN that copies the functionality of the software, and a generative NN to generate the modified image A' but I haven't got good results.
The software can only receive binary images, so I cannot use a loss function because my last layer of the generator being a softmax, if I apply a threshold, I will lose the track of the gradients, so I cannot apply gradient descent.
Someone told me that when you cannot calculate the gradient of the loss with respect to the weights, reinforcement learning with policy gradients is a good solution.
I'm new to this field, so I want to be sure I'm going in the right direction.

This really doesn't sound like an RL problem to me. If the only issue you have is backpropagating through your thresholded softmax layer, I recommend you look into Gumbel Softmax. Maybe that can solve your problem?

Thank you for your reply harwiltz! I checked the Gumbel Softmax paper you mentioned, it is very interesting, I took a look into tensorflow implementation, tfp.RelaxedOneHotCategorical, the problem is that when you sample from the Gumbel distribution you get just the approximation of the one hot encoded vector, and my external software cannot receive any approximation, i'm forced to either Cast to int or modify the variables in such a way that I cannot do backward propagation

Ah, that's unfortunate. In PyTorch the implementation gives you the option to get the one hot representation. However I think you can actually get away with casting, I believe this is called "straight through sampling" or something.


text:
Can we give a command to an AI and wait for it to do the job without explicitly telling it how to do it?
I am a computer science student. I learned about programming languages recently, but I don't know much about artificial intelligence.
I want to know, why don't we program something in a way that we could tell the program

Hey! Do this for me!

And then just sit down and wait that the AI does the job?
Is this currently possible to do?

Normally when you write a program, you are acting like a boss that micromanages the job, telling the workers how to accomplish a task, perhaps without even letting them know what the purpose is. What you are hoping to be is a boss that gives the workers a goal and allows them to determine how to accomplish it.
In many ways, that is one of the aims of AI.
We already have small examples, such as a smart washing machine that weighs the clothes, monitors the amount of dirt in the water, and continually decides how much water to add or drain, when to agitate, when to rinse, and when to spin.  All you had to tell it was "clean these", and perhaps say what kind of material they are made of.
As a much larger example, automobiles were traditionally operated by turning the steering wheel to cause it to change direction and by pressing the pedals to increase or decrease the speed, but now we are working on car controllers that can respond to "go to Cleveland" by determining the best speed and direction by itself.
But note that those two examples (the first requiring only a little "intellegence", the second a lot) were for very specific requests that could be expressed in a few words.
As soon as the request becomes even slightly more complicated, the difficulty of creating a solution becomes much much more difficult.
Ask your "Do this for me!" request of a human being.
The first thing they'll do is ask "what does this mean?".
And then you'll have to give a lot of details.
And then they'll have questions about what you really want.
And so on.
Providing the requirements for "this" will be neither simple nor easy.
Human intellegence is still vastly superior to current AI.
In particular, it is capable of not only recognizing that additional information is required, but of having the intuitive ability to know what that missing information must be like.

Not writing my own answer because this hits many of the points I wanted to make, but I feel you're missing an important one. AI at the moment can not infer - it has no intuition. If you give it a complicated set of instructions, unless it has immense amounts of experience with similar instructions, the AI will fail miserably. An AI that could adequately decipher complicated instructions would (in my opinion) be self aware, so until we reach that point, I don't think a general AI for performing actions from complicated instructions can be a thing.

@Recessive, I've expanded the answer.  If I missed the point you wanted, feel free to edit the answer.

This is still a long way off and would require what is known as Artificial General Intelligence, which is probably the best descriptive term for AI with cognitive capacity akin to humans.
(Another term often used is "superintelligence", but it's less precise in regard to general intelligence, in that it could be understood to connote superior intelligence that is not general.  The same problem exists for Searle's "Strong AI", an AI that matches or exceeds humans capacity, since AI can now be "strong but narrow", exceeding human capability in a single task or a set of tasks, such as a game, or managing fully definable systems such as air-conditioning.)
Part of the problem is we still don't really understand how human cognition works, so the famous projection by an incredibly important figure in AI, Herbert A. Simon, that:

"Machines will be capable, within twenty years, of doing any work a man can do."

is still, like fusion power, potentially always 20 years in the future.  (Worth noting that if a general superintelligence is developed, it could not only do anything humans could do, but things we can't even conceive of, because its intelligence would greatly exceed our own, and would likely be capable of continual self-optimization.)


text:
Are there any existing ontologies that model engineering data?
Are there any existing ontologies available for "engineering" data?
By "engineering" I mean pertaining to the fields of electrical, mechanical, thermal, etc., engineering.

Can you provide more information on the motivation for your question?

@tmaric I wast looking for a structured data resource for engineering, so far have found wikidata and dbpedia but not sure how engineering specific they are and what level of completeness exists in either of them.

I'm asking because there's an effort at the Uni where I work to define a metadata standard for (mechanical) engineering, which is the answer to your question, so I would also like to know if there is something out there already.

@tmaric good to know that, so what level of completion is that project, is it available to the public or only for internal use?

We are just starting with this, but there are some preliminary results from Stuttgart: https://www.ub.uni-stuttgart.de/forschen-publizieren/forschungsdatenmanagement/projekte/dipl_ing/index.en.html  There is some information in English but a lot of the info is in German.

Here are the slides from that project in English: https://www.ub.uni-stuttgart.de/forschen-publizieren/forschungsdatenmanagement/projekte/dipl_ing/materials/output/MTSR2018_14_SchemberaIglezakis_EngMeta.pdf


text:
Are policy-based methods better than value-based methods only for large action spaces?
In different books on reinforcement learning, policy-based methods are motivated by their ability to handle large (continuous) action spaces. Is this the only motivation for the policy-based methods? What if the action space is tiny (say, only 9 possible actions), but each action costs a huge amount of resources and there is no model for the MDP, would this also be a good application of policy-based methods?