text:
Does SARSA(0) converge to the optimal policy in expectation if the Robbins-Monro conditions are removed?
The conditions of convergence of SARSA(0) to the optimal policy are :

The Robbins-Monro conditions above hold for $α_t$.
Every state-action pair is visited infinitely often
The policy is greedy with respect to the policy derived from $Q$ in the limit
The controlled Markov chain is communicating: every state can be reached from any other with positive probability (under some policy).
$\operatorname{Var}{R(s, a)} < \infty$, where $R$ is the reward function

The original proof of the convergence of TD(0) prediction (page 24 of the paper Learning to Predict by the Method of Temporal Differences) was for convergence in the mean of the estimation to the true value function. This did not require the learning rate parameter to satisfy Robbins-Monro conditions.
I was wondering if the Robbins-Monro conditions are removed from the SARSA(0) assumptions would the policy converge in some notion of expectation to the optimal policy?


text:
How should I define the state space for this life science problem?
I would like to ask for a piece of advice with regard to Q-learning. I am studying RL and would like to do a basic project applied to life science and calculate the reward. I have been trying to get my head around how to define all possible states of the environment.
My states are $S = ( \text{health } (4 \text{ levels}), \text{shape } (3 \text{ levels}) \}$. My actions are $A=\{a_1, a_2, \dots, a_4 \}$. My possible states are $60=4 * 3 * 5$. Could you advise whether these are correct?
$(s_{w_0, sh_0}, a_1, s'_{w_1, sh_1})$ is a tuple of the initial state $s_{w_0, sh_0}$, the first action $a_1$ and the next state $s'_{w_1, sh_1}$, where $w$ is the health level, $sh$ is the shape of the tumor.

Hi and welcome to AI SE! What is "w0,sh0" and "w1,sh1"? Also, when you say that your states are a set composed of "health" and "shape", does this mean that all states have these components and that you have 48 possible states?

S(w0,sh0) is the initial state.  The 48 possible states are the 4X3X5 , health levels(w), shape of a malign tumor and  actions of different doses of treatment.  Reading some manuscripts the description of the states are very vague. In my case I have a tuple of variables and the variables also have different levels. The agent is supposed to find the best combination to cure/improve the disease(health and and reduction of the shape of the tumor).

This is a fictitious scenario with only one patient to illustrate my understanding of Q-learning and compute the Q matrix manually.  I want to be sure that I got the states correct before moving to use a neural network for complex real data. Any help or comments would be highly appreciated.

It looks like you have 12 possible states not 48 (435 is 60 by the way). What do these 5 different doses of treatment mean? Are those different doses of actions $a_i$. For example does that mean there are 5 different doses of action $a_1$? If yes then that is an action and not part of the state so you would have 12 states (43) and 20 actions (45).

Thanks Brale, yes these are different doses of a1 and effectively it was four no five. Thanks a lot for your help.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to deal with approximate states when doing path planning?
If one is interested in implementing a path planning algorithm that is grid-based, one needs to consider the fact that your grid points will never represent the true state of the robot.
How is this dealt with?
Suppose we're doing path planning using a grid-based search on the side of the control for a desired grid position as an output state.
How would you handle the discrepancy between your actual starting position and your discretized starting position?
I understand that normally you may use an MPC instead, which continually recalculates an optimal path using some type of nonlinear solver, but suppose we don't do this - suppose we restrict ourselves to only a grid search and suppose at after every action the state of the robot has to be considered as living in a particular grid point.

So, will your robot use some kind of approximate odometry information that can be subject to drift and you will be performing path planning as the robot navigates the environment?

Not just drift, in other words, suppose your robot only knows what cell it is in but not its precise location within the cell. I guess the issue is "robust planning. After each action you can always assign the robot to it's closest cell, thus you could repeatedly call the planner after every action to address this problem but it will be VERY computationally expensive. So how does one do robust path planning?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Should the RL agent be trained in an environment with real-world data or with a synthetic model?
I want to train a reinforcement learning agent in an environment with parameters (for example, the wind speed, sun irradiation, etc.) that change over time. I have recorded a limited amount of data for these time series.
Should the RL agent be trained in an environment, which replays the recorded time series over and over, or should I model the time series with a generative model first and train the agent in an environment with these synthetic time series?
On the one hand, I think the RL algorithm will perform better with the synthetic data, because there are more diverse trajectories. On the other hand, I don't really have more data, because it is modelled after the same data the RL algorithm could learn from in the first place.
Are there any papers that elaborate on this topic?


text:
Can the cross-entropy loss be used for a NLP task with LSTM?
I am trying to build an LSTM model to generate Shakspeare-like poems. I have training set $\{s_1,s_2, \dots,s_m\}$, which are sentences of Shakespeare poems, and each sentence contains words $\{w_1,w_2, \dots,w_n\}$.
To my understanding, each sentence $s_i$, for $i=1, \dots,m$ is a random sequence containing the words $w_j$, for $j=1, \dots,n$. The LSTM model is estimated by applying the maximum likelihood (MLE) method, which will use cross-entropy loss for optimization. The use of MLE requires that the samples in the random sequence be independent and identically distributed (i.i.d), however, the word sequence $w_j$ is not i.i.d (since it is non-Markov). Therefore, I am suspicious about using cross-entropy loss for training an LSTM for the NLP task (which seems to be the common practice).

The negative log-likelihood should indeed be equivalent to the cross-entropy, provided that the independence assumption is made. In fact, I have derived their equivalence in my notes by making this assumption. So, if you use the cross-entropy, you're implicitly making the independence assumption.

Thanks for the edit and reply. I agree that negative log-likelihood is equivalent to cross-entropy when independence assumption is made. But for a NLP task, where the distribution for the next word is clearly not independent and identical to that of previous words, I am very suspicious on the adoption of cross-entropy loss. I wonder if there are any reasoning for using it or we can use any other loss for such task.

Well, while training your model, you will maybe provide the sentences (or words) in random order, this should have a "decorrelation effect". However, I don't know if this is sufficient. I would need to review my notes about CE and NLL and where exactly do we assume independence. Also, in some cases (e.g. naive Bayes), the independence assumption is made (only for simplicity and because it sometimes works), while it often doesn't hold.


text:
Why does the error ensemble use the ceiling of the number of classifiers?

What is $y$? Why is $k$ the ceil of $n/2$? What is $y \geq k$?

Where did you get this text/screenshot from? Which book, article, etc.?

This SE:AI post will shed light into the application of the binomial probability to this problem of ensembles.

Y ensemble size voting wrong
k = 50% or majority threshold

If you have 11 models. Then the majority of models is anything bigger than 50% of the number of ensemble models. In the example where you have 11 base models. The majority would be anything bigger than N/2 or 11/2. But since 11 is an odd number and cannot be divided by 2. We have to use the python ceiling function to round 5.5 to 6. The in other words. For your ensemble to be wrong. We must look for the probability that Y of them are wrong. Y is ≥ than K and K is ceil(N/2). Thus we must calculate this probability(6 wrong) like shown above summating the possibility of each of these discrete combinations into one


text:
Is an oracle that answers only with a "yes" or "no" dangerous?
I was thinking about the risks of Oracle AI and it doesn't seem as safe to me as Bostrom et al. suggest. From my point of view, even an AGI that only answers questions could have a catastrophic impact. Thinking about it a little bit, I came up with this Proof:
Lemma

We are not safe even by giving the oracle the ability to only answer yes or no.

Proof

Let's say that our oracle must maximize an utility function $\phi$, there is a procedure that encodes the optimality of $\phi$. Since a procedure is, in fact, a set of instructions (an algorithm), each procedure can be encoded as a binary string, composed solely of 0 and 1, Therefore we will have $\phi \in {\{0,1\}^n}$,  assuming that the optimal procedure has finite cardinality.  Shannon's entropy tells us that every binary string can be guessed by answering only yes/no to questions like: is the first bit 0? and so on, therefore we can reconstruct any algorithm via binary answers (yes / no).

Is this reasoning correct and applicable to this type of AI?

Can you clarify the relationship between your proof that uses Shannon's entropy and the fact that the oracle could be dangerous? How would the ability of guessing make an oracle dangerous?

My proof is simple: you can implement every procedure just by answering yes/no in the same way that you can compose every string just by answering yes/no.

How can an oracle that just guesses and cannot do anything else (e.g. move something) be dangerous? She may be used to retrieve information, etc., but, if she cannot act, apart from guessing questions, how can it be dangerous? Well, if someone uses the oracle to do something, that could be dangerous, but the oracle alone will not. What do you mean by "you can implement every procedure just by answering yes/no..." How can you compose a string by answering yes or no? Someone needs to ask the question anyway. Are you talking about any theory of computation concept (that I don't recall now)?

It can by fooling the humans that make the questions. Of course the procedure will be exponentially longer in respect to a "free and cross domain ai" but it can nonetheles reach its final goal.

Your argument is interesting. However, I think that it misses some details (and therefore could be flawed). For example, what is your definition of an oracle? When proving something, you precisely define your terms, so that everyone knows what you're talking about. I encourage you to define the oracle.

@nbro How reconstruction of an algorithm is related to safety?


text:
Why do momentum techniques not work well for RNNs?
AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy.
I've recently learned that it is not as helpful for RNNs, where plain SGD is preferred.
For example, Deep Learning by Goodfellow et. al says (section 10.11, page 401):

Both of these approaches have largely been replaced by simply using SGD (even without momentum) applied to LSTMs.

The author talks about LSTMs and "both of these approaches" refer to second-order and first-order SGD methods with momentum methods, respectively, according to my understanding.
What causes this discrepancy?

The momentum does not always help on CNN either

@mirror2image Sure, but I suppose it's less common to use it for RNNs than CNNs.


text:
How can I merge outputs of two separate layers so that the overall performance improves?
I am training a combined model (fine-tuned VGG16 for images and shallow FCN for numerical data) to do a binary classification. However, the overall AUC score is not what I expected it to be.
Image-only mean AUC after 5-fold cross-validation is about 0.73 and numeric data only 5-fold mean AUC is 0.65. I was hoping to improve the mean AUC by combining the models into one and merging output layers using concatenate in Keras.
img_output = Dense(256, activation="sigmoid")(x_1)

and
numeric_output = Dense(128, activation="relu")(x_2)

are the output layers of the two models. And,
concat = concatenate([img_output, numeric_output])
hidden1 = Dense(64, activation="relu")(concat)
main_output = Dense(1, activation='sigmoid', name='main_output')(hidden1)

is the way I concatenated them.
Since image-only performance was better I decided that it might be reasonable to have more dense layers for image_output (256) and ended up using 128 in numeric_output.I could only reach up to mean AUC of 0.67 using a combined model. I think I should rearrange the concatenation of two outputs somehow (by introducing another learnable parameter (like the formula (10) at 3.3 section of this work?, bias?, or something else) to get more boost on mean AUC. However, I was not able to find what options were available.
Hope you have some ideas worth trying.

I'm not sure it's possible to help much because this is an experimental question. I'm afraid the only answer comes with testing many different options.
I see a little thing that might be making your model a little worse, though:

You're concatenating "relu" with "sigmoid".

Placing two different nature values in the same array may make it more difficult for updating weights properly.
A few independent suggestions:

Make the output of the images model have "relu" activation as well before the concatenation. Preferrably, use a batch normalization before the image relu and before the number relu (this way you concatenate values that are in very similar ranges).
Instead of concatenating, you can try Multiply()([img_output, numeric_output]), in this case, both outputs must have the same size, one of them uses "relu" or "linear", and the other uses "sigmoid".

Now, something important when using AUC: you need big batch sizes, because AUC is dependent on the whole data, it's not like usual metrics/losses that you can take the mean from the results of each batch.

Thank you for insights. let me try them. BTW, why do you think, if multiplied, one should have "relu" or "linear", and the other "sigmoid"?

To keep roughly the same range as the other layers. Sigmoid is between 0 and 1, and ReLU is from 0 to unknown. Multiplying two ReLUs may produce blowing values, too high, multiplying two Sigmoids may decrease the values too much. Use a balanced multiplication.


text:
Understanding the derivation of the first-order model-agnostic meta-learning
According to the authors of this paper, to improve the performance, they decided to

drop backward pass and using a first-order approximation

I found a blog which discussed how to derive the math but got stuck along the way (please refer to the embedded image below):

Why
 disappeared in the next line.
How come  (which is an Identity matrix)

Update: I also found another math solution for this. To me it looks less intuitive but there's no confusion with the disappearance of  as in the first solution.

Regarding your 1st question, what is $\theta$? In all other cases, $\theta$ has a subscript.

θ was in the Algorithm 1, but then in the blog it uses both θ and $\theta_{0}$. Maybe they're both "initial model parameter" and the notation was not consistent. I'm working on understanding it too ..

@Long what is the rationality behind ignoring the 2nd-order derivative in FOMAML and regarding it as an identity matrix?

@S.EB it is common in many research that people ignore high-order derivatives to avoid computation overhead and simpler implementation. See this
The meta-optimization step above relies on second derivatives. To make the computation less expensive, a modified version of MAML omits second derivatives, resulting in a simplified and cheaper implementation, known as First-Order MAML (FOMAML)

@Long Thank you very much for your explanation. I have two questions: 1) why meta-learning is using two steps of gradients, also called bi-level optimization? 2) why it is said gradients of gradients? what is the exact meaning of it?

@S.EB I think these questions should be in a new post, the comment section will not be beneficial to those who have similar questions as you do. I will try to answer them and I think there are many people with better knowledge (than I have) would do so too.

$\nabla_{\theta_{i-1}} \theta_{i-1} = \mathbf{I}$ in a similar way that $\frac{d f}{dx} = 1$ for $f(x) = x$. Strictly speaking, $\mathbf{I}$ should be a vector of $1s$ with the same dimensionality as $\theta_{i-1}$, but they are probably abusing notation here and putting such a vector as the diagonal elements of a matrix. Alternatively (actually, the most likely reason!), they are computing the partial derivative of $\theta_{i-1}^j$ with respect to $\theta_{i-1}^k$, for all $k$, for all $j$, which will make up an identity matrix.
Regarding your first question, $\nabla_{\theta} \theta_{0}$ probably becomes 1, but I am not familiar enough with the math of this paper to tell you why. Maybe it's because $\nabla_{\theta} \theta_{0}$ actually means $\nabla_{\theta_0} \theta_{0}$. I would need to dive into it.

@DuttaA Well, I guess that what you say is consistent with my explanation. If you look at that part of the formula, you see terms where the gradient with respect to the previous parameters is taken of the current parameters. However, initially, we only have $\theta_0$ (i.e. no previous parameters), so the derivative of $\theta_0$ with respect to itself is $1$.

I updated my question with a second solution. Please have a look and give some comment about it. It seems like in this case we don't need to worry about the $\theta_{0}$  anymore.


text:
Is there a way to add "focus" on parts of the image when using CNNs?
I'm building a CNN/3DCNN model that classifies hand gestures. The problem is that the actual gesture occupies only like 1% of the whole image. That means that an enormous amount of convolutional operations is done on the "empty" parts of the image, which is useless.
Is there a way to solve this problem? I was thinking about a MaxPooling layer with a giant pool size, but near features that are extracted from the gesture will be probably "compressed" in only 1 feature.

I think you can enforce such behavior by modifying the loss function. However, is cropping images out of question in your case? Removing empty parts seem like a simpler approach to me.

Maybe you could use object detection to locate the hand first. Which technique you use really depends on the background. What is the background by the way?

By the way, "adding focus" is essentially what Faster R-CNN does by incorporating an RPN into the model.

I was also thinking about object detection first, but that would resize the hand to another resolution, causing images to have different sizes
EDIT: Nevermind, actually, after the hand gets detected, you can just resize the image to the resolution you want. The only problem is that I need datasets both for gestures and hand classification, wich can be a little tricky


text:
Is it possible to do token classification using a model such as GPT-2?
I am trying to use PyTorch's transformers as a part of a research project to do sentiment analysis of several types of review data (laptop and restaurant).
To do this, my team is taking a token-based approach and we are using models that can perform token analysis.
One problem we have encountered is that many of the models in PyTorch's transformers do not support token classification, but do support sequence classification. One such model we wanted to test is GPT-2.
In order to overcome this, we proposed using sequence classifiers on single tokens which should work in theory, but possibly at reduced accuracy.
This raises the following questions:

Is it possible to do token classification using a model such as GPT-2 using PyTorch's transformers?
How do sequence classifiers perform on single token sequences?


text:
How does positional encoding work in the transformer model?
In the transformer model, to incorporate positional information of texts, the researchers have added a positional encoding to the model. How does positional encoding work? How does the positional encoding system learn the positions when varying lengths and types of text are passed at different time intervals?
To be more concrete, let's take these two sentences.

"She is my queen"
"Elizabeth is the queen of England"

How would these sentences be passed to the transformer? What would happen to them during the positional encoding part?
Please explain with less math and with more intuition behind it.

There already two other similar questions on this site (both of them with no answer): this and this.


text:
Is a basic neural network architecture better with small datasets?
I'm currently trying to predict 1 output value with 52 input values. The problem is that I only have around 100 rows of data that I can use.
Will I get more accurate results when I use a small architecture than when I use multiple layers with a higher amount of neurons?
Right now, I use 1 hidden layer with 1 neuron, because of the fact that I need to solve (in my opinion) a basic regression problem.

I'm not aware of a direct way for finding the best NN architecture for a given task, but the recommended way, as far as I know, is to devise a network that can overfit the training data, and then apply regularization on top of it.
That way, you can be almost sure you're not underfitting/underperforming due to network capacity.

It's harder to overfit it certainly!
I mean practically speaking there has to be some assumptions on the generation model of your data, either explicit or implicit.
I would try probably 1-2 layer network first(maybe your data is linearly separable if you're lucky).


text:
How is the probability transition matrix populated in the Markov process (chain) for a board game?
Following on from my other (answered) question:
With regards to the Markov process (chain), if an environment is a board game and its states are the various position the game pieces may be in, how would the transition probability matrix be initialised? How would it be (if it is?) updated?

Check out this fun project on reddit. It tries to calculate probability of arriving each square in the game of Monopoly.

Transition probability matrix cannot be initialized. Your game world has some rules. Since we do not know these rules we can approximate them. To do this, we should run the game over and over and collect the outcome of each action.
Here is a toy example: assume we control a plane. We want the plane to fly forward, but sometimes there may be wind in a random direction. So the plane won't end up in the desired position. Instead, it has some probability to fly left, right and forward. After the data is collected, we can estimate this probabilities.
Another example is estimating n-gram model: assume we want to train a model that will suggest a word in search field. We would analyze a corpus and calculate all 2 word sequences (for bigram model). Then, if a user starts typing I the model would suggest the word am, because I am occurs more frequently then I did, for example, and hence, the transition probability from I to am is greater.
Here is good book on Language Modeling
But your question suggest me that you want something like reinforcement learning. Check out this video wich is a part of a great lectures series on RL by David Silver

authors: พงศกร พรรณโนพรรดิ์


text:
Is it recommended to remove stop words before named entity recognition?
Removing stop words can significantly speed up named entity recognition (NER) modeling by reducing the number of tokens in a document.
Are stop words critical to get correct NER performance?

It depends how you recognise the entities.
If you do a simple gazetteer lookup, then it could be faster, as you have fewer tokens to deal with.
However, if you use contextual rules, then stop words might be vital to identify certain contexts, so by removing stop words you lose information about the entity's environment. For example, if [work] at {organisation} is a rule you use to identify companies etc, then this wouldn't work if you take out the at.
You will also have problems if the stop words are part of an entity, eg the town Stoke-on-Trent. If hyphens are used to split tokens, then you won't be able to recognise it if the on is discarded.
In general I think stop words should mostly be kept; apart from information retrieval where they are not all that useful in an inverted index you will always lose something. If stop words were really pointless, then they would have been discarded from languages a long time ago. In the old days they were a useful way of reducing the demand on computational resources without losing too much, but nowadays I would say this is not really a problem anymore.


text:
Does the performance of a model increase if dropout is disabled at evaluation time?
I know dropout layers are used in neural networks during training to provide a form of regularisation in an attempt to mitigate over-fitting.
Would you not get an increased fitness if you disabled the dropout layers during evaluation of a network?

Keras turns off dropout when computing validation accuracy during training. Also when making predictions

Follow-up question: https://ai.stackexchange.com/questions/18397/why-is-my-validation-test-accuracy-higher-than-my-training-accuracy

Dropout is a technique that helps to avoid overfitting during training. That is, dropout is usually used for training.

units may change in a way that they fix up the mistakes of the other
units. This may lead to complex co-adaptations. This, in turn, leads to
overfitting because these co-adaptations do not generalize to unseen
data.

If you want to evaluate your model, you should turn off all dropout layers. For example, PyTorch's model.eval() does this work.
Note that in some cases dropout can be used for inference, e.g. to add some stochasticity to the output.
More about dropout:

Improving neural networks by preventing co-adaptation of feature detectors
Dropout: A Simple Way to Prevent Neural Networks from Overfitting

I did not know PyTorch automatically deactivated drop out layers

Dropout is usually disabled at test (or evaluation) time. For example, in Keras, dropout is disabled at evaluation time by default, although you can enable it, if you need to (see below). The purpose of dropout is to decorrelate the units (or feature detectors) so that they learn more robust representations of the data (i.e. a form of regularisation).
However, there's also Monte Carlo (MC) dropout, i.e., you train the network with dropout and you also use dropout at test time in order to get stochastic outputs (i.e. you will get different outputs, for different forward passes, given the same inputs). MC dropout is an approximation of Bayesian inference in deep Gaussian processes, which means that MC dropout is roughly equivalent to a Bayesian neural network.

Does the performance of a model increase if dropout is disabled at evaluation time?

Yes, possibly. However, MC dropout provides an uncertainty measure, which can be useful in certain scenarios (e.g. medical scenarios), where a point estimate (i.e. a single prediction or classification) is definitely not appropriate, but you also need a measure of the uncertainty or confidence of the predictions.


text:
Can you provide some pseudocode examples of what constitutes an AI?
After years of learning, I still can't understand what is considered to be an AI. What are the requirements for an algorithm to constitute Artificial Intelligence? Can you provide pseudocode examples of what constitutes an AI?

Here's a very similar question What are the minimum requirements to call something AI?. I won't close it as a duplicate because here you are also asking for pseudocode examples.

Philosophically, my own research has led me to understand AI as any artifact that makes a decision.  This is because the etymology of "intelligence" strongly implies "selecting between alternatives", and these meanings are baked in all the way back to the proto-Indo-European.
(Degree of intelligence, or "strength" is merely a measure of utility, typically versus other decision making mechanisms, or, "fitness in an environment", where an environment is any action space.)
Therefore, the most basic form of automated (artificial) intelligence is:
if [some condition]
then [some action]

It is worth noting that narrow AI which matches or exceeds human capability, in the popular sense, manifested only recently when we had sufficient processing and memory to derive sufficient utility from statistical decision making algorithms. But Nimatron constitutes perhaps the first functional strong-narrow AI in a modern computing context, and the first automated intelligence are simple traps and snares, which have been with us almost as long as we've used tools.
I will leave it to others to break down all the various forms of modern AI.

The expression strong-narrow AI is not a standard one and may be misleading or confusing. You're talking about narrow AIs that are really good, such as AlphaGo, but maybe you could use another adjective other than "strong". Btw, I think that "strong AI" is really a bad expression. AGI is definitely a better expression.

@nbro I noticed that certain scholars started using "strong narrow" after the advent of AlphaGo.  I think you're technically correct that "strong" is not strictly required, since it references degree of intelligence, as opposed to the nature of intelligence.  Here though, I am using it to connote intelligence that matches or exceeds human intelligence.  Perhaps I need to define that?  (PS—I agree fully that the traditional use of "strong AI" is archaic, and that AGI should be used, because it is more explicit—general intelligence is only implied in the former term.)

AI is not a simple term. There are different types, ranging from the most simplistic rule-based AI to black-box AI's so complicated it's unreasonable for a human to understand exactly what they're doing.
There's no pseudocode that if used in a program automatically constitutes it as an AI. It's not that black and white. But I can give examples:
Here's a rule-based chess AI that forfeits if it's too far behind, and plays aggressively if it's far enough ahead.
if player.score - my.score > 10:
    forfeit
elif my.score - player.score > 10:
    agressive = True

for each piece of my.pieces:
    for each square of board.squares:
        if noThreats(square) and agressive is True:
            move(piece, square)
            return

This is considered an "AI" because it feigns intelligence - appearing to have a true understanding of chess while simply following a set of rules, making it an "Artificial" Intelligence.
Here's another more complicated AI:
decisionNet = NeuralNetwork(64 inputs, 2 outputs)

choice = decisionNet(board.squares) // Returns a chess square with one of my pieces and desitnation
move(choice)

This uses a neural network to make the decision, which could have been trained on a bunch of example games or against itself. Due to this "training phase", humans can't understand precisely what the network is doing without extensive effort, so it's an even gives an even more convincing understanding of chess. But if we want, we could still understand the nuances of this network, and show it doesn't possess an intelligence, it again only feigns it.
I should mention that virtually any code that has an if statement can be considered AI. The examples I provided are just easier to pass off as understanding a very complicated concept (chess), as opposed to, say, verifying a user login. They both have the same fundamentals, it's just one appears more complicated on the surface than the other.

Do all AIs include the if statement or it depends on the type of AI?

Another question is:
"How to build intelligent behavior from natural (like) language pseudo code?"
I am layering very natural like language above Python code and modules.
This will generate and execute code from english NLL pseudo code.
But, pseudo code is just code if the BNF definition works.
You can see the basics below.
I'm adding this NLL to my Iceberg/Sagent intelligent agent AI.
Theoretically, that will be more like what you mean by "AI"
As the first answer says, "AI as any artifact that makes a decision"
I would elaborate by saying, "An AI performs an action by making a decision"
My Iceberg/Sagent AI uses a layer above python modules turning them into Sagents (or Intelligent agents).
Sagents process data using a type of decision tree methodology.
This will be programmed via instincts, basically built-in (bootstrap) NLLs.
Answer 1 said, "sufficient processing and memory to derive sufficient utility from statistical decision making"
By sufficient, I would agree there needs to be a large body of
instinctive or bootstrapped performance to get the AI to the point
where it can grow its knowledge in an unattended way.
But just as you should never leave a child unattended,
the same is true for an AI. I've had unattended experiments that have gone a bit bat shit. I'm worried about this aspect of AI, that someone will create a dangerous AI and turn it loose. "AI Robot" could be a real issue at some point in the next few years.
Anyway, below is a simplistic thought starter on NLL pseudo code:

Built-Ins
|"execute python", code|
|"python",    code|
|"execute c", code|

ai_def( "a human sentence" ):
    sentence = |"python","input("Enter a sentence: " |
    return( sentence)

ai_def( "remove stopwords from", tokens ):
    stopwords = |"python","nltk.corpus.stopwords.words('english')"|
    non_stops = |"python","tokens.difference(stopwords)"|
    return( non_stops )

ai_def( "keywords from", sentence ):
    tokens = |"python","nltk.word_tokenize( sentence)"|
    keys = |"remove stopwords from", tokens|
    return( keys )

ai_def( "add to sayings"):
     keys = |"keywords from",saying=|"a human sentence"||
     |"add to sayings dictionary using", keys, saying|

ai_def( "get saying for human" ):
   subject, verb = |"get subject verb of", saying=|"a human sentence"||
   match = |"find saying matched from", subject, verb|
   report( match)

Anybody interested?
Or, if something already exists, let me know.


text:
How can I implement the reward function for an 8-DOF robot arm with TRPO?
I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to write the code for the reward functions and the algorithm for the joint space motion.

The most important part of the RL is the reward function. If we want an agent to do something speciﬁc, we must provide rewards to it in such a way that it will achieve the goal. It is thus very important that the reward function accurately indicates the exact behavior.
Assume, the robot's goal is to reach the desired position as fast as possible. You can construct your reward function so, that it will take into account the Euclidean distance to the position. If the arm moves to the position directly, you will reward the agent with a positive value, otherwise, you will punish it with a deviation from the direct line. You probably have other parameters of the joints, such as position and velocity. It can be also included in your reward function, in order to find optimal movements.

Check out this video from the free udacity overview course on RL and this paper "Setting up a Reinforcement Learning Task with a Real-World Robot"
Here is also related DeepMind's article and paper
I also have a project on github, where I implemented custom Gazebo environment for OpenAI Gym. This allows you to run the test even on a Jupyter Notebook. Check out my example


text:
What are examples of approaches to create an AI for a fighting robot in an MMO game?
I have an MMO game where I have players. I wanted to invent something new to the game, and add player-bots to make the game be single-playable as well. The AI I want to add is simply only for fighting other players or other player-bots that he sees around at his level.
So, I thought of implementing my fighting strategy, exactly how I play, to the bot which is basically using if statements and randoms. For example, when the opponent has low health, and the bot has enough special attack power, he will use this chance and use his special attack power in order to try to knock the opponent down, or if the bot has low health he will eat in time but not too much because there is a point in risking fights, if you eat too much your opponent will do too. Or, for example, if the bot detects the opponent player is eating too much and gains health, he will do the same.
I told this idea of the implementation to one of my friends and he simply responded with: This is not AI, it's simply just a set of conditions, it does not have any heuristic functions.
For that type of game, what are some ideas to create a real AI to achieve these conditions?
Basically, the AI should know what to do in order to beat the opponent, based on the opponent's data such as current health, Armour and weapons, and level, if he risks his health or not and so on.
I am a beginner and it really interests me to do it in the right way.

I would guess that your friend has not heard of expert systems https://en.wikipedia.org/wiki/Expert_system - a lot of video game AI is expert systems plus pathfinding algorithms. Whether or not someone calls it "real" AI is kind of secondary to is the game playing fun, but I am sure someone here can help you with some guidance. You might also consider searching on https://gamedev.stackexchange.com/ for advice

An expert system is really old tech, and extremely tedious to code (code a pile of conditions, test, fix bugs, improve, test, fix bugs, improve, etc). If the OP is new, he might as well go straight into what Oliver mentioned instead of coding an expert system. There are enough open source AI systems out there that people really should go into it if they're interested.

What does "make the player be single-player" mean?

Also, game programming people would call it an AI. Machine learning people would not call it an AI. Players don't care whether your game AI has machine learning or not, only how good it is.

@user253751 I meant that with proper bot AI you can make your game be playable as a singleplayer, no need for other players

But if I think about it, when I play they game, my mind also kinda performs a condition, like I personally will pull out my special weapon if he has a low health, or i will eat more food if he eats too to make me safe aswell as he does. but the thing is, my mind somehow knows that this is the right way or atleast thinks so because it helps me win fights if I do so. Thats what interests me aswell, how can I achieve that kind of a machine learning?

I think this question has potential for deeper answers, I started a bounty because this concept really interests me, to see what other answers I can get for this.

I would set up a list of goals for your bot. These could be 'maintain a minimum level of health', 'knock out human player', 'block way to location X', etc. This obviously depends on the domain of your MMO.
Then you can use a planner to achieve these goals in the game. You define a set of actions with preconditions and effects, set the current goal, and the planner will work out a list of actions for the bot to achieve the goal. You can easily express your actions (and the domain) in PDDL.
Examples for actions would be 'move to location X', 'eat X', 'attack player X'. A precondition of 'attack player X' could be 'health(X) is low', and an effect could be 'health(X) is reduced by Y'. There are different ways of expressing these depending on the planner's capabilities.
The beauty of this is that you don't actually have to explicitly code any behaviour. You describe the domain, and tell the bot what it should achieve, and what capabilities it has. The actual behaviour then emerges out of that description. If the bot only attacks a player if the player has lower health, then observing the player eat (and thus up their health) could result in the bot eating (to push its own health above the player's so that it can attack) — but you have not told the bot directly to do that.
For a starting point, go to http://education.planning.domains/ for a list of resources.
If you only have a few actions available, it might appear predictable to a human user, but with a variety of goals and actions, this will quickly become more complex and seem more 'intelligent'.
Update: Here is a link to a paper, Applying Goal-Oriented Action Planning to Games, which describes how this can be applied in a game.

Oliver Mason's answer is great for specific methods and tools to use, but I wanted to pull out a more general principle which was mentioned in a comment.
The distinction your friend is making is not one that would be generally recognised. One of my university lecturers defined AI as something like "an artificial system that exhibits behaviour that resembles how an intelligent being would behave".
If an intelligent being would always use the special attack in a particular situation, then an algorithm that always does so in the same situation is behaving intelligently, even though the algorithm behind it is incredibly simple. If you can come up with a complete description of an intelligent player, you have what is called an expert system, i.e. a system which captures the decision-making process of a real expert.
Your friend is not even correct that your proposed AI "does not have any heuristic functions". When you write a condition like "if the AI's health is below 50%, it will eat food", you're approximating the rule a human would use. You can make the heuristic more complex by increasing the probability of eating in proportion to current health; that might in turn make the heuristic closer to optimal.
You can only really find out how "good" your AI is by putting it into different situations and observing it - sometimes, a simple set of rules gives rise to "emergent behaviours" that look surprisingly intelligent. As you build up more complex rules - i.e. more optimal heuristics - the emergent behaviour will change, and you can tweak it for the desired effect.

Yes, I agree. Presumably the issue here hinges on 'predictability', in the sense that a very predictable behaviour does not appear very 'intelligent', whereas a more complex and varied behaviour which is less predictable seems to be more intelligent (when it isn't really).

You can train your bot using reinforcement learning (in particular Q-Learning).

The most important part of the RL is a reward function. If we want agent to do some thing speciﬁc, we must provide rewards to it in such a way that it will achieve our goals. It is thus very important that the reward function accurately indicates the exact behaviour

So you can construct your own reward function that will satisfy your requirements. If the bot does something you want, you will reward it with higher score, otherwise you will punish it with negative reward.
AlhpaGo and OpenAI teams used a similar technique to train their model which could then beat humans in games like Go, StarCraft 2 and Dota 2
Also, check out this Deep Reinforcement Learning Free Preview on udacity.

What about GANs or genetic algorithms?
The first idea (GAN) is that you basically create 2+ random bots who fight each other, and they keep adjusting their weights so that they can beat the other bot. That means, that those 2+ bots keep improving their "fighting performance" for as long as you want, eventually being even better than humans.
The second idea (genetic algorithms) is to generate a lot of bots who genetically differ from each other just for a slight mutation. You make them fight, and the best perforing/last standing will become the new clone where the new "lot of bots" gets generated from.

If you are willing to take an evolutionary approach, you may employ the NEAT algorithm (Neural Evolution of Augmenting Topologies) to train your bot. It will take some work setting it up and all, but it then will gradually improve over time.
Check out the following:

http://gekkoquant.com/2016/03/13/evolving-neural-networks-through-augmenting-topologies-part-1-of-4/
https://www.youtube.com/watch?v=WiPZSieT6qs
https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies

That should be enough to pique your interests and get you started. That last link links to a number of NEAT implementations available in a number of languages.


text:
How does one continue the pre-training in BERT?
I need some help with continuing pre-training on Bert. I have a very specific vocabulary and lots of specific abbreviations at hand. I want to do an STS task. Let me specify my task: I have domain-specific sentences and want to pair them in terms of their semantic similarity. But as very uncommon language is used here, I need to train Bert on it.

How does one continue the pre-training (I read the GitHub release from google about it, but don't really understand it) Any examples?
What structure does my training data need to have, so that BERT can understand it?
Maybe training BERT from scratch would be even better. I guess it's the same process as continuing the pretraining just the starting checkpoint would be different. Is that correct?

Also, very happy about all other tips from you guys.

I’m looking for answers to the same question. This may help: https://github.com/allenai/dont-stop-pretraining Not dug into the code yet, but it’s all I’ve found so far. Did you have more luck? Cheers
B


text:
Is there any way of generating fixed-length sequences with RNNs?
Is there any way of generating fixed-length sequences with RNNs? I want to tell my character level RNN to generate a name of length 3, 4, 5 and so on. I haven't found anything online like this, but my intuition tells me that, if I append the sequence length (e.g. 5) at each RNN input, the RNN should be able to do this. Does anyone know if this task is possible?

What is the downstream case? Are you just generating from nothing, but want the random generated name to be a specific length?

Yeah that's exactly it


text:
How can I process neural network with 25000 input nodes?
I'm trying to build a neural network between protein sequence and its drug fingerprint. My input size is 20000. The output size is 881. The sample size is 610.
Can I process this huge neural network? But how? And in which tool?

it seems sort data sample to adjust a NN and get good predictions from it...when you adjust and try please let us know if you have got useful trained patterns from this data. Also, take into account that NN recognize patterns, I doubt you can find a correlation between slices in your proteins and drug fingerprint slices.

Yes, it should be no problem.
When you decide to use a CNN, you have to make sure that this makes sense. Another answer mentioned using 3x3 convolutions -- which I would recommend against. For that to work, you would need to turn your vector into a rectangular array, and you would be implying a structure that isn't there.
Use one-dimensional convolutions instead.

It sure is possible, imagine a CNN can handle way bigger number of inputs. An image with size of 512x512 has already 262144 input nodes when re-arranged to a one-row vector. The trick sicne 2012/2014 is to use Convolutions, and deep ones, so stacking a lot of 3x3 Convolutions for example. Its way less sensitive than a fully-connected Dense network and needs a siginificant amount of less parameters. For more check this out, chapter 9 : Ian-Goodfellow, Deep Learning
Tools for that are tensorflow and keras based on python, or tensorflow-js on java, you can also use pytorch but the community is rather small on comparison.

The 3x3 convolution only makes sense for data that is inherently two-dimensional. When you fold a one-dimensional vector into a two-dimensional array, you are imposing a structure that is not really there: two points that are vertical neighbors don't necessarily have any relationship to each other; only horizontal neighborhoods matter.


text:
Is there any wrong in my focal loss derivation?
Assume $\mathbf{X} \in R^{N, C}$ is the input of the softmax $\mathbf{P} \in R^{N, C}$, where $N$ is number of examples and $C$ is number of classes:
$$\mathbf{p}_i = \left[ \frac{e^{x_{ik}}}{\sum_{j=1}^C e^{x_{ij}}}\right]_{k=1,2,...C} \in R^{C} \mbox{ is a row vector of } \mathbf{P}$$
Consider example $i$-th, because softmax function $\mathbf{p}:R^C \mapsto R^C$ (eliminate subscript $i$ for ease notation), so the derivative of vector-vector mapping is Jacobian matrix $\mathbf{J}$:
$$\mathbf{J}_{\mathbf{p}}(\mathbf{x}) = \left[ \frac{\partial \mathbf{p}}{\partial x_1}, \frac{\partial \mathbf{p}}{\partial x_2}, ..., \frac{\partial \mathbf{p}}{\partial x_C}  \right] =
\begin{bmatrix}
    \frac{\partial p_1}{\partial x_1} & \frac{\partial p_1}{\partial x_2} & \dots & \frac{\partial p_1}{\partial x_C} \\
    \frac{\partial p_2}{\partial x_1} & \frac{\partial p_2}{\partial x_2} & \dots & \frac{\partial p_2}{\partial x_C} \\
    \dots & \dots & \dots & \dots \\
    \frac{\partial p_C}{\partial x_1} & \frac{\partial p_C}{\partial x_2} & \dots & \frac{\partial p_C}{\partial x_C}
\end{bmatrix}
\in R^{C, C}
$$
$\mathbf{J}_{\mathbf{p}}(\mathbf{x})$ is called the derivative of vector ${\mathbf{p}}$ with respect to vector $\mathbf{x}$
$$\mbox{1) Derivative in diagonal:}\frac{\partial p_{k}}{\partial x_{k}} = \frac{\partial}{\partial x_{k}}\left( \frac{e^{x_k}}{\sum_{j=1}^C e^{x_j}} \right)$$
$$ = \frac{\left( \frac{\partial e^{x_k}}{\partial x_k} \right)\sum_{j=1}^C e^{x_j} - e^{x_k}\left(\frac{\partial \sum_{j=1}^C e^{x_j}}{\partial x_k} \right)}{\left(\sum_{j=1}^C e^{x_j}\right)^2} \mbox{  (Quotient rule) }$$
$$ = \frac{e^{x_k}\sum_{j=1}^C e^{x_j} - e^{x_k} e^{x_k}}{\left(\sum_{j=1}^C e^{x_j}\right)^2} = \frac{e^{x_k}(\sum_{j=1}^C e^{x_j} - e^{x_k})}{\left(\sum_{j=1}^C e^{x_j}\right)^2}$$
$$ = \frac{e^{x_k}}{\sum_{j=1}^C e^{x_j}} \left(\frac{\sum_{j=1}^C e^{x_j} - e^{x_k}}{\sum_{j=1}^C e^{x_j}}\right) = \frac{e^{x_k}}{\sum_{j=1}^C e^{x_j}} \left(1 - \frac{e^{x_k}}{\sum_{j=1}^C e^{x_j}}\right)$$
$$\rightarrow \frac{\partial p_{k}}{\partial x_{k}} = p_{k}(1-p_{k})$$
$$\mbox{2) Derivative not in diagonal } k \neq c \mbox{ :} \frac{\partial p_{k}}{\partial x_{c}} = \frac{\partial}{\partial x_{c}}\left( \frac{e^{x_k}}{\sum_{j=1}^C e^{x_j}} \right)$$
$$ = \frac{\left( \frac{\partial e^{x_k}}{\partial x_c} \right)\sum_{j=1}^C e^{x_j} - e^{x_k}\left(\frac{\partial \sum_{j=1}^C e^{x_j}}{\partial x_c} \right)}{\left(\sum_{j=1}^C e^{x_j}\right)^2} \mbox{  (Quotient rule) }$$
$$ = \frac{0 - e^{x_k} e^{x_c}}{\left(\sum_{j=1}^C e^{x_j}\right)^2} = -\frac{e^{x_k}}{\sum_{j=1}^C e^{x_j}}\frac{e^{x_c}}{\sum_{j=1}^C e^{x_j}}$$
$$\rightarrow \frac{\partial p_{k}}{\partial x_{c}} = -p_{k}p_{c}$$
$$\rightarrow \mathbf{J}_{\mathbf{p}}(\mathbf{x}) =
\begin{bmatrix}
    p_1(1-p_1) & -p_1p_2 & \dots & -p_1p_C \\
    -p_2p_1 & p_2(1-p_2) & \dots & -p_2p_C \\
    \vdots & \vdots & \ddots & \vdots \\
    -p_Cp_1 & -p_Cp_2 & \dots & p_C(1-p_C)
\end{bmatrix}$$

Focal Loss: $\displaystyle FL = -\sum_{k=1}^C y_{k} \alpha_{k}(1-p_k)^\gamma \log (p_k)$

$$\nabla_{\mathbf{x}} FL = \nabla_{\mathbf{p}}FL (\mathbf{J}_{\mathbf{p}}(\mathbf{x}))^T $$
$$\nabla_{\mathbf{p}} FL =
\begin{bmatrix}
    \frac{\partial FL}{\partial p_1}\\
    \frac{\partial FL}{\partial p_2}\\
    \vdots \\
    \frac{\partial FL}{\partial p_C}
\end{bmatrix} \mbox{ where } \frac{\partial FL}{\partial p_k}
= - y_k\alpha_k \left(-\gamma(1-p_k)^{\gamma-1} \log(p_k) + \frac{(1-p_k)^\gamma}{p_k}  \right) = y_k \alpha_k\gamma(1-p_k)^{\gamma-1}\log(p_k) - y_k\alpha_k\frac{(1-p_k)^\gamma}{p_k}$$
$$\nabla_{\mathbf{x}} FL =
\begin{bmatrix}
    \frac{\partial FL}{\partial p_1}\\
    \frac{\partial FL}{\partial p_2}\\
    \vdots \\
    \frac{\partial FL}{\partial p_C}
\end{bmatrix}^T
\begin{bmatrix}
    \frac{\partial p_1}{\partial x_1} & \frac{\partial p_1}{\partial x_2} & \dots & \frac{\partial p_1}{\partial x_C} \\
    \frac{\partial p_2}{\partial x_1} & \frac{\partial p_2}{\partial x_2} & \dots & \frac{\partial p_2}{\partial x_C} \\
    \vdots & \vdots & \ddots & \vdots \\
    \frac{\partial p_C}{\partial x_1} & \frac{\partial p_C}{\partial x_2} & \dots & \frac{\partial p_C}{\partial x_C}
\end{bmatrix} =
\begin{bmatrix}
    \sum_{k=1}^C \left(\frac{\partial FL}{\partial p_k}\frac{\partial p_k}{\partial x_1}\right)\\
    \sum_{k=1}^C \left(\frac{\partial FL}{\partial p_k}\frac{\partial p_k}{\partial x_2}\right)\\
    \vdots \\
    \sum_{k=1}^C \left(\frac{\partial FL}{\partial p_k}\frac{\partial p_k}{\partial x_C}\right)
\end{bmatrix}^T \in R^C
$$
$$\mbox{Case 1: }\displaystyle \frac{\partial FL}{\partial p_k}\frac{\partial p_k}{\partial x_k} \forall k=1,2,...,C$$
$$\frac{\partial FL}{\partial p_k} \frac{\partial p_k}{\partial x_k} = y_k \alpha_k\gamma(1-p_k)^{\gamma-1}\log(p_k)p_k(1-p_k) - y_k\alpha_k\frac{(1-p_k)^\gamma}{p_k}p_k(1-p_k)$$
$$ = y_k \alpha_k (1-p_k)^{\gamma}(\gamma p_k \log(p_k) - 1 + p_k) $$
$$\mbox{Case 2: } (k \neq c)\displaystyle \frac{\partial FL}{\partial p_k}\frac{\partial p_k}{\partial x_c}$$
$$\frac{\partial FL}{\partial p_k} \frac{\partial p_k}{\partial x_c} = - y_k\alpha_k\gamma(1-p_k)^{\gamma-1}\log(p_k)p_kp_c + y_k\alpha_k\frac{(1-p_k)^\gamma}{p_k}p_kp_c$$
$$ = - y_k\alpha_k (1-p_k)^{\gamma-1}p_c(\gamma p_k \log(p_k) - 1 + p_k) $$
$$\mbox{For each } d=1,2,...,C \mbox{ : }\sum_{k=1}^C \left(\frac{\partial FL}{\partial p_k} \frac{\partial p_k}{\partial x_d}\right) = y_d \alpha_d (1-p_d)^{\gamma}(\gamma p_d \log(p_d) - 1 + p_d) + \sum_{c \neq d}^C \left( - y_d\alpha_d (1-p_d)^{\gamma-1}p_c(\gamma p_d \log(p_d) - 1 + p_d) \right) = y_d\alpha_d(1-p_d)^{\gamma-1}(\gamma p_d \log(p_d) - 1 + p_d)\left(1-p_d -\sum_{c \neq d}^C(p_c)\right) $$
$$\rightarrow \nabla_{\mathbf{x}} FL = \left[ y_d\alpha_d(1-p_d)^{\gamma-1}(\gamma p_d \log(p_d) - 1 + p_d)\left(1-p_d -\sum_{c \neq d}^C(p_c)\right) \right]_{d=1,2,...,C}$$
However, the problem is $\left(1-p_d -\sum_{c \neq d}^C(p_c)\right) = 0$ (because sum of all probabilities is 1) then the whole expression collapses to $0$.
Is there any wrong in my focal loss derivation?
Reference:

https://en.wikipedia.org/wiki/Softmax_function
https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant
https://arxiv.org/abs/1708.02002


text:
How do we minimize loss for a single neuron with a feedback?
Suppose we had a series of single-dimensional data points $X = \{x_1, x_2, \dots, x_n \}$, where $n$ is the number of data points and there corresponding output values $T = \{t_1, t_2, \dots, t_n \}$.
Now, I want to train a single neuron network given below to learn from the data (the model is bad, but I just wanted to try it out as an exercise).

The output function of this neuron would be a recursive function as:
$$
y = f(a_0 + a_1x + a_2 y)
$$
where
$$
f(x) = \frac{1}{1 + e^{-x}}
$$
for a given $x$.
The error function for such a model would be:
$$
e = \sum_{i=1}^N (y_i - t_i)^2
$$
How should I minimise this loss function? What are the derivatives that I need to use to update the parameters?
(Also, I am new to this problem, therefore it would be really helpful if you tell me sources/books to read about such problems.)


text:
Which deep reinforcement learning algorithm is appropriate for my problem?
My task is to solve an optimization problem with deep reinforcement learning. I read about several algorithms like DQN, PPO, DDPG, and A2C/A3C but use cases always seem to be problems like video games (sparse rewards, etc.) or robotics (continuous action spaces, etc.). Since my problem is an optimization issue, I wonder which algorithm is appropriate for my setting:

limited number of discrete actions (like 20)
high-dimensional states (like 250 values)
instant reward after every single action (not only at the end of an episode)
a single action can affect the state quite a lot

There's no "goal" like in a video game, an episode ends after a certain number of actions. I'm not quite sure which algorithm is appropriate for my use case.

How long is the average episode? And on how many episodes will you train?

Getting feedback from the environment is pretty slow so there's a max. number of actions in each episode. The number greatly depends on the hyperparameters so I can't really tell for now.

Theoretically video games and robotics problems are also about optimization(getting maximum reward). So, just like other reinforcement learning problems, I would expect PPO to be the most efficient in your case too. I don't think a "goal" is necessary for rl, all you need is the rewards.

In RL, the reward defines the goal. RL isn't the most appropriate approach to solve many optimization problems, because it can be very expensive to train good agents for real-world problems. Games are actually a very limited scenario. Even games like Go, which are combinatorially difficult to solve, are limited, in the sense that there aren't many external sources of noise, etc.


text:
How are the weights retained for filters for a particular class in a CNN?
I am new to CNN. What I have learned so far about the filters is that when we are giving a training example to our model, our model updates the weights by gradient descent to minimize the loss function.
So my question is how the weights are retained for a particular class label?
The question is vague as my knowledge is vague. It's my 4th hour to CNN.
For example, if I am talking about the MNIST dataset with 10 labels. Let's say I am giving 1 image to my model initially. It will have a bigger loss for the forward pass. Let's say now it came for the back pass and adjusted the weights for and minimized the loss function for that label.
Now, when a new label arrives for training, how will it update the weights for filters which have already been updated according to the previous label?

Hi and welcome to AI SE! Your question is indeed unclear and vague. Are you familiar with gradient descent and back-propagation?

Hey I read about it from some links. Can you provide me an easy to go source ?

Besides the last layer rest of the weights are shared among all classes. When an image is passed to the network all weights are updated accordingly. The only weights that are directly responsible for one specific class are the ones of the final layer. The rest of the weights are updated to find the best values to minimize the average loss for all classes.
To rephrase there aren't "filters" in convolutional layers that are specific for a single class. They are used to extract features so that the final layer can make the final prediction (which has weights for each specific class).
I'd suggest you look a bit into how gradient descent and backpropagation works.


text:
How to add a dense layer after a 2d convolutional layer in a convolutional autoencoder?
I am trying to implement a convolutional autoencoder with a dense layer at the bottleneck to do some dimensional reduction. I have seen two approaches for this, which aren't particularly scalable. The first was to introduce 2 dense layers (one at the bottleneck and one before & after that has the same number of nodes as the conv2d layer that precedes the dense layer in the encoder section:
input_image_shape=(200,200,3)
encoding_dims = 20

encoder = Sequential()
encoder.add(InputLayer(input_image_shape))
encoder.add(Conv2D(32, (3,3), activation="relu, padding="same"))
encoder.add(MaxPooling2D((2), padding="same"))
encoder.add(Flatten())
encoder.add(Dense(32*100*100, activation="relu"))
encoder.add(Dense(encoding_dims, activation="relu"))

#The decoder
decoder = Sequential()
decoder.add(InputLayer((encoding_dims,)))
decoder.add(Dense(32*100*100, activation="relu"))
decoder.add(Reshape((100, 100, 32)))
decoder.add(UpSampling2D(2))
decoder.add(Conv2D(3, (3,3), activation="sigmoid", padding="same"))

It's easy to see why this approach blows up as there are two densely connected layers with (32100100) nodes each or more or in that ballpark which is nuts.
Another approach I have found which makes sense for b/w images such as the MNIST stuff is to introduce an arbitrary number of encoding dimensions and reshape it (https://medium.com/analytics-vidhya/building-a-convolutional-autoencoder-using-keras-using-conv2dtranspose-ca403c8d144e). The following chunk of code is copied from the link, I claim no credit for it:
#ENCODER
inp = Input((28, 28,1))
e = Conv2D(32, (3, 3), activation='relu')(inp)
e = MaxPooling2D((2, 2))(e)
e = Conv2D(64, (3, 3), activation='relu')(e)
e = MaxPooling2D((2, 2))(e)
e = Conv2D(64, (3, 3), activation='relu')(e)
l = Flatten()(e)
l = Dense(49, activation='softmax')(l)
#DECODER
d = Reshape((7,7,1))(l)
d = Conv2DTranspose(64,(3, 3), strides=2, activation='relu', padding='same')(d)
d = BatchNormalization()(d)
d = Conv2DTranspose(64,(3, 3), strides=2, activation='relu', padding='same')(d)
d = BatchNormalization()(d)
d = Conv2DTranspose(32,(3, 3), activation='relu', padding='same')(d)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(d)

So, is there a more rigorous way of adding a dense layer after a 2d convolutional layer?

For me, this worked perfectly. I encoded with conv2d and dense and then I flatten I and reshape in the decoder after the dense layer so the encoder and decoder are symmetrical. The only difference is that in my case I use images (224,224,1)
# create encoder
# 28,28 -> 1st conv2d (filter 3x3,relu activation, padding, strides == 'jumps')
self.encoder = tf.keras.Sequential([layers.Input(shape=(224,224,1)),
                                    layers.Conv2D(16,kernel_size=3,activation='relu',padding='same',strides=2),
                                    layers.Conv2D(8,kernel_size=3,activation='relu',padding='same',strides=2),
                                    layers.Conv2D(4,kernel_size=3,activation='relu',padding='same',strides=2),
                                    layers.Flatten(),
                                    layers.Dense(units=3136,activation='sigmoid')]) # (28,28,4)

# deconvolution -> decoding
self.decoder = tf.keras.Sequential([layers.Input(shape=(3136)),
                                    layers.Dense(units=3136,activation='sigmoid'),
                                    layers.Reshape((28,28,4)),
                                    layers.Conv2DTranspose(4,kernel_size=3,activation='relu',padding='same',strides=2),
                                    layers.Conv2DTranspose(8,kernel_size=3,strides=2,activation='relu',padding='same'),
                                    layers.Conv2DTranspose(16,kernel_size=3,strides=2,activation='relu',padding='same'),
                                    layers.Conv2D(1,kernel_size=(3,3),activation='sigmoid',padding='same')])


text:
What is the difference between training a model with RGB images and using only the color channels separately?
What is the difference between training a model with RGB images and using only the color channels separately (like only the red channel, green channel, etc.)? Would the model also learn patterns between the different colors in the first case?
If for me the single-channel results are relevant but also the patterns between different channels are relevant, it would be beneficial to use them together?
I am asking this because I want to apply this to signals of an accelerometer that has x, y, z-axis data. And I want to increase the resolution of the data. Will the model learn to combine all features from different axis if I input (1024, 3) length, channels of a one-dimensional signal into my one-dimensional CNN?


text:
Are there any commonly used discontinuous activation functions?
Are there any commonly used activation functions (e.g. that take values in $(0,.5)\cup (.5,1)$)? Preferably for classification?
Why? I was looking for commonly used activation functions on Google, and I noticed that all activation functions are continuous. However, I believe this is not needed in Hornik's paper.
When I did a bit of testing myself, with a discontinuous activation function on the MNIST dataset, the results were good. So I was curious if anyone else used this kind of activation function.

I'm looking for a commonly used discontinuous bounded activation function.  Somethign I can use for classification..


text:
What are examples of commonly used feature and readout maps?
It is well-known that deep feedforward networks can approximate any continuous function from $\mathbb{R}^k$ to $\mathbb{R}^l$, (uniformly on compacts).
However, in practice feature maps are typically used to improve the learning quality and likewise, readout maps are used to make neural networks suited for specific learning tasks.
For example:

Classification: networks are composed with the softmax (readout) function so they take values in $(0,1)^l$.

What are examples of commonly used feature and readout maps?

@nbro I have reformulated the question to reduce generality and clarify what I'm looking for... (letting you know in advance)

You gotta be kidding me! This is a completely different question! Now, you're not even talking about graph neural networks. Please, stop editing your posts every time you have a new question. If you have a new question, open a new thread!


text:
Will structured knowledge bases continue to be used in question answering with the likes of BERT gaining popularity?
This may come across as an open and opinion-based question, I definitely want to hear expert opinions on the subject, but I am also looking for references to materials that I can read deeply.
One of the ways question answering systems can be classified is by the type of data source that they use:

Structured knowledge bases with ontologies (DBPedia, WikiData, Yago, etc.).

Unstructured text corpora that contain the answer in natural language (Wikipedia).

Hybrid systems that search for candidate answers in both structured and unstructured data sources.

From my reading, it appears as though structured knowledge bases/knowledge graphs were much more popular back in the days of the semantic web and when the first personal assistants (Siri, Alexa, Google Assistant) came onto the scene.
Are they dying out now in favor of training a deep learning model over a vast text corpus like Bert and/or Meena? Do they have a future in question answering?


text:
Would a new human-like general artificial intelligence be more similar, in terms of eduction, to a toddler or an adult human?
The naive concept of a general AI, or strong AI, or artificial general intelligence, is some kind of software that can answer questions like

What is the volume of a cube that is 1 m wide?

or even

Why are there only two political parties in the US?

The second question requires external knowledge and high-level reasoning. For example that US means USA in the context, the constitution and that having two parties is caused by the mathematical properties of the election system.
But I would expect that newborn human child is intelligent in the sense of intelligence that is used in general artificial intelligence, but toddlers can not answer these questions.
That is not because an infant is not intelligent, but because it is not educated, I think.
What is the apparent level of education of an artificial intelligence that could be called human-like?

A human-level AI should be able to learn and behave in the same way that humans learn and behave, otherwise, we shouldn't be calling it a human-level AI. So, either if it starts with more or less knowledge of a baby human, it should be able to learn similarly to a human and acquire more knowledge with experience.

What is the apparent level of education of an artificial intelligence that could be called human-like?

To answer you question more directly, the level of education of a human-like AI (even newly created ones) can potentially be variable (i.e. different human-like AIs could potentially have different levels of education), but the human-like AI will necessarily need to be able to increase its level of education and, in general, knowledge (because humans can also do this).
Moreover, note that AGI is not necessarily restricted to human-level AI. So, other AGIs may not follow the same principles of humans.


text:
What is the difference between batch and mini-batch gradient decent?
I am learning deep learning from Andrew Ng's tutorial Mini-batch Gradient Descent.
Can anyone explain the similarities and dissimilarities between batch GD and mini-batch GD?

It is really simple.
In gradient descent not using mini-batches, you feed your entire training set of data into the network and accumulate a cost function based on this full set of data. Then you use gradient descent to adjust the network weights to minimize the cost. Then you repeat this process until you get a satisfactory level of accuracy. For example, if you have a training set consisting of 50,000 samples, you would feed all 50,000 samples along with the 50,000 labels into the network, then perform gradient descent and update the weights. This is a slow process because you have to process 50,000 inputs to do just one step of gradient descent.
To make things go faster instead of running all 50,000 inputs through the network, you split up the training set into "batches". For example, you could break the training set up into 50 batches each containing 1000 samples. You would feed the network the first batch of 1000 samples, accumulate the loss value then perform gradient descent and adjust the weights. Then you feed in the next batch of 1000 samples and repeat the process. So, now, instead of only getting one step of gradient descent for 50,000 samples, you get 50 steps of gradient descent. This method of using batches leads to a much faster convergence of the network.

The other answer provides a correct description of what is often called "gradient descent" and "mini-batch gradient descent", respectively, but doesn't clarify the terminology used, so let me add a few notes about that.
From my experience, "batch GD" and "mini-batch GD" can refer to the same algorithm or not, i.e. some people may use "batch GD" and "mini-batch GD" interchangeably, but other people may use "batch GD" to refer to what the author of the other answer calls "gradient descent not using mini-batches", i.e. you use all training data before performing a GD step, which is sometimes just called "gradient descent" (as I wrote above).

Batch Gradient Descent(BGD):

can do gradient descent with a whole dataset, taking only one step in one epoch. For example, a whole dataset has 100 samples(1x100), then gradient descent happens only once in one epoch which means model's parameters are updated only once in one epoch.
's pros:

The computation is stable(not fluctuated).
It's strong in noise(noisy data). *Noise(noisy data) means outliers and anomalies.
It gets an accurate value.

's cons:

It's not good at a large dataset because it takes much memory slowing down the computation.
It needs the repreparation of a whole dataset if you want to update a model.
It cannot easily escape local minima or saddle points.

Mini-Batch Gradient Descent(MBGD):

can do gradient descent with splitted dataset(the small batches of a whole dataset) one small batch by one small batch, taking the same number of steps as the small batches of a whole dataset in one epoch. For example, the whole dataset which has 100 samples(1x100) is splitted into 5 small batches(5x20), then gradient descent happens 5 times in one epoch which means model's parameters are updated 5 times in one epoch.
's pros:

It's good at a large dataset because it takes small memory not slowing down the computation.
It's good at online learning. *Online learning is the way which a model incrementally learns from a stream of dataset in real-time.
It doesn't need the repreparation of a whole dataset if you want to update a model.
It can more easily escape local minima or saddle points than BGD.

's cons:

The computation is less stable than BGD.
It's less strong in noise(noisy data) than BGD.
It can more often escape local minima or saddle points than BGD.


text:
Can a fully convolutional network always return an image of the same size as the original?
I'm trying to perform a segmentation task on images of multiple sizes using fully convolutional neural networks.
Currently, I'm using EfficientNet as a feature extractor, and adding a deconvolution/backwards convolution/transposed convolution layer as described in the original Fully Convolutional Networks for Semantic Segmentation paper.
But this transposed convolution layer doesn't return a filter of a size equivalent to the original image for images of varying sizes.
For example, let's assume the original image is $100 \times 100$, and the last layer contains filters of size $50 \times 50$. To get a filter of the same size as the original, you would need a transposed convolution layer of size $51 \times 51$.
Now, assume you passed in an image of size $200 \times 200$. The last layer would contain filters of size $100 \times 100$. That same transposed convolutional filter of size $51 \times 51$ would result in an output of size $150 \times 150$.
Is there any way to make it so that a fully convolutional network always returns an image of the same size as the original?

Have you solved this issue? If yes, how? You may want to write a formal answer below, for reference.

I ended up using a work around.
I set up the network so that an C x C (i.e. 320 x 320) input would output a C x C mask for some constant C (in my case it was 320).
I then resized the image I wanted to pass in to C x C, and then resized the output back to the original size of the Image.


text:
Is ReLU a non-linear activation function?
According to this blog post

The purpose of an activation function is to add some kind of non-linear property to the function

The sigmoid is typically used as an activation function of a unit of a neural network in order to introduce non-linearity.
Is ReLU a non-linear activation function? And why? If not, then why is it used as an activation function of neural networks?

ReLU is non-linear by definition

In calculus and related areas, a linear function is a function whose graph is a straight line, that is a polynomial function of degree one or zero.

Since the graph of the ReLU function $f(x) = \max(0,x)$ is not a straight line (equivalently, it cannot be expressed in the form $f(x) = mx + c$), by definition it is not linear.
ReLU is piecewise linear
ReLU is piecewise linear on the bounds $(-\inf,0]$ and $[0,\inf)$:
$$
f(x) = \max(0,x) = \begin{cases}
               0               & x \le 0\\
               x               & x \gt 0\\
       \end{cases}
$$
But this is still non-linear on the entire domain:

Short Answer: Yes
Visually:

if you see the image from wikipedia, it shown that ReLU (the blue line) is non-Linear (the line is not straight, it turns in 0). You can also check "visual" definition of linear function in wikipedia:

"In calculus and related areas, a linear function is a function whose graph is a straight line"

Mathematically:
Linear function of one variable can be defined as:
$ f(x) = ax + b $
If you plot that function in 2D, it will give you a straight line. Then, the form of linear function with multi variables:
$ f(x_1, x_2, ..., x_n) = a_1x_1 + a_2x_2 + ... + a_nx_n + b $
If you again plot that function in the correct dimension it also give you a straight line. And if you that function carefully, it similar with calculation that happen in a neuron. That's why neuron addition and multiplication is a linear function:
$ f(x_1, x_2, ..., x_n) = w_1x_1 + w_2x_2 + ... + w_nx_n + b $
Adding more layer of linear functions doesn't make the function become "complex" for example, if you have $f(x)$ like below and then you put another layer of linear function $g(x)$ on top of it:
$f(x) = ax + b$
$g(x) = cf(x) + d = cax + cb + d$
as the neural network is trained to find the value of $a,b,c,d$, we can group the constant from the formula above, and then rewrite to:
$h(x) = mx + n$
with $m=ca$ and $n=cb+d$. So without non-linear function the layer of neural network is useless, it only give you another "simple" linear function
ReLU formula is a $f(x)=max(0,x)$, it produces non-linearity as you can't write to linear function format. Using this function will give you "complexity" when you add more layer on top of it.


text:
Is there any programming language designed by deep learning?
I know that AI can be used to design printed circuit boards (PCBs), so it can be used to solve complex tasks.
Is there any programming language designed by deep learning (or any other AI technique)?

There are certainly things like this.
I'd say a strong example is layered learning approaches, descended from Peter Stone's work.
A programming language is essentially a collection of useful shorthands for assembly-level instructions. Ultimately, everything you do in a programming language eventually gets executed in assembly. So making a programming language amounts to learning how to write short, reusable, assembly language programs that you can then use as building blocks to solve harder problems.
An example of this in action is Kelly & Heywood's approach to constructing 'Tangled Program Graphs' for reinforcement learning (IJCAI 2018). Here an evolutionary algorithm is used to learn short assembly programs, that can be combined into a graph to make more complex programs. This is similar to graphical programming languages like J.


text:
Are PreLU and Leaky ReLU better than ReLU in the case of noisy labels?
Let's assume I want to build a semantic segmentation algorithm, based on Multires-UNET. My GT-masks are messy and generated by a GAN, but they are getting better and better over time. The goal is knowledge expansion (based on the paper Noisy-Student).
Can you generally say that PreLU and Leaky Relu are better for noisy labels (or imperfect ones), like the situation in GANs in general?

To my experience ReLU is superior to LeakyReLU.

The provably more-reliable approach to learning with noisy labels/label errors (instead of altering the model architecture or loss function) is to find the label errors and train without them on cleaned data -- a result shown in the confident learning paper (Northcutt, Lu, Chuang; 2021)). I am an author on this paper.
Re-phrasing your question to two questions: (1) "how can I generally find label errors in any dataset for any model?" and (2) "how can I train a model on noisy data as if it was clean data?".
There are one line of code answers to each of these questions using the cleanlab python package which implements the algorithms and theory in the confident learning paper and works for any dataset you can train a classifier on for most data formats, ML and deep learning frameworks, and data modalities, e.g. image, text, tabular, and audio data.
link to python package: https://github.com/cleanlab/cleanlab
Find label issues in 1 line of code
from cleanlab.classification import CleanLearning
from cleanlab.filter import find_label_issues

# Option 1 - works with sklearn-compatible models - just input the data and labels ツ
label_issues_info = CleanLearning(clf=sklearn_compatible_model).find_label_issues(data, labels)

# Option 2 - works with ANY ML model - just input the model's predicted probabilities
ordered_label_issues = find_label_issues(
    labels=labels,
    pred_probs=pred_probs,  # out-of-sample predicted probabilities from any model
    return_indices_ranked_by='self_confidence',
)

Train a model as if the dataset did not have errors -- 3 lines of code
from sklearn.linear_model import LogisticRegression
from cleanlab.classification import CleanLearning

cl = CleanLearning(clf=LogisticRegression())  # any sklearn-compatible classifier
cl.fit(train_data, labels)

# Estimate the predictions you would have gotten if you trained without mislabeled data.
predictions = cl.predict(test_data)

Documentation and runnable tutorials for cleanlab: https://docs.cleanlab.ai/


text:
What are the pros and cons of deep learning and machine learning to develop a trading system?
As I want to start coding a new Trading AI this year (first based on Python and later maybe in C++) I stumbled over the following question:
Today, I would like to make a pro/contra list with you in the area of deep learning vs machine learning. The difference should be clear to most of you. If not, here is a nice explanation from Hackernoon.
Up to now, I was convinced that this future project will be based on Tensorflow, Keras, etc. However, the following came to my mind afterward.
Most of you will probably have heard of Pluribus already. Dr. Sandholm and Mr. Brown (as work for his Ph.D.) were the first to program an AI that won against 6 poker world champions in No-Limit-Texas Holdem Poker. This seemed impossible because poker is a game of imperfect information. If you haven't read/seen their work until now, here's a link to a Facebook blog post Facebook, Carnegie Mellon build first AI that beats pros in 6-player poker. See also the paper Superhuman AI for multiplayer poker and this video.
From the work, it is clear that they wrote the whole thing in C++ and WITHOUT the use of any deep learning library but exclusively on the basis of machine learning. This was also confirmed to me by both of them via email. So it was possible to bring the AI in under 24H to a level that could beat 6 Poker World Champions without any problems.
The stock and crypto market is nothing else. A game of imperfect information. The prices of a crypto coin or stock are influenced by an incredible number of factors. This includes of course prices from the past but also current media (as currently seen with covid-19) and data from the economy.
We want to grab the data out of the "Big Players", like CoinCap API, CryptoAPI.io for all kinds of historical and new charts prices, etc. The same we will do with the Yahoo Finance API to grab data out of the stock market. Depending on the size of this project and how it will develop, we want to implement also some kind of NLP to grab the most out of Economy Data like dailyfx news to predict some in/decreases for some stocks but this is a future feature.
So, basically, the main question is: should we use neural networks (deep learning) or machine learning?
All this leads me to the conclusion that I am not sure what the better option for a trading bot would be. I know that the training of AI-based on deep learning would take much longer than based on machine learning, but is it safe to say that the results are really better?
What are the pros and cons of deep learning and machine learning to develop a trading system?


text:
How can I develop a reinforcement learning agent that plays memory cards game?
I am new to RL, and I am thinking of doing a little project. The goal of the project is to learn an agent play the memory game with cards.
I already created the program for detecting the cards on the table (with YOLO) and classifying them what kind of object they are.
I want an agent to be able to play the memory game by itself, without being explicitly told the rules and such.
Any tips on how to get started to make the RL process easier?

Hi and welcome to AI SE! Are you talking about this game: https://en.wikipedia.org/wiki/Concentration_(card_game)?

Hello.  Yes, this is the one.

Environment is not fully observable you cant model it as MDP and work in standard RL framework, you would have to model it as POMDP and then research what RL algorithms are available for such environments or try to use LSTMs maybe with standard RL algorithms. Also, environment is very dynamic, card layout changes every game so it will be quite difficult to learn something meaningful. You picked very hard problem for beginner project.

The main thing to keep in mind when designing a reinforcement learning agent is that you need to develop an interactive environment in which the agent can learn and define the possible moves the agent can make. In your case, the environment is the memory cards.
Next you need to define how the agent can interact with the environment, that is choosing a card or flipping a card etc. Also how the environment will behave when the agent chooses a move.
Finally you need to define the fitness function, in this case the score (number of pairs created) should be fine. You should have a look at some online RL competitions to get a better understanding of the architecture. AWS DeepRacer


text:
What are pros and cons of Bi-LSTM as compared to LSTM?
What are the pros and cons of LSTM vs Bi-LSTM in language modelling? What was the need to introduce Bi-LSTM?

I would say that the logic behind the introduction was more empirical than technical. The only difference between LSTM and Bi-LSTM is the possibility for Bi-LSTM to leverage future context chunks to learn better representations of single words. There is no special training step or units added, the idea is just to read a sentence forward and backward to capture more information.
And as trivial as the idea sounds, it works, in fact, in the original paper the authors managed to achieve state-of-the-art scores in three tagging tasks, namely part-of-speech tagging, chunking and named entity recognition.
Even though it must be said that these scores were not dramatically higher compared to other models, and also the complete architecture included a Conditional Random Field on top of the Bi-LSTM.
Probably the most important aspect to stress out is that the authors performed two interesting comparison tests: one using random embedding initialisation and another one using only words (unigrams) as input features. Under these two test conditions, Bi-LSTM (with CRF on top) outperformed significantly all other architectures, proving that Bi-LSTM representations are more robust than representation learned by other models.
I would like also to make a side note regarding human reading. It makes sense to consider unidirectional sequence models as the most reasonable to emulate human reading, because we experience reading as a movement of the eyes that goes from one direction to the opposite. But the reality is that saccades (really rapid unconscious eye movement) and other eye movements play an enormous rule in reading. Which means that also we humans do continuously look to past and future words as well in order to understand the purpose of a word or sentence we're processing. Of course in our case these movements are directed by implicit knowledge and habits that allow us to direct our attention only to important words/parts (for example we barely read conjunctions) and it is interesting to notice that now state-of-the-art models based on transformers try to learn exactly this, where to pay attention rather than single probabilities for each word in a vocabulary.


text:
Is there an up-to-date list of suitable kernels for Gaussian processes?
Consider a stochastic process $\{X_t \colon t \in T\}$ indexed by a set $T$. We assume for simplicty that $T \in \mathbb{R}^n$. We assume that for any choice of indexes $t_1, \dots, t_n$, the random variables $(X_{t_1}, \dots, X_{t_n})$ are jointly distributed according to a multivariate Gaussian distribution with mean $\mu = (0, \dots, 0)$, for a given covariance matrix $\Sigma$.
Under these assumptions, the stochastic process is completely determined by the 2nd-order statistics. Hence, if we assume a fixed mean at $0$, then the stochastic process is fully defined by the covariance matrix. This matrix can be defined in terms of the covariance function
$$
k(t_i, t_j) = \mbox{cov}(X_{t_i}, X_{t_j}).
$$
It is well-known that functions $k$ as defined above are admissible kernels. This fact is often used in probabilistic inference, when performing regression or classification.
Several functions can be suitable kernels, but only a few are used in practice, depending on the application.
Given a large amount of related literature, can someone provide an up-to-date list of functions commonly used as kernels in this context?


text:
Agents meeting in a directed connected graph
We have a directed connected graph, where nodes are places one can go to and edges are the "roads" between places. We have K agents whose goal is to meet in one node. Agents start in different nodes. Note that more than one agent can be at one node at the same time and that all agents move by one node at every turn(they move synchronously).
We have to variants of this task:

in each turn every agent must move

an agent may pass on moving.

For a chosen variant, I have to find an algorithm to complete this task, but it cannot be the state-space search algorithm.
I've been sitting on this for a while, but I cannot think of anything.
I've been thinking if agents could know each other positions in order to choose where to go, but it's a state-space search. I also thought that if agents meet, they could continue together. But I'm looking for an alternative to state-space search algorithms.

It's not possible to solve version 1) of the problem in general. To see why, consider a graph with 2 cities, and 2 agents, where the agents start in opposite nodes. Since both agents need to move every turn, they will never meet in the same city.
For version 2), I'm going to make some assumptions that aren't completely clear from your text:

Agents have only local information. They can't see the global structure of the graph, and just all agree to meet at city X, because they don't know that X exists. They also can't see where other agents are initially.
Cities are labelled, or otherwise identifiable to the agents. That is, an agent can tell whether they've visited a city before or not.
Agents can tell how many agents are in their current city, and in all adjacent cities.
There are a finite number of cities, and it's always possible to get from city A to city B, for any pair of cities (A,B).
Agents have no way to communicate with each other, except by being present in the same space or not.

In this setup, the following heuristic will always work:

If you've never seen another agent, move to a random adjacent city, or stay in place, with a uniform probability of taking each possible action.
If you've never seen another agent before, but you see one or more agents now, then switch permanently to the following strategy:

If you can see an adjacent city with more agents than your city, move there now. If you can see several such cities, move to the largest one. Pick randomly in case of a tie.
If an adjacent city has the same number of agents as yours, flip a coin to decide whether to go or stay. Eventually, one city ends up with more than the other, and then step 1 causes all agents to end up in the same city.
Otherwise, if you can see other adjacent cities with fewer agents, just stay put. They'll come to you.
If all adjacent agents are empty and there's a total of $n$ agents you can see in your city right now, then remember $n$.
As long as you still can't see more than $n$ agents in total, stay put with probability $(n-1)/n$ and move to a random adjacent city with probability $1/n$.
If you stayed put per step 5, and you still can only see n agents, and all but one of them is in your current city, move to the city with the single agent who moved. Otherwise, if 0 or more than 1 agent moved, stay put. The agents who moved come back, and we go to step 5.
After successfully moving, go to step 1 again.

You can see that once agents meet up in a group, that group never shrinks. Further, groups of agents are always trying to move around together. You can prove that a group moves an average of about $\frac{1}{2e}$ every step. It may take a while, but eventually every group will bump into every other group, resulting in all the agents being in the same city.

Thanks a lot, that's a great idea that I didn't think of. I came up with another solution. I just pick a node for them to meet in. Now interesting would be how to pick the best city. But I really like your idea too, it's much more complex.

Isn't strategy 5 going to eventually make a group to split?


text:
Why does GAN loss converge to log(2) and not -log(2)?
In Goodfellow's paper, he says:

Hence, by inspecting Eq. 4 at $D^*_G (\mathbf{x}) = \frac{1}{2}$, we find $C(G) = \log \frac{1}{2}+ \log \frac{1}{2} = − \log 4$. To see that this is the best possible value of $C(G)$

i.e. $D$ and $G$ loss should converge to $\log \frac{1}{2}$. This makes perfect sense. When I train a GAN in PyTorch with BCEloss, the loss for $D$ and $G$ converge to $\log(2)$, the negative of what Goodfellow states and what I'd expect.
What am I missing?

Hi and welcome to AI! $\log 2$ is not the negative of $-\log 4$. Can you clarify this?

Sorry, individual loss for D and G are log(½) = -log(2). When added together the total loss is -2log(2) = -log(4), whereas I am getting log(2) + log(2) = log(4)


text:
What effect does batch norm have on the gradient?
Batch norm is a technique where they essentially standardize the activations at each layer, before passing it on to the next layer. Naturally, this will affect the gradient through the network. I have seen the equations that derive the back-propagation equations for the batch norm layers. From the original paper: https://arxiv.org/pdf/1502.03167.pdf

However, I have trouble understanding if there is an intuitive understanding of what effect it actually has on the network. For instance, does it help with the exploding gradient problem, since the activations are rescaled, and the variance of them is constrained?

"Naturally, this will affect the gradient through the network." this statement is only partially true, let's see why by starting explaining the real aim of batch normalisation.
As the title of the paper suggest, the aim of batch normalisation is to decrease training time by reducing covariance shift. What is covariance shift? We can conceive it as the variation that can happen between between the values of two layers of a network. We are all familiar with the concept that if we have input features with different unite scale, like kilos and euros, most likely lot of values will have different magnitude orders, like thousands for the weight could appear often with hundred of thousands for the money. When applying the activation function to different order of magnitude this discrepancy will remain, causing the values in the first layer to assume a really broad range of values. This is not good, since high fluctuation means more time to converge to stable values. This is why values fed to neural nets are always standardised.
The authors applied the same logic to the hidden layers, arguing that a deep neural net can be conceived as a repetition of itself (every hidden layer is an input layer that send features to another hidden layer) ergo the features should be normalised at every layer. How can we do it? Normalising every batch is the most natural way to procede, but in this way there is the risk to end up transforming the internal representation of a layer, because normalising is not a linear transformation. This is way the authors propose a clever way to perform hidden layers normalisation, which consists in a classic normalisation followed by a linear scaling performed with two trainable parameters $\beta$ and $\gamma$ (which appear in the last step of the batch norm scratch code below).

A really important thing to notice is that the mean and variance used to perform the classic normalisation are mean and variance calculated on the mini batch. I will explain why this is important in a sec, first I want to stress out that the $\beta$ parameter can actually bring to increase overfitting when batch norm is randomly stucked on top of other layer. The reason why is that, as we see in the scale and shift step, $\beta$ is nothing but a biased term, added to perform the shift of the mean of the batch hidden values. So, to avoid having an extra bias that lead to overfit, the bays on the previous layer should be removed, leaving only the classic matrix of weights parameters.
Back to the gradient problem, we can see that in itself doesn't necessarily lead to increased performances, but it does provide an advantage in terms of hidden layer values convergence. The x axis on the two right sub plots of the figure below represent the variation of the hidden values of net trained with and without batch norm. When trained with batch norm the hidden values reach stable ranges after few iteration. This help the net to reach high accuracy in less iterations (first subplot on the left) but we can see that even without batch norm the net reach eventually high accuracy.

The only help provided by batch norm to the gradient is the fact that, as noticed before, the normalisation is firstly performed by calculating the mean and variance on individual batches. This is important because this partial estimation of mean and variance introduces noice. Similarly to drop out, which has a regularisation effect due to the noice generated by randomly deactivating part of the weights, batch norm can introduce regularisation by adding noice due to the larger or smaller mean and variance estimated on individual batches. But still, batch norm was not introduced as a regularisation technique, and the equation you put on the question simply prove that it is possible to calculate the derivatives of the equations applied to perform the batch norm.

This seems consistent with my knowledge (although, to be honest, I am not really an expert on the topic). However, I think you don't really address why and how batch normalization affects the gradient, if it mitigates or not the vanishing and/or exploding gradients.

It is also dubious as to whether the "covariate shift" is indeed why batch norm helps.  http://gradientscience.org/batchnorm/ and https://blog.paperspace.com/busting-the-myths-about-batch-normalization/

@information_interchange Thank's for the links! Had a chance to read them yesterday and they are really interesting, it's a shame that I was not taught them during real classes, apparently, even some professors give for granted some explanations without digging deeper.