text:
Deep Q-learning is not performing well when there are several enemies
I am playing with a deep Q-learning algorithm in my own environment. The network can perform well as long as there is only one enemy. My agent can perform the following actions:

do_nothing
prepare_for(e)
attack(e)

where e is some enemy.
In the case of two enemies, the action vector has 5 elements:
|   0       |      1          |      2      |        3         |     4      |
-----------------------------------------------------------------------------
|do_nothing | prepare_for(e1) |  attack(e1) |  prepare_for(e2) | attack(e2) |
-----------------------------------------------------------------------------

After a couple of episodes, the agent always starts picking the first do_nothing action, which is not desired. Changing reward for do_nothing action is not helping, even using significantly higher negative reward, than for other actions.
There is no problem with the environment with only one enemy. (Only using columns 0, 1, 2). I feel like my action encoding can be the issue, but I can't figure it out, how to fix it. Any suggestions?

You have to provide a bit more details. What is the shape of your neural network? What are you using as input, state features or images? Are you providing state observation + action as the input or only state observation? How does the output look like, are there 5 nodes each for specific action or is it only a single output node for the specific action you provided as a part of the input. What are the rewards for your environment? Are you using exploratory policy? Are you using experience replay? Are you using separate networks? Are you providing enough training time?

1) Vary the number of enemies during the training process.
2) Use other generalisation methods that are appropriate for neural networks such as dropout or L1 and L2 regularisation.

Thank you for really fast responses. I have been stuck on this issue for the last 3 days and now, out of nowhere, it started working. Combination of two things has been critical to making it work: slightly more neurons in the hidden layer + significantly higher learning rate. Changing the learning rate from 0.0001 to 0.01 made it work.

@Savco Thanks for the update, it is always useful to know which solutions people have found.


text:
Can traditional neural networks be combined with spiking neural networks?
Can traditional neural networks be combined with spiking neural networks? And can there be training algorithms for such hybrid network? Does such hybrid network model biological brains?
As I understand, brains contain only spiking networks and traditional networks are more or less crude approximation of them. But we can imagine that evolutionary computing can surpass the biological evolution and so the new structure can be created that are better than mind. And that is why the question about such tradition-spiking hybrid neural networks should be interesting.

Yes, they can. Backpropagation through-time (BPTT) and gradient
descent is usually used to train such networks. The networks are are
arranged in orderly layers which makes them unlike biological
brains. And BPTT is not how real brains learn.
For example, see the paper Online Training Through Time for Spiking
Neural Networks. It demonstrates how regular artificial neural
networks for image classification can be "spikified" and then trained
using the method the authors proposes, online training through
time. The figure shows their VGG11 spiking architecture for CIFAR10:

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Can we derive the distribution of a random variable based on a dependent random variable's distribution?
In the diagram below, there are three variables: X3 is a function of (depends on) X1 and X2, X2 also depends on X1. More specifically, X3 = f(X1, x2) and X2 = g(X1). Therefore, X3 = f(X1, g(X1)).

If the probabilistic distribution of X1 is known, is it possible to derive the probabilistic distribution of X3?

I don't think you can know the probability distribution of $X_3$ without knowing the functions $f$ and $g$, which define how $X_3$ is dependent on $X_1$ and $X_2$.

No, it is not possible. We could derive the most probable $x_3$ by calculating the maximum likelihood: $x^*_3=\underset{x_3}{\arg\max} p(x_1,x_2|x_3)$. We are unable to calculate this as you only stated that there is a correlation, but we don't know how it looks like.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is the difference between automatic transcription and automatic speech recognition?
What is the difference between automatic transcription and automatic speech recognition? Are they the same?
Is my following interpretation correct?
Automatic transcription: it converts the speech to text by looking at the whole spoken input
Automatic speech recognition: it converts the speech to text by looking into word by word choices

They are both the same. There are different algorithms to recognise speech, but essentially they all aim to identify the content of the spoken input and convert it into written text.
Automatic transcription is then done, whereas the output of more general ASR is often passed on to further processing, such as recognising entities or commands expressed in the speech.

automatic transcription is a system to change phonemes to graphemes automatically, it's more like syllabel recognition, used for build speech recognition in another language base on stable and existing version of speech recognition.
see this picture to more understanding: see this picture
one think that you must understood, sound is different with meaning, computer need tools to understood the sound, it's speech or music or another form of sound. all speech sound already mapped in International Phonetic Alphabet (IPA).
and with little computation to combine the graphemes (call dictionary), resulting understanding speech of specific language.
example: you already know the sound of "spoon" or 'key', the grapheme is split by s and 'p-o-on'.
in other language like Indonesia's language, another developers use this grapheme identification system (transcription system) to build a speech recognition, for words 'meskipun'.
in grapheme, 'meskipun' words is: 'mehs-key-poon'.
with a simple computation, we can make computer understood the 'meskipun' words.
only said if "mesh + key + poon" show, the words is 'meskipun'
meskipun (=althought, english)
The big problem in building speech recognition with using automated transcription system, there is no 100% similar for IPA map in every language.
so the developers should use several 'transfer learning' database of language to make their speech recognition have more high accuracy. except, he decided to build it from scratch.
automated speech recognition is an end to end speech recognition that set to understood a specific speech language, which in the middle of the system contain automatic transcription system.


text:
Can the rewards be stochastic when the transition model is deterministic?
Suppose we have a deterministic environment where knowing $s,a$ determines $s'$. Is it possible to get two different rewards $r\neq r'$ in some state $s_{\text{fixed}}$? Assume that $s_{\text{fixed}}$ is a fixed state I get to after taking the action $a$. Note that we can have situations where in multiple iterations we have: $$(s,a) \to (s_1, r_1) \\ (s,a) \to (s_{\text{fixed}}, r_1) \\ (s,a) \to (s_{\text{fixed}}, r_2) \\ (s,a) \to (s_3, r_3) \\ \vdots$$
My question is, would $r_1 =r_2$?

My question is, would $r_1 =r_2$?

That's usually up to you as the designer of the system.
Usually when you declare that you have "a deterministic environment", you imply that both $s'$ and $r$ are fixed values depending on $(s,a)$. So in your examples, you would expect your observations to also have $r_1 = r_2$
However, it is possible to define a MDP where transition to state $s'$ is deterministic, but $r$ is not. For instance, you could define reward in a game equal to the sum of a number of dice rolled, with better rewards (on average) resulting in more dice. This is still a valid MDP and can be solved using RL techniques.
A real-world example of this might be managing a queue of work, where you want to minimise lead time, but don't know for certain how long each task will take. Your state progression moves deterministically - you have a queue of pending tasks, current tasks and workers, and assigning a task to a worker is completely deterministic. However, you don't know how efficiently tasks will be performed until after they are done, so you don't know the reward perfectly from the assignment (whether you can treat this as random or hidden state is a more complex issue - it is often pragmatic to treat such unknown data as random though).


text:
Where does the expectation term in the derivative of the soft-max policy come from?
At slide 17 of the David Silver's series, the soft-max policy is defined as follows
$$
\pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta}
$$
that is, the probability of an action $a$ (in state $s$) is proportional to the "exponentiated weight".
The score function is then defined as follows
$$
\nabla_\theta \log \pi_\theta (s, a) = \phi(s, a) -
\mathbb{E}_{\pi_{\theta}}[\phi(s, \cdot)]
$$
Where does the expectation term $\mathbb{E}_{\pi_{\theta}}[\phi(s, \cdot)]$ come from?

I'm not 100% sure what you are asking but I'll try to answer it anyway.
The expectation term comes from the derivative of the softmax function. This video from Geoffrey Hinton should help: https://www.youtube.com/watch?v=mlaLLQofmR8

The confusion here is that, when we take log of pi, then it will log(sum x_i) in dem, where x_i is that expression. Now my doubt is here this term log(sum_i  x_i) is replace by sum_i (log x_i), how? Is this inequality valid?

see this answer over on the math.stackexchange.com site: https://math.stackexchange.com/a/2340848/414001


text:
Is it possible to use AI to denoise noisy documents?
I have some documents containing some text (machine writing text) that I intend to apply OCR on them in order to extract the text.
The problem is that these documents contain a lot of noise but in different ways (some documents have noise in the middle, others in the top, etc.), which means that I can't apply simple thresholding in order to remove the noise (i.e applying simple threshold does not only remove the noise, but it removes some parts of the text).
For these reasons, I thought about using AI to de-noise the documents.
Does anyone know if it is possible to do that with AI or any alternative way?

This is also the topic of Image Processing (which has analytical solutions instead of learning) mostly through predesigned filters. The filter depends on the type of noise, (salt & pepper, Gaussian, etc.) i.e., for salt & pepper choosing the median in a window. There are a lot of denoising research in literature. There are also more recent learning based denoising applications, but it requires data so that you can train.

thank you for you clarification, Do you recall any articles or links to start with?
It is OK for the data, I can provide data for training and testing.

This seems as a good article: http://papers.nips.cc/paper/4686-image-denoising-and-inpainting-with-deep-neural-networks  . Also, inpainting might be needed if occlusions are too much. If you want analytic solutions, you can also look up sparse representations/dictionary learning methods for denoising/inpainting.

Also, I forgot to mention, but if you specifically target documents, then you can narrow down your research. Some lookup returned these that look promising: Kaggle competition on what you ask. https://www.kaggle.com/c/denoising-dirty-documents , A paper on removing corruption in scanned documents. https://unitec.researchbank.ac.nz/handle/10652/2730

The Kaggle competion sounds promising, thank you for the sources!


text:
DQN stuck at suboptimal policy in Atari Pong task
I am in the process of implementing the DQN model from scratch in PyTorch with the target environment of Atari Pong. After a while of tweaking hyper-parameters, I cannot seem to get the model to achieve the performance that is reported in most publications (~ +21 reward; meaning that the agent wins almost every volley).
My most recent results are shown in the following figure. Note that the x axis is episodes (full games to 21), but the total training iterations is ~6.7 million.

The specifics of my setup are as follows:
Model
class DQN(nn.Module):
    def __init__(self, in_channels, outputs):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=in_channels, out_channels=32, kernel_size=8, stride=4)
        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2)
        self.conv3 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1)
        self.fc1 = nn.Linear(in_features=64*7*7 , out_features=512)
        self.fc2 = nn.Linear(in_features=512, out_features=outputs)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = x.view(-1, 64 * 7 * 7)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x    # return Q values of each action

Hyperparameters

batch size: 32
replay memory size: 100000
initial epsilon: 1.0
epsilon anneals linearly to 0.02 over 100000 steps
random warmstart episodes: ~50000
update target model every: 1000 steps
optimizer = optim.RMSprop(policy_net.parameters(),  lr=0.0025, alpha=0.9, eps=1e-02, momentum=0.0)

Additional info

OpenAI gym Pong-v0 environment
Feeding model stacks of 4 last observed frames, scaled and cropped to 84x84 such that only the "playing area" is visible.
Treat losing a volley (end-of-life) as a terminal state in the replay buffer.
Using smooth_l1_loss, which acts as Huber loss
Clipping gradients between -1 and 1 before optimizing
I offset the beginning of each episode with 4-30 no-op steps as the papers suggest

Has anyone had a similar experience of getting stuck around 6 - 9 average reward per episode like this?
Any suggestions for changes to hyperparameters or algorithmic nuances would be greatly appreciated!

Hi @Mink , I am working on the same project right now, but my average score is capped at -10. Some differences I can see between our implementations are:

"PongDeterministic-v4" environment which takes care of frame skipping.
did not crop the score part on the top, but my frames are 84 x 84.
do not treat losing a volley as a terminal state.
replay buffer size is 50000. RAM fills up and overflows if I use any more than that. No idea how you managed to store 100000 experience tuples.
using l2 loss, was planning to switch to huber loss though.

Do you have a Github repo?

@hridayns, to prevent memory overflow, I save the images as uint8 type in my replay buffer and cast (and divide by 255) right before I forward pass it to my model.

By the way, the GitHub repo is https://github.com/MatthewInkawhich/learnRL

I have done the same uint8 conversion. I was planning to normalize them by 255 too, but I believe they take up the same memory, whether divided or not. I was thinking I might lose information if I did that. What do you think? Also, I have followed you on github. In regards to your issue, I was thinking maybe it has something to do with your target network update frequency. It could be too low, causing instability during learning? Another thing you could try is a Prioritized Replay buffer.

@hridayns thanks for the suggestions, I will try that. Also, I would recommend always normalizing your image inputs between 0 and 1, as it tends to play nicer with the initial weights and activation function.

Thanks. I will probably implement that then. Along with better initialization schemes for the networks, gradient and reward clipping. You can check out my repo here if you like: https://github.com/hridayns/Reinforcement-Learning/

According to this answer here, using a squared loss function with gradient clipping is equivalent to using a Huber loss function without the gradient clipping. Gradient clipping step may not be required for you since you are using something close to Huber loss already.

Incase you still haven't been able to resolve the problem, here's a link to the answer to my own question, which has the step-wise changes I made to achieve a +18 average score saturation using just a 10000 replay buffer size and a normal Double DQN (DDQN), trained for about 700-800 episodes. The updated code can also be found here.
No fancy changes like Prioritized Replay Buffer or any secret hyperparameter changes are required. It's usually something simple, like a small problem with the input preprocessing step.


text:
Can a machine learning model predict the pattern of given sequence?
I am curious if it is possible to do so.
For example, if I supply

$[0, 1, 2, 3, 4, 5]$, the model should return "natural number sequence",

$[1,3,5,7,9,11]$, it should return "natural number with step of $2$",

$[1,1,2,3,5]$, it should return "Fibonacci numbers",

$[1,4,9,16,25]$, it should return "square natural number"

and so on.

Like this? https://www.wolframalpha.com/widgets/view.jsp?id=a3af2e675c3bfae0f2ecce820c2bef43 (note some sequences cause broken behaviour, but it is marked "beta")

Those all fit into a single quadratic, auto-correlated model.
$$ x_0 = a \\ x_i = b i^2 + c x_{i-1} + d i + e $$
The sequences can be curve fitted producing a set of $n$ perfect fits of the form $(a, b, c, d, e)$ given the above model. A rules engine given the correct parameterized rules can produce the most desirable verbal description from among the $n$ fits in the set. The rules can also be prioritized by a simple feed forward network trained to simulate the most natural selection of string descriptions from any set of fits where $n > 1$.
This will work well for the examples in the question and many more, however, if the sequence $\{1, 4, 1, 5, 9\}$ is fed into the system, it will produce some weird description based on the quadratic, auto-correlated model it was given rather than, "digits of $\pi$ to the right of the decimal place."
The only way to produce the most common response a university freshman math student would produce would be to extend the boundaries of AI engineering first. For example, once an AI system is developed that can handle natural language and cognition like a child, several of them can be separately trained in a simulation of primary and secondary school mathematics. The median response for each sequence given to the class of AI students (class made up of artificial students studying math, not class of humans studying AI) will then be a reasonable prediction of what human university freshmen would produce as a median response.

This can be framed as a classification problem where a model is supervised on a dataset containing finite-length number sequences $x^{(i)}_1, \cdots, x^{(i)}_n$ and the sequence name $y_i$. For example, the dataset could look like this:

([0,1,2,3,4], 0)
([1,3,5,7,9], 1)
([1,1,2,3,5], 2)
([1,4,9,16,25], 3)

where the numbers on the right are integer representations of the sequence name. Given intuitive sequences, plentiful training data, and training examples of reasonable length, this problem would not be that difficult to solve. Sequence models from deep learning, such as recurrent networks (LSTM, GRU) or temporal convolutional networks, are well-suited for tasks such as this one.
Of course, this is only possible within certain constraints. The models are only good at what they are trained to do, so it would be impossible to use them to infer whether a sequence skips by 2 or 3 without having that information explicitly present in the training data. It would be interesting to see whether unsupervised models could detect this sort of information, although I don't think this work has been done in the present.


text:
How can we create eXplainable Artificial Intelligence?
Currently, we can build the Artificial Intelligence (AI) approaches that respectively explain their actions within the use of goal trees 1. By moving up and down across the tree, it keeps tracking the last and next movements. Therefore, giving the ability to the machine for "explain" the actions.
Explainability regarding human levels, requires some cognitive effort, such as self-awareness, memory retrieval, a theory of mind and so on 2. Humans are adept at selecting several causes from an infinite number of causes to be the explanation. However, this selection is influenced by certain cognitive biases. The idea of explanation selection is not new in eXplainable Artificial Intelligence (XAI) [3, 4]. But, as far as we are aware, there are currently no studies that look at the cognitive biases of humans as a way to select explanations from a set of causes.
Despite a clear definition and description of the XAI field, several questions remain present. The issues are summarized in just one sentence and listed as follows.
That said, our question is:

How can we create and build XAI?

References
1 Hadoux, Emmanuel, and Anthony Hunter. Strategic Sequences of Arguments for Persuasion Using Decision Trees. AAAI. 2017.
2 Miller, T., 2018. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence.
3 Gunning, D., 2017. Explainable artificial intelligence (XAI). Defense Advanced Research Projects Agency (DARPA), nd Web.
4 Samek, W., Wiegand, T. and Müller, K.R., 2017. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708.08296.

Can you confirm that this (your) question is a duplicate of this one? If you think it's not a duplicate, can you explain why? That supposed duplicate is asking for XAI techniques, which is exactly what you're asking, i.e. "How can we create XAI", i.e. the techniques are suggestions/answers for your answer.

Despite being true that this question is similar to that one, your question was published on Jun 15 '19, while this question was published on Jan 25 '19. Making it an earlier contribution. Please consider my last politely claims and try to understand the reasoning here.

Yes, but that question has 2 objectively good answers. It would not make sense to mark the other question as a duplicate of this one, when this does not have any good answer. Do you agree? So, if you think it's a duplicate, I will mark this as a duplicate of the other, even though this is a slightly older post. Btw, I upvoted your question, which will just be marked as a duplicate, i.e. it will not be deleted, so you will not lose the reputation points.

Although I do not agree with this level of justice, I can understand your logic. However, I would like to slightly specify the question for the topic of medical imaging, which will make the extent of questions/answers totally different. Please give me some time to change the question and reconsider not flagging it as duplicate.

I suggest that you ask a completely different question than changing this one. A new question can become "hot" and attract a lot of users, while an older question cannot.


text:
How do I classify measurements into only two classes?
I am a member of a robotics team that is measuring the amount of reflected IR light to determine the lightness/darkness of a given material. We eventually hope to be able to use this to follow a line using a pre-set algorithm, but the first step is determining whether the material is one of the binary options: light or dark.
Given a large population of values between 0 and 1023, probably in two distinct groupings, how can I best go about classifying a given point as light or dark?

The way to categorize measurements into two separate populations is through what ML people currently term unsupervised learning. Such a process is part of the AI tool chest. Statistics is part, but not all, of the mathematics involved in the theory that leads to algorithms that learn without labeling the array of points light or dark in advance.
Reflected IR light can be specular or mirror in nature and significant levels of IR emission will also occurs, without raising the surface temperature of materials under most common circumstances. There is also refraction and transmission to be considered in the robotics space.
Materials will not exhibit lightness. The material composition, coating, and surface texture will exhibit various reflectivities, emission dependent on the absolute temperature raised to the fourth power1, and transmission of light entering the material from various directions, possibly diffused internally.

We eventually hope to be able to use this to follow a line using a pre-set algorithm.

That is one of the primary objectives of computer vision and has been since the first digital signal processing in the mid twentieth century.

The first step is determining whether the material is one of the binary options, light or dark.

That may be an experimental first step, but the loss of information from taking a number that discretely represents a cube in horizontal-vertical-frame space, each of which has $\tau$ channels of $\beta$ bits, is not a good computer vision strategy. The total number of bits of IR information is given by
$$ \tau \beta \; \text{,} $$
so the data loss will be, in percent,
$$ \dfrac {100 \, (\tau \beta - 1)} {\tau \beta} \; \text{.} $$
Object recognition and thus collision avoidance will be frustrated by that much data loss, even if only one IR spectral range is contained in the cube ($\tau = 1$).

Given a large population of values between 0 and 1023, probably in two distinct groupings, how can I best go about classifying a given point as light or dark?

It appears that $\tau$ does equal one and $\beta = \log_{2} (1023 - 0 + 1) = 10$, thus the information in bits per cube (pixel in a frame) is 10. If the project, for some reason, requires that 9 bits be discarded in a way to conserve as much of the original 10 bits as possible (perhaps to maintain a particular throughput through a transmission or processing bottleneck), then the way to do it is through a one bit feature extraction, meaning that the output of the artificial network should be one bit. These are a few unsupervised learning options to investigate.

RBMs (restricted Boltzmann machine)
K-means clustering
Gaussian mixture models

Most AI and artificial network frameworks and libraries will have examples of these in the example directory. It is advisable to approach the project like this.

Find web pages or articles using the above as search terms until an explanation that is clear to the team is found.
Study all three for at least a few hours each as a team.
Find a library or framework that contains all three.
Install the necessary prerequisites for all three on the lab computer(s).
Produce a working example of the three types so that selection is not a function of which one is easiest to get running (something which would qualify as an research anti-pattern).
Make a choice.
Change the working code, one verifiable step at a time into the desired unsupervised learning program.
Eventually, when thoroughly ready, make the jump from the example data set to the robotic IR light frame set.

Footnotes
[1] Stefan-Boltzmann law

This looks like part of what I was looking for but it is not particularly feasible for our project and I don’t understand all of it.  We can’t use any libraries and have to rely on our own code.  All I need to do is classify a few points.  Would it be easier to simply have a hard-coded cutoff value?

Under ordinary circumstances I would certainly agree with you but we are doing this as a high school project on a platform with very minimal computing power.  We simply want to follow a line with a sensor to improve the accuracy of our autonomous algorithm on game boards that may be smudged or of varying quality and we wanted to do it in a higher-level way than the typical method of deciding on an arbitrary value somewhere in the middle of the available range.


text:
Why is the reward signal normalized in openAI's REINFORCE?
Pytorch's example for the REINFORCE algorithm for reinforcement learning has the following code:
import argparse
import gym
import numpy as np
from itertools import count

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.distributions import Categorical

parser = argparse.ArgumentParser(description='PyTorch REINFORCE example')
parser.add_argument('--gamma', type=float, default=0.99, metavar='G',
                    help='discount factor (default: 0.99)')
parser.add_argument('--seed', type=int, default=543, metavar='N',
                    help='random seed (default: 543)')
parser.add_argument('--render', action='store_true',
                    help='render the environment')
parser.add_argument('--log-interval', type=int, default=10, metavar='N',
                    help='interval between training status logs (default: 10)')
args = parser.parse_args()

env = gym.make('CartPole-v0')
env.seed(args.seed)
torch.manual_seed(args.seed)

class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
        self.affine1 = nn.Linear(4, 128)
        self.affine2 = nn.Linear(128, 2)

        self.saved_log_probs = []
        self.rewards = []

    def forward(self, x):
        x = F.relu(self.affine1(x))
        action_scores = self.affine2(x)
        return F.softmax(action_scores, dim=1)

policy = Policy()
optimizer = optim.Adam(policy.parameters(), lr=1e-2)
eps = np.finfo(np.float32).eps.item()

def select_action(state):
    state = torch.from_numpy(state).float().unsqueeze(0)
    probs = policy(state)
    m = Categorical(probs)
    action = m.sample()
    policy.saved_log_probs.append(m.log_prob(action))
    return action.item()

def finish_episode():
    R = 0
    policy_loss = []
    rewards = []
    for r in policy.rewards[::-1]:
        R = r + args.gamma * R
        rewards.insert(0, R)
    rewards = torch.tensor(rewards)
    rewards = (rewards - rewards.mean()) / (rewards.std() + eps)
    for log_prob, reward in zip(policy.saved_log_probs, rewards):
        policy_loss.append(-log_prob * reward)
    optimizer.zero_grad()
    policy_loss = torch.cat(policy_loss).sum()
    policy_loss.backward()
    optimizer.step()
    del policy.rewards[:]
    del policy.saved_log_probs[:]

def main():
    running_reward = 10
    for i_episode in count(1):
        state = env.reset()
        for t in range(10000):  # Don't infinite loop while learning
            action = select_action(state)
            state, reward, done, _ = env.step(action)
            if args.render:
                env.render()
            policy.rewards.append(reward)
            if done:
                break

        running_reward = running_reward * 0.99 + t * 0.01
        finish_episode()
        if i_episode % args.log_interval == 0:
            print('Episode {}\tLast length: {:5d}\tAverage length: {:.2f}'.format(
                i_episode, t, running_reward))
        if running_reward > env.spec.reward_threshold:
            print("Solved! Running reward is now {} and "
                  "the last episode runs to {} time steps!".format(running_reward, t))
            break

if __name__ == '__main__':
main()

I am interested in the function finish_episode():
the line
 rewards = (rewards - rewards.mean()) / (rewards.std() + eps)

makes no sense to me.
I thought this might be baseline reduction, but I can't see why divide by the standard deviation.
If it isn't baseline reduction, then why normalize the rewards, and where should the baseline reduction go?
Please explain that line

It's just a hack to normalize the rewards to make optimization less problematic in cases where the reward might have very large/small values. It's fine because scaling the rewards by a constant factor won't change what the optimal policy is. You can think of the mean reward as the baseline.

The mean reward is the baseline, i get it, but why divide by the standard deviation?

Also, that constant does change between episodes. Could you please elaborate on how that actually works?


text:
Eligibility trace In Model-based Reinforcement Learning
In model-based reinforcement learning algorithms, the model of the environment is constructed to efficiently use samples, models such as Dyna, and Prioritize Sweeping. Moreover, eligibility trace helps the model learns (action) value functions faster.
Can I know if it is possible to combine learning, planning, and eligibility traces in a model to increase its convergence rate? If yes, how it is possible to use eligibility traces in the planning part, like Prioritize Sweeping?


text:
How to use SLAM on other sensor other than camera?
I have a sensor that reads electromagnetic field strength from each position.
And the field is stable and unique for each position. So the reading is simply a function of the position like this: reading = emf(x,y,z)
The reading consists of 3 numbers (not position).
I want to find the inverse function of emf function. This means I want to find function pos that is defined like this: x,y,z = pos(reading)
I don't have access to both emf and pos function. I think that I want to gradually estimate the pos function using a neural network.
So, I have the input reading and acceleration ax,ay,az of the sensor through space from an IMU. The acceleration is not so accurate. I want to use these 2 inputs to help me figure out the position of the sensor over time. You can assume that the starting position is at 0,0,0 on the first reading.
In short, input is reading and ax,ay,az on each timestep, the output will be an adjustment on the weights of pos function or output will be position directly.
I've been reading about SLAM (simultaneous localization and mapping) algorithm and I think that it might help in my case because my problem is probabilistic. If I know accurately the acceleration, I would not need any probability, but the acceleration is not accurate.
So, I want to know how do I model this problem in terms of SLAM?
I don't have a camera to do vision-based SLAM though.
Why do I think this is tractable?
If the first reading is 1,1,1 and the position is at origin 0,0,0, and I move the sensor, the position can drift because the sensor has never seen other reading before, but after I go back to the origin, the reading will be 1,1,1 again so the sensor should report the origin 0,0,0 as output. During the movement of the sensor, the algorithm should filter the acceleration so that all the previous positions make sense.


text:
Why is this PyTorch implementation of the actor-critic algorithm inconsistent with the mathematical formulas?
This PyTorch implementation of the actor-critic algorithm calculates the losses like so:
actor_loss = -log_prob * discounted_reward
policy_loss = F.smooth_l1_loss(value, torch.tensor([discounted_reward]))

Both are different from the regular formulas which are, in the case of the actor loss (parameterized by $\theta$):
$$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$
and, in the case of the critic loss (parameterized by $w$):
$$r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t}),$$
where $r(s_t,a_t)$ is the immediate reward following taking the action.
For the actor, "the immediate critic evaluation of the transition" was replaced with "the discounted reward". For the critic, the discounted evaluation of the value from the next state $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1})$ was replaced by "the discounted reward". The $L_1$ loss is then calculated, effectively discarding the sign of the (equation) loss.
Questions:

Why did they make these changes?

Why is the sign discarded for the critic loss?

There is no 'regular' formula for calculating policy loss, the regular thing when calculating policy gradient is to multiply gradient with advantage function which can be many things. Look at section 2 of this paper for coverage on basic advantage functions. Also, the expected discounted reward is the same thing as the state-action value function (Q value).
$$Q^\pi(s_t, a_t) = \mathbb{E}_{s_{t+1:\infty}, a_{t+1:\infty}}[\sum_{l=0}^\infty \gamma^lr_{t+l}]$$
So, the variations you posted roughly calculate the same thing.
Regarding the negative sign, in policy gradient methods, we want to maximize our performance function which has the following form:
$$J(\theta) = \sum_\limits{\substack{a}} \pi(a \mid s, \theta)A(s, a)$$
So, the higher our performance the better it is. When people write code, they use minimizers that minimize the objective function, but, in this case, we want to maximize it, so maximizing the objective function is the same thing as minimizing the negative objective function therefore the negative sign.

Thanks! However, there isn't actually a negative sign in the equations. There is an absolute value (l1_loss), which makes the sign, either positive or negative, lose its meaning. Why does it make sense to only care about the distance of the actual reward to the expected reward, but not about the direction?

The loss that you are refering to, judging by the code, is the value loss, that is critic loss, not actor (policy) loss. Taking the example you provided in the question, objective function of the critic is to minimize TD error which is the difference between $Q_w(s_{t},a_{t})$ and $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1})$ . We don't care about which of those values is bigger we only care to minimize difference between them, therefore l1 loss. You could have used l2 loss as well .

I still don't see how this satisfies the equations (in the original question), in which clearly, there is significance to the sign. Is the absolute value missing from the equation?

Are you refering to the equation from the article where they update critic with this: $r(s_t,a_t) + \gamma Q_w(s_{t+1},a_{t+1}) - Q_w(s_{t},a_{t})$ ? If yes, then like I said we want to minimize that difference so the sign doesn't really matter, maybe sometimes we take a step in negative direction and sometimes in positive direction, but as the difference reduces we wouldn't be taking any steps, we would converge to parameter vector $w$ , it's same as basic tabular q-learning

Yes, I am referring to that equation. Please explain again why the sign doesn't matter. If we don't take sign into account (and the difference is not VERY small), then it could very easily cause divergence, as we could very likely change the weights in the opposite direction

divergence will happen only if you set very large step parameter $\beta$ , if step is small (less than 0.1) then divergence is unlikely to happen. If in one iteration TD target is negative and we move in negative direction and if we somehow 'overshoot' the true value, in next update iteration TD target will be positive and we will move in opposite direction.  We will keep moving until we converge to the true value. It's pretty much like linear regression and LMS algorithm, that's what we are basically doing here.


text:
Can I use deterministic policy gradient methods for stochastic policy learning?
Can I treat a stochastic policy (over a finite action space of size $n$) as a deterministic policy (in the set of probability distribution in $\mathbb{R}^n$)?
It seems to me that nothing is broken by making this mental translation, except that the "induced environment" now has to take a stochastic action and spit out the next state, which is not hard using on the original environment. Is this legit? If yes, how does this "deterministify then DDPG" approach compare to, for example, A2C?

Can you clarify what you mean by "treat"? This question is unclear to me.


text:
What is the difference between search and planning?
I'm reading the book Artificial Intelligence: A Modern Approach (by Stuart Russell and Peter Norvig).
However, I don't understand the difference between search and planning. I was more confused when I saw that some search problems can be determined in a planning way. My professor explained to me in a confusing way that the real difference is that search uses a heuristic function, but my book says that planning uses a heuristic too (in chapter 10.2.3).
I read this Stack Overflow post that says in a certain way what I'm saying.
So, what is the difference between search and planning?

What is planning?
Planning (aka automated planning or AI planning) is the process of searching for a plan, which is a sequence of actions that bring the agent/world from an initial state to one or more goal states, or a policy, which a function from states to actions (or probability distributions over actions). Planning is not just the use of a search algorithm (e.g. a state-space search algorithm, such as A*) to find the plan, but planning also involves the modelling of a problem by describing it in some way, e.g. with an action language, such as PDDL or ADL, or even with propositional logic.
There are different planning problems, such as

classical planning
planning in Markov Decision Processes (which are mathematical models that describe stochastic environments/problems, i.e. environments where an action $a$ may not always lead to the same outcome)
temporal planning (i.e. when multiple actions can be taken at the same time and/or their duration may be different/variable)
hierarchical vs non-hierarchical planning

There are different planning approaches, such as

planning by state-space search (with or without heuristics), i.e. formulate the planning problem so that you can apply a state-space search algorithm  (examples of state-space search algorithms are A*, IDA*, WA*, DFS or BFS); this is probably the approach to planning that makes people confuse planning with search (which typically implicitly refers to state-space search)

planning by SAT, i.e. formulate the planning problem so that you can you a SAT solver to solve it (e.g. SATPLAN)

planning by solving constraint satisfaction problems (similar to SAT)

planning by "symbolic" search methods (such as Binary Decision Diagrams)

There are many different planning algorithms or planners (of course, each of these corresponds to some approach or is used to solve a particular planning problem), such as

policy iteration in the context of Markov decision processes
GraphPlan
STRIPS (which stands for Stanford Research Institute Problem Solver, but note that STRIPS also and maybe more commonly refers to the corresponding description language used to describe the planning problem)

What is search?
In the context of AI, when people use the word search, they can either implicitly refer to state-space search (with algorithms like A*) or, more vaguely/generally, to any type of search, e.g. gradient descent can also be viewed as a search algorithm, typically, in a continuous state space (rather than discrete state space), e.g. the space of real-valued vectors of parameters for a certain model (e.g. a neural network).
What is the difference between planning and search?
The difference should more or less be clear from my first paragraph above that describes planning: search is used for planning, but the planning process/study also includes other things, such as describing the problem with an action language, such as PDDL. You can also view planning as an application of search algorithms of different types. In any case, planning is about searching for plans or policies. You should probably check this course for more info. This definition of (automated) planning is consistent with our usual notion of planning.

Often terminology is vague and if this will be a test item for school I would circle back with teacher with my response before accepting it.
They key difference is prior knowledge.
Search - when one searches they explore without prior knowledge. This would likely be a stochastic process of trial and error. There is no expectation about state transitions. Example is exploring an abandoned house. You have no idea what you'll find next.
Plan - you have a model of state transitions and can make predictions. One step or multi step prediction capabilities allow you to predict an outcome and therefore choose an outcome from several options.

You can search with prior knowledge. This is typically done with heuristics and this is known as informed search.

Your over thinking this person's question and not directly answering it at the level he is asking it. My answer was purposely simplistic because the question is simplistic

But it's wrong to say that search cannot use domain knowledge. A* (a search algorithm) is an informed search algorithm that can use domain knowledge. I'm not overthinking anything. My answer is based on my knowledge of the topic and the info that I've found after having researched the topic even further for a few hours.

Although your answer has correct information and mine leaves out lots of information I think that the person asking it needed a simple answer. If you boiled down your answer into a single qualitative description of the difference im curious what it would be.

The main difference between planning and search is given in my first paragraph. In other words, I would say that planning is an application of search (or applied search) or a "specialized" search in order to achieve a goal starting from some initial conditions. One thing that needs to be super clear is: planning definitely involves some notion of "search", but (almost) any problem-solving technique involves search anyway (maybe apart from random guessing).


text:
What does learning mean?
Can someone explain what is the process of learning? What does it mean to learn something?

This question is unclear. It's not clear whether you were interested in the concept of learning in machine learning or more generally. So, I closed it as "needs more details" to avoid poor and open-ended answers with different interpretations. Please, next time, before asking a question, do a little bit of research and tell us your findings. I would also recommend that you do not edit this post anymore and, if you're still interested in this question, ask a new question with more clarity and details.

That is a wonderfully fundamental question.

Learning is the use of a system to change another system so that, instead of doing what it did before, which may have been nothing, it does something else.

In the human brain, the system is the way that genetic expression caused the directed mutability of that brain so that human intentions and responses to external stimuli would alter. The direction of alteration is based on incentives and deterrents. In classes of children, there are things a teacher may do or a culture in the school that motivate learning. Misbehavior is deterred through other mechanisms. In that case, the first system is the educational process and the second system is the brain's ability to use what it was taught.
Even without classes, children learn things that create comfort, so comfort is the internal incentive. It is wired into us as a kind of teacher to dislike groin moisture, so we learn to use bathroom receptacles. It is wired into us to like praise from parents and teachers, so we tend to perform to get it. Things like that are complex losses and rewards and primary systems to incentivize learning.
Learning in artificial networks is not nearly as complex as in human brains. In some cases the artificial results are better. In other cases the abilities of the human brain cannot be approached by artificial networks yet.
The functioning of an artificial network often begins arbitrary and completely useless, but it's a parametric function, meaning the function can be changed by changing numbers called parameters. Each time the function is used to process an example drawn from a data set, the result is evaluated and the evaluation is used to modify the parameters. Care is taken to not over-modify the parameters, otherwise confusion can occur. Mathematicians call this type of system confusion chaos.
Repeating this carefully directed and moderated process ideally leads to something called convergence. The learning system is set up so that the result of convergence is a set of parameters that minimize losses, maximize gains, or both.
Sometimes initial learning is not enough and there are other related things to do later.

Adjusting learned behavior to adapt to new conditions or information
Unlearning things that no longer produce benefit so that new learning can replace them
Relearning things that had faded from memory
Developing greater confidence in what was learned so that unlearning requires greater impetus

There are additional technical terms for the above concepts, more details that can be grasped, many categories of learning system types, varieties within those, and the mathematics that was used as a foundation to construct all this and make it work. Because the question was fundamental, those details and technicalities were omitted.

Turing had much difficulty explaining how a computer could learn, and in his 1950 paper  asked, How could a machine learn? Its possible behavior is completely defined by its rules of operation (program), whatever the machine's history (past, present, future) might be. His proposed solution was ephemerally valid rules, whatever they might be - and he doesn't say.
So learning can be understood as the machine acquiring new behaviors but not by the behaviors being programmed into the machine by a human.
Perhaps a better way to look at this is causally. A human can define the causation (possible behavior) of a computer by programming it into the machine. But this is a case of the human using their knowledge to define how the machine will react to situations. Learning is the case where the machine itself acquires now behaviors or possible behaviors (not by a human using their knowledge of the world). And any such alleged intrinsic learning can easily be tested. If it helps the machine survive in a complex and hostile world, then it really is learning.

To be precise, Turing did not ask "How could a machine learn?", but "Can machines think?", at least, in section 1 of the paper. I didn't yet fully read the paper and there's a section "7. Learning Machines". In any case, I don't find the exact question "How could a machine learn?" in the paper. Maybe you can clarify the specific part of the paper that address the "learning". Yes, sure, the imitation game is framed as a way of assessing "human-like behaviour", but that doesn't necessarily imply learning.


text:
How should I predict which characters are going to die in a certain movie?
How would one go about predicting which characters (actors) are going to die e.g. in the next Avengers movie?
To elaborate a bit, given all leaked scripts (fake or not), interviews of different actors and directors, contracts with different actors and directors. How would/should one go about predicting if an actor is going to die in the upcoming movie or not.
NOTE: I am not sure, but I think a similar effort has already been made for Game of Thrones.

This is more of a probability question. Having said that, based on each leaked script, you generate the possibilities of each of the characters dying (for example yes 0.2%, no 0.7%, unknown 0.1%) - you also give a weight for each leaked script based on on its perceived authenticity (from 0.1 to 1). You multiply the weight of each script by each of the possibilities and then you add the result for each possibility and each character across all scripts. So, for example, say you have 2 scripts. The first script has perceived authenticity of 20% (0.2) and the second script has perceived authenticity of 40% (0.4). In the first script, Character A has 50% (0.5) chance of dying, 20% (0.2) of not dying, and 30% (0.3) chance of unknown outcome. In the second script, character A has 20% (0.2) chance of dying, 50% (0.5) chance of not dying, and 30% (0.3) chance of unknown outcome. So the following applies for Character A:

Dying: (0.2 x 0.5) + (0.4 x 0.2) = 0.18 (18% chance of dying)
Not dying: (0.2 x 0.2) + (0.4 x 0.5) = 0.24 (24% chance of not dying)
Unknown outcome: (0.2 x 0.3) + (0.4 x 0.3) = 0.20 (20% unknown outcome)

The probability of not dying is 24%, which is the highest.
Note: I am not saying that probability has nothing to do with AI (in fact, it is very important in AI).


text:
Can I filter barking sounds on the television?
My dog goes bonkers every time the sound of a barking dog is heard on a television program.  I never noticed this before but literally every movie or show with an outdoors setting eventually includes the sound of a barking dog.
Is it possible to develop a real-time filter that blocks or masks these sounds?

I mean, it's certainly possible in theory. But is that your question?  Do you have the skills needed to do data science, collect data sets, cleanse them, write the final program, and ultimately design and build a physical circuit and rewire your television to include it as a pre-speaker filter? No one here is going to do that on your behalf, if that's what you're trying to ask.

It's not a bad idea actually. This is a common problem. As this appears to be a wholly theoretical question, lets see if it garners some answers about AI approaches to this type of problem.

While I've studied data science enough to attempt model development, the other aspects are beyond my scope.  Maybe some student looking for a senior project will be inspired to take on this world changing challenge.

OK, here is one approach.

Acquire a data set of 'clean' audio samples without barking dogs and an data set of barking dogs sounds.
Generate a training set by mixing random selections of clean audio with a random selections of barking dog noises at various volume levels. This is your input data. Your output data is the clean audio before you added the barking.
Find a suitable neural network. This might be a good starting point: https://github.com/andabi/music-source-separation
Train the network.
Apply the network to a load of the clean+bark mixes and check the result by ear. If the results are not good enough, go to 3 to tweak the network. If the results are still poor go to 2 and generate more training data, perhaps adding some distortions or other effects the the barking. Happy with the result? Go to 6.
Publish the results online for the betterment of humanity. (And/or write the network to  a miniature GPU board (such as the NVIDIA Jetson https://developer.nvidia.com/buy-jetson) and work out how to wire that up to the decoded audio in your telly before it heads to the speakers), then blog about it because that would be a great project.

Thanks for the roadmap.  I'm sure if I record audio while I'm watching TV with my dog in the room for a few hours I can capture enough training data.

You'll need more than a few hours of barks to train a network to distinguish between barks and humans making short exclamations or car tire screeches, assuming the frequent elimination of false positives from the audio is not desirable.

The initial part of this plan 1,2 is similar to how audio systems can be trained to respond to activation phrases (e.g. "OK Google, "  or "Alexa, ")


text:
What is the definition of "soft label" and "hard label"?
In semi-supervised learning, there are hard labels and soft labels. Could someone tell me the meaning and definition of the two things?

Where have you come across these terms? Could you provide a source for that?

According to Galstyan and Cohen (2007), a hard label is a label assigned to a member of a class where membership is binary: either the element in question is a member of the class (has the label), or it is not.
A soft label is one which has a score (probability or likelihood) attached to it. So the element is a member of the class in question with probability/likelihood score of eg 0.7; this implies that an element can be a member of multiple classes (presumably with different membership scores), which is usually not possible with hard labels.
One use of soft labels in semi-supervised learning could be that the training set consists of hard labels; a classifier is trained on that using supervised learning. The classifier is then run on unlabelled data, and adds soft labels to the elements. This enlarged data set is then used for further training, where the algorithm can treat hard and soft labels differently.

Hi, My question is what models in NLP can treat differently the soft and the hard labels in the same dataset. Can they recognize the difference and treat the "hard" as the source of truth? A link also would be  very helpfull about these models. Thank you!


text:
How can a specific connectivity pattern be stored in an optimally compact representation?
I am interested in optimizing the memory capacity of an AGI. Given a specific complex input an AI can create a simplified model. This is a problem that can be solved using sparse coding [1]. However, this solves only the problem of encoding and not the maintenance of online representations—in cognitive terms: the state of mind.
A default cognitive model of short-term memory can be separated in three different stages:
Encoding → Maintenance → Retrieval

One solution is to use specialized hardware [2][3], but I am interested in software approaches to this problem and I would thus like to emphasize that it is the digital representation, which I am most interested in.
With the exception of qubits, the smallest possible representation are binary digits. However, additional architecture is required to represent phase spaces (i.e. floating point precision memory) and higher-order representations maybe include arrays and dictionaries. (Optimizing these are trivial... or simply to be postponed until needed, according to Knuth).

How can a specific connectivity pattern be stored in an optimally
compact representation?
Is there an implementation with concrete example code?
What is the state-of-the-art?*

* I will not specify "real-time" here, but the context is humanoid AGI.

[1] Papyan, V., Romano, Y., & Elad, M. (2017). Convolutional Neural Networks Analyzed via Convolutional Sparse Coding. Journal on Machine Learning Research, 18(83): 1–52. arXiv:1607.08194
http://jmlr.org/papers/volume18/16-505/16-505.pdf

[2] LeGallo et al. (2018). Mixed-precision in-memory computing. https://www.nature.com/articles/s41928-018-0054-8

[3] IBM. (2018). IBM Scientists Demonstrate Mixed-Precision In-Memory Computing for the First Time; Hybrid Design for AI Hardware. https://www.ibm.com/blogs/research/2018/04/ibm-scientists-demonstrate-mixed-precision-in-memory-computing-for-the-first-time-hybrid-design-for-ai-hardware/

Welcome to SE:AI!  I assume this is a hypothetical AGI we are discussing, but I feel like this is a deep encoding question involving binary representation.  (Continuous learning also seems ancillary, but is good context for the problem:) I tweaked the title to reflect the central question.

Thanks @DukeZhou! It's actually some theoretical groundwork for a basic AGI that I wish to implement IRL. Unfortunately, "deep encoding" doesn't seem to be a popular keyword. Thanks for pointing me to "Continuous learning". I think information-theory is applicable, but requires two Shannon-streams as the input destination is the temporary memory storage. Just pointing me to the right keywords is very helpful for my research.

Solution
In learning systems, the output of learning is a set of parameters. These parameters can be a tensor in the case of a CNN, a representation of a directed graph in the case of recursive network designs, or any other structure holding the results of training.
The general case of compactly storing the result of training involves four process elements. These elements can be combined into one algorithm if needed.

Encoding of ordinals
Lossless conversion of IEEE floats
Serialization of the structure
Lossless compression of the stream

Upon retrieval, the elements are reversed.

Decompression of the stream
Parsing to reassemble structure
Expansion of IEEE float representations
Decoding of ordinals

The encoding of ordinals requires the mapping of values to the minimal number of bytes or, most optimally, bits. The mapping must be reversible, meaning that there can be no information loss, so that decoding is completely reliable.
If the encoding and serialization packs the information tightly, there may be little gain with a general compression algorithm such as LZ4.
Notions of General Intelligence
Before devoting much time on developing humanoid AGI and modeling cognition and states of mind, it may be useful to curb expectations away from trendy ideas. There is much conflict between the naive conceptions of general intelligence and the realities of neurological systems in animals and people, which are far more sophisticated than implied in the trendy media hype. The most important considerations are these.

The complexity of networks in the brain
That there is a significant varieties of neurons in brains
The impact of neuroplasticity
Complexity of chemical signaling independent from nerves
Whether intelligence is in fact multi-dimensional
Large number of genes that correlate with intelligence
The academic slant of concepts regarding intelligence
Difference between brain signal topology and computing models
Data-centricity

There are many articles on the web on these topics, along with critiques of concepts such as the g-factor and the idea of general problem solving and general intelligence. There are several questions here on this topic.

Why did Artificial Intelligence projects fail?
Can the IQ of an AI program be measured?
Can a brain be intelligent without a body?
Is the singularity concept mathematically flawed?
Is there a ceiling to intelligence itself?
Self-conscious artifical intelligence
Is the Turing Test, or any of its variants, a reliable test of artificial intelligence?

There is no value in discouraging curiosity and serious research into understanding the nature of intelligence or efforts to expanding the boundaries of computing. The above comments are to assist in bringing the starting point of such inquiry into the realm of enthusiasm anchored by scientific methodology. Conjecture makes for great movie plots, but does not further science and does not lead to real world technology products and services.

I am well aware of the caveats concerning the construct validity of intelligence, the nature of consciousness, and neurophysiological realism. I am currently focusing on aspects of implementation. What is possible with current software engineering and ML knowledge? For example, in your answer to "Why did [AI] projects fail?" you mention the importance of parallel over serial processing. However, when I read open code that solves specific problems I am concerned that those solutions are too slow and not suitable for AGI. There is a bias toward exact solutions and less emphasis on heuristics.


text:
Why do we have to dot product in the Low-rank Bilinear Pooling?
I was reading this paper Hadamard Product for Low-rank Bilinear Pooling. I understand what they are trying to say, but I don't know why we have to convert the element-wise multiplication into a scalar (using the dot product)
$$
\mathbb{1}^{T}\left(\mathbf{U}_{i}^{T} \mathbf{x} \circ \mathbf{V}_{i}^{T} \mathbf{y}\right)+b_{i} \tag{2}\label{2}
$$
Why do we have to multiply the resulting vector by the one vector? We would still use the multiplicative interaction between elements if we did not consider multiplying by that one vector.

The paper (which I was not familiar with) proposes a new pooling method, which they name "Low-rank Bi-linear Pooling". Because it is a pooling method, its output should be a single value over some input spatial region (the goal of pooling is to down-sample an input representation). They do not want to use the resulting vector from the element-wise multiplication, they want to get a single scalar. Notice their initial formula:
$$
f_{i}=\sum_{j=1}^{N} \sum_{k=1}^{M} w_{i j k} x_{j} y_{k}+b_{i}=\mathbf{x}^{T} \mathbf{W}_{i} \mathbf{y}+b_{i}\label{1}\tag{1}
$$


text:
Can someone help me to understand the alpha-beta pruning algorithm?
I understand the minimax algorithm, but I am unable to understand deeply the minimax algorithm with alpha-beta pruning, even after having looked up several sources (on the web) and having tried to read the algorithm and understand how it works.
Do you have a good source that explains alpha-beta pruning clearly, or can you help me to understand the alpha-beta pruning (with a simple explanation)?

You can explore how alpha-beta prunes a tree here, but it doesn't animate the steps performed by alpha-beta: https://movingai.com/ab/

Have you read the following article: http://web.cs.ucla.edu/~rosen/161/notes/alphabeta.html? It should give you some intuition behind the algorithm.

Suppose that you have already search a part of the complete search tree, for example the complete left half. This may not yet give you the true game-theoretic value for the root node, but it can already give you some bounds on the game-theoretic value that the player to play in the root node (let's say, the max player) can guarantee by moving into that part of the search tree. Those bounds / guarantees are:

$\alpha$: the minimum score that the maximizing player already knows it can guarantee if we move into the part of the search tree searched so far. Maybe it can still do better (get higher values) by moving into the unsearched part, but it can already definitely get this value.
$\beta$: the maximum score that the minimizing player already knows it can guarantee if we moves into the part of the search tree searched so far. Maybe it can still do better (get lower values) by moving into the unsearched part, but it can already definitely get this value.

The intuitive idea behind alpha-beta pruning is to prune chunks of the search tree that become uninteresting for either player because they already know they can guarantee better based on the $\alpha$ or $\beta$ bounds.

For a simple example, suppose $\alpha = 1$, which means that the maximizing player already has explored a part of the search tree such that it can guarantee at least a value of $1$ by playing inside that part (the minimizing player has no options inside that entire tree to reduce the value below $1$, if the maximizing player plays optimally in that part).
Suppose that, in the current search process, we have arrived at a node where the minimizing player is to play, and it has a long list of child nodes. We evaluate the first of those children, and find a value of $0$. This means that, under the assumption that we reach this node, the minimizing player can already guarantee a value of $0$ (and possibly get even lower, we didn't evaluate the other children yet). But this is worse (for the maximizing player) than the $\alpha = 1$ bound we already had. Without evaluating any of the other children, we can already tell that this part of the search tree is uninteresting, that the maximizing player would make sure that we never end up here, so we can prune the remaining children (which could each have large subtrees below them).

I think this answer gives some intuition behind the algorithm, but it lacks a few clarifications. For example, you do not explain the reason behind searching with respect to the root player (i.e. because the first player is the one that plays first...). Or what happens if the root player is not the maximizing player but the minimixing player? Also, what happens if one or both players do not act optimality?

I think these things can confuse people. The idea of pruning parts of the tree because they will not be "explored", if both players act optimality, is clear, but it is not clear which players will play or not, what is the relation between the players, etc.

Minimax
As the question author may already understand, minimax is an approach before it is an algorithm. The goal is to minimize the advantage of a challenger and maximize the advantage of the participant presently applying the minimax approach. This is a subset of a wider strategy of summing benefits and subtracting the sum of costs to derive a net value.
Alpha-beta Pruning
The strategic goal of alpha beta pruning is to produce uncompromized decision making with less work. This goal is usually driven by the cost of computing resources, the impatience of the person waiting on results, or a missed deadline penalty.
The principle involved is that if there is a finite range to the net gain that can aggregate as a result of traversing any subtree in a search. It can be proven, using the algebra of inequalities, that no knowledge can be lost in the search if the subtree cannot, under any conditions, show any net advantage over the other options.
Some Graph Theory to Clarify Pruning
Vertices are the graph theory name for what are often called tree nodes. They are represented as $\nu$ below, and they correspond to states. The connections between vertices are called edges in graph theory. They are represented as $\epsilon$ below, and they correspond to actions.
The minimax net gain thus far accumulated at an vertex can be calculated as the tree is traversed.
As the limitations to net gain at further depths are applied to these intermediate net gain values, it can be inferred that one edge (action) will, in every case, be advantageous over another edge. If an algorithm reveals that one edge leads to disadvantage in every possible case, then there is no need to traverse that edge. Such dismissal of never-advantageous edges saves computing resources and likely reduces search time. The term pruning is used because edges are like branches in a fruit orchard.
The Condition of Pruning and the Algorithm To Accomplish It
Initialization for the algorithm begins by setting the two Greek letter variables to their worst case values.
$$ \alpha = - \infty \\
   \beta = \infty $$
The pruning rule for edge $\epsilon$ leading to a vertex $\nu$ is simply this.
$$ \alpha_\nu \geq \beta_\nu \Rightarrow \text{prune}(\epsilon_\nu) $$
Source code doesn't usually directly implement this declaration, but rather produces its effect in the algorithm, which can be represented in simplified pseudo-code. The algorithm to do such is simple once recursion is understood. That is an conceptual prerequisite, so learn about recursion first if not already known.
ab_minimax ν, α, β

    if ν.depth = search.max_depth or ν.is_leaf
        return ν.value

    if ν.is_maximizing
        for ε in ν.edges
            x = max(β, ab_minimax ε.child, α, β)
            if α ≥ β
                ν.prune ε
                return α
        return α
    else
        for ε in ν.edges
            x = min(α, ab_minimax child, α, β)
            if α ≤ β
                ν.prune ε
                return β
        return β


text:
How do I update the Q values of a Deep Q Network when exploring?
I am trying to implement a Deep Q Network to play Asteroids. Unfortunately, I am not sure how to calculate the Q value exactly, if I am exploring. For example, the agent is exploring for 1 second (otherwise makes no sense; I cannot let it just explore one step). Unfortunately, it makes a mistake at 0.99s, and the reward collapses.
At the moment, I am using the following formula to evaluate or update the Q value:
$$Q_{new,t} = reward + \gamma Q_{max,t+1}$$
But how do I know the max Q value of the next step? I could consider the best Q value the network says, but this is not necessarily true.
You can see the current implementation at the following URL:
https://github.com/SuchtyTV/RLearningBird/blob/master/src/main/java/rlgame/Brain.java.

For tabular Q-learning, the q-values for state s and action a are updated according to
$$
Q(s, a) \gets Q(s, a) + \alpha [(r + max_{a'} Q(s', a')) - Q(s,a)]
$$
where $\alpha$ is the learning rate and $(r + max_{a'} Q(s', a')) - Q(s,a)$ is the difference between the current estimate of the q-value, $Q(s,a)$, and the target, $r + max_{a'} Q(s', a')$.
The target q-value is based on the greedy policy, not the exploratory policy. Q-learning is theoretically guaranteed to converge to the optimal policy for any behavior policy (like $\epsilon$-greedy) that is guaranteed to visit every state and action pair an infinite number of times.  See Section 6.5 of the Sutton and Barto book for more details.
In contrast to Q-learning, the target q-value for SARSA is $r + Q(s', a')$, where $a'$ is chosen from an exploratory behavior policy like $\epsilon$-greedy. For SARSA the learned q-values are dependent on the behavior policy and therefore not guaranteed to converge to the optimal policy. A behavior policy that intentionally acted randomly for multiple consecutive actions, as in your example Asteroids exploratory policy, would likely lead to learning different q-values than would be learned for an $\epsilon$-greedy behavior policy.
Unfortunately Q-learning's theoretical guarantees of convergence to an optimal policy go out the window when nonlinear function approximation is introduced, as is the case for deep neural networks. Nevertheless, in the Deep Q-Networks paper, the q-value function is updated using a target value based on the maximum q-value for the next state. Specifically, if $Q(s, a, w)$ is a q-value function parameterized by weights $w$, then the weights are updated by
$$
w \gets w + \alpha [(r + max_{a'} Q(s', a', w^-)) - Q(s, a, w)] \nabla_w Q(s,a,w)
$$
where $w^-$ are the parameters of the target network used to stabilize training. (See the paper for more details). This update rule is chosen to minimizes the loss function
$$
L(w) = E[(r + max_{a'} Q(s', a', w^-)) - Q(s, a, w)]^2
$$
For your own implementation, it may be helpful to see a code example of the Deep Q-Networks parameter updates.  A tensorflow implementation is available in the function build_train in the OpenAI Baselines DeepQ code.

Thanks for your answer. Unfortunately, reading the python code of tensorflow is beyond my grasp.
Maybe you can help me with this concrete example.
A Game lasts 3 steps.
First: the NN goes left then right, then up
It gets the rewards 0,0,1.
It calculated the q value of the first action to be -1. So what I would have to do step by step to calculate the new q value.


text:
What kind of AI technique can I use to play the "Lines" game?
I am trying to find a good approach to create a computer player for the game "Lines" from gamious on Android. The concept of the game is pretty straightforward :

Lines is an abstract ‘zen’ game experience where form is just as important as function. Place or remove Dots to initiate a colourful race that fills a drawing. The colour that dominates the race wins.

The game starts with a drawing (that can be described as a set of "blank" lines, with connection to other lines). Dots of different colour are placed somewhat randomly on the lines. The player get a colour assigned. When the game start, paint start flowing from the dots and filling the (at first blank) lines of the drawing. You win if your colour dominates.
The game gives you different tools to win (the game starts when all of them have been used) :

[0 to 2] scissor to cut lines
[0 to 5] additional dot of your own color to place on the drawing
[0 to 4] enemy dots eraser
[0 to 3] additional straight lines to connect different part of the drawing

A quick example: the first image is the initial state of a round. "My" colour is the yellow (1 enemy = brown) and I have 4 tools (2 eraser and 2 lines). The second image shows the game running after I used the tools to put my colour in a winning position (yes, we can do better)

If I try to approach this as a classical optimization problem, things get messy pretty fast :

highly non-linear
high number of dimensions

AI seems to be the right way to go, but I would like your help to get in the right direction: what would be your approach to create an AI to play this game?
To limit the scope of this question, you can consider that I already have a data structure to represent the game initial state, the use of different tools and the game "physics". What I really want to do is finding how to create an AI which can learn how to efficiently use the tools.
Regarding my experience, I took 2 semesters of AI classes during the last year getting my engineering degree and have used non-linear optimization tools for a while: you can go technical, but I am not fully understand it.

Could you explain how your input is structured? Are you starting with a bitmap image of the game, so your first challenge is requiring a system that can comprehend the game based on a screen image? If not, what data structure are you using?

@NeilSlater To reduce the scope of the question, you can consider that the game is already represented by the data structure you prefer (2d image recognition would be fairly easy). What do you mean by "input system" ?

I didn't say "input system". I was using the word "system" in a general hand-wavy way - could also be "component" or "pre-processor" or whatever. Either way the problem is more complex if you add a requirement for machine vision and that the AI has to solve the game using the same I/O as a human player. You have clarified that you have that solved already though, thanks for the edit

If you can get the game state representation to be the most convenient one possible (probably a graph with all the relevant points and their connections, plus their current colours), and you can efficiently implement the "rules" of the game... my go-to algorithm would be Monte-Carlo Tree Search. If you have to work with raw pixels, you're likely better off learning a policy end-to-end from pixels (i.e. Deep Reinforcement Learning, with extra precautions due to it being a two-player game rather than single-player)

@DennisSoemers Thanks for your input. What bothers me the most is the "add new line" tool. "Scissor", "eraser" and "new dots" are easier to score and to optimize. But "new line" is harder... Regarding the MCTS algorithm, it seems similar to optimization with random restart, the hard part is to choose the moves with some intelligence (instead of playing randomly)... Right know I am trying to find for each tool a matrix linking the tool position to the final score, so I could optimize tool by tool, but no luck so far. I will try the MCTS next week :)

@Olivier MCTS became popular specifically in domains where it is difficult/impossible to construct good heuristic evaluation functions by hand (such as the game of Go, and General Game Playing where agents should be able to play pretty much any game without knowing which games they're gonna be playing in advance). The basic intuition behind it is to play lots of games (semi-)randomly, use the average outcomes (wins/losses) of many such simulations as evaluation functions. Plus some tree building to ensure convergence to minimax when given an infinite amount of time.

@Olivier That does assume actually that you have access to a "simulator" or a "model" of the game though. That you can simulate the outcomes that would result from given trajectories / sequences of actions.

Writing a game AI is difficult in general and it's even harder when there's no game to play. Like in your case.
While "Lines" is pretty neat, there's no (playing) opponent and there are no moves. apart from the setup. So what we have is an optimization problem where we need to get more point than the "opponents".
Game playing algorithms like Reinforcement learning or MCTS are IMHO out of questions as there's no game to play. Still, it's an interesting problem, though I can't see how to make any learning AI. Anyway, what you need first is a simulator and an optimizer choosing promising actions.
Some ideas concerning the actions
Some actions are trivial, e.g., the eraser as there are just a few dots to erase. Trying all combination should be good enough for the start.
The scissors can be in theory placed anywhere, but the only interesting places are just next to the oponent piece or a crossing. Additionally, you only need to consider crossings which gets "won" (reached first) by an opponent. Moreover, they only matter if they get won by an opponent having more points than you or blocking you.
The dots to insert should similarly by placed near an oponent or so that you just win a crossing.
The additional lines are worst, as they may be used to let you reach unconnected areas, or reach a crossing faster or simply give you additional points (then you want a long line somewhere).
A model
Most of the game can be seen as a non-directed graph with edges labeled by their length, where the crossing and the initial dots are nodes. The simulator doesn't need to work "pixel by pixel", but may instead jump to the next time, when a crossing gets reached or two "flows" meat each other. The points of a player grow linearly with time, where the slope is given by the number of their extending "flows".
So far, I could imagine modelling it as an Integer Linear Program, but with the additional lines, it's probably no more possible.
I'd probably let the optimization be simulator-guided, i.e., analyse whether I could get closer to winning by reaching a crossing sooner. The details are still unclear....


text:
If deep learning is a black box, then why are companies still investing in it?
If deep learning is a black box, then why are companies still investing in it?

Companies invest because deep learning has been very effective for many companies in achieving desired goals. The 'black box' neural network itself is the easy part. The hard part is deciding what inputs to use, what representation should those inputs be presented in, what should be the expected outputs and representation, what utility functions will achieve the best results and in many cases collecting a set of training data that will actually lead to good results. That's what they are investing in. I disagree there are only small numbers of systems doing something real.

Easy answer: utility.
The strength and applicability of "black box" NNs has been regularly validated in the past few years, and business is concerned with results.  (i.e. they don't care how the sausage is made, so long as it gets made.)

:Very useful answer. One of my friend ask me the question about iPhone facial recognition .Can we define this process to lay person?How things are happening at backend to tag my photos?How iPhone facial recognition works.Any mathematical evidance?.How can we inspire someone who is machine learning enthusiast and needs some facts to enter into this research.I need to motivate my junior researchers with useful answers

Yes, I think you're right, but I think it would be also nice to at least mention that, in certain areas, like healthcare, people are reluctant to use deep learning (or ANNs) because of its "black box" nature. See also explainable AI.

I think that the universal approximation theorem plays a large role in why companies and governments are investing in deep learning, it states that theoretically an ann can approximate any continuous function with n-dimension input variables. Specifically it states that feed forward nets with a single hidden layer can do this lending credence to the implication that rnns and cnns are also capable of universal function approximation. So they are investing because they have continuous functions that need to be approximated and really the best tool for the job is neural networks.

RNNs have been proven to be Turing complete and CNNs have also been proven to be universal approximators, so the powerfulness of RNNs and CNNs is not just hypothetical. See https://ai.stackexchange.com/a/13319/2444. Anyway, I don't think companies are investing in neural networks because they are theoretically powerful. AIXI is also theoretically an optimal agent, but nobody uses it.

Well, I think AIXI isn't used due to the fact that it is incomputable and the approximations of it being less effective than other comparable RL algorithms.


text:
What is an adversarial attack?
I'm reading this really interesting article CycleGAN, a Master of Steganography. I understand everything up until this paragraph:

we may view the CycleGAN training procedure as continually mounting an adversarial attack on $G$, by optimizing a generator $F$ to generate adversarial maps that force $G$ to produce a desired image. Since we have demonstrated that it is possible to generate these adversarial maps using gradient descent, it is nearly certain that the training procedure is also causing $F$ to generate these adversarial maps. As $G$ is also being optimized, however, $G$ may actually be seen as cooperating in this attack by learning to become increasingly susceptible to attacks. We observe that the magnitude of the difference $y^{*}-y_{0}$ necessary to generate a convincing adversarial example by Equation 3 decreases as the CycleGAN model trains, indicating cooperation of $G$ to support adversarial maps.

How is the CycleGAN training procedure an adversarial attack?
I don't really understand the quoted explanation.

Adversarial examples are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake; they’re like optical illusions for machines.

Source: Attacking Machine Learning with Adversarial Examples
You create an input and test against the output, tuning the input to maximize the error. There are different criteria for tuning the input, sometimes you might want the shortest possible input to create the largest error, often you would want very similar input to cause the greatest error.
Example: I take a \$1 bill and write a few more zeros after the one; you accept it as a \$1000 bill.

I guess they are talking about adversarial attacks in the same way that Szegedy et al. did in "Intriguing properties of neural networks"
They described "adversarial attacks" or "adversarial examples" as images with hardly perceptible perturbations that change the network's prediction.
For example, imagine you've trained a CNN to classify between a variety of classes. You take a picture of a dog $X_1$, and your CNN correctly classified it as a "dog", everything is fine so far.
Then you can add some small perturbation $p$ to your image $X_1$, so now you have a new image $X_2 = X_1 + p$. This new image still looks like a dog, because your perturbation was so small that is almost imperceptible.
The problem is that your CNN will classify your picture $X_2$ as something that is not a dog, for example, "fish".
Here, $X_2$ is an adversarial example created after using an adversarial perturbation $p$.
What is interesting about these adversarial perturbations $p$ is that they are not random. Actually, CNNs are very robust to random perturbations (noise), but adversarial perturbations $p$ are not like them. They are computed to fool a classifier (not only CNNs).
You can refer to figure 5 of the aforementioned paper for more examples.

Simple answer is tweaking an image in unnoticeable ways that completely fool software. Eg a cat that is identified as 99% likely "to be guacamole" https://mashable.com/2017/11/02/mit-researchers-fool-google-ai-program/#CU7dSAfQ5sqY


text:
How can I use a 2-dimensional feature matrix as the input to a neural network?
How can I use a 2-dimensional feature matrix, rather than a feature vector, as the input to a neural network?
For a WWII naval wargame, I have sorted out the features of interest to approximate the game state $S$ at a given time $t$.

they are a maximum of 50 naval task forces and airbases on the map

each one has a lot of features (hex, weather, number of ships, type, distance to other task forces, distance to objective, cargo, intelligence, combat value, speed, damage, etc.)

The output would be the probability of winning and the level of victory.

I have seen that AlphaGo and AlphaGo Zero have been processing data as an image
19 x 19 x (17 or 48) features
and then a CNN

The most obvious way to do this would be to simply "unroll" your matrix into a vector. Your example input matrix would get turned into the following input vector:
$$\left( \begin{array}{} a_1 & a_2 & \dots & a_t & b_1 & b_2 & \dots & b_t & c_1 & c_2 & \dots & c_t \end{array} \right)$$
I don't think there are any other clear ways to use an "input matrix" really. The only benefit I could see in using an input matrix rather than an unrolled vector (if it were possible to do so in whatever way) would be if doing so would somehow enable the learning algorithm to exploit the "domain knowledge" that certain input features are related to each other in special ways (i.e. features in the same row belong to the same unit, and features in the same column are the same "type" of feature, or other way around). Intuitively, I suspect something like this could be accomplished by restricting the number of connections you make to the next layer. For example, you could make a part of the next layer only be connected to all the $a_i$ features, a different part connected only to all the $b_i$ features, etc. Similarly, you could have a part that is connected only to the $a_1, b_1, c_1, \dots$ features, a different part only connected to the $a_2, b_2, c_2, \dots$ features, etc. I don't know for sure how well this would work though... just think that it could.

Then it would make sense to remove the dependency of the units:  for instance instead of keeping all distance between them, then add a feature representing the density of friendly units for the given unit.
It would still mean 2000 features. Is it too much for feed forward neural network and single computer ?

@CarrierBattles Hmmm... difficult for me to judge if it would be "too much". 2000 features is a lot though. I think it may be too much, definitely wouldn't be surprised if it were... but not 100% sure. Best way is to try and see. Your problem does sound like a large and complex problem in and of itself though (controlling many different units at the same time, like in Real-Time Strategy games, simply is a complex problem). When talking about such complex problems, it often becomes difficult to avoid solutions that are "too much" for a small amount of hardware.

(Split the answer from Dennis in two)
A very different approach would be to use a network architecture with recurrence, for example an LSTM. You could treat your input as a "sequence" rather than a matrix (just like a sentence in language processing would be a sequence of inputs), providing your feature vectors for different units as inputs one at a time. This would remove the need of having a giant input layer (with support for 50 units) in cases where only a small portion of them would be used (e.g., if you only have 5 units). There is not really a concept of "time" in your inputs though... ideally the output of your network would be invariant to the order in which you provide it with the different inputs for the different units, but that would not typically be the case with these kinds of architectures in practice.

This seems a tweak of the usual way to use Recurrent Neural Networks but why not. I see the advantage of having only the right numbers of unit vectors instead a big block filled with zero for missing units. For the relationships between the units, I may strive to simplify the features using some friendly/enemy units density for each unit for instance. For each S(t), shall I always keep logical order (always the same unit first) or a Radom order ?

By using this method and removing nice to have features, I may end up with less than 700 features in average
What are the performance of such network compared to the classical perceptron ?


text:
Why isn't the Credit Card Fraud Detection dataset from Kaggle already balanced?
I am working on a credit card fraud detection problem using autoencoders. I have a question regarding the dataset I'll be using.
I've downloaded the dataset for the above problem from Kaggle, which is highly imbalanced: it contains only 492 frauds out of 284,807 transactions. Why isn't the dataset already balanced? Should I balance it before applying the auto-encoder?

I've edited your post to leave only one question because, otherwise, your post should have been closed as too broad. Furthermore, I've added a link to the supposed dataset you are talking about and I've updated the info about this dataset.

I believe that the idea is to have a similar ratio of fraud/"normal transaction" as to the ones that bank encounter on real life.
If you balance it you will probably have a lot of false positive once you apply your solution to real world's data and, if that may be fine for you to play with, it's not what a bank would like as they can't block too much of the "normal" transaction or the client will change of bank. According to this post (https://www.quora.com/How-many-transactions-do-typical-banks-process-everyday) just the mastercard alone represent 3.4billion transaction/day, just imagine if 1% of the daily transaction where blocked every day, it would represent 34 million of transactions blocked without any valid reason.
It's different from a lot of classification problem where you want to have balanced dataset, here you try to detect anomalies and, by definition, they are rare so they should be as rare in your dataset.


text:
How to crossover chromosomes composed of genes that are tuples such that the elements of the tuples do not appear twice in the chromosome?
Each chromosome contains an array of genes, each gene contains a letter and a number, both letter and number can only exist once in each chromosome.
Parent A = {a,1}{c,2}{e,3}{g,4}
Parent B = {a,2}{b,1}{c,4}{d,3}

What would be the best crossover operator to create a child that doesn't break the rule described above?

First, store the two parent chromosomes into a sorted dictionary (in terms of implementation, std::map in C++ might be a good option) where the key is the letter and the value is a pair of ints (the second gene element). When you populate the map, add all the letters from each chromosome as keys and take 0 (or a negative number) to indicate that a gene with the corresponding letter does not exist in one of the parents.
Next, create a second map, this time with the number as a key. Iterate over the first map. If both parent chromosomes contain a gene with the same letter, the value field in the map would contain two positive numbers - pick one at random and insert the {number, letter} combination into the second map only if the number doesn't exist as a key. If the gene exists only in one parent, either pick the gene as is or choose to skip that gene altogether.
This would guarantee that 1) your genomes contain a {letter, number} gene only once and 2) the genes in the offspring's chromosomes would be automatically sorted by letter. If you don't need the genes to be sorted but still need them to be unique, you can use an unordered map (a hash table).
In your example, the map would look like this:
Mapped parent chromosomes
{a: (1,2)}
{b: (0,1)}
{c: (2,4)}
{d: (0,3)}
{e: (3,0)}
{g: (4,0)}

Offspring chromosome (OC)
{}

Where 0 indicates that the corresponding gene does not exist in that parent. Iterating over this table from top to bottom, you might end up doing the following:

Key a: pick 1; 1 is not a key in OC, so insert {1,a}
Key b: pick 1; 1 is a key in OC, so skip letter b
Key c: pick 4; 4 is not a key in OC, so insert {4,c}
Key d: choose to skip
Key e: pick 3; 3 is not a key in OC, so insert {3,e}
Key g: pick 4; 4 is a key in OC, so skip letter g

Final offspring chromosome:
{a,1}{c,4}{e,3}

Thanks for the reply I really appreciate it. However, the offspring you created is invalid because the number 4 exists twice in the offspring and that would be invalid in my solution.

I see - I have updated the answer to take into account this clarification.


text:
How should we choose the dimensions of the encoding layer in auto-encoders?
How should we choose the dimensions of the encoding layer in auto-encoders?

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so it should be small enough to be trainable in a reasonable time.


text:
How can I design a reinforcement learning model for a game with multiple complex actions taken at a time?
I have a steady hex-map and turn-based wargame featuring WWII carrier battles.
On a given turn, a player may choose to perform a large number of actions. Actions can be of many different types, and some actions may be performed independently of each other while others have dependencies. For example, a player may decide to move one or two naval units, then assign a mission to an air unit or not, then adjust some battle parameters or not, and then reorganize a naval task force or not.

Usually, boardgames allow players to perform only one action each turn (e.g. go or chess) or a few very similar actions (backgammon).
Here the player may select

Several actions
The actions are of different nature
Each action may have parameters that the player must set (e.g. strength, payload, destination)

How could I approach this problem with reinforcement learning? How would I specify a model or train it effectively to play such a game?
Here is a screenshot of the game.

Here's another.

This an old question, but could you please provide the name of this game? Moreover, how are you actually interacting with the game? Is there an available public API that the game developers provide so that users can program their own AI? Maybe this is a game you're developing. It seems that the game is this one: https://apprecs.com/ios/961994680/carrier-battles-4-guadalcanal or https://carrier-battles.com/.

The master's thesis Action space representation in combinatorial multi-armed bandits (2015) seems to provide an answer to my question.
Several algorithms can be used

Naive Monte-Carlo Sampling (NMC)
Linear Side Information (LSI)
Monte Carlo Tree Search with Hierarchical Expansion (MCTS-HE)
MCTS with Dimensional Expansion

The idea is to divide and conquer.
Here's a screenshot of the HE algorithm from section 4.2.

One very popular RL algorithm that is capable of predicting multiple action outputs concurrently is Proximal Policy Optimization. In that algorithm, one or more, say $n$, tuples of outputs, $(\mu, \sigma)$, can be predicted at once (having $2*n$ output nodes), where each tuple is used to parameterize a Gaussian distribution from which a respective action value is sampled. The thus sampled action values are then applied in the simulation/game. By slightly modifying this procedure, this can of course also be applied to discrete action spaces equally well.
To get you started, multiple high-quality implementations of PPO are available for rapid prototyping, e.g. OpenAI baselines or stable-baselines.