text:
Generalising performance of Q-learning agent through self-play in a two-player game (MCTS?)
I'm using Q-learning (off-policy TD-control as specified in Sutton's book on pg 131) to train an agent to play connect four. My goal is to create a strong player (superhuman performance?) purely by self-play, without training models against other agents obtained externally.
I'm using neural network architectures with some convolutional layers and several fully connected layers. These train surprisingly efficiently against their opponent, either a random player or another agent previously trained through Q-learning. Unfortunately the resulting models don't generalise well. 5000 episodes seems enough to obtain a high (> 90%) win rate against whichever opponent, but after > 20 000 episodes, they are still rather easy to beat by myself.
To solve this, I now train batches of models (~ 10 models per batch), which are then used in group as a new opponent, i.e.:

I train a batch of models against a completely random agent (let's call them the generation one)
Then I train a second generation of agents against this first generation
Then I train a third generation against generation two
...

So far this helped in creating a slightly stronger/more general connect four model, but the improvement is not as good as I was hoping for. Is it just a matter of training enough models/generations or are there better ways for using Q-learning in combination with self-play?
I know the most successful techniques (e.g. alpha zero) rely on MCTS, but I'm not sure how to integrate this with Q-learning? Neither how MCTS helps to solve the problem of generalisation?
Thanks for your help!

To solve this, I now train batches of models (~ 10 models per batch), which are then used in group as a new opponent,

This seems quite a reasonable approach on the surface, but possibly the agents will still lose generalisation if the solutions in each generation are too similar. It also looks like from your experiment that learning progress is too slow.
One simple thing you could do is progress through the generations faster. You don't need to train until agents win 90% of games before upping the generation number. Yuo could set the target as low as 60% or even 55%.
For generalisation, it may also help to train against a mix of previous generations. E.g. if you use ten opponents, have five from previous generation, two from each of two iterations before that, and one even older one.
Although the setup you have created plays an agent you are training against another agent that you have created, it is not quite self-play. In self-play, an agent plays against itself, and learns as both players simultaneously. This requires a single neural network function that can switch its evaluation to score for each player - you can either make it learn to take the current player into account and make the change in viewpoint itself, or in zero-sum games (which Connect 4 is one) it can be more efficient to have it evaluate the end result for player 1 and simply take the negative of that as the score for player 2. This is also equivalent to using $\text{max}_a$ and $\text{argmax}_a$ for player 1's action choices and  $\text{min}_a$ and $\text{argmin}_a$ for player 2's action choices - applying the concept of minimax to Q learning.
You can take minimax further to improve your algorithm's learning rate and performance during play. Essentially what Q learning and self-play does is learn a heuristic for each state (or state/action pair) that can guide search. You can add search algorithms to your training and play in multiple ways. One simple approach during training is to perform some n-step look ahead using negamax with alpha-beta pruning (an efficient variant of minimax in zero-sum games), and if it finds the end of the game:

when training, use the result (win/draw/lose) as your ground truth value instead of the normal Q-learning TD target.
when evaluating/playing vs human, prefer the action choice over anything the Q function returns. In practice, only bother with the Q function if look-ahead search cannot find a result.

In the last few months, Kaggle have been running a "Connect X" challenge (which is effectively only Connect 4 at the moment). The forums and example scripts (called "Kernels") are a good source of information for writing your own agents, and if you choose to compete, then the leaderboard should give you a sense for how well your agent is performing. The top agents are perfect players, as Connect 4 is a solved game. I am taking part in that competition, and have trained my agent using self-play Q-learning plus negamax search as above - it is not perfect, but is close enough that it can often beat a perfect playing opponent when playing as player 1. It was trained on around 100,000 games of self-play as I described above, plus extra training games versus previous agents.

I know the most successful techniques (e.g. alpha zero) rely on MCTS, but I'm not sure how to integrate this with Q-learning? Neither how MCTS helps to solve the problem of generalisation?

MCTS is a variant of search algorithm, and could be combined with Q-learning similarly to negamax, although in Alpha Zero it is combined with something more like Actor-Critic. The combination would be similar - from each position in play use MCTS to look ahead, and instead of picking the direct action with the best Q value, pick the one with the best MCTS score. Unlike negamax, MCTS is stochastic, but you can still use its evaluations as ground truth for training.
MCTS does not solve generalisation issues for neural networks, but like negamax it will improve the performance of a game-playing agent by looking ahead. Its main advantage over negamax in board games is a capability to scale to large branching factors. MCTS does work well for Connect 4. Some of the best agents in the Kaggle competition are using MCTS. Howver, it is not necessary for creating a "superhuman" Connect 4 agent, Q-learning plus negamax can do just as well.

Thanks for this really useful answer!
As for your first point, I already mix up opponents of several generations, but didn't mention it in my question to not over-complicate things. Good suggestion to not train all the way to 90% win rate and up quicker, I'll look into it.
From your answer, it's clear that I was still missing some subtleties related to the zero-sum aspect of this problem. Your answer relates it to techniques negamax and MCTS far clearer than I've come across anywhere else. This will definitely help!

MCTS does not help with generalization directly, but it enables the agent to plan ahead (see depth-first search or breadth-first search). Having the state space search embedded in the algorithm is very important for playing zero sum games (we also plan ahead in our head when making moves right?). Now Q-learning is generally good for simple environments, but to achieve superhuman performance on board games you would need HUGE amounts of data without using any planning algorithm. I don't even know if practically achieving superhuman performance by only Q-learning is even possible.


text:
What are bag-of-features in computer vision?
In computer vision, what are bag-of-features (also known as bag-of-visual-words)? How do they work? What can they be used for? How are they related to the bag-of-words model in NLP?

Introduction
Bag-of-features (BoF) (also known as bag-of-visual-words) is a method to represent the features of images (i.e. a feature extraction/generation/representation algorithm). BoF is inspired by the bag-of-words model often used in the context of NLP, hence the name. In the context of computer vision, BoF can be used for different purposes, such as content-based image retrieval (CBIR), i.e. find an image in a database that is closest to a query image.
Steps
The BoF can be divided into three different steps. To understand all the steps, consider a training dataset $D = \{x_1, \dots, x_N \}$ of $N$ training images. Then BoF proceeds as follows.
1. Feature extraction
In this first step, we extract all the raw features (i.e. keypoints and descriptors) from all images in the training dataset $D$. This can be done with SIFT, where each descriptor is a $128$-dimensional vector that represents the neighborhood of the pixels around a certain keypoint (e.g. a pixel that represents a corner of an object in the image).
If you are not familiar with this extraction of computer vision (sometimes known as handcrafted) features, you should read the SIFT paper, which describes a feature (more precisely, keypoint and descriptor) extraction algorithm.
Note that image $x_i \in D$ may contain a different number of features (keypoints and descriptors) than image $x_j \neq x_i \in D$. As we will see in the third step, BoF produces a feature vector of size $k$ for all images, so all images will be represented by a fixed-size vector.
Let $F= \{f_1, \dots, f_M\}$ be the set of descriptors extracted from all training images in $D$, where $M \gg N$. So, $f_i$ may be a descriptor that belongs to any of the training examples (it does not matter which training image it belongs to).
2. Codebook generation
In this step, we cluster all descriptors $F= \{f_1, \dots, f_M\}$ into $k$ clusters using k-means (or another clustering algorithm). This is sometimes known as the vector quantization (VQ) step. In fact,  the idea behind VQ is very similar to clustering and sometimes VQ is used interchangeably with clustering.
So, after this step, we will have $k$ clusters, each of them associated with a centroid $C = \{ c_1, \dots, c_k\}$, where $C$ is the set of centroids (and $c_i \in \mathbb{R}^{128}$ in the case that SIFT descriptors have been used). These centroids represent the main features that are present in the whole training dataset $D$. In this context, they are often known as the codewords (which derives from the vector quantization literature) or visual words (hence the name bag-of-visual-words). The set of codewords $C$ is often called codebook or, equivalently, the visual vocabulary.
3. Feature vector generation
In this last step, given a new (test) image $u \not\in D$ (often called the query image in this context of CBIR), then we will represent $u$ as a $k$-dimensional vector (where $k$, if you remember, is the number of codewords) that will represent its feature vector. To do that, we need to follow the following steps.

Extract the raw features from $u$ with e.g. SIFT (as we did for the training images). Let the descriptors of $u$ be $U = \{ u_1, \dots, u_{|U|} \}$.
Create a vector $I \in \mathbb{R}^k$ of size $k$ filled with zeros, where the $i$th element of $I$ corresponds to the $i$th codeword (or cluster).
For each $u_i \in U$, find the closest codeword (or centroid) in $C$. Once you found it, increment the value at the $j$th position of $I$ (i.e., initially, from zero to one), where $j$ is the found closest codeword to the descriptor $u_i$ of the query image.
The distance between $u_i$ and any of the codewords can be computed e.g. with the Euclidean distance. Note that the descriptors of $u$ and the codewords have the same dimension because they have been computed with the same feature descriptor (e.g. SIFT).
At the end of this process, we will have a vector $I \in \mathbb{R}^k$ that represents the frequency of the codewords in the query image $u$ (akin to the term frequency in the context of the bag-of-words model), i.e. $u$'s feature vector. Equivalently, $I$ can also be viewed as a histogram of features of the query image $u$. Here's an illustrative example of such a histogram.

From this diagram, we can see that there are $11$ codewords (of course, this is an unrealistic scenario!). On the y-axis, we have the frequency of each of the codewords in a given image. We can see that the $7$th codeword is the most frequent in this particular query image.
Alternatively, rather than the codeword frequency, we can use the tf-idf. In that case, each image will be represented not by a vector that contains the frequency of the codewords but it will contain the frequency of the codewords weighted by their presence in other images. See this paper for more details (where they show how to calculate tf-idf in this context; specifically, section 4.1, p. 8 of the paper).

Conclusion
To conclude, BoF is a method to represent features of an image, which could then be used to train classifiers or generative models to solve different computer vision tasks (such as CBIR). More precisely, if you want to perform CBIR, you could compare your query's feature vector with the feature vector of every image in the database, e.g. using the cosine similarity.
The first two steps above are concerned with the creation of a visual vocabulary (or codebook), which is then used to create the feature vector of a new test (or query) image.
A side note
As a side note, the term bag is used because the (relative) order of the features in the image is lost during this feature extraction process, and this can actually be a disadvantage.
Further reading
For more info, I suggest that you read the following papers

Video Google: A Text Retrieval Approach to Object Matching in Videos (2003) by Sivic and Zisserman
A Bayesian Hierarchical Model for Learning Natural Scene Categories (2005) by Fei-Fei and Perona
Introduction to the Bag of Features Paradigm for Image Classification and Retrieval (2011) by O'Hara and Draper
Bag-of-Words Representation in Image Annotation: A Review (2012) by Tsai

Can I infer then for matching images you compare the histogram vectors of different images with examplars?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to measure sample efficiency of a reinforcement learning algorithm?
I want to know if there is any metric to use for measuring sample-efficiency of a reinforcement learning algorithm? From reading research papers, I see claims that proposed models are more sample efficient but how does one reach this conclusion when comparing reinforcement learning algorithms?

First of all, let's recall some definitions.
A sample in the context of (deep) RL is a tuple $(s_t, a_t, r_t, s_{t+1})$ representing the information associated with a single interaction with the environment.
As for sample efficiency, it is defined as follows [1]:

Sample efficiency refers to the amount of data required for a learning system to attain any chosen target level of performance.

So, how you measure it is closely related to the way it is defined.
For example, one way to do it would be as shown in the figure below:

On the y-axis you have the performance of your RL algorithm (e.g. in terms of average return over episodes as done in [2], or mean total episode reward across different environment runs as done in [3])

On the x-axis you have the number of samples that you took.

The dashed line corresponds to your performance baseline (e.g. the performance at which a certain game or any other RL environment is considered solved).

So, you can measure sample efficiency at the intersection, where you get the number of samples needed to reach the performance baseline. So, an algorithm that requires less samples would be more sample-efficient.

Another way to do it would be the other way around, i.e. the RL agent is provided with a limited budget for the number of samples it could take. As a result, you can measure sample efficiency my measuring the area under the curve, as illustrated below. So, that would be how much performance you got just by using those samples in the budget. An algorithm that achieves a higher performance than another with the same amount of samples would then be more sample efficient.

I am not aware if there exist RL libraries that would provide you with this measure out-of-the-box. However, if you're using Python for example, I believe that using libraries like scipy or scikit-learn along with matplotlib could do the job.
NB: Image credits go to the following presentation: DLRLSS 2019 - Sample Efficient RL - Harm Van Seijen

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Two DQNs in two different time scales
I have the following situation. An agent plays a game and wants to maximize the accumulated reward as usual, but it can choose its adversary. There are $n$ adversaries.
In episode $e$, the agent must first select an adversary. Then for each step $t$ in the episode $e$, it plays the game against the chosen adversary. Every step $t$, it receives a reward following the chosen action in step $t$ (for the chosen adversary). How to maximize the expected rewards using DQN? It is clear that choosing the "wrong" (the strongest) adversary won't be a good choice for the agent. Thus, to maximize the accumulated rewards, the agent must take two actions at two different timescales.
I started solving it using two DQNs, one to decide the adversary to play against and one to play the game against the chosen adversary. I have two duplicate hyperparameters (batch_size, target_update_freq, etc), one for each DQN. Have you ever seen two DQNs like this? Should I train the DQNs simultaneously?
The results that I am getting is not that good. The accumulated reward is decreasing, the loss isn't always decreasing...

Are episodes in any way connected so that a series of episodes form a "meta-episode" with clear time sequence, consequences for picking a particular opponent in episode 1 as opposed to episode 3, and an accumulation of reward over multiple episodes that matters (e.g. winning a series of games)?

I think they should be connected right? but I don't know how. Right now in my modeling, once I choose an adversary and play against it, I see the reward and move to the next episode by resetting the environment. Should I count the number of times I won against an adversary as a meta-episode or something?

It depends what your goal is. Is there any larger structure in the game other than an episode? Does the choice of opponent have any other effect than influencing the results of an individual episode? For instance, does it affect the list of opponents possible for the next episode? If not, then perhaps there is no connection and the answer will be different. That is why I am checking in comments for clarification

Yes, right now I am thinking of no connection. I thought of playing it with $n$ independent DQNs---one for each adversary---and then select the one with the best reward. But this does not work for large $n$.

From comments, you say there is no "outer" goal for picking an adversary other than  scoring highly in an individual episode.
You could potentially model the initial adversary choice as a partially separate Markov Decision Process (MDP), where choosing the opponent is a single-step episode with return equal to whatever reward the secondary MDP - which played the game - obtains. However, this "outer" MDP is not much of an MDP at all, it is more like a contextual bandit. In addition, the performance of the inner game-playing agent will vary both with the choice of opponent, and over time as it learns to play better against each opponent. This makes the outer MDP non-stationary. It also requires the inner MDP to know what opponent it is facing in order to correctly predict correct choices and/or future rewards.
That last part - the need for any "inner" agent to be aware of the opponent it is playing against - is likely to be necessary whatever structure you chooose. That choice of opponent needs to be part of the state for this inner agent, because it will have an impact on likely future rewards. A characterisation of the opponents also needs to be part of whatever predictive models you could use for the outer agent.
A more natural, and probably more useful, MDP model for your problem is to have a single MDP where the first action $a_0$ is to select the opponent. This matches the language you use to describe the problem, and resolves your issue about trying to run a hierarchy of agents. Hierarchical reinforcement learning is a real thing, and very interesting for solving problems which can be broken down into meaningful sub-goals that an agent could discover autonomously, but it does not appear to apply for your problem.
This leaves you with a practical problem of creating a model that can switch between choosing between two sets of radically different actions. The select an opponent action only occurs at the first state of the game, and the two sets of actions do not overlap at all. However, in terms of the theoretical MDP model this is not an issue at all. It is only a practical issue of how you get to fit your Q function approximator to the two radically different action types. There a a few ways around that. Here are a couple that might work for you:
One shared network
Always predict for all kinds of action choice, so the agent still makes predictions for switching opponents all the way to the end of the game. Then filter the action choices down to only those available at any time step. When $t=0$ only use the predictions for actions for selecting an opponent, for $t \ge 1$ only use predictions relating to moves in the game.
Two separate approximators
Have two function approximators in your agent, use one for predicting reward at $t=0$ that covers different opponent choices, and use the other for the rest of the game. If $n$ is small and there is no generalisation between opponents (i.e. no opponent "stats" that give some kind of clue towards the end results), then for the first approximator, you could even use a Q table.
For update steps you need to know whether any particular action value was modelled in one or other of the Q functions - and this will naturally lead you to bootstrap
$$\hat{q}_{o}(s_0, a_0, \theta_0) \leftarrow r_1 + \gamma \text{max}_{a'}\hat{q}_{p}(s_1, a', \theta_1)$$
where $\hat{q}_{o}$ is your approximate model for action values of selecting opponents (and $a_0$ must be an opponent choice) at the start of the game, and $\hat{q}_{p}$ is the nodel you use for the rest of it (and $a'$ must be a position play in the game). I've misused $\leftarrow$ here to stand in for whatever process used to update the action value towards the new estimate - a tabular method that would be a rolling average with current estimate, in neural networks of course that is gradient descent using backpropagation.

I fully agree with your first three paragraphs, but not sure about the rest. Yeah, that could work, but I feel like completely separating the problems would work better; treat the opponent selection as a MAB, and the playing as an MDP. Inside the MDP, you could augment the state representation to include a one-hot vector to represent the opponent, or even some other kind of representation if meaningful features are available that could allow for generalisation across similar opponents. Treating everything as a single MDP would just exacerbate the credit assignment problem?

@DennisSoemers: "Treating everything as a single MDP would just exacerbate the credit assignment problem?" I don't think so. That issue is a core part of the problem definition and would either appear as "credit assignment" in a single MDP view or high variance and non-stationarity in a two MDP view. In terms of number of samples required, and the values passing between steps for value updates, the two views (single MDP with differing action spaces vs two MDPs) are practically identical.

Say I use MAB to select opponents. Should I do (1) or (2)? Here is (1): in each round of the MAB, I select an opponent, and train the DQN (given the chosen opponent). After training, I get the best reward from the trained DQN, and update MAB probabilities and go to the next round of the MAB. Here is (2):  in each round of the MAB, I select an opponent, and play one episode of the DQN (given the chosen opponent). After the terminal state is reached, I get the reward, and update MAB probabilities and go to the next round.

@zdm: If you are able to do this as an experiemnt, then you can test your idea. If you want some theoretical or practical advice from an expert then I think that you should ask a new question on the site. On a short think through, with (1) you will need to train N DQNs (one for each opponent) in sequence and keep them. It may be more reliable than (2) in the long term, but (2) may find your best initial choice earlier and require a lot less training as it will start to focus on the probably best opponents before needing to fully converge.


text:
How to know if my DQN is optimized?
I made a DQN that controls a traffic light. The observation states are the number of vehicles of each lane in the intersection. I trained it for 500 episodes and saved the model every 50th episode. I plotted the reward curve of the model after the training and found out that around the 460th episode, the reward curve become unstable. Does it mean that the optimized DQN model is the 450th model? If not, how do I know if the my DQN is really optimized?

There is a good chance that your DQN is already optimized but you would have to take a look at its performance to really check and see whether its actions seem up to par.
Reasons it may not be optimized: If you are tracking the reward after every episode, unstable rewards are very common just due to random chance, but if you are averaging the reward over the past 50 episodes or so, then it may also be your learning rate or epsilon.
If your learning rate is too high or low you may either never be able to reach a fully optimized DQN or be stuck in a local minimum. An easy way to solve a problem like this would be to just add a simple learning rate decay, so that the learning rate would start off high as to not get stuck in local minima yet decay to a small enough number where you know that the agent has found the global minimum.
The other problem could be that your epsilon may be too high or low. A high epsilon will never allow the agent to fully optimize while a low epsilon doesn't allow the agent to explore and discover better strategies, so it may be a good idea to mess around with this as well.
The only way to really gauge the agent's performance would be to look at it making some decisions through a video or by analyzing some of its predictions. And, if it seems to be doing well, then it may very well be optimized, but, if the agent is not performing as well as it should, then it may be a good idea to try out some of the strategies above.


text:
How to handle changing goals in a DQN?
I created a virtual 2D environment where an agent aims to find a correct pose corresponding to a target image. I implemented a DQN to solve this task. When the goal is fixed, e.g. the aim is to find the pose for position (1,1), the agent is successful. I would now like to train an agent to find the correct pose while the goal pose changes after every episode. My research pointed me to the term "Multi-Objective Deep Reinforcement Learning". As far as I understood, the aim here is to train one or multiple agents to achieve a policy approximation that fits all goals.
Am I on the right track or how should I deal with different goal states?

The simplest thing you can do is to add data regarding the target pose to the state vector. This will allow any generalisations that the agent learns that apply to similar poses to be used directly.
Clearly in normal use, where the target pose remains fixed during the episode, then that part of the state information will not change either during the episode. You will also need to train with a large variety of target poses - so training will take longer.
Multi-Objective Deep Reinforcement Learning is slightly different in that it attempts to resolve prioritising between multiple sub-goals. It would also be a more complex solution, whilst augmenting the state vector should allow you to continue using a very similar DQN set up as you have already.

Thanks for your explanation. So the state vector would maybe look like (...,current_pose, target_pose), where target_pose changes after every episode? I already included those information in my current implementation. I think my mistake was that I changed the target_pose after the agent found the previous one during my first tries. Very appreciated your help.

Yes. If you can, then you should randomise target_episode according to expected distribution in production for each episode whilst training. If the initial pose can vary then you should randomise that too - again the ideal is to match distribution it would work with in production.


text:
Why does the BERT NSP head linear layer have two outputs?
Here's the code in question.
https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491
class BertOnlyNSPHead(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.seq_relationship = nn.Linear(config.hidden_size, 2)

    def forward(self, pooled_output):
        seq_relationship_score = self.seq_relationship(pooled_output)
        return seq_relationship_score

I think it was just ranking how likely one sentence would follow another? Wouldn't it be one score?

This seems to be inherited from the original Google implementation, which also uses 2 outputs (https://github.com/google-research/bert/blob/master/run_pretraining.py#L293). I can see two possible reasons that the original implementation uses 2 outputs:

They are using the cross entropy loss, which typically works with log probabilities. To get probabilities they use softmax activation, which requires an output for each class. It is possible, of course, to compute cross entropy from sigmoid activations (which would correspond to a 1-output architecture), but there seems to be some confusion as to whether the output of the sigmoid function should be used as a probability.

Using 2 outputs can simplify the computation of the binary cross entropy loss, which, in typical Google fashion, is computed using low-level tensorflow ops rather than with tf.nn.softmax_cross_entropy_with_logits. Specifically,

-tf.reduce_sum(one_hot_labels * log_probs, axis=-1)

where one_hot_labels and log_probs are $\mathbb{R}^{N \times 2}$, is much easier to read than
-tf.reduce_sum(binary_labels * tf.math.log(probs) + (1 - binary_labels) + tf.math.log(1 - probs))

where binary_labels and probs are $\mathbb{R}^N$.


text:
Is CNN capable of extracting the descriptive statistics features
I was trying to build a CNN model. I used time series data of daily temperature to predict if there is risk of an event, say bacteria growth. I calculated the descriptive statistics of the time series, ie. mean, variance, skewness, kurtosis etc for each observation and added them to input data.
My question:
Is CNN capable of extracting the effect of the descriptive statistics the label, meaning that adding these descriptive statistics features manually does not make a difference?
(I will still try this later, but like to hear what you think about it). Thanks

CNNs learn convolutional filters that get trained on finding local, recurring patterns in some kind of image/volume data. 1D convolution is actually a thing, but I think what would be more suitable for your case is using Recurrent Neural Nets. They are specifically designed for working on time series-es of heterogeneous data.
Update:
I would like to reconsider the answer I gave earlier.
First of all, in case of dealing with time series data where it is uncertain over which time span a given event to be detected lasts, I'd generally consider using Recurrent Neural Networks (RNNs) rather than CNNs, since RNNs maintain so-called hidden states that can carry potentially useful information over time, i.e. past inputs may influence present outputs. So, either if you know that the event to be detected spans many time steps (which might not even be consecutive time steps) or if you are uncertain about how long exactly the time span of an event to be detected is, then I'd suggest going for RNNs. If you decide to use RNNs, adding extra statistics to your data might make the learning task easier for the network, but is not strictly required since you would expect an RNN to learn relevant time-series statistics on its own.
However, there is a case where using CNNs might actually suffice and yield good results as well. Consider the case of the figure shown below, where the white tiles are time series inputs and the blueish area is the area that a CNN filter/kernel (or multiple ones) is working on right now.

As you can see, there is a time series of data points and some statistics derived from that time series that have been appended to the input data stream. So, in the figure, the augmented time series input consists of the measurements themselves as well as mean, variance, and skewness values derived from the time series data (over consecutive time steps). Under the assumption that the type of event you are interested in is bounded, in terms of time steps, to the width of the CNN filter(s)/kernel(s) that you slide over your input data, then a simple CNN might suffice to detect the events you are interested in. In other words, if the width of your filter spans over at least as many time steps as an event of interest lasts, then your CNN will be able to learn to detect the event (given proper training). In that case, adding further statistics to your input data might indeed help or even be necessary to detect certain events, since it implicitly widens the time window that your CNN filter can observe at any given time step, since a mean (or other statistics) may also considers past data that the filter itself (or multiple filters...) might not span over at a given time step where you want to detect an event.

But couldn't this require a potentially very long kernel? It may actually require a variable-length kernel, given that the relevant pasts even might have happened at different time steps in the past, from time step to time step.

Yes, that's definitely an issue with this approach and the reason why I advertised the RNN idea instead in the first place. It's also why I put the condition that the kernel must be wide enough in bold letters. But in theory, applying CNNs in this way is possible. That's why I altered my answer. On the other hand, I'd personally probably also not use CNNs in this way, but resort to RNNs or more traditional approaches, instead.

It may be worthing linking to some research work that used CNNs for problems that involve time-series.

Previously, I had only heard of people sometimes applying CNNs to time series tasks. But judging from the abstract of this paper, the body of research on applying CNNs to time series tasks is not insignificant. Actually, searching on Google Scholar for "cnn for time series" yields quite some findings.


text:
Should illegal moves be excluded from loss calculation in DQN algorithm?
I'm implementing DQN algorithm to train my agent to play a turn-based game. The action space for the game is small, but not all moves are available at all the states. Therefore, when deciding on which action to pick, agent sets Q-values to 0 for all the illegal moves while normalizing the values of the rest.
During training, when the agent is calculating the loss between policy and target networks, should the illegal actions be ignored (set to 0) so that they don't affect the calculations?

I've implemented this exact scenario before; your approach would most likely be successful, but I think it could be simplified.

Therefore, when deciding on which action to pick, agent sets Q-values to 0 for all the illegal moves while normalizing the values of the rest.

In DQN, the Q-values are used to find the best action. To determine the best action in a given state, it suffices to look at the Q-values of all valid actions and then take the valid action with highest Q-value. Setting Q-values of invalid actions to 0 is unnecessary once you have a list of valid actions. Note that you would need that set of valid actions to set invalid Q-values to 0 in the first place, so the approach I'm suggesting is more concise without worsening the performance.
Since the relative order of the Q-values is all that is required to find the best action, there is no need for normalization. Also, the original DQN paper uses $\epsilon$-greedy exploration. Keep in mind to only sample from valid actions in a given state when exploring this way.

During training, when the agent is calculating the loss between policy and target networks, should the illegal actions be ignored (set to 0) so that they don't affect the calculations?

As noted in one of your previous questions, we train on tuples of experiences $(s, a, r, s')$. The definition of the Q-learning update is as follows (taken from line 6.8 of Sutton and Barto):
$$Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \left[R_{t+1} + \gamma\max\limits_aQ(S_{t+1}, a) - Q(S_t, A_t)\right].$$
The update requires taking a maximum over all valid actions in $s'$. Again, setting invalid Q-values to 0 is unnecessary extra work once you know the set of valid actions. Ignoring invalid actions is equivalent to leaving those actions out of the set of valid actions.


text:
q learning appears to converge but does not always win against random tic tac toe player
q learning is defined as:

Here is my implementation of q learning of the tic tac toe problem:
import timeit
from operator import attrgetter
import time
import matplotlib.pyplot
import pylab
from collections import Counter
import logging.handlers
import sys
import configparser
import logging.handlers
import unittest
import json, hmac, hashlib, time, requests, base64
from requests.auth import AuthBase
from pandas.io.json import json_normalize
from multiprocessing.dummy import Pool as ThreadPool
import threading
import time
from statistics import mean
import statistics as st
import os
from collections import Counter
import matplotlib.pyplot as plt
from sklearn import preprocessing
from datetime import datetime
import datetime
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
import matplotlib
import numpy as np
import pandas as pd
from functools import reduce
from ast import literal_eval
import unittest
import math
from datetime import date, timedelta
import random

today = datetime.today()
model_execution_start_time = str(today.year)+"-"+str(today.month)+"-"+str(today.day)+" "+str(today.hour)+":"+str(today.minute)+":"+str(today.second)

epsilon = .1
discount = .1
step_size = .1
number_episodes = 30000

def epsilon_greedy(epsilon, state, q_table) :

    def get_valid_index(state):
        i = 0
        valid_index = []
        for a in state :
            if a == '-' :
                valid_index.append(i)
            i = i + 1
        return valid_index

    def get_arg_max_sub(values , indices) :
        return max(list(zip(np.array(values)[indices],indices)),key=lambda item:item[0])[1]

    if np.random.rand() < epsilon:
        return random.choice(get_valid_index(state))
    else :
        if state not in q_table :
            q_table[state] = np.array([0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0])
        q_row = q_table[state]
        return get_arg_max_sub(q_row , get_valid_index(state))

def make_move(current_player, current_state , action):
    if current_player == 'X':
        return current_state[:action] + 'X' + current_state[action+1:]
    else :
        return current_state[:action] + 'O' + current_state[action+1:]

q_table = {}
max_steps = 9

def get_other_player(p):
    if p == 'X':
        return 'O'
    else :
        return 'X'

def win_by_diagonal(mark , board):
    return (board[0] == mark and board[4] == mark and board[8] == mark) or (board[2] == mark and board[4] == mark and board[6] == mark)

def win_by_vertical(mark , board):
    return (board[0] == mark and board[3] == mark and board[6] == mark) or (board[1] == mark and board[4] == mark and board[7] == mark) or (board[2] == mark and board[5] == mark and board[8]== mark)

def win_by_horizontal(mark , board):
    return (board[0] == mark and board[1] == mark and board[2] == mark) or (board[3] == mark and board[4] == mark and board[5] == mark) or (board[6] == mark and board[7] == mark and board[8] == mark)

def win(mark , board):
    return win_by_diagonal(mark, board) or win_by_vertical(mark, board) or win_by_horizontal(mark, board)

def draw(board):
    return win('X' , list(board)) == False and win('O' , list(board)) == False and (list(board).count('-') == 0)

s = []
rewards = []
def get_reward(state):
    reward = 0
    if win('X' ,list(state)):
        reward = 1
        rewards.append(reward)
    elif draw(state) :
        reward = -1
        rewards.append(reward)
    else :
        reward = 0
        rewards.append(reward)

    return reward

def get_done(state):
    return win('X' ,list(state)) or win('O' , list(state)) or draw(list(state)) or (state.count('-') == 0)

reward_per_episode = []

reward = []
def q_learning():
    for episode in range(0 , number_episodes) :
        t = 0
        state = '---------'

        player = 'X'
        random_player = 'O'

        if episode % 1000 == 0:
            print('in episode:',episode)

        done = False
        episode_reward = 0

        while t < max_steps:

            t = t + 1

            action = epsilon_greedy(epsilon , state , q_table)

            done = get_done(state)

            if done == True :
                break

            if state not in q_table :
                q_table[state] = np.array([0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0])

            next_state = make_move(player , state , action)
            reward = get_reward(next_state)
            episode_reward = episode_reward + reward

            done = get_done(next_state)

            if done == True :
                q_table[state][action] = q_table[state][action] + (step_size * (reward - q_table[state][action]))
                break

            next_action = epsilon_greedy(epsilon , next_state , q_table)
            if next_state not in q_table :
                q_table[next_state] = np.array([0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0])

            q_table[state][action] = q_table[state][action] + (step_size * (reward + (discount * np.max(q_table[next_state]) - q_table[state][action])))

            state = next_state

            player = get_other_player(player)

        reward_per_episode.append(episode_reward)

q_learning()

The alogrithm player is assigned to 'X' while the other player is 'O':
    player = 'X'
    random_player = 'O'

The reward per episode:
plt.grid()
plt.plot([sum(i) for i in np.array_split(reward_per_episode, 15)])

renders:

Playing the model against an opponent making random moves:
## Computer opponent that makes random moves against trained RL computer opponent
# Random takes move for player marking O position
# RL agent takes move for player marking X position

def draw(board):
    return win('X' , list(board)) == False and win('O' , list(board)) == False and (list(board).count('-') == 0)

x_win = []
o_win = []
draw_games = []
number_games = 50000

c = []
o = []

for ii in range (0 , number_games):

    if ii % 10000 == 0 and ii > 0:
        print('In game ',ii)
        print('The number of X game wins' , sum(x_win))
        print('The number of O game wins' , sum(o_win))
        print('The number of drawn games' , sum(draw_games))

    available_moves = [0,1,2,3,4,5,6,7,8]
    current_game_state = '---------'

    computer = ''
    random_player = ''

    computer = 'X'
    random_player = 'O'

    def draw(board):
        return win('X' , list(board)) == False and win('O' , list(board)) == False and (list(board).count('-') == 0)

    number_moves = 0

    for i in range(0 , 5):

        randomer_move = random.choice(available_moves)
        number_moves = number_moves + 1
        current_game_state = current_game_state[:randomer_move] + random_player + current_game_state[randomer_move+1:]
        available_moves.remove(randomer_move)

        if number_moves == 9 :
            draw_games.append(1)
            break
        if win('O' , list(current_game_state)) == True:
            o_win.append(1)
            break
        elif win('X' , list(current_game_state)) == True:
            x_win.append(1)
            break
        elif draw(current_game_state) == True:
            draw_games.append(1)
            break

        computer_move_pos = epsilon_greedy(-1, current_game_state, q_table)
        number_moves = number_moves + 1
        current_game_state = current_game_state[:computer_move_pos] + computer + current_game_state[computer_move_pos+1:]
        available_moves.remove(computer_move_pos)

        if number_moves == 9 :
            draw_games.append(1)
#             print(current_game_state)
            break

        if win('O' , list(current_game_state)) == True:
            o_win.append(1)
            break
        elif win('X' , list(current_game_state)) == True:
            x_win.append(1)
            break
        elif draw(current_game_state) == True:
            draw_games.append(1)
            break

outputs:
In game  10000
The number of X game wins 4429
The number of O game wins 3006
The number of drawn games 2565
In game  20000
The number of X game wins 8862
The number of O game wins 5974
The number of drawn games 5164
In game  30000
The number of X game wins 13268
The number of O game wins 8984
The number of drawn games 7748
In game  40000
The number of X game wins 17681
The number of O game wins 12000
The number of drawn games 10319

The reward per episode graph suggests the algorithm has converged? If the model has converged shouldnt the number of O game wins be zero ?

The primary issue I see is that in the loop through time steps t in every training episode, you select actions for both players (who should have opposing goals to each other), but update a single q_table (which can only ever be correct for the "perspective" of one of your two players) on both of those actions, and updating both of them using a single, shared reward function.
Intuitively, I guess this means that your learning algorithm assumes that your opponent will always be helping you win, rather than assuming that your opponent plays optimally towards its own goals. You can see that this is likely indeed the case from your plot; you use $30,000$ training episodes, split up into $15$ chunks of $2,000$ episodes per chunk for your plot. In your plot, you also very quickly reach a score of about $1,950$ per chunk, which is almost the maximum possible! Now, I'm not 100% sure what the winrate of an optimal player against random would be, but I think it's likely that that should be lower than 1950 out of 2000. Random players will occasionally achieve draws in Tic-Tac-Toe, especially taking into consideration that your learning agent itself is also not playing optimally (but $\epsilon$-greedily)!

You should instead pick one of the following solutions (maybe there are more solutions, this is just what I come up with on the spot):

Keep track of two different tables of $Q$-values for the two different players, and update each of them only on half of the actions (each of them pretending that actions selected by the opponent are just stochastic state transitions created by "the environment" or "the world"). See this answer for more on what these scheme would look like.
Only keep track of a $Q$-value for your own agent (again only updating it on half the actions as described above -- specifically only on the actions your agent actually selected). Actions by the opposing player should then NOT be selected based on those same $Q$-values, but instead by some different approach. You could for instance have opposing actions selected by a minimax or alpha-beta pruning search algorithm. Maybe selecting them to minimise instead of maximise values from the same $Q$-table could also work (didn't think this idea fully through, not 100% sure). You probably could also just pick opponent actions randomly, but then your agent will only learn to play well against random opponents, not necessarily against strong opponents.

After looking into the above suggestions, you'll probably also want to look into making sure that your agent experiences games in which it starts as Player 1, as well as games in which it starts as Player 2, and trains for both of those possible scenarios and learns how to handle both of them. In your evaluation code (after training), I believe that you always make the Random opponent play first, and the trained agent play second? If you don't cover this scenario in your training episodes, your agent may not learn how to properly handle it.

Finally, a couple of small notes:

Your discount factor $\gamma 0.1$ has an extremely small value. Common values in literature are values like $\gamma = 0.9$, $\gamma = 0.95$, or even $\gamma = 0.99$. Tic-Tac-Toe episodes tend to always be very short anyway, and we tend to not care too much about winning quickly rather than winning slowly (a win's a win), so I would tend to use a high value like $\gamma = 0.99$.
A small programming tip, not really AI-specific: your code contains various conditions of the form if <condition> == True :, like: if done == True :. The == True part is redundant, and these conditions can be written more simply as just if done:.

I have an answer to your question: what is the optimal score against a random player.  See here:  https://ai.stackexchange.com/questions/31536/what-is-the-optimal-score-for-tic-tac-toe-for-a-reinforcement-learning-agent-aga/31537#31537


text:
Is it possible to count the number of squats with Computer Vision techniques?
I am planning to build an app which will count the number of sqauts from videos. Assuming that the user and camera do not move, are there ways I can count the number of squats? Do such models to understand human activity and pose exist?

You can do that pretty easily using Posenet or Openpose. Train the keypoints for Squats and then count it. :)

Yes, this is a rather straightforward problem that should be able to be solved using openCV or ANNs.
Check out this paper to get you started: https://arxiv.org/pdf/1701.08936.pdf

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Text classification of non-equal length texts, should I pad left or right?
Text classification of equal length texts works without padding, but in reality, practically, texts never have the same length.
For example, spam filtering on blog article:
thanks for sharing    [3 tokens] --> 0 (Not spam)
this article is great [4 tokens] --> 0 (Not spam)
here's <URL>          [2 tokens] --> 1 (Spam)

Should I pad the texts on the right:
thanks for     sharing --
this   article is      great
here's URL     --      --

Or, pad on the left:
--   thanks  for    sharing
this article is     great
--   --      here's URL

What are the pros and cons of either pad left or right?

For any model that does not take a time series approach like an RNN does, the padding shouldn't make a difference.
I prefer padding right simply because there also might be text you need to cut-off. Then padding is more intuitive as you either cut-off a text if it's too long or pad a text when it's too short.
Either way, when a model is trained a certain way, it shouldn't make a difference so long the testing is also padded the way it was presented in training.

the problem with padding right is the last value for time of RNN unit is mostly always zero, i tackle it by return sequence from the recurrent layer, so it's not affected by the last zero pad. supposed to be correct?


text:
Why are decision trees and random forests scale invariant?
Feature scaling, in general, is an important stage in the data preprocessing pipeline.
Decision Tree and Random Forest algorithms, though, are scale-invariant - i.e. they work fine without feature scaling. Why is that?

what problem are you trying to solve?  this is a very generalized question.  Are you trying to understand the limitations of the Random Forest Classifier and Regressor

Scaling only makes sense when there is something that reacts to that scale. Decision Trees though, just make a cut at a certain number.
Imagine: For a feature that goes from 0 to 100 a cut at 50 may be improving performance. Scaling this down to 0 to 1 making the cut a 0.5 doesn't change a thing.
Now on the other hand NN have some kind of activation function (leaving RELu aside) that react differently to input that is above 1. Here Normalization, putting every feature between 0 and 1 makes sense.

Feature scaling happens to be a problem when a model is characterized by having a distance metric (or another kind of numerical evaluation for that matter). Therefore models such as support vector machines, neural networks, distance based clustering methods (e.g. k means) and linear/logistic regression are prone to changes by feature scaling.
Those which are based on probability rather than distances are not scale variant. These include Naive Bayes Classifiers, or decision trees.


text:
What's the best practice for Boltzmann Exploration temperature in RL?
I'm currently modeling DQN in Reinforcement Learning. My question is: what are the best practices related to Boltzmann Exploration? My current thoughts are: (1) Let the temperature decay through training and finally stop at 0.01, when the method will always select the best practice, with almost no randomness. (2) Standardize the predicted Q values before feeding into the softmax function.
Currently, I'm using (2), and the reward is suffering from high variance. I'm wondering whether it has something to do with the exploration method?

Do you have to use Boltzmann exploration, strictly? There is a modification for Boltzmann exploration called Mellow-max. It, basically, provides an adaptive temperature for Boltzmann exploration.
Here is the link for the paper for tuning mellow-max with deep reinforcement learning (DQN is often mentioned): http://cs.brown.edu/people/gdk/pubs/tuning_mellowmax_drlw.pdf
Here is the link for mellow-max implemented with SARSA (I recommend reading this first, to get an understanding of mellow-max): https://arxiv.org/pdf/1612.05628.pdf


text:
How to "forward" updated NN model to a transferred model?
I've trained a robot to walk in a straight line for as long as it can (using TD3), and now I'm using that pre-trained model for two new models with separate purposes: 1. Walk to a specific point and halt (adding target position to the NN inputs); 2. Walk straight at a specified velocity (adding a target velocity to NN inputs).
Now let's say I retrain the original model again to walk properly after changing, say, the mass of the robot. How can I approach "forwarding" this update to the two transfer-learned models? The purpose of this is to minimize re-training time for all future models transfer-learned from the original.
(What strikes me as particularly challenging is the fact that the input layer of the transfer-learned models have additional features, so this may re-wire the majority of the NN, making a "forwarded update" completely incompatible...)

I think there is no simple way to transfer knowledge changes between different models.
If you take your initial model and create a new version of it which you use to learn some other task (like "Walk to a specific location"), then the values copied from the first (original) model change in the second model.
From that moment on, training the former model on another task will have different effects on its weights than continuing the training of the second model, whose parameter have been changed already.
Consider, for example, that you had changed the mass of the robot and trained the initial model on that new task already. Then, if you took all the re-trained parameters from the first model and implanted them into the second model (trained on walking to a certain location), then you would essentially overwrite the additional knowledge the second model had gained already during its initial transfer-learning-process (not even taking into consideration any additional parameters appended to the list of parameters in the second model).
So, you will have to re-train all three models (the original one and the two transfer-learning models) if you change the mass of the robot.
Edit:
There might be an option to apply the same knowledge changes to another model architecture if you refrain from pure transfer learning. This can be achieved with a more modularized model architecture.
Consider that you train your first model on walking straight head. Let's call this model $m_{walk}$.
Then you intend to recycle $m_{walk}$ for another task, like walking straight ahead to a given location. Such a model architecture could be realized in two ways:

You apply real transfer learning, retraining a copy of $m_{walk}$ to walk straight ahead until it reaches a certain location
You take $m_{walk}$, don't change it, but add a second model (let's call it $m_{navigator}$), which is trained on predicting $go$ vs. $stop$.

In the second case, your overall model architecture (let's call it $m_{go\_to}$) consists of both one model used for walking (i.e. $m_{walk}$) and one model architecture which is used for predicting $go$ vs. $stop$ (i.e. $m_{navigator}$).
The idea then is that the robot executes the actions suggested by $m_{walk}$ until $m_{navigator}$ suggests stopping, upon which prediction the suggestions by $m_{walk}$ will be ignored.
Then, whenever you retrain your model $m_{walk}$ (e.g. because the mass of some robot changes), you can simply apply the changes to $m_{go\_to}$ by replacing $m_{walk}$ by a new version, leaving the rest of $m_{go\_to}$ intact.
If you generalize $m_{walk}$ to not only walk straight ahead, but also to take turns etc. and you generalize $m_{navigator}$ to predict going $left$, $right$, $straight\ ahead$, $back$, or $stop$ (being predicted when a certain destination has been reached), you can generalize $m_{go\_to}$ to walk where-ever you want it to go.


text:
Which paper introduced the term "softmax"?
Nowadays, the softmax function is widely used in deep learning and, specifically, classification with neural networks. However, the origins of this term and function are almost never mentioned anywhere. So, which paper introduced this term?

The paper that appears to have introduced the term "softmax" is Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters (1989, NIPS) by John S. Bridle.
As a side note, the softmax function (with base $b = e^{-\beta}$)
$$\sigma (\mathbf {z} )_{i}={\frac {e^{-\beta z_{i}}}{\sum _{j=1}^{K}e^{-\beta z_{j}}}}{\text{ for }}i=1,\dotsc ,K {\text{ and }}\mathbf {z} =(z_{1},\dotsc ,z_{K})\in \mathbb {R} ^{K}$$
is very similar to the Boltzmann (or Gibbs) distribution
$$
p_i=\frac{e^{- {\varepsilon}_i / k T}}{\sum_{j=1}^{M}{e^{- {\varepsilon}_j / k T}}}
$$
which was formulated by Ludwig Boltzmann in 1868, so the idea and formulation of the softmax function is quite old.


text:
Is the size of a neural network directly linked with an increase in its inteligence?
Just came across this article on GPT-3, and that lead me to the question:
In order to make a certain kind of neural network architecture smarter all one needs to do is to make it bigger?
Also, if that is true, how does the importance of computer power relates with the importance of fine-tuning/algorithmic improvement?

Maybe you should summarise what the article says (for those not interested in reading it) that made you think that a bigger architecture leads to more "intelligence". The thing that increases if the number of parameters of the model increases is the capacity (i.e. roughly, the number of functions that the neural network can represent). So, what's the definition of intelligence you are using here?

First of all, there is no real 'intelligence' innate to artificial Neural Networks (NNs).
All they do is trying to approximate a mathematical function with a certain degree of generalization (hopefully without learning a given dataset by heart, i.e. hopefully without overfitting).
The more nodes (or neurons) you include into the network, the more complex a function can be that a network can learn to approximate. It's similar to high-school math: The higher the degree of some polynomial, the better the polynomial can be adjusted to fit some observation to be modeled; with the only difference being that NNs commonly include non-linearities and are trained via some kind of stochastic gradient descent.
So, yes. The more nodes a model possesses, the higher the so-called model capacity, i.e. the higher the degree of freedom a NN-model has to fit some function. After all, NN are said to be universal function approximators - given they have enough internal nodes in their hidden layer(s) to fit some given function.
In practice, however, you don't want to blow up a model architecture unnecessarily, since this commonly results in overfitting if it doesn't cause some instabilities of the training procedure instead.
Generally, the larger the model to be trained, the higher the computational cost to train the network.
A common suggestion is to reduce the number of nodes in a network at the expense of increasing a network's depth, i.e. the number of hidden layers. Often, that can help reduce the demand for excessively many nodes.


text:
Do correlations matter when building neural networks?
I am new to working with neural networks. However, I have built some linear regression models in the past. My question is, is it worth looking for features with a correlation to my target variable as I would normally do in a linear regression or is it better to feed the neural network with all the data I have?
Assuming that the data I have is all related to my target variable of course. I am working with this dataset and building a neural network regressor for it.
https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DL0101EN/labs/data/concrete_data.csv
Here is a snippet of the data. The target variable is the concrete strength rate given a certain combination of materials for that concrete sample.

I greatly appreciate any tips and explanations. I excuse me if this is too noob of a question but unfortunately I did not find any info about it on google. Thanks again!

If there is a correlation between some of the data, that means a couple of things. 1, you might be better of just doing linear regression or some simpler fitting as a NN might be overkill and 2, a neural network should learn very easily off this data. But it ultimately comes down to testing; if using all the data increases accuracy go with that, if not than just use a subset as it should make the model faster.

If there is some correlation between features, that is what the network will ideally find out on its own and learn to utilize. So, in general, don't take correlated samples or features out of the training loop only because they look correlated. After all, they could convey a lot of valuable information.
When it comes to correlation between data samples during training, this correlation is commonly broken up by training a network on randomly selected mini-batches of training data samples. So, you randomly sample e.g. 16 or 32 (or so) training examples based on which you apply a single update of the weights using some Stochastic Gradient Descent variant. Since the members of a mini-batch are sampled at random, chances for finding highly correlated training samples in some mini-batch shall be sufficiently minimized in order not to negatively affect the training outcome.
Having said that, if you are concerned about overfitting of your model or weights that would overly weight just a small subset of all available input features, you could try applying regularization techniques like L1 (encouraging sparse representations) or L2 (encouraging low weights in general) regularization or dropout.
In your particular case, since the main concern is an excessive contribution of only a small set of input features, L2 shall yield better results (avoiding excessively large weights that would be required to excessively much weight just a small number of features).
Besides that: Commonly, you split your training dataset into 3 parts:

Data used for fitting the model (actual training data)
Data used for assessing the training progress & possibly for determining when to apply early stopping (validation data)
Test data used to assess the performance of the system after all training & intermediate testing is done

The final evaluation on the test dataset shall reveal then the generalization ability of your trained model to novel data.
So, with regularization in place during training and relatively low error rates on the validation and test datasets, you are pretty much save even without checking for correlated data beforehand. Only when you really struggle decreasing the validation loss, it might be worth to further inspect what exactly is going wrong in terms of correlations and such.

Hey Daniel, thanks for the great explanation. Greatly appreciated!


text:
Do the order of the features ie channel matter for a 1d convolutional network?
Do the test dataset feature order and inference (real world) feature order have to be the same as the training dataset? For example, if features are in the order (a,c,b,e,d) for the training dataset, does that particular order have to match for the inference and test dataset?

Generally, order matters. A (trained) Neural Network (NN) is just a mathematical function trained on taking some given input and producing the corresponding output. So, if you train a certain node on producing large output if (and only if) an animal is present in a picture (for example), but later you give it the numeric evidence for a car being present in the image, it will still produce large output, indicating an animal being present in the image. This is simply because a node doesn't know what it is receiving or supposed to detect. It just follows its standard mathematical procedure.
So, if you train your network on one kind of input data, you must provide the same kind of input data during testing (or at least you cannot simply exchange inputs or outputs of certain nodes without manually correcting for that in the remainder of the network).
In the most simple case, consider whether you could simply swap the inputs to that simple function: $f(x,y) = x^2 + y$. If you swap the inputs, the output will be different. The exactly same applies to NNs.
Addition:
I think this post explains and especially illustrates the intuition nicely.


text:
Detect data in tables of roughly the same structure
I would like to train a model that serializes a table of nutrition facts into it's values.
The tables can vary in form and colour, but always contain the same set of keys (e.g. carbs, fats).
Examples for these tables can be found here.
The end goal is to be able to take a picture of such a table and have it's values added to a database.
My initial idea was to train a model on finding subpictures of the individual key/value pairs and then using OCR to find out which value it actually is.
As I am relatively new to ML, I would love to have some ideas about how one could try to build this, so I can do further research on it.
Thanks

Assuming all of the tables will be oriented in similar ways (label and value running horizontally) and that all writing will be printed rather than handwritten, one solution method would be to use an image segmentation method such as edge detection to segregate these horizontal (label, value) pairs and then use a library like Tesseract for OCR.
There are many types of image segmentation methods that may all have value, but if my assumption holds regarding the neat, structured nature of the tables, then I think simple edge detection methods could be sufficient.

Traditional edge detection may not be enough because the tables could have no lines between cells. The last link gave me a good idea about image segregation though. I might try to train a model to detect the various segments and use Tesseract to read the value.

Ah, my mistake @kapuetze - I didn't see the examples of tables with no horizontal lines. Still, glad you found something useful.


text:
Number of LSTM layers needed to learn a certain number of sequences
Theoretically, number of units for a LSTM layer is the number of hidden states or the max length of sequences as per my practice.
For example, in Keras:
Lstm1 = LSTM(units=MAX_SEQ_LEN, return_sequences=False);

However, with lots of sequences to train, should I add more LSTM layers? because increasing MAX_SEQ_LEN is not the way as it doesn't help make the network better since the extra number of hidden states isn't useful any more.
I'm considering increasing number of LSTM layers, but how many are enough?
For example, 3 of them:
Lstm1 = LSTM(units=MAX_SEQ_LEN, return_sequences=True);
Lstm2 = LSTM(units=MAX_SEQ_LEN, return_sequences=True);
Lstm3 = LSTM(units=MAX_SEQ_LEN, return_sequences=False);

From my personal experience, the units hyperparam in LSTM is not necessary to be the same as max sequence length. Add more units to have the loss curve dive faster.
And about the number of LSTM layers, trying out a single LSTM layer is a good start point, the model trains better with more LSTM layers.
For example, MAX_SEQ_LEN=10, in Keras:
Lstm1 = LSTM(units=30, return_sequences=True); #Time sequence, to feed to next layer
Lstm2 = LSTM(units=20, return_sequences=True); #Time sequence, to feed to next layer
Lstm3 = LSTM(units=10, return_sequences=False);

Output = Lstm3(Lstm2(Lstm1));
```


text:
In what situations ELUs should be used instead of RELUs?
I always use RELUs actication functions when I need to and I understand limitations of ELUs. So in what situation do I need to consider ELUs over RELUs?

ELU does not suffer from dying neurons issue, unlike ReLU. While ELU can help you to achieve better accuracy, it is slower than ReLU because of its non-linearity in its negative range.
Choosing the right activation function totally depends on the situation but you need to also consider other similar types of activation functions such as leaky ReLU.
Check this link out. It could be useful.

The answer above makes some great comparisons/trade-offs. To help address the non-linearity issue with eLU units that the previous answer brings up, you can also use Leaky-ReLU units, which are linear in both the positive and negative range, and piecewise linear across the whole real domain.
Please see the link here for more details.

Note that the answer you mention already mentions leaky ReLU.


text:
Why L2 loss is more commonly used in Neural Networks than other loss functions?
Why L2 loss is more commonly used in Neural Networks than other loss functions?
What is the reason to L2 being a default choice in Neural Networks?

By "L2 loss" do you mean L2 penalty/regularisation?

I'll cover both L2 regularized loss, as well as Mean-Squared Error (MSE):
MSE:

L2 loss is continuously-differentiable across any domain, unlike L1 loss.  This makes training more stable and allows for gradient-based optimization, as opposed to combinatorial optimization.
Using L2 loss (without any regularization) corresponds to the Ordinary Least Squares Estimator, which, if you're able to invoke Gauss-Markov assumptions, can lead to some beneficial theoretical guarantees about your estimator/model (e.g. that it is the "Best Linear Unbiased Estimator").  Source: https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem.

L2 Regularization:

Using L2 regularization is equivalent to invoking a Gaussian prior (see https://stats.stackexchange.com/questions/163388/why-is-the-l2-regularization-equivalent-to-gaussian-prior) on your model/estimator.  If modeling your problem as a Maximum A Posteriori Inference (MAP) problem, if your likelihood model (p(y|x)) is Gaussian, then your posterior distribution over parameters (p(x|y)) will also be Gaussian.  From Wikipedia: "If the likelihood function is Gaussian, choosing a Gaussian prior over the mean will ensure that the posterior distribution is also Gaussian" (source: https://en.wikipedia.org/wiki/Conjugate_prior).

As in the case above, L2 loss is continuously-differentiable across any domain, unlike L1 loss.

I think it's important to note that L2 loss definitely isn't used more commonly than other loss functions. In fact if anything I would say cross-entropy has the largest appearance as it ties in with softmax activations which is all around excellent for classification.

@Recessive I agree

I think this answer doesn't clarify an important point. An "l2 loss" would be any loss that uses the "l2 norm" as a regularisation term (and, in that case, you will get MAP). This loss can be the MSE  or it can e.g. the cross-entropy, i.e. l2 norm can be used for regression or classification.

@nbro MSE loss specifically is sometimes referred to as L2 loss, so that one was probably meant in the question (e.g., see https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#mse-l2)

@DennisSoemers Right. I suspected that L2 loss couldn't just refer to any loss that uses L2 norm, but, well, I guess it's a little bit misleading to call MSE L2 loss when it doesn't use L2 regularisation, but I guess the 2 stands for the square in that case. Anyway, why not just calling it MSE? I don't get it.


text:
How are mujoco environments used for meta-rl?
Afaik, investigating meta reinforcement learning algorithms requires a collection of two or more environments which have similar structure but are still different enough.
When I read this paper it was unclear to me what the meta-training and meta-testing environments were.
For eg., a graph is given for Ant-Fwd-Bkwd showing its performance with number of gradient steps. I'm guessing these are the meta-testing performances. So, which environment was it 'meta-trained' on?
Was it meta-trained on the same Ant-Fwd-Bkwd environment?

According to this paper (PEARL):

These locomotion task families require adaptation across reward
functions (walking direction for Half-CheetahFwd-Back, Ant-Fwd-Back,
Humanoid-Direc-2D, target velocity for Half-Cheetah-Vel, and goal
location for Ant-Goal2D) or across dynamics (random system parameters
for Walker-2D-Params).

It looks like different versions of the same environment with differing reward functions are used. For eg., Forward direction might be rewarded positively in one version, Negative in another, Both directions rewarded positively in yet another.


text:
Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?
Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?
I could not find a clear answer on this. I would say yes it is Multi Agent Learning, as there are two Agents playing against each other.

Depends on perspective.
On one hand, you have an agent playing in an environment with another agent also evolving. This falls under the definition of Multi-Agent Learning, as can be seen with works such as

Michael Bowling and Manuela Veloso. Multiagent learning using a variable learning
rate. Artificial Intelligence, 136(2):215 – 250, 2002.

Michael Bowling. Convergence and no-regret in multiagent learning. In Proceedings of
the 17th International Conference on Neural Information Processing Systems, NIPS’04,
pages 209–216, Cambridge, MA, USA, 2004. MIT Press.

M. D. Awheda and H. M. Schwartz. Exponential moving average q-learning algorithm. In
2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning
(ADPRL), pages 31–38, April 2013.

Sherief Abdallah and Victor Lesser. A multiagent reinforcement learning algorithm with
non-linear dynamics. Journal of Artificial Intelligence Research, 33:521–549, 2008.

However, you can also claim that you simply have a single agent learning on a non-stationary environment (the environment contains both the game rules and the opponent), and you simply learn on that basis. From this perspective, there is no multi-agent learning at all.

Thanks for your answer. But could not one argue with the second argument, that all multi agent learning is just single agent learning, as long as the agents do not need to act cooperatively? I mean it's just an agent in an non stationary environment?

Yes, you could argue that any "fully competitive environment" can be considered a non-stationary single. Tbh I don't like this interpretation, it was a discussion I had during my PhD defense, but it is still a PoV


text:
Get Neural Network to predict a tag/class on a certain word using the surrounding words as context [PyTorch]?
I am somewhat a novice at the topic of Neural Netoworks and PyTorch.
I am trying to create a model that takes a word (that I have modified very slightly) and a 'window' of context around it and predicts one of 5 tags (the tags relate to what sort of action I should perform on that word to get its correct form).
For example, here's what I would call a window of size 7 and it's tag (what it means isn't too important, it's just the 'target'):
        Sentence                    Label
here is a sentence for my network     N

sentence is the word that I want the network to predict the label for, but the 3 words on either side provide contextual meaning. My problem is, how would I get a network to know I want it to predict for that central word but not outright ignore the others? I am familiar with more normal NLP tasks such as NMT and character level classification.
I have already gotten my dataset 'padded' out so they're all of equal size.
Any help is appreciated

I don't know how to do this specifically with pytorch, but for a recent project, we used BERT to make embeddings for the sentences, and then used a k-neighbors classifier to classify the words.

@VarunVejalla So will that basically focus on my 'centre' word while taking the others into account? Thanks for the help also!

You're welcome! That's pretty much right. I explained more in my answer.

You may want to take a look at this article, but I'll summarize. You can use BERT (or some other tool) to make embeddings of every word in every sentence. Then for each word, make a contextualized embedding vector using the rest of the sentence. bert-embedding does all of this itself. Then keep the embedding vector for the important words.
For each important word, you would then have two pieces of information: the embedding vector and the correct label (which could easily be made into an integer from $0$ to $4$). Depending on the size of the embedding vectors, you could use PCA to reduce the size, although this may not be needed. Using this data, you can then train a neural network or use a k-nearest neighbors classifier.
There is more information in the article, which I suggest that you read. They do a better job explaining than me and also have some actual code you may want to look at.


text:
How much computing power does it cost to run GPT-3?
I know it cost around $4.3 million dollars to train, but how much computing power does it cost to run the finished program? IBM Watson chatbot AI only costs a few cents per chat message to use, OpeenAI Five seemed to run on a single gaming PC setup. So I'm wondering how much computing power does it need to run the finished ai program.

This is a programming/hardware issue, so it's off-topic here. Please, read https://ai.stackexchange.com/help/on-topic. This question is maybe more appropriate for Data Science SE.

Not a direct answer to the question but this is also interesting in this context: https://www.reddit.com/r/MachineLearning/comments/i49jf8/d_biggest_roadblock_in_making_gpt4_a_20_trillion/

I can't anwser your question on how much computing power you might need, but you'll need atleast a smallgrid to run the biggest model just looking at the memory requirments (175B parameters so 700GB of memory). The biggest gpu has 48 GB of vram  I've read that gtp-3 will come in eigth sizes, 125M to 175B parameters. So depending upon which one you run you'll need more or less computing power and memory. (https://lambdalabs.com/blog/demystifying-gpt-3/)
For an idea of the size of the smallest, "The smallest GPT-3 model is roughly the size of BERT-Base and RoBERTa-Base."

I think it is premature to answer your question as OpenAI has not made GPT-3 available yet other than via a web-based API.  For more information see OpenAI API.
From OpenAI will start selling its text-generation tech, and the first customers include Reddit, by James Vincent:
Access to the GPT-3 API is invitation-only, and pricing is undecided.
You can join the OpenAI wait list here: https://beta.openai.com/
I read somewhere that to load GPT-3 for inferencing requires 300GB if using half-precision floating point (FP16).  There are no GPU cards today that even in a set of four will provide 300GB of video RAM.  For example, the best I believe you can do in a single desktop box is four NVLinked Nvidia RTX 8000 cards on a single motherboard.  Each card has 48GB of VRAM each.  That would only provide a total of 192GB of VRAM.


text:
What is layer freezing in transfer learning?
Transfer learning consists of taking features learned on one problem and leveraging them on a new, similar problem.
In the Transfer Learning, we take layers from a previously trained model and freeze them.
Why is this layer freezing required and what are the effects of layer freezing?

Why is this layer freezing required?
It's not.
What are the effects of layer freezing?
The consequences are:
(1) Should be faster to train (the gradient will have far less components)
(2) Should require less data to train on
If you do unfreeze the weights, I'd think your performance would be better because you are adjusting (i.e., fine-tuning) the parameters to your specific problem at hand. I am not sure what the marginal improvements are in practice, as I have not experiemented much with fine-tuning (like are the improvements typically a 0.01% reduction in error rate? Not sure.)

Layer freezing means that the layer weights of the trained model do not change when reused on a subsequent downstream mission, they remain frozen. Basically, when backpropagation is performed during training, these layer weights aren't compromised.


text:
Finding the optimal policy from a set of fixed policies in reinforcement learning
This is an open-ended question.Suppose I have a reinforcement learning task that is being solved using many different fixed policies, one of which is optimal. The goal of the agent is not to figure out what the optimal policy is but rather which policy (from a set of predefined fixed policies) is the optimal one.
Are there any algorithms/methods that handle this?
I was wondering if meta learning is the right area to look into?

you could check out a paper I just read called deep exploration with bootstrapped DQN. They kind of do what you say -- they have $k$ Q-functions and one is chosen uniformly at random to choose actions for the policy and they store the tuple in the replay buffer with a mask to determine whether the $k$th network should be updated with said tuple.

Thanks. I'll look into this. It seems like an analogy for this would be like solving a multi-armed bandit but rather than single lotteries you're finding the best policy.
Would it be fair to classify this approach as meta-learning?

Sorry, I don't know anything about meta-learning so I can't comment!

The quickest way to do this would be to use policy evaluation methods. Most of the standard optimal control algorithms consist of policy evaluation plus a rule for updating the policy.
It may not be possible to rank arbitrary policies by performance when considering all states. So you will want to rank them according to some fixed distribution of state values. The usual distribution of start states would be a natural choice (this is also the objective when learning via policy gradients in e.g. Actor-Critic).
One simple method would be to run multiple times for each policy, starting each time according to the distribution of start states, and calculate the return (discounted sum of rewards) from each one. A simple Monte Carlo run from each start state would be fine, and is very simple to code. Take the mean value as your estimate, and measure the variance too so you can establish a confidence for your selection.
Then simply select the policy with the best average value in start states. You can use the variance to calculate a standard error for this, so you will have a feel for how robust your selection is.
If have a large number of policies to select between, you could do a first pass through with a relatively low number of samples, and try to rule out policies that perform badly enough that even adding say 3 standard errors to the estimated value would not cause them to be preferred. Other than that, the more samples you can take, the more accurate your estimates of mean starting value for each policy will be, and the more likely you will be to select the right policy.

I was wondering if meta learning is the right area to look into?

In general no, but you might want to consider meta learning if:

You have too many policies to select between by testing them all thoroughly.

The policies have some meaningful low dimension representation that is driving their behaviour. The policy function itself would normally be too high dimensional.

You could then use some form of meta-learning to predict policy performance directly from the representation, and start to skip evaluations from non-promising policies. You may need your fixed policies to number in the thousands or millions before this works though (depending on the number of parameters in the representation and complexity of mapping between parameters and policy function), plus you will still want to thoroughly estimate performance of candidates selected as worth evaluating by the meta-learning.
In comments you suggest treating the list of policies as context-free bandits, using a bandit solver to pick the policy that scores the best on average. This might offer some efficiency over evaluating each policy multiple times in sequence. A good solver will try to find best item in the list using a minimal number of samples, and you could use something like UCB or Gibbs distribution to focus more on the most promising policies. I think the main problem with this will be finding the right hyperparameters for the bandit algorithm. I would suggest if you do that to seed the initial estimates with an exhaustive test of each policy multiple times, so you can get a handle on variance and scale of the mean values.


text:
My Deep Q-Learning Network does not learn for OpenAI gym's cartpole problem
I am implementing OpenAI gym's cartpole problem using Deep Q-Learning (DQN). I followed tutorials (video and otherwise) and learned all about it. I implemented a code for myself and I thought it should work, but the agent is not learning. I will really really really appreciate if someone can pinpoint where I am doing wrong.
Note that I have a target neuaral network and a policy network already there. The code is as below.
import numpy as np
import gym
import random
from keras.optimizers import Adam
from keras.models import Sequential
from keras.layers import Dense
from collections import deque

env = gym.make('CartPole-v0')

EPISODES = 2000
BATCH_SIZE = 32
DISCOUNT = 0.95
UPDATE_TARGET_EVERY = 5
STATE_SIZE = env.observation_space.shape[0]
ACTION_SIZE = env.action_space.n
SHOW_EVERY = 50

class DQNAgents:

    def __init__(self, state_size, action_size):
        self.state_size = state_size
        self.action_size = action_size
        self.replay_memory = deque(maxlen = 2000)
        self.gamma = 0.95
        self.epsilon = 1
        self.epsilon_decay = 0.995
        self.epsilon_min = 0.01
        self.model = self._build_model()
        self.target_model = self.model

        self.target_update_counter = 0
        print('Initialize the agent')

    def _build_model(self):
        model = Sequential()
        model.add(Dense(20, input_dim = self.state_size, activation = 'relu'))
        model.add(Dense(10, activation = 'relu'))
        model.add(Dense(self.action_size, activation = 'linear'))
        model.compile(loss = 'mse', optimizer = Adam(lr = 0.001))

        return model

    def update_replay_memory(self, current_state, action, reward, next_state, done):
        self.replay_memory.append((current_state, action, reward, next_state, done))

    def train(self, terminal_state):

        # Sample from replay memory
        minibatch = random.sample(self.replay_memory, BATCH_SIZE)

        #Picks the current states from the randomly selected minibatch
        current_states = np.array([t[0] for t in minibatch])
        current_qs_list= self.model.predict(current_states) #gives the Q value for the policy network
        new_state = np.array([t[3] for t in minibatch])
        future_qs_list = self.target_model.predict(new_state)

        X = []
        Y = []

        # This loop will run 32 times (actually minibatch times)
        for index, (current_state, action, reward, next_state, done) in enumerate(minibatch):

            if not done:
                new_q = reward + DISCOUNT * np.max(future_qs_list)
            else:
                new_q = reward

            # Update Q value for given state
            current_qs = current_qs_list[index]
            current_qs[action] = new_q

            X.append(current_state)
            Y.append(current_qs)

        # Fitting the weights, i.e. reducing the loss using gradient descent
        self.model.fit(np.array(X), np.array(Y), batch_size = BATCH_SIZE, verbose = 0, shuffle = False)

       # Update target network counter every episode
        if terminal_state:
            self.target_update_counter += 1

        # If counter reaches set value, update target network with weights of main network
        if self.target_update_counter > UPDATE_TARGET_EVERY:
            self.target_model.set_weights(self.model.get_weights())
            self.target_update_counter = 0

    def get_qs(self, state):
        return self.model.predict(np.array(state).reshape(-1, *state.shape))[0]

''' We start here'''

agent = DQNAgents(STATE_SIZE, ACTION_SIZE)

for e in range(EPISODES):

    done = False
    current_state = env.reset()
    time = 0
    total_reward = 0
    while not done:
        if np.random.random() > agent.epsilon:
            action = np.argmax(agent.get_qs(current_state))
        else:
            action = env.action_space.sample()

        next_state, reward, done, _ = env.step(action)

        agent.update_replay_memory(current_state, action, reward, next_state, done)

        if len(agent.replay_memory) < BATCH_SIZE:
            pass
        else:
            agent.train(done)

        time+=1
        current_state = next_state
        total_reward += reward

    print(f'episode : {e}, steps {time}, epsilon : {agent.epsilon}')

    if agent.epsilon > agent.epsilon_min:
        agent.epsilon *= agent.epsilon_decay

Results for first 40ish iterations are below (look for the number of steps, they should be increasing and should reach a maximum of 199)
episode : 0, steps 14, epsilon : 1
episode : 1, steps 13, epsilon : 0.995
episode : 2, steps 17, epsilon : 0.990025
episode : 3, steps 12, epsilon : 0.985074875
episode : 4, steps 29, epsilon : 0.9801495006250001
episode : 5, steps 14, epsilon : 0.9752487531218751
episode : 6, steps 11, epsilon : 0.9703725093562657
episode : 7, steps 13, epsilon : 0.9655206468094844
episode : 8, steps 11, epsilon : 0.960693043575437
episode : 9, steps 14, epsilon : 0.9558895783575597
episode : 10, steps 39, epsilon : 0.9511101304657719
episode : 11, steps 14, epsilon : 0.946354579813443
episode : 12, steps 19, epsilon : 0.9416228069143757
episode : 13, steps 16, epsilon : 0.9369146928798039
episode : 14, steps 14, epsilon : 0.9322301194154049
episode : 15, steps 18, epsilon : 0.9275689688183278
episode : 16, steps 31, epsilon : 0.9229311239742362
episode : 17, steps 14, epsilon : 0.918316468354365
episode : 18, steps 21, epsilon : 0.9137248860125932
episode : 19, steps 9, epsilon : 0.9091562615825302
episode : 20, steps 26, epsilon : 0.9046104802746175
episode : 21, steps 20, epsilon : 0.9000874278732445
episode : 22, steps 53, epsilon : 0.8955869907338783
episode : 23, steps 24, epsilon : 0.8911090557802088
episode : 24, steps 14, epsilon : 0.8866535105013078
episode : 25, steps 40, epsilon : 0.8822202429488013
episode : 26, steps 10, epsilon : 0.8778091417340573
episode : 27, steps 60, epsilon : 0.8734200960253871
episode : 28, steps 17, epsilon : 0.8690529955452602
episode : 29, steps 11, epsilon : 0.8647077305675338
episode : 30, steps 42, epsilon : 0.8603841919146962
episode : 31, steps 16, epsilon : 0.8560822709551227
episode : 32, steps 12, epsilon : 0.851801859600347
episode : 33, steps 12, epsilon : 0.8475428503023453
episode : 34, steps 10, epsilon : 0.8433051360508336
episode : 35, steps 30, epsilon : 0.8390886103705794
episode : 36, steps 21, epsilon : 0.8348931673187264
episode : 37, steps 24, epsilon : 0.8307187014821328
episode : 38, steps 33, epsilon : 0.8265651079747222
episode : 39, steps 32, epsilon : 0.8224322824348486
episode : 40, steps 15, epsilon : 0.8183201210226743
episode : 41, steps 20, epsilon : 0.8142285204175609
episode : 42, steps 37, epsilon : 0.810157377815473
episode : 43, steps 11, epsilon : 0.8061065909263957
episode : 44, steps 30, epsilon : 0.8020760579717637
episode : 45, steps 11, epsilon : 0.798065677681905
episode : 46, steps 34, epsilon : 0.7940753492934954
episode : 47, steps 12, epsilon : 0.7901049725470279
episode : 48, steps 26, epsilon : 0.7861544476842928
episode : 49, steps 19, epsilon : 0.7822236754458713
episode : 50, steps 20, epsilon : 0.778312557068642

I have also tried by setting the epsilon (parameter for epsilon-greedy) to 0.1 from the beginning. The learning still remain worse than what it is when epsilon starts from 1. Apparently there is some issue in updating the best Q value for the selected action. But I cannot find what exactly I am missing, doing wrong, or what I should add.

You know that your epsilon needs to be high in the beginning since it's the way you search though your search space... (eploitation vs exploration)

There is a really small mistake in here that causes the problem:

for index, (current_state, action, reward, next_state, done) in enumerate(minibatch):
            if not done:
                new_q = reward + DISCOUNT * np.max(future_qs_list) #HERE
            else:
                new_q = reward

            # Update Q value for given state
            current_qs = current_qs_list[index]
            current_qs[action] = new_q

            X.append(current_state)
            Y.append(current_qs)

Since np.max(future_qs_list) should be np.max(future_qs_list[index]) since you're now getting the highest Q of the entire batch.
Instead of the getting the highest Q from the current next state.
It's like this after changing that (remember an epsilon of 1 means that you get 100% of your actions taken by the a dice roll so I let it go for a few more epochs, also tried it with the old code but indeed didn't get more then 50 steps (even after 400 epochs/episodes))
episode : 52, steps 16, epsilon : 0.7705488893118823
episode : 53, steps 25, epsilon : 0.7666961448653229
episode : 54, steps 25, epsilon : 0.7628626641409962
episode : 55, steps 36, epsilon : 0.7590483508202912
episode : 56, steps 32, epsilon : 0.7552531090661897
episode : 57, steps 22, epsilon : 0.7514768435208588
episode : 58, steps 55, epsilon : 0.7477194593032545
episode : 59, steps 24, epsilon : 0.7439808620067382
episode : 60, steps 46, epsilon : 0.7402609576967045
episode : 61, steps 11, epsilon : 0.736559652908221
episode : 62, steps 14, epsilon : 0.7328768546436799
episode : 63, steps 13, epsilon : 0.7292124703704616
episode : 64, steps 113, epsilon : 0.7255664080186093
episode : 65, steps 33, epsilon : 0.7219385759785162
episode : 66, steps 33, epsilon : 0.7183288830986236
episode : 67, steps 39, epsilon : 0.7147372386831305
episode : 68, steps 27, epsilon : 0.7111635524897149
episode : 69, steps 22, epsilon : 0.7076077347272662
episode : 70, steps 60, epsilon : 0.7040696960536299
episode : 71, steps 40, epsilon : 0.7005493475733617
episode : 72, steps 67, epsilon : 0.697046600835495
episode : 73, steps 115, epsilon : 0.6935613678313175
episode : 74, steps 61, epsilon : 0.6900935609921609
episode : 75, steps 43, epsilon : 0.6866430931872001
episode : 76, steps 21, epsilon : 0.6832098777212641
episode : 77, steps 65, epsilon : 0.6797938283326578
episode : 78, steps 45, epsilon : 0.6763948591909945
episode : 79, steps 93, epsilon : 0.6730128848950395
episode : 80, steps 200, epsilon : 0.6696478204705644
episode : 81, steps 200, epsilon : 0.6662995813682115

I think the problem is with openAI gym CartPole-v0 environment reward structure.
The reward is always +1 for each time step. So if pole falls reward is +1 itself.
So we need to check and redefine the reward for this case.
So in the train function try this:
if not done:
    new_q = reward + DISCOUNT * np.max(future_qs_list)
else:
    # if done assign some negative reward
    new_q = -20

(Or change the reward during replay buffer update)
Check the lines 81 and 82 in Qlearning.py code in this repo for further clarification.

You really don't need to edit the reward structure. The reward structure is fine. Such a high negative reward can even have negative effects.

@hal I’m not sure that that’s true (that you don’t need to change it). If the environment gives +1 on termination then that can be problematic for the agent, it is better to change it to 0 or -1.

yeah, I have to agree on that, 0 or -1 seems reasonable to me too but the reward structure is fine enough to be able to solve without doing that.

@hal I experimented with different reward stuctures for cartpole using classical Q-learning(Not DQN). I found that reward between -10 to -20 for terminal state working better. My experimentation code and graph can be found here. But I found this code having reward of -200 for terminal state in official gym website.