text:
Is it possible to use an RNN to predict a feature that is not an input feature?
I came across RNN's a few minutes ago, which might solve a problem with sequenced data I've had for a while now.
Let's say I have a set of input features, generated every second. Corresponding with these input features, is an output feature (also available every second).
One set of input features does not carry enough data to correlate with the output feature, but a sequence of them most definitely does.
I read that RNN's can have node connections along sequences of inputs, which is exactly what I need, but almost all implementations/explanations show prediction of the next word or number in a text-sentence or in a sequence of numbers.
They predict what would be the next input value, the one that completes the sequence. However, in my case, the output feature will only be available during training. During inference, it will only have the input features available.
Is it possible to use RNN in this case? Can it also predict features that are not part of the input features?
Thanks in advance!

Is it possible to use RNN in this case? Can it also predict features that are not part of the input features?

Yes.
No changes are required to a RNN in order to do this. All you need is correctly labelled data mapping a sequence of $x$ to correct $y$ in order to train, and of course a RNN architecture which has input vectors matching shape of $x$ and output vectors matching shape of $y$. The case where $x$ and $y$ are the same data type is just a special case of RNN design, and not a requirement.
You may need to consider some details:

If the relationship between $x$ and $y$ is complex and non-linear even accounting for accumulated hidden state during the sequence, you may need to add deeper layers. The output of the LSTM can be some vector $h$ and you can add fully-connected layers to help with predicting $y$ from $h$. This, or adding more LSTM layers, is a choice of hyperparameter that you may want to experiment with. Start with a basic LSTM to see how that goes first.
If you wish to predict a sequence of output features that is either not the same length as the input feature sequence, or logically should come after the whole sequence (think language translation) then this may need a slight change in setup to get best results. For a predict-same-kind sequence you can feed your predicted output value into the next input, but if input and output have different data types, this will not work. Instead, you will need to have some dummy input or other setup for creating sequences of $y$.

In your specific case the second point does not seem to apply, as you want to predict a single $y$ immediately after a sequence of $x$.

As an aside, I recommend the course https://www.coursera.org/learn/nlp-sequence-models/home/welcome if you are new to RNNs.


text:
How to design a classifier while the patterns of positive data are changing rapidly?
In some situation, like risk detection and spam detection. The pattern of Good User is stable, while the patterns of Attackers are changing rapidly. How can I make a model for that? Or which classifier/method should I use?

The phenomenon where the prediction targets (in your case, behaviour) change over time is referred to as "concept drift".
If you search for that term, you'll find that there have been many publications attempting to tackle that over multiple decades, way too many papers to all summarize here in a single answer. It's still a difficult problem though, by no means a "solved" problem.
Two different, broad directions for ideas are:

Frequently re-training (offline) static models on the most recent training data
Using online learning approaches that can continuously be updated from a data stream, online as new labelled data becomes available.

This github page contains a large list of papers on credit card fraud detection, where the problem you describe occurs because fraudsters change their behaviour in an attempt to evade detection. Most of those papers discuss variants of the first approach. Basically, many of those papers use an ensemble of multiple Random Forests. Every day, new labelled data becomes available. They often then remove the oldest of multiple Random Forests, and add a new Random Forest trained on the most recent data made available that day.
There are also some variants where they don't always train new models at a fixed schedule (e.g., every day), but try to detect when the statistical properties of the data have changed using statistical tests, and only train new models when it is "necessary" (due to such changes).
For the second idea, you'll often be thinking of approaches that use Stochastic Gradient Descent-like approaches for learning; with a non-decreasing learning rate / step size, such techniques will naturally, slowly "forget" what they have learned from old data, and focus more on the latter data.
If you have some method to obtain accurate labels for certain instances relatively quickly, you could consider an approach like the one proposed in this paper (disclaimer: I'm an author on this paper). For example, in that paper the assumption is that human experts can relatively quickly investigate and obtain accurate labels for a small selection of transactions, and this can be exploited to quickly learn in an online manner.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to build a commercial image captioning system?
Image Captioning is a hot research topic in the AI community. There are considerable image captioning models for research usage such as NIC, Neural Talk 2 etc. But can these research models be used for commercial purpose? Or we should build much more complex structured ones for commercial usage? Or if we can make some improvements based these models to meet the business applications situation? If so, what improvements should we take? Are there any existing commercial Image Captioning applications can be referenced?

Thinking Commercially
A commercial solution will need to be able to ascertain, continuously verify, and utilize the best of options for captioning learning models.
Each fairly successful image captioning learning models could be placed in an adapter to provide a common training, optimizing, testing, evaluation, and usage interface.  The hot swappable container-addon mega pattern used in peripheral device installation, RAID, J2EE containers, browsers, and other containment sub-systems can be applied.
System Description
The system acceptance criteria being as follows.

A new model could be added without stopping or starting the system if tauted as successful by others
Any model can be deleted without starting or stopping the system if it has not been performing well
Each possible state of each model added can be given some percentage of the total system resources
Various training processes and hyper parameter tuning processes can be applied to any of those model-state combinations
A-B testing can be done on any model-state combination
Stats can be displayed on any model-state combination any model overall or any state across all added models
The interface for each model requires that its associated model produces an array of caption-reliability pairs with zero ore more pairs
The reliability measure that is included in the pair with the caption is an indication of the model's assessment of appropriateness to the image
The most appropriate caption is picked from the proposals of the various models

States can be idle or any of these.

Training
Testing
Evaluation
Hyper parameter tuning
In use

For example, for the two learning models usable in captioning systems suggested in this question, NIC and Neural Talk 2 we could have a system resource allocation like this:

30% NIC Training
5% NIC Hyper parameter tuning
45% NT2 In use
15% NT2 Evaluation
5% NT2 Hyper parameter tuning

Samples may be pulled from a pool of samples that have been vetted.  That pool may be augmented by real images passing through the system, filtered in accordance with security criteria to avoid external attempts at control.
In the assignment of resources, the sample pool selection criteria must be specified.  If the system is already at 100%, the model-state combinations from which the resources shall be drawn must also be specified.
Handling Multiple Output Options
Since there may be more than one model in use and each model may have zero, one, or multiple caption suggestions for each image, each with a reliability measure, the outputs must be analyzed to provide the best choice to associate with the image being analyzed.  Additional system criteria must cover this process scenario.  For any given image, final evaluation must follow the following general guidelines.

If multiple models produce similar or exactly the same captions, they most be weighted higher in final evaluation.
If the model's reliability is proven in actual use based on feedback from end users, the model's output must be weighed higher accordingly
Re-entrant models (such as reinforcement learning network models) must have access to the end user feedback for additional learning even in in-use state
Clear winners are chosen
Close races are disambiguated through trained functionality
Exact ties are broken through pseudo random index generation

Another artificial network may be placed at the output and appropriate encoding and normalization may be applied before training so that a properly trained network, converged using a quantification of the above additional criteria, can select the best caption from the options for each image.
Phased Development Approach
Phase one of such a system would likely require manual handling of model-state allocation.  Phase two would be semi-automation.  The location of new models would still require expert attention.  Perhaps further in the future a hunt for new models could be automated too.

What about the dataset? MSCOCO is just for research usage, right? How to collect a dataset for commercial use? What about the dataset's quantity, quality and collecting method?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Does balancing the training data set distribution for a neural network affect its understanding of the original distribution of data?
I have a very imbalanced dataset of two classes: 2% for the first class and 98% for the second. Such imbalance does not make training easy and so balancing the data set by undersampling class 2 seemed like a good idea.
However, as I think about it, should not the machine learning algorithm expect the same data distribution in nature as in its training set? I know, for sure, that the distribution of data in nature matches my imbalanced dataset. Does that mean that the balanced dataset will negatively affect the neural net performance with testing? when it assumed a different distribution of data caused by my balanced data set.


text:
SEIF motion update algorithm doubt
I want to implement Sparse Extended information slam. There is four step to implement it. The algorithm is available in Probabilistic Robotics Book at page 310, Table 12.3.

In this algorithm line no:13 is not very clear to me. I have 15 landmarks. So $\mu_t$ will be a vector of (48*1) dimension where (3*1) for pose. Now $H_t^i$ is a matrix whose columns are dynamic as per the algorithm it is (3j-3) and 3j. J is the values of landmarks 1 to 15. Now how could I multiply a dynamic quantity with a  static one. There must be a error that matrix dimension mismatch when implement in matlab.
Please help me to understand the algorithm better.

You are right, that pseudocode is not correct. In particular, the definition of $H_t^i$ in line $11$ should be changed; all the way on the right-hand side, it should have $3N - 3j$ columns of $0$s, rather than $3j$ columns of $0$s.
With that change, every matrix $H_t^i$ will have the same number of columns:
$$6 + 3j - 3 + 3N - 3j = 3 + 3N,$$
which evaluates to a total of $48$ in your case (because you have $N = 15$ landmarks). That's precisely the correct dimensionality required for matrix multiplication with your $\mu_t$ vector.

The version of the book that you linked to appears to be a fairly old draft. This webpage contains errata for the third edition for the book, in which page 393 corresponds to what was page 310 in your version of the book. The errata for that third edition of the book can be downloaded at the following URL: http://probabilistic-robotics.informatik.uni-freiburg.de/corrections/pg393.pdf
There you'll find the fix that I described above, but also some other fixes (most of them are just notational, adding bars over the $\mu$ vectors, but it looks like a more serious issue was additionally fixed in line 13, where a minus was changed to a plus).

Oh Thank You sir. It help me a lot. It's really save my code. Can you please elaborate what is the need of $\H_t^i$ matrix? In Table 12.5 of Probabilistic Robotics what is the significant of $\mu_t$. I think it content pose and landmarks coordinates.If I am right then at Table 12.3 (the picture I given with this question)line no. 5,6 also give the co ordinate of observed landmarks. Which one should be taken.?

@EllenaMori Your understanding of the contents of $\mu$ is correct. In lines 5, 6 of Table 12.3, they are initialized to the initial observations in cases where a landmark was never observed before. In Table 12.5, those estimates are updated using subsequent observations. I'm not 100% sure on an intuitive explanation of the $H$ matrix. I have once implemented some variant of SLAM during my studies multiple years ago, but am by no means an expert on SLAM unfortunately :(


text:
Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?
Many have examined the idea of modifying learning rate at discrete times during the training of an artificial network using conventional back propagation.  The goals of such work have been a balance of the goals of artificial network training in general.

Minimal convergence time given a specific set of computing resources
Maximal accuracy in convergence with regard to the training acceptance criteria
Maximal reliability in achieving acceptable test results after training is complete

The development of a surface involving these three measurements would require multiple training experiments, but may provide a relationship that itself could be approximated either by curve fitting or by a distinct deep artificial network using the experimental results as examples.

Epoch index
Learning rate hyper-parameter value
Observed rate of convergence

The goal of such work would be to develop, via manual application of analytic geometry experience or via deep network training the following function, where

$\alpha$ is the ideal learning rate for any given epoch indexed by $i$,
$\epsilon$ is the loss function result, and
$\Psi$ is a function the result of which approximates the ideal learning rate for as large an array of learning scenarios possible within a clearly defined domain.

$\alpha_i = \Psi (\epsilon, i)$
The development of arriving at $\Psi$ as a closed form (formula) would be of general academic and industrial value.
Has this been done?

Has this been done?

Difficult to prove a negative, but I suspect although plenty of research has been done into finding ideal learning rate values (the need for learning rate at all is an annoyance), it has not been done to the level of suggesting a global function worth approximating.
The problem is that learning rate tuning, like other hyperparameter tuning, is highly dependent on the problem at hand, plus the other hyperparamater values currently in use, such as size of layers, which optimiser is in use, what regularisation is being used, activation functions.
Although you may be hoping for $\Psi(\epsilon, i | P)$ to exist where P is the problem domain, it likely does not except as a mean value over all $\Psi(\epsilon, i | D, H)$ for the problem domain, where D is the dataset and H all the other hyperparameters.
It is likely that such a function exists, of ideal learning rate for best expected convergence per epoch. However, it would be incredibly expensive to sample it with enough detail to make approximating it useful. Coupled with limited applicability (not domain-specific, but linked to data and other hyperparameters), a search through all possible learning rate trajectories looks like it would give poor return on investment.
Instead, the usual pragmatic approaches are:

Include learning rate in hyperparameter searches, such as grid search, random search, genetic algorithms and other global optimisers.
Decay learning rate using one of a few approaches that have been successfully guessed and experiments have shown working. These have typically been validated by plotting learning curves of loss functions or other metrics, and the same tracking is usually required in new experiments to check that the approach is still beneficial.
Some optimisers use a dynamic learning rate parameter, which is similar to your idea but based on reacting to measurements during learning as opposed to changes based on an ideal function. They have a starting learning rate, then adjust it based on  heuristics derived from measuring learning progress. These heuristics can be based on per-epoch measurements, such as whether a validation set result is improving or not. One such approach is to increase learning rate whilst results per epoch are improving, and reduce learning rate if results are not improving, or have got worse.

I have tried this last option, on a Kaggle competition, and it worked to some extent for me, but did not really improve results overall - I think it is one of many promising ideas in ML that can be made to work, but that has not stayed as a "must have", unlike say dropout, or using CNNs for images.
Some optimisers store a multiplier per layer or even per weight - RMSProp and Adam for example track rate of change of each parameter, and adjust the rate for each weight during updates. These can work very well in large networks, where the issue is not so much needing a specific learning rate at any time, but that a single learning rate is too crude to cover the large range of gradients and differences in gradients across the index space of all the connections. With RMSProp and Adam, the need to pick specific learning rates or explore them is much reduced, and often a library's default is fine.


text:
Can a basic CNN (Conv2D, MaxPooling2D, UpSampling2D) find a good approximation of a product of its input channels?
Let's assume I want to teach a CNN some physics. Starting with a U-Net, I input images A and B as separate channels. I know that my target (produced by a very slow Monte-Carlo code) represents a signal such as f(g(A) * h(B)), where f, g and h are fairly "convolutional" operations -- meaning, involving mostly blurring and rescaling operations.
I feel safe to state that this problem would not be too difficult for the case of f(g(A) + h(B)) -- but what about f(g(A) * h(B))? Can I expect a basic CNN such as the U-Net to be able to represent the * (multiplication) operation?
Or should I expect to be forced to include a Multiply layer in my network, somewhere where I expect that the part before can learn the g and h parts, and the part after can learn the f part?

I think U-Net is already a quite complex network that probably should (as of my experience) be able to approximate this multiplication. However this would still be an approximation that might not be accurate and maybe only solves a function that looks like a multiplication for the range of the shown input samples defined by your dataset (therefore potentially overfitting on your training dataset).
So in general if you know that your target function does have this multiplication then you should definetly excplicitly enforce it in your network. If you know so much about the wanted function it's always better to build a good fitting architecture than using a general neural network architecture. That will ease the optimization and should generalize much better.
However, it's hard or impossible to tell you for sure how much depth or complexity you need to solve this task within your wanted accuracy. Eventually you should just try it out.


text:
Is the cube root function suitable as a n activation function?
I am trying to design a neural network on Python.
Instead of the sigmoid function which has a limited range, I am thinking of using the cube root function which has the following graph:

Is this suitable?

There are a few traits that you want the activation function to have, and cube roots rate as OK-ish:

Nonlinear – check.
Continuously differentiable – no. There is a problem at $x=0$. Unlike other discontinuous functions like ReLU, although the gradient can be calculated near zero, it can be arbitrarily high as you approach $x=0$, because $\frac{d}{dx}x^{\frac{1}{3}} = \frac{1}{3x^{\frac{2}{3}}}$
Range considerations – limited range functions are more stable, large/infinite range functions are more efficient. You may need to reduce learning rates compared to e.g. tanh.
Monotonic – check.
Monotonic derivative - no.
Approximates identity near the origin – no, the approximation is bad near the origin.

If you look through the list of current, successful activation functions, you will see a few that also fail to provide one or more desirable traits, yet are still used routinely.
I would worry about the high gradients near $x=0$, but other than that I think the function could work ok. It may sometimes be unstable during learning, as small changes near zero will result in large changes to output. You might be able to work around the high gradients simply in practice, by clipping them. If the raw calculation returns value greater than $1$ (or less than $-1$) then treat the gradient as if it were $1$ (or $-1$) for the rest of backpropagation.
The only way of finding out if the function is competitive with other more standard activation functions is to try it on some standard data sets and make comparisons.


text:
Creating a self learning Mario Kart game AI?
I will be undertaking a project over the next year to create a self learning AI to play a racing game, currently the game will be Mario Kart 64.
I have a few questions which will hopefully help me get started:

What aspects of AI would be most applicable to creating a self learning game AI for a racing game (Q-Learning, NEAT etc)
Could a ANN or NEAT that has learned to play Mario Kart 64 be used to learn to play another racing game?
What books/material should i read up on to undertake this project?
What other considerations should i take throughout this project?

Thank you for your help!

Do you have access to a programmable environment for Mario Kart? Regardless of approach to AI/learning, you will need to have a system that can start the game, control it and receive feedback.

I can read the game screen in using OpenCV, apply preprocessing to produce an input for the NN and then use a python module to simulate keypresses for the game, does that sound reasonable?

If you use the game screen as input, I guess your model won't be able to play other game than Mario Kart (but I am not an expert so maybe I am wrong...)

What aspects of AI would be most applicable to creating a self learning game AI for a racing game (Q-Learning, NEAT etc)

In general, you are looking at a problem that involves sequential decision making, in a machine learning context.
If you are wanting to build an agent that can learn by receiving screen images, then NEAT cannot scale to that complexity directly. Although there might be clever combinations of deep learning and evolutionary algorithms you could apply, the most heavily explored and likely successful solution will be found in Deep Reinforcement Learning. Algorithms like DQN, A3C, A2C, PPO . . . there are dozens to consider, but all are based around agents using samples of experience to update functions that measure either the "value" of acting in a certain way (a policy) or estimate the best policy directly.

Could a ANN or NEAT that has learned to play Mario Kart 64 be used to learn to play another racing game?

Within limits, yes. You will have built a system that takes pixel inputs, and outputs controller messages. If you re-start training from scratch on any compatible N64 game (with same screen resolution and same controller outputs) there is a chance it could learn to play that new game well. As other driving games on the N64 are a subset of all games, and more similar to each other than, say, a scrolling shooter or adventure game, then an agent that can successfully learn one will likely learn another too.
It is unlikely that a Mario Kart agent will immediately be good at another game without re-training. The visual and control differences will probably be too much. You could try though. An interesting experiment would be to take your trained agent, or some part of it, and see if starting with that improves learning time on a new game. This is called Transfer Learning.

What books/material should i read up on to undertake this project?

You will need at least the first parts of an introduction to Reinforcement Learning. If your goal is to aim to just make the agent, then you can skip theory-heavy parts, but within limits the more theory you understand, the easier it will be to change code features towards getting something working. I can suggest the following:

The book Sutton & Barto, Reinforcement Learning: An Introduction - this covers theory behind basic RL, and the online draft is free
A course of lectures on Reinforcement Learning from UCL, by David Silver. The video lectures cover same topics as the Sutton & Barto book.
The book Deep Reinforcement Learning Hands-On, which you will need to buy. As the title suggests, it covers more practical aspects of building RL systems. It uses Python and PyTorch. As RL has become more popular in recent years, there are plenty of other similar books - I have no reason to recommend this one over them, but I am reading it at the moment, and the quality is adequate.

What other considerations should i take throughout this project?

Before you get started, you should know this could be an ambitious project, requiring a lot of compute time using currently-available toolkits. You need to think ahead a little, as you will be faced with these decisions:

Is it more important to you to make a working bot on this problem, or more important to understand the underlying theories so that you understand what is going on. You will need to do at least a little of both, but you can go strongly in either direction:

There are enough pre-built learning agents available on the internet that you might have a successful project by learning just enough to wire up a copy of the game, and then letting the learning agent do its job. Would this be enough for you, would you feel that you had solved your problem?
There is plenty of educational material about RL and how the maths behind it works. If you are interested in understanding that, and perhaps generating your own ideas for improvements to current algorithms, then you need to study that material harder. However, this may lead you away from your Mario Kart player goal, and you may never actually solve it, because far simpler problems that require less computing power are still actually interesting academically.

In addition, I can think of the following:

Building your Mario Kart player is a long-term goal. You will need to start with simpler agents in order to understand the techniques that you hope to apply to Mario Kart.
You may need more compute power to solve this game than you have available.
You will need to solve the issue of automating control of a N64 emulator. There is at least an existing emulator Mupen64Plus - I do not know whether it will be adequate for you, but at least one person has attempted to wrap this for automated learning, in the gym-mupen64plus project.


text:
Why do you not see dropout layers on reinforcement learning examples?
I've been looking at reinforcement learning, and specifically playing around with creating my own environments to use with the OpenAI Gym AI. I am using agents from the stable_baselines project to test with it.
One thing I've noticed in virtually all RL examples is that there never seems to be any dropout layers in any of the networks. Why is this?
I have created an environment that simulates currency prices and a simple agent, using DQN, that attempts to learn when to buy and sell. Training it over almost a million timesteps taken from a specific set of data consisting of one month's worth of 5-minute price data it seems to overfit a lot. If I then evaluate the agents and model against a different month's worth of data is performs abysmally. So sounds like classic overfitting.
But is there a reason why you don't see dropout layers in RL networks? Is there other mechanisms to try and deal with overfitting? Or in many RL examples does it not matter? e.g. there may only be one true way to the ultimate high score in the 'breakout' game, so you might as well learn that exactly, and no need to generalise?
Or is it deemed that the chaotic nature of the environment itself should provide enough different combinations of outcomes that you don't need to have dropout layers?

Have you tried adding dropout to a RL network to see what happens? I have (for a LunarLander_v2 solver that works well otherwise), and the results were terrible, it goes from solving in 400 episodes, to completely failing to solve it ever (and I tried plenty of variations in other params). Other regularisation is OK, and even beneficial. I don't know why the problems with dropout yet though, was thinking of asking a question here . . .

Neil, yes, it seem that your experience there echos what is said below by Dennis. As I expected, in the 'solve this puzzle' type scenario you really are looking to overfit as you do want to solve that specific puzzle. And you want to know if you are encountering a blue alien vs a red alien as they might behave differently in your game.

@MattHamilton I just happened to see a thread of tweets someone made about the "RL researchers are evaluating on the training data" story. I've edited a link to that into my answer. The point is still the same when it comes to answering your question, but I do agree with the chain of tweets I linked to there that the truth is a bit more nuanced.

Dropout essentially introduces a bit more variance. In supervised learning settings, this indeed often helps to reduce overfitting (although I believe there dropout is also already becoming less.. fashionable in recent years than in the few years before that; I'm not 100% sure though, it's not my primary area of expertise).
In Reinforcement Learning, additional variance is not really what we're looking for. There already tends to be a large amount of variance in the learning signals that we get, and this variance already tends to be a major issue for learning stability and/or learning speed. For example:

Randomness in action selection leads to variance in returns that we observe
There may be randomness inherent to the environment itself, leading to extra variance in our observations (some environments are nondeterministic)
Unlike Supervised Learning settings, in Reinforcement Learning we often actually use our own predictions as a part of our loss function / training signal. For example, in temporal-difference learning (like Q-learning / DQN), the target that we update towards looks like $r + \max_{a'} Q(s', a')$. In that term, only the $r$ is a ground-truth observation (like we would use in supervised learning), and the other term is our own prediction. During a learning process, those latter parts (our own predictions) are changing over time. This is a "moving target'' problem, which can be viewed as additional variance in our learning signals.

Many important parts of Deep RL algorithms (without which our training processes empirically turn out to destabilize and break down) are very much tailored towards reducing that variance. For example, Target Networks in DQN were introduced specifically to reduce the moving target problem. From this point of view, it's not surprising that if we were to add more artificial variance through other means again (such as dropout), that this would hurt performance / destabilize learning.

Is there other mechanisms to try and deal with overfitting? Or in many RL examples does it not matter? e.g. there may only be one true way to the ultimate high score in the 'breakout' game, so you might as well learn that exactly, and no need to generalise?

In the majority of current (Deep) Reinforcement Learning research, overfitting is indeed not viewed as a problem. The vast majority of RL research consists of training in one environment (for example Cartpole, or Breakout, or one particular level in Pacman, or navigating in one specific maze, etc.), and either constantly evaluating performance during that learning process, or evaluating performance after such a learning process in the same environment.
If we were to compare that evaluation methodology to what happens in supervised learning... we are basically evaluating performance on the training set*. In supervised learning, this would be absolutely unacceptable, but in RL it is very much treated as acceptable and more rule than exception. Some say this is simply a problem in current RL research, something that needs to change. It could also be argued that it's not necessarily a problem; if we really are able to train the agent in precisely the same environment that we wish to deploy it in later... well, then what's the problem with it overfitting to that environment?
So, when we're using the evaluation methodology described above, indeed we are overfitting to one specific environment, but overfitting is good rather than bad according to our evaluation criteria. It is clear that this methodology does not lead to agents that can generalize well though; if you consistently train an agent to navigate in one particular maze, it will likely be unable to navigate a different maze after training.
*Note: the truth, in my opinion, is slightly more nuanced than that we are really "evaluating on the training set" in RL. See, for example, this nice thread of tweets: https://twitter.com/nanjiang_cs/status/1049682399980908544

I have created an environment that simulates currency prices and a simple agent, using DQN, that attempts to learn when to buy and sell. Training it over almost a million timesteps taken from a specific set of data consisting of one month's worth of 5-minute price data it seems to overfit a lot. If I then evaluate the agents and model against a different month's worth of data is performs abysmally. So sounds like classic overfitting.

Note that your evaluation methodology described here indeed no longer fits the more "common" evaluation methodology. You have a problem with concept drift, with nonstationarity in the environment. This means overfitting may be a problem for you.
Still, I'm not sure if dropout would help (it's still additional variance which may hurt). First and foremost, you'd want to make sure that there's some way to keep track of the time / month in your inputs, such that you'll at least have a chance of learning a policy that adapts itself over time. If you have a clear, solid boundary between "training phase" and "evaluation phase", and you know that concept drift occurs across that boundary (you know that your environment behaves differently in the training phase from the evaluation phase)... you really don't have much hope of learning a policy only from experience in the training phase that still performs well in the evaluation phase. I suspect you'll have to get rid of that clear, solid boundary. You'll want to keep learning throughout the evaluation phase as well. This enables your learning algorithm to actually collect experience in the changed environment, and adapt to it.

Dennis, thank you very much for your very detailed reply! A lot of it confirms I think my suspicions: ie that more RL is trying to solve a very specific problem in a specific environment in which there is often one clear 'best' solution to that problem.

@MattHamilton Note that there is research towards RL for more general environments. If you're interested in that stuff, you'll want to search for combinations of "Transfer learning" and "Reinforcement Learning", or things like Multi-Task RL (Multi-Objective RL may also be interesting, but is probably slightly different). Those kinds of efforts do still tend to be significantly less performant than the traditional approach of training and evaluating on one environment at a time though.

Overfitting might also be a problem in RL, resulting in compounding errors and catastrophic forgetting: https://ai.stackexchange.com/questions/28079/deep-q-learning-catastrophic-drop-reasons


text:
How to detect a Neural Network will work with the whole dataset?
I want to implement a neural network on a big dataset. But training time is long (~1h30 per epoch). I'm still in the development process, so I don't want to wait such long time just to have poor results at the end.
This and this suggest that overfitting the network on a very small dataset (1 ~ 20 samples) and reach a loss near 0 is a good start.
I did it and it works great. However, I am looking for the next step of validating my architecture. I tried to overfit my network over 100 samples, but I can't reach a loss near 0 in reasonable time.
How can I ensure the results given by my NN will be good (or not), without having to train it on the whole dataset ?

you should look at other metric than the loss. Maybe presicsion recall, accuracy....

You can try to train it on 1% data, then on 2%, 3%, etc... Then plot it and see if increasing data increases the performance and how it is changing. Not sure if that's correct answer, but at least you can iterate this method pretty fast.

How can I ensure the results given by my NN will be good (or not), without having to train it on the whole dataset ?

You can't.
If you're interested in diagnostic techniques for neural networks, read section 2.5 of my Master's thesis

Don't know anything about your dataset, but maybe by using clustering* on it, you can get N "most distinct" examples and train only on them. This obviously will not give you same performance as if network would have seen all examples, but this way at least you will show it "diverse" examples.
*That is, of course, if you have time for that.


text:
Interpretation of a good overfitting score
As shown below, my deep neural network is overfitting :

where the blue lines is the metrics obtained with training set and red lines with validation set
Is there anything I can infer from the fact that the accuracy on the training sets is really high (almost 1) ?
From what I understand, it means that the complexity of my model is enough / too big. But does it means my model could theoretically reach such a score on validation set with same dataset and appropriate hyperparameters ? With same hyperparameters but bigger dataset ?
My question is not how to avoid overfitting.

"But does it means my model could theoretically reach such a score on validation set with same dataset and appropriate hyperparameters ? With same hyperparameters but bigger dataset ?" It's a question about how to avoid overfitting ;)

@JérémyBlain Sorry for my poor english, I'm not native. My sentence might not convey the sens I wished. I was asking some interpretation based on the (overfitting) training accuracy. I know the methods to avoid overfitting. I was more asking about experience with overfitting, if it is possible to reach such a good accuracy without applying all the counter-overfitting methods.

See chapter 2.5 of Analysis and Optimization of Convolutional Neural Network Architectures

It doesn't tell you very much, to be honest. It does mean that (assuming your training and validation distributions are similar) your model could get the same results on your validation set should you train on that, but that would still be overfitting.
Really, the only useful thing overfitting tells you is that you don't have enough regularisation.

overfitting tells you that your model don't generalize well. Regularization is not the only method to reduce overfitting (think of a bigger dataset)

I would argue that using more data, augmenting data, reducing the size of the net, etc. are all forms of regularisation.


text:
Is 1mb an acceptable memory size for images being trained in a CNN?
I am using Tensorflow CNN to build an image classification/prediction model. Currently all the images in the dataset are each about 1mb in size.
Most examples out there use very small images.
The image size seems large, but I not too sure.
Any thoughts on the feasibility of 1mb images? If not what can I do to compress programmatically?

I don't heard of any network trained on high resolution images sized, but I guess it will be slow to train... Don't forget the more pixels you have, the more computation you need. (and maybe more deeper the network need to be, and so the computation time increase again...)

@JérémyBlain Ah I see. Thank you, any idea of acceptable resolution size?

I saw a paper (or an article) from google which stated that low resolution image are fine for prediction. I don't have the actual size, nor a good limit, but I think they resized image to a 32x32 size (or something similar) and the prdiction was actually almost the same. I don't remember the paper (or article) so I can't link it there, unfortunatly.... Think of the CIFAR dataset with 32x32 images size... it's enough for a good prediction, even for the 100 classes one.

@JérémyBlain Ok interesting. For example, I imagine for a model that does tree/plant specification where a lot of classes (leaves/trees) are similar yet completely different classifications, that reducing the image quality will most certainly have a detrimental effect on the quality of the model. Not entirely sure about that, just my intuition. I'll need to do more research.

I'm going to try and use image size of 336X252 and see how that performs. The image quality will certainly have a correlation to the training quality in the type of dataset I am using.

Yes, maybe for classes that are similar like trees, maybe the prediction on low resolution image affect the prediction a lot. I guess 336x252 is enough, though ! Imagenet, with a lot of classes which are similar (dogs for example) has images of resolution 224x224. As usual, in machine learning, you need to try to see the result ! good luck

1 MB by images is too much. It means that you have a lot of pixels to compute in the inputs, and your images have more features that are not very useful for the classification (we human don't need high resolution images for recognize objects, it's the same for model)
It means that maybe you need also a more deeper network, and then, more computation.
In your particular case, you have a dataset of trees, so images are somewhat similar and then, maybe the classification is harder. So, images need enough information in pixels to have a good prediction.
You should resize your whole datsets to more common size because 1mb are too much (I don't know the resolution, but I think it could be around thousand for both dimension).
Imagenet have images of size 224x224, and there are 1000 classes, which some of them are close. For examples, there are many dogs classes (boxer, american terrier, ...), cats classes (tiger, egyptian, ....), thus I think size of a few hundred would be enough.
In your comment you said you will try with 336x252, I think it can be a good start.
Of course you will need some experimentation in this regard. Maybe you can try to train some models in a few epoches with different image sizes, to see the best one, and keep it to train the model further !


text:
Is it suitable to find inverse of last layer's activation function and apply it on the target output?
I have a neural network with the following structure:

I am expecting specific outputs from the neural network which are the target values for my training. Let's say the target values are 0.8 for the upper output node and -0.3 for the lower output node.
The activations function used for the first 2 layers are ReLu or LeakyReLu while the last layer uses atan as an activation function.
For back propogation, instead of adjusting values to make the network's output approach 0.8,-0.3. is it suitable if I use the inverse function for atan -which is tan itself- to get "the ideal input to the output layer multiplied by weights and adjusted by biases".
The tan of 0.8 and -0.3 is 0.01396 and -0.00524 approximately.
My algorithm would then adjust weights and biases of the network so that the "pre-activated output" of the output layer -which is basically (sum(output_layer_weight*output_layer's inputs)+output_layer_biases)- approaches 0.01396 and -0.00524.
Is this suitable

As mentioned in the topic of the question I am only considering to apply this on the last layer since there is no 'nesting' of multiple activation functions.

Your third point on over fitting might be invalid since the method i suggest only finds inverse values of the last activation function to apply back propogation to the layers before it so that the input values of the last layer approaches the inverse function values of the target.

What do you want to achieve? What is your goal?

For the above stated artificial network, these two training scenarios are similar.

Training to converge to the ideal output vector at the point after the last layer's activation functions are applied
Training to converge to the vector formed by applying $tan$ functions to each component of the ideal output vector, when convergence occurs at the point just after the vector-matrix multiplication with the last layer's parameters, prior to the last layer's $atan$ activation functions

Distinctions between them include these.

The applications of gradients and associated code must be adapted to the modification of the starting point of back propagation to before the final $atan$ activation functions.
The slope and curvature of the loss function will differ if the same loss function is used for the two scenarios, so the accuracy, speed, and reliability of convergence will also be different.

The main difference your change would have is to allow you to apply a loss function to a different part of the network. This may affect training.
If you keep the same loss function (e.g. MSE), but apply it to the pre-transformed values, then you will have changed the objective of the network, perhaps significantly. Whether or not this is a good thing depends on how much you needed the original loss function. However, the fact that it would result in a different training target is usually going to be a bad thing if your original training target was correct. This will also be true if you pick a new arbitrary loss function that seems to fit the pre-transform representation better.
If you engineer a "correct" loss function such that the objective of the network remains unchanged, then the behaviour of the network will not change much - probably not at all. However, in some cases this can lead to more stable and/or faster training - it is often used for classifiers to avoid need to use exponentiation, see tf.nn.softmax_cross_entropy_with_logits in TensorFlow, which does exactly this.

Be careful to specify exactly what you mean by adjust weights and biases of the network so that the "pre-activated output" ... approaches ....
When training a neural network, one minimizes a loss function. This loss function is determines how important the deviation from 0.01396 is compared to the deviation of the other node from -0.00524. By transforming the target labels backwards you should also express the original loss function in terms of the back-transformed labels.

What one can do in some cases is to combine the input to the last layer's activation function with the loss function and algebraically simplify the resulting expression.
This concept is for example implemented in Tensorflow's tf.nn.sigmoid_cross_entropy_with_logits. This function can be used for the case of a single output with sigmoid activation function and binary cross-entropy loss (a similar function also exists for the case of multiple output nodes with a softmax activation function).
Instead of first passing the values through a sigmoid activation and then calculating the binary cross-entropy with respect to the target label, it combines the two expressions and uses an equivalent but simpler expression.
If you look at the documentation of this function, you'll see that the number of transcendental functions (which are computationally expensive to calculate) can be reduced from
loss(x,z) = z * -log(1 / (1 + exp(-x))) + (1 - z) * -log(exp(-x) / (1 + exp(-x)))

to
loss(x,z) = max(x, 0) - x * z + log(1 + exp(-abs(x)))

where x is the input to the (sigmoid) nonlinearity (corresponding to the output of the green nodes in your diagram), also called 'logits' and z is the target label.

So, based on this discourse, is it suitable or not?

it is not clear from the question whether the OP is transforming his loss function to account for the missing tanh nonlinearity or not -- as I mentioned above he did not write how he defines the new minimization goal before the last nonlinearity. If he keeps the same loss as at the output nodes, the method is finding a different type of minimum (and therefore the method is unsuitable). If he adapts the loss to account for the removed tanh nonlinearity then it depends if one gets some simplification in the new loss function or not. It is not clear what loss function he uses and why tanh .

I think your idea would work... fine, but I don't necessarily see any advantages to it. I haven't actually tried it (that'd be the best way for you to also see whether it works!), so I'm mostly going by first thoughts and intuition here.
Anyway, what you are essentially doing with your idea is "cutting off" the last layer of the Neural Network from the perspective of your learning algorithm (typically backpropagation). Whatever weights you have between the last hidden layer and the output layer will be fixed to their initial values. The last hidden layer can actually be viewed as an "output" layer, since you also have fixed targets that you want to converge towards for them.
Whether this makes your learning process better/faster/easier, or worse/slower/harder seems to be very much dependent on how the weights between your last hidden layer and your output layer are initialized. For example:

If those weights are initialized to all-zero, your "real" output layer is doomed to always predict zeros, so your problem becomes impossible to solve.
If those weights are initialized to implement the identity function, this becomes 100% equivalent to the case you would have if you'd simply cut off the last layer and train that in the traditional sense (i.e. you effectively have one layer less than you really do).
If those weights are initialized randomly, it looks to me like you have a post-processing step consisting of a random projection. Such a random projection may be beneficial for training (random projections can be useful for dimensionality reduction, or for, in combination with the subsequent non-linear function, turning an otherwise linear function into a non-linear function).

I don't think it'd very often be better than actually have an extra "real", trainable layer with a non-linear activation function though. I suspect such a "non-trainable" extra layer can sometimes be better than not having anything there, but I don't think it'd often be better than having a real, trainable layer.


text:
Learning an arbitrary function using a feedforward net
I would like to get a simple example running in matlab that will use a neural net to learn an arbitrary function from input output data (basically model identification) and then be able to approximate that function from just the input data. As means of training this net I have implemented a simple back propagation algorithm in matlab but I was not able to get anywhere close to satisfactory results. I would like to know what I may be doing wrong and also what approach I may use instead.
The goal is to have the network represent an identified function f(x) which takes a series x as input and outputs the learned mapping from x -> y.
Here is the GNU octave code I have so far:
pkg load control signal

function r = sigmoid(z)
    r = 1 ./ (1 + exp(-z));
end

function r = linear(z)
    r = z;
end

function r = grad_sigmoid(z)
    r = sigmoid(z) .* (1 - sigmoid(z));
end

function r = grad_linear(z)
    r = 1;
end

function r = grad_tanh(z)
    r = 1 - tanh(z) .^ 2;
end

function nn = nn_init(n_input, n_hidden1, n_hidden2, n_output)
    nn.W2 = (rand(n_input, n_hidden1) * 2 - 1)'
    nn.W3 = (rand(n_hidden1, n_hidden2) * 2 - 1)'
    nn.W4 = (rand(n_hidden2, n_output) * 2 - 1)'
    nn.lambda = 0.005;
end

function nn = nn_train(nn_in, state, action)
    nn = nn_in;

    [out, nn] = nn_eval(nn, state);

    d4 = (nn.a4 - action) .* grad_linear(nn.W4 * nn.a3);
    d3 = (nn.W4' * d4) .* grad_tanh(nn.W3 * nn.a2);
    d2 = (nn.W3' * d3) .* grad_tanh(nn.W2 * nn.a1);

    nn.W4 -= nn.lambda * (d4 * nn.a3');
    nn.W3 -= nn.lambda * (d3 * nn.a2');
    nn.W2 -= nn.lambda * (d2 * nn.a1');
end

function [out,nn] = nn_eval(nn_in, state)
    nn = nn_in;

    nn.z1 = state;
    nn.a1 = nn.z1;

    nn.a2 = tanh(nn.W2 * nn.a1);
    nn.a3 = tanh(nn.W3 * nn.a2);
    nn.a4 = linear(nn.W4 * nn.a3);

    out = nn.a4;
end

nn = nn_init(1, 100, 100, 1);
t = 1:0.1:3.14*10;
input = t;
output = sin(input);
learned = zeros(1, length(output));

for j = 1:500
    for i = 1:length(input)
        nn = nn_train(nn, [input(i)], [output(i)]);
    end
    j
end

for i = 1:length(input)
    learned(i) = nn_eval(nn, [input(i)]);
end

plot(t, output, 'g', t, learned, 'b');

pause

Here is the result:

The result is not even close to where I want it to be. Has it got something to do with my implementation of back propagation?
What changes do I need to do to the code to get a better approximation going?

You need to scale the input. Neural networks work best with a limited input domain, and train badly when it is exceeded.
For statistical data, you would typically scale your input to have mean 0, standard deviation 1.
Here, you will be better off fitting the input to roughly -1 to 1.
Up to you where you scale the values, but usually this is done outside of the NN code. So I would do something like:
nn_input = (t - 15)/15

And then use nn_input in the training and evaluation loops. As you are putting these directly into a sorted array for plotting, you won't need to do any further re-mapping back or maintain a conversion function. However, in the more common case of arbitrary inputs, you would need to store the conversion factors somewhere (in this case just hardcoded as a function perhaps) in order to make use of the trained NN.
Another thing that may help is shuffling your input/output data pairs during training to remove correlation between sequence of input pairs.


text:
Should the reward or the Q value be clipped for reinforcement learning
When extending reinforcement learning to the continuous states, continuous action case, we must use function approximators (linear or non-linear) to approximate the Q-value.  It is well known that non-linear function approximators, such as neural networks, diverge aggressively.  One way to help stabilize training is using reward clipping.  Because the temporal difference Q-update is a bootstrapping method (i.e., uses a previously calculated value to compute the current prediction), a very large previously calculated Q-value can make the current reward relatively minuscule, thus making the current reward not impact the Q-update, eventually leading the agent to diverge.
To avoid this, we can try to avoid the large Q-value in the first place by clipping the reward between [1, -1].
But I have seen some other people say that instead of clipping the reward itself, we can instead clip the Q-value between an interval.
I was wondering which method is better for convergence, and under what assumptions / circumstances.  I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better.

You could clip the TD Error, which has the benefit of not modifying the objective, just the size of update step that you can take at one time. That might in fact be related to what you are referring to - perhaps give a reference/link to the reward or q value clipping document(s) that you read, in order to clarify that.

I'll start with the last question in your post:

I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better.

I highly doubt there will be any such theoretical work. The problem is that these variants of clipping (clipping rewards and clipping $Q$ values) fundamentally modify the task / the original objective. Once you clip either of those things, you fundamentally change what your agent is trying to optimize for from what the original goal was. I don't think it's ever going to be possible to get any rigorous, theoretical proofs about which one would be better in general. You'd likely have to start out with some very strong assumptions on the reward structure in the original task to have any hope of proving anything here, but such strong assumptions make you lose generality.

Intuitively... I think reward clipping feels "safer" to me more often than clipping $Q$-values. Clipping $Q$-values seems more aggressive, it could be viewed as some combination of clipping rewards (if you clip $Q$-values to $[-1, 1]$, you're still at the very least also clipping all rewards to that range), but additionally also putting a constraint on how far in the future you're looking (in some sense). This whole argument is very handwavy though
I suppose, slightly less handwavy, you could say that reward clipping is definitely "better" (in the sense that you don't deviate as much from the original objective) in environments where rewards of similar magnitudes can be collected frequently. I struggle to really think of a situation where clipping $Q$-values would be a clear favorite based on intuition. I wouldn't be surprised if clipping $Q$-values may turn out to be better after empirical evaluation in some cases, but it's difficult to say where that would be. It will also very much depend on what range is chosen. Clipping rewards to a range of $[-1, 1]$ is very different from clipping $Q$-values to the same range.

Thanks for the answer Dennis.  Yes, the clipping Q values would indeed affect the "prediction horizon" for RL.  That is a great point!  I will probably try both and see which one is more interesting, but for Q clipping, I will try to clip it at a higher interval to not affect the prediction horizon.


text:
Would this work to prevent forgetting: train a neural net with N nodes. Then, add more nodes and stop training the original nodes
Would this work at all?
Idea is to start training a neural net with some number of nodes. Then, add some new nodes and more layers and start training only the new nodes (or only modifying the old nodes very slightly). Ideally, we would connect all old nodes to the new layer added since we might have learned many useful things in the hidden layers. Then repeat this many times.
Intuition is that if the old nodes give bad information the new layer of nodes will weight the activations of old nodes close to zero and learn new/better concepts in the new nodes. The benefit is that we will keep old knowledge forever.
Caveat is that the network can still temporarily "forget" concepts if a new layer weights old information close to zero, but it can potentially remember it again too.
If this completely fails, I'm curious if there's some known way to prevent a neural network from forgetting concepts it learned.

Could you clarify what kind of forgetting scenario you hope this will address? The problem usually occurs in online learning due to temporal sampling bias or population drift. At what point would you add new nodes/layers?

The scenario is that when training a network, you might end up with catastrophic forgetting. That is, if you haven't seen an input in a while, the changes in node weights might make it forget how to recognize the input.

There's a big difference between addressing this issue in offline and online training, or in Reinforcement Learning. I most often see the term "catastrophic forgetting" in RL, is that what you are asking about?

I'm referring to forgetting in the following sense: If you train a neural network to recognize digits from 0-9, and if you show it no examples of the number 5 for a very long time (but all the other digits), it will forget how to recognize it as the neural network weights change.

This is getting a little circular: I am asking about the higher-level scenario, not the data flow. I.e. "How does it occur that you have a training scenario where relevant data is not available?". For an offline supervised learning on MNIST using the data set, this is not going to happen, and is also trivial to fix if it is made to occur artificially, as there are no real consequences to a dip in performance. To analyse the consequences of your idea requires understanding the context in which it will be used.

Per Neil's line of questioning:

If you're doing offline, supervised or reinforcement learning, where you have the dataset and decide what to show to the network next, there's no reason to do this. It's better to just adjust your training to show rare examples to the network more often.
If you're doing online reinforcement learning, then the agent typically controls which examples it sees next through its choice of actions. Increasing the agent's exploration rate will cause it to see a more diverse set of examples, but these examples are also less likely to be useful for solving the task the agent is working on.
If you're doing supervised, online, learning, and you don't control the order in which examples appear, it might be useful to freeze some of the weights, but it might be better, and would certainly be simpler, to increase the learning rate for rare classes instead (in effect, showing the network a given rare example repeatedly when it does show up, so that it takes longer to forget it).


text:
Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?
I am reviewing a statement on the website for ES regarding structured exploration.
https://blog.openai.com/evolution-strategies/

Structured exploration. Some RL algorithms (especially policy
  gradients) initialize with random policies, which often manifests as
  random jitter on spot for a long time. This effect is mitigated in
  Q-Learning due to epsilon-greedy policies, where the max operation can
  cause the agents to perform some consistent action for a while (e.g.
  holding down a left arrow). This is more likely to do something in a
  game than if the agent jitters on spot, as is the case with policy
  gradients. Similar to Q-learning, ES does not suffer from these
  problems because we can use deterministic policies and achieve
  consistent exploration.

Where can I find sources showing that policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies?
Also, what does "max operation" have to do with epsilon-greedy policies?

@NeilSlater I at least need an explanation that is relevant to the question. How do policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies? Also, what does "max operation" have to do with epsilon-greedy policies? Thank you

Where can I find sources showing that policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies?

You can find example algorithms for Q learning and policy gradients in Sutton & Barto's Reinforcement Learning: An Introduction - Q learning is in chapter 6, and policy gradients explained in chapter 13.
Neither of these things are strictly true in all cases. However, both are very common situations for the two kinds of learning agent:

Policy gradient solvers learn a policy function $\pi(a|s)$ that gives the probability of taking action $a$ given observed state $s$. Typically this is implemented as a neural network. Neural networks are initialised randomly. For discrete action selection using softmax output layer, the initial function will roughly be choosing evenly from all possible actions. So if some actions oppose and undo each other, e.g. move left and move right are options - then the situation as described in your quote can easily happen. Nowadays there are many other solutions using policy gradients that don't suffer as much with this effect. For instance, deterministic policy gradient methods such as A2C or DDPG.
For Q learning, and many variants of it, $\epsilon$-greedy is a very commonly used action selection mechanism. It is convenient and simple, and allows a simple parameter to control balance between exploration and exploitation whilst learning. However, Q learning can work with any action selection mechanism that has some possibility of acting optimally at least occasionally.

The best approach to action selection during the learning process in both policy-based methods and value-based methods is an active area of research. So if you read the RL literature you may find a lot of variation. The blog you quoted from has identified two quite common choices.

Also, what does "max operation" have to do with epsilon-greedy policies?

The "max operation" is finding the maximum value of some function produced when varying a parameter. The related "argmax operation" is finding the value of the parameter that produces the maximum value. Q learning can use both types of operation, but specifically uses $\text{argmax}$ for $\epsilon$-greedy action selection.
An $\epsilon$-greedy policy requires acting optimally (according to current estimates of value function) with probability $p=1-\epsilon$, and randomly with equal probability of each action with probability $p=\epsilon$
In order to do this, the algorithm usually (with probability $p=1-\epsilon$) needs to know what the current best guess at an optimal solution is. That is the greedy action with respect to the current Q values i.e. $\text{argmax}_a Q(s,a)$  -


text:
How to recognize non-circular radial symmetry in images?
This is a question about pattern recognition and feature extraction.
I am familiar with Hough transforms, the Fast Radial Transform and variants (e.g., GFRS), but these highlight circles, spheres, etc.
I need an image filter that will highlight the centroid of a series of spokes radiating from it, such as the center of a asterix or the spokes of a bicycle wheel (even if the round wheel is obscured.  Does such a filter exist?

What do you have in hand? Single object of interest or a complete scene? Binary thresholded image or a picture? If it is a binary image of a single shape, the answer would be extremely simple.

It would generally be an unthresholded image with a complex background.  Another example would be a dandelion in grass.  The edges on the grass lie in more or less random directions, while the dandelion leaves radiate out from a center.

For the dandelion case, it is easy to segment out the broad part of the leaves, but the center area isn't similar.

The Hough Transform extended to orthogonal ellipses uses this model, accumulating on $\theta$ for all $\{x, y\}$ with parameter matrix
\begin{Bmatrix}
c_x & c_y \\
r_x & r_y
\end{Bmatrix}
where
$$1 = \dfrac {(x - c_x) \, \cos \theta} {r_x} + \dfrac {(y - c_y) \, \sin \theta} {r_y}$$
The question is looking to detect the normal lines, so any of the several algorithms for the above model can be modified to accumulate on $r$ for all $\{x, y\}$ with parameter matrix
\begin{Bmatrix}
c_x & c_y  \\
r_x & r_y
\end{Bmatrix}
where
$$0 = \dfrac {x - c_x} {r_x} + \dfrac {y - c_y} {r_y}$$
Lines that intersect $(c_x, c_y)$ don't rely on $r_x$ or $r_y$.  However, it may be useful to recognize that, if radially equally spaced, viewing the lines from a position other than one that projects into the plane of the lines at $(c_x, c_y)$ will present a line density that is a function of $\arctan (r_x, r_y)$.

Yes, I soon realized that I can just modify the hough transform to remove the orthogonality (or rather, add another) before accumulation. I will post an update when I have a proof -of-concept coded. Thanks!

First step would be getting the object out of the scene. This bit is not trivial in your case, however, there are many methods to choose from. I suggest reading about watershed threshold algorithm.
Second part is easier. Once you have a single segmented object at hand, perform noise removal. Next step is to extract the contours. Find the center of gravity, transform coordinates to polar and represent these contours as a function which has the x axis as degrees and y axis as distance from the center. Take the Fourier transform of this function. If the shape is symmetrical, there will be few non-zero entries, and a large spike in the spectrum.


text:
How to combine input from different types of data sources?
I've to train a neural network using microphone data (wav files), accelerometer sensor data and light sensor data.
Right now the approach I thought was to convert all data into images and combine them into a single image and train my neural network.
Another approach was to convert wav files into arrays and combine them along with sensor data and train my neural net.
Are my approaches correct or is there a better way to do this?
Any suggestions/ideas are welcome.

Handling Multiple Input Types
Multiple input types are common as learning technology moves from academic labs and open source examples into the real world.  The design process includes the below four steps, which related directly to this question.  There are other steps that don't centrally related not mentioned here.
First Step — List Inputs in Most Usable Form
In the case of this question, there are three in the list.

Monophonic or multichannel audio data as an array of samples or sample vectors acquired at a fixed sampling rate
Acceleration data as an array of samples in $\mathbb{R}^n$ where $n \in \{1, 2, 3\}$ acquired at a fixed sample rate.
Visible light or IR detection data as an array of samples of $c$ channels at a fixed sample rate.

Fixed sample rate is listed with each type.  Variable sample rates pose additional burden on learning.  If any of the three are acquired with a variable sample rate, they can be converted to fixed sample rate arrays provided there is a time stamp or a time-since-last-sample channel available with each sample vector.
Variable to fixed sample rate conversion can be accomplished easily via ffmpeg, programmatically via its library, or through any equivalent software designed for this purpose.  The acceleration and visible light signals can also be adapted for ffmpeg's input.
When a specific dynamic characteristic must be preserved, finer control over the interpolation can be obtained using cubic spline algorithms.  Such algorithms were originally written in FORTRAN or C and have been ported to Java and Python.
Second Step — When Spectral Data is Practically Required
With time series data, the phenomena being learned is often largely a function of vibration frequencies and amplitudes.  This is almost always the case with audio data.  The exception is when transient detection is the primary teller of the conditions being measured rather than more continuous sounds.
A note is spectral in description.  A knock is a transient and both spectra and actual rise characteristics are indicative of source.
Presenting digital samples from the transducer circuit (in this case the microphone diaphragm's position in response to pressure waves) to the learning system directly is not a best practice.  Unconverted vibration information forces the learning system to learn transformations for which optimized algorithms already exist.  This slows learning and generally pushes complexity beyond the learning system's capacity.
When the spectral features best characterize the phenomena being learned, as in hums, screeches, vowel sounds, music, horns, or beeps, the best practice is to convert the samples in the time domain to frequency domain data.
This is usually done using one of the forms of the FFT (fast Fourier transform) algorithm in combination with a RMS (root mean square) calculation of amplitude and windowing.  Windowing attenuation forms, such as the Hamming window or the cosine window, are available in most packages that have FFT capabilities.
The sample rate is also important.  Nyquist's sample rate criteria must be met by the A-to-D converter (in the sound card or instrumentation circuitry) to ensure that the spectral range includes the best indicators of the phenomena to be learned.  In machine learning, such indicators are called features.
In most cases, audio or vibration information is presented to a learning system or sub-system as spectral frequencies and amplitudes.  Vibration recognition systems and speech to text systems generally execute FFTs (with RMS and windowing) on the raw input.  In human ears, the conversion to spectral information is mechanical, using the tapering geometry of the ear's cochlea.
There also exists algorithms that can directly develop spectral information from variable sample rate data to amplitudes and frequencies, combining two transformations into one for efficiency of computing resources and additional input accuracy.
In summary of this section, whether light, IR, acceleration, or other data is treated as audio and converted to spectral information is based on whether

The signals are vibrations that have rises and falls within specific frequency ranges like hums, notes, and speech or
They are transients (as with tapping the mechanical package on which the accelerometer is mounted) or
They are binary in nature (as in the case of linear encoder signals).

Third Step — When Topology of Types
The topological orientation of the input source types to one another is key to feeding the information into any learning system.  These are the two pure extremes.

The signals from the source types are describing the same physical or data mining property.
The signals are measuring independent properties that together describe the state of the system being measured.

In the case of (1), a normalized sample from one type of input channel is, from a learning point of view, indistinguishable from a normalized sample from another.  This would be the case if the accelerometer is measuring the same vibration that the microphone is measuring.  This is the less usual case, but is more common when the data is for study of vibration to project mechanical or materials failures in advance.
With Background Clarified, the Q&A Becomes Clearer
The question asked this.

How [does one best] combine input from different types of data sources?

Whether the source data is case (1) or case (2) or some midpoint between them is key to answering this question.  Samples from different source types that measure single phenomena can be phase adjusted, spectrally adjusted, and otherwise normalized to present a single data type at the input.  Merging into a single serial data stream would be the optimal pre-processing for training in case (1).
More often, the topological orientation of the different input source types conforms to case (2) above.  In that case, different input types should be presented in parallel to the learning system inputs.  The merging would be counterproductive in this case, for a reason that will be clarified in the next section below.
To avoid sparsity in input data, which creates additional training burden, the sample rates should be equalized for all channels of all data source types.  This may require conversion of the sample rate, which can be done using the same techniques (above) for converting from variable to fixed sample rates.
Images of Data
The question stated this.

Right now, the approach I [was considering is] to convert all data into images and combine them into a single image and train my neural network.

The attempt to normalize and combine into one input stream is the correct approach if (1) above is the case.  However, presenting data as images requires the learning system restore the data to a less redundant, more feature encoded form, so conversion to images is not recommended.
If (2) above is the case, the provision of unique and property-independent data through the same input tensors requires the artificial network sort the input samples by type, which can be done with recurrent or attention based artificial networks, but it is unnecessary.  Here's the rule.

If the signals from different sources are measuring independent properties to which learning will be applied, do not merge the input streams.  Instead, present them in parallel to the learning system so that the learning system does not need to also learn how to discriminate sample types.

The reason topology is mentioned in the subtitle of this section is that there is a third case, (3), which is a hybrid of (1) and (2) as hinted above.
Many excellent detection and control systems process the input channels in relation to one another prior to the actual learning system components.  This pre-processing takes advantage of known relationships in the data and the topology of signals in phase space.
The phase space of two or three signals can be visualized in 2D or 3D where each channel is assigned to a Cartesian coordinate system's axis and the time domain is not represented by any axis.  Mechanical fault detection can be optimized using phase space analysis.
An example of this more complex case (3) is in the human recognition of both consonant and vowel sounds and various other non-speech audio phenomena.  After the cochlear conversion to spectra, the transient information (as from a 'k' sound or a click) and tone information (as from an 'e' or a whistle) is presented to speech recognition nets in the brain in an integrated form.
This decreases the learning time newborns take to learn voice and household sounds and probably improved evolutionary viability in times of global stress or local predatory activity.
In automated vehicles and smart robotics, the same is true.  Some sounds, accelerations, and visible light changes are transient and some are vibrations.  Studying the topology of the surface created by the input signal types and their spectra in phase space provides much information.  Armed with the results of such studies allows excellence in the preparation of input to a learning system.
Fourth Step — Normalizing and Standardizing
Once the above is complete, understanding how to present the data to the input layer of an artificial network becomes clearer.  The goal of this design step is to have relatively equal ranges for the components of each input tensor, whether presented serially or in parallel depending on the above criteria.
Doing so reduces the burden on the learning system.  All signals are presented at full strength but within the clipping range of the signal.  This is a general principle and is important for both data center (batch) learning and real time learning and adaptive systems.


text:
Best way to create an image dataset for CNN
I am creating a dataset made of many images which are created by preprocessing a long time series. Each image is an array of (128,128) and the there are four classes. I would like to build a dataset similar to the MNIST in scikit-learn.database  but I have no idea how to do it.
My aim is to have something that I can call like this:
(x_train, y_train), (x_test, y_test) = my_data()

Should I save them as figures? or as csv?
Which is the best way to implement this?

I found this solution and will be happy about any improvements or suggestions:
First, I create a random dataset of images, which are 28x28 pixels, and corresponding random labels (just for sake of clarification, I have another image dataset, this is just for explaining). Then I use a sklearn module for splitting the data:
import numpy as np
from sklearn.model_selection import train_test_split

# create the data and labels
def pixel_dataset(n_data=10, dpi = 28):

    ary = np.zeros((n_data, dpi, dpi))
    label = []

    for i in np.arange(n_data):
        labels =['a', 'b', 'c']
        ary[i,::] = np.reshape(np.array(np.random.randint(0, 255, 28*28)),(28,28))
        label.append(np.random.choice(labels, 1)[0])

    return ary, label

# create the test train split
def mydata(test_size=0.3):

    X, y = pixel_dataset()
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = test_size)
    return (X_train, y_train), (X_test, y_test)


text:
Reason for issues with correlation in the dataset in DQN
From the paper Human level Control through DeepRL,  the correlation in the data causes instability in the network and may causes the network to diverge. I wanted to understand what does this instability and divergence mean ? And why correlated data causes this instability.

I wanted to understand what does this instability and divergence mean ?

These are with reference to learning curves for the neural network. If a neural network is stable and converges, it means that the value of the error or cost function reduces consistently over time and reaches a stable point of minimum error.
In practice this is often a noisy process and not smooth. There are degrees of noise that are expected and acceptable when solving real problems. However, an instable learning curve will oscillate wildly, and a divergent learning curve will get consistently worse error function during training.
A typical stable, converging learning curve, cost function versus training data consumed, might look like this:

Whilst an instable, diverging learning curve might look like this:

These are not using the same vertical scale - the lowest point of the unstable curve will typically be higher than most or even all of the stable curve.

And why correlated data causes this instability.

This is because for gradient descent to work, it needs gradient samples that it uses on every weight update step to be unbiased estimates of a true gradient. In RL you have either an online learning process or non-stationary targets (and usually both), so you must use stochastic or mini-batch gradient descent, working with a few samples at a time. You need those samples to be independent, not related to each other other than by random chance, otherwise the gradient value will be biased and gradient descent will consistently make updates in the wrong overall direction.
A good way to illustrate the difference is to use a really simple example, using gradient descent updates to estimate a mean value (this is roughly equivalent to training a neural network with one neuron, with weight fixed to zero and learning a bias value to represent the mean of the target- no actual input is required).
Say we have an array of values from 0 to 200 inclusive (example code is in Python):
import numpy as np
train_y = np.arange(0, 201)

If this array is kept sorted, then sequential values are highly correlated. If you plot consecutive pairs against each other, you will get a straight line.
We can estimate the mean value by setting a "bias" value to some arbitrary number, and running an update rule based on MSE (between current bias and observed value):
mean_estimate = 0.0
alpha = 0.1
for y in train_y:
    mean_estimate += alpha * (y - mean_estimate)
print(mean_estimate)

This prints (roughly) $191$ as the estimate for the mean, almost double.
However, if we shuffle the array first, it removes the correlation. If you plot consecutive pairs against each other, you will get a scatter graph with no apparent pattern. Adding the one line np.random.shuffle(x) to do this changes the results radically:
mean_estimate = 0.0
alpha = 0.1
np.random.shuffle(x)
for y in train_y:
    mean_estimate += alpha * (y - mean_estimate)
print(mean_estimate)

We get much better estimates (typically between $90$ and $110$), closer to the true value of $100$, and not biased to be higher or lower (run it enough times and you would find the expected result of this algorithm is very close to the true value).
This is because in the first version of the code, the gradient of the error was not sampled fairly - it kept pointing "up" even for high estimates due to the correlation. In the shuffled version, gradients are likely to be in either direction depending only on the current estimate, and will appear roughly in the ratios necessary to find the correct value.
As an exercise you could extend this simple example with mini-batches and repeated "epochs" to show that the effect persists with those changes, and the shuffling is the most important change here for better estimates.

Thank you for the detailed answer. "The bias in gradient", "fair sampling", "Shoots up in one direction" were the key take aways. Again, appreciate your time.


text:
How to overcome overfitting to single player styles in reinforcement learning?
I am implementing an actor-critic reinforcement learning algorithm for winning a two player tic-tac-toe like game. The agent is trained against a min-max player and after a number of episodes is able to learn a set of rules which lead it to winning a good majority of games.
However, as soon as I play against the trained agent by using even a slightly different playing style, it looses miserably. In other words, it is evident the agent overfitted with respect to the deterministic behaviour of the min-max player. It is clear to me what are the roots of the problem, but I would like to get an overview of the different methodologies which can be applied to overcome (or mitigate) this issue.
The two solutions I would like to try are the following:
1. Training the agent with different opponents for fixed amounts of episodes (or time) each. So that for example I train the agent by using a depth 2 min-max player for the first 10000 episodes, then I use a random playing agent for the next 10000 episodes, then I use a depth 4 min-max player for other 10000 episodes and repeat the process.
2. Starting episodes from different initial configurations. In this way the agents will play a much wider set of sampled games and will be more difficult for the agent to overfit.
Are these two reasonable approaches? Are there other tricks/good practices to try out?

Both of the solutions you suggest seem to be built around the intuition that it's good to ensure that there is sufficient variety in the experiences that you provide to your RL algorithm.
That intuition is good, but it should not come at (too much of) a cost in playing strength of the opponent. I'm afraid that your first solution may break down because of this; Tic Tac Toe is such a simple game, any agent that doesn't play optimally can really be viewed as a very poor agent... and I think minimax agents with very low limits on search depth may end up playing suboptimally. Your second solution seems better in this regard, that could help.
For this particular case of Tic Tac Toe, I suspect you should be able to train just fine against only optimal minimax agents, as long as you make sure that those optimal minimax agents break ties randomly. In some situations, there may be multiple different actions that are all "equally optimal". Then, you'll want to make sure that your minimax agents break ties randomly, rather than always deterministically picking the same action. This can be done, for example, by making sure to always shuffle your lists of legal moves in minimax after they are generated.

Thank you for your reply. Excluding these two approaches do you have any other hint?

@aprospero Hmm nope don't think I got any other ideas now. By far the most common approach is still randomization of opponents selected for training - but, importantly, not complete randomization. Typically, in a self-play learning setting, they randomly sample one agent from a set of relatively recent "checkpoints" (the assumption being that all those relatively recent agents are still somewhat similar in strength, not going all the way back to agents in the earliest stages of a learning process).


text:
Reinforcement learning objective as conditional expectations
In one of his lectures Levine describes the objective of reinforcement learning as: $$J(\tau) = E_{\tau\sim p_\theta(\tau)}[r(\tau)]$$
where $\tau$ refers to a single trajectory and $p_\theta(\tau)$ is the probability of having taken that trajectory so that $p_\theta(\tau) = p(s_1)\prod_{t = I}^T \pi_{\theta}(a_t, s_t)p(s_{t+1}|s_t, a_t))$.
Starting from this definition, he writes the objective as $J(\tau) =\sum_{t=1}^T E_{(s_t, a_t)\sim p_\theta(\tau)}[r(s_t, a_t)]$ and argues that this sum can be decomposed by using conditional expectations, so that it becomes:
$$J(\tau) = E_{s_1 \sim p(s_1)}[E_{a_1 \sim \pi(a_1|s_1)}[r(s_1, a_1) + E_{s_2 \sim p(s_2|s_1, a_1)}[E_{a_2 \sim \pi(a_2|s_2)}[r(s_2, a_2)] + ...|s_2]|s_1,a_1]|s_1]]$$
Can anyone explain this last step? I guess the law of total expectation is involved, but I can not figure out how exactly.

I tried to sketch a mathematical justification of the equality. So we have:
$$J(\tau) = E_{\tau\sim p_\theta(\tau)}[r(\tau)]$$ where $p_\theta(\tau) = p(s_1)\prod_{t = 1}^T \pi_{\theta}(a_t, s_t)p(s_{t+1}|s_t, a_t))$.
Now, $p_\theta(\tau)$ can be re-written in terms of the Markov Chain transition probabilities, namely:
$p_\theta(\tau)= \prod_{t = 1}^T \pi_{\theta}(a_t, s_t)p(s_{t}|s_{t-1}, a_{t-1})) = \prod_{t = 1}^T p(s_{t}, a_{t})$.
Here we focus on T = 2 (I guess it is not difficult to prove with a general T, maybe by induction over T). The following holds:
$$J(\tau) = E_{\tau\sim p_\theta(\tau)}[r(\tau)] = E_{(s_1, a_1, s_2, a_2)\sim p(s_1,a_1)p(s_2,a_2)}[r(s_1,a_1) + r(s_2,a_2)] \\
= \int_{(s_1,a_1)} \int_{(s_2,a_2)} (r(s_1,a_1) + r(s_2,a_2))p(s_1,a_1)p(s_2,a_2) d(s_1,a_1)d(s_2, a_2) \\
= \int_{(s_1,a_1)} \left( \int_{(s_2,a_2)} (r(s_1,a_1)  + r(s_2,a_2))p(s_2,a_2) d(s_2,a_2)\right)p(s_1,a_1)d(s_1, a_1) \\
= E_{(s_1, a_1)\sim p(s_1,a_1)}\left[E_{(s_2, a_2)\sim p(s_2,a_2)}[r(s_1,a_1) + r(s_2,a_2)]\right] \\
= E_{(s_1, a_1)\sim p(s_1,a_1)} \left[E_{(s_2, a_2)\sim p(s_2,a_2)}[r(s_1,a_1)] + E_{(s_2, a_2)\sim p(s_2,a_2)}[r(s_2,a_2)] \right] \\
= E_{(s_1, a_1)\sim p(s_1,a_1)} \left[ r(s_1,a_1) + E_{(s_2, a_2)\sim p(s_2,a_2)}[r(s_2,a_2)] \right] $$
where the second line is due to the definition of the expectation, in the third line we just changed the order of the terms, in the fourth we used the definition of expectation, in the fifth we used the linearity property of the expectation and the last line is because $r(s_1, a_1)$ is constant with respect to the integration over $(s_2, a_2)$. I preferred to move to the integral form of the expectation in line two and three because it was not clear to me how exactly the expectation could factorise.
To conclude the justification, note that we defined $p(s_t, a_t) = \pi_{\theta}(a_t, s_t)p(s_{t}|s_{t-1}, a_{t-1})$, so we have another product here which can be factorised in the same way we did in line two and three of the set of equations above. This brings us the to conditional expectation form of the objective.
In general, every term we have in the original trajectory distribution $p_\theta(\tau)$ corresponds to an expectation and every reward can be moved up to the first outer expectation which it depends on.

In the YouTube depiction of CS294-112 fall 2017 lecture 3 Reinforcement Learning, Levine, the transition of the finite horizon expected reward to a form where each transition is decoupled from the entire Markov chain of state-action marginals is explained between $t_{video}$ = 44:04 and $t_{video}$ = 45:22.
At t=44:29, the probabilistic expectation where no actions take place is given as a starting point.  It's a distribution already known, the initial state distribution.
The statement, "Inside that expectation ...," at $t_{video}$ = 44:39, begins the description of how to build an expectation expression for the first two vertices of the Markov chain.  This is the beginning of a process where expectations $E$ are factored recursively.  The purpose of the expansions is to decouple the expectation at each vertex from those of other vertices in the Markov chain, clarifying dependencies and exposing the mathematical features of the recursion.
The right-hand factor is the integrated expectation of reward over the achievement of the next state-action marginal, where the action is distributed according to the fixed policy.
How a state and action pair fits into a state-action marginals which can then form a Marcov Chain, provided the policy is fixed and known, was explained at $t_{video}$ = 21:40 in this same lecture.  Each vertex in the chain is taken to be the aggregation of a state with the proposed action to be applied to it, leading to a reward $r(s, a)$.
Returning to $t_{video}$ = 44:39, notice the plus sign without the right-hand addend, waiting for another recursion.  When the recursion is expressed later in the process, the right-hand factor becomes the integrated expectation of reward over the achievement of the finite sequence represented by the remainder of the Markov chain.
Multiplication is implied because we are now conditioning on both probabilities of the initial state and a transition from it.  Notice also that the second vertex in the Marcov Chain is a state-action marginal.
Notice now that rewards are additive because the goal of learning was defined as a sum of products, which is a simplifying assumption.
Also notice that each expectation distributions of reward after the first one can be expressed as a product of single-vertex expectation distributions.  Each factor is a localized expectation along the sequence of vertices in the Markov chain of state-action marginals.  This is the factoring of expectation distributions.
This factoring is the application of Product distribution, which can be proven using the "law" of total expectations.
At $t_{video}$ = 44:51, the pattern recursion is expressed with a single ellipsis.  Recognition of the recursion pattern can be facilitated by re-expressing with a few modifications of mathematical style.

Use of the original variable to represent the value objective
Extending the equality
Using two ellipses instead of one clarify the recursion is a continuation of conditions from right to left as well as continuation of rewards from left to right
Strategically placing line returns

$$\pi_{\theta}(\tau) = \sum_{t=1}^T \operatorname{E}_{(s_t, a_t)\sim p_\theta(\tau)}[r(s_t, a_t)] \\ = E_{s_1 \sim p(s_1)}[E_{a_1 \sim \pi(a_1|s_1)}[r(s_1, a_1) \\ + E_{s_2 \sim p(s_2|s_1, a_1)}[E_{a_2 \sim \pi(a_2|s_2)}[r(s_2, a_2)] \\ + \; ... \\ ...|s_2]|s_1,a_1]|s_1]]$$
Inverting the transformation, recombining the expansion back to its original form may also be illustrative.