text:
Understanding an execution of the Monte Carlo tree search algorithm
I have the execution of the Monte Carlo Tree Search (MCTS) below. I need to expand it, but I don't understand steps 1 and 2.
Why does it go to the first node and then make a new node, instead of going to the deepest left leaf? I thought it needs to go to the most probable leaf.

In step 1 of MCTS, it added a new node. Now, there are 9 cases.

In step 2 of MCTS, it added a new node. Now, there are 2 cases under the first node:


text:
How do I model the blocked N queens problem as a search problem?
The blocked N-queens is a variant of the N-queens problem. In the blocked N-queens problem, we also have a NxN chess board and N queens. Each square can hold at most one queen. Some squares on the board are blocked and can not hold any queens. Conditionality is that queens do not dare to attack each other. At the entrance to this problem are the queues and blocked areas.
How do I model the blocked N queens problem as a search problem, so that I can apply search algorithms like BFS?

In general, the process of modelling a problem as a search problem consists in creating a graph which contains nodes, which represent the possible states in your problem, and edges, which represent the relations between these states (that is, you will have an edge between nodes $A$ and $B$ if it is possible to go from state $A$ to state $B$, and vice-versa, in your problem).
In the case of the 8 (or, in general, N) queens problem (or blocked 8 queens problem), the states are all possible configurations of the board (that is, all possible combinations of the positions of all the 8 queens). In the graph that represents this problem, you will have a node for all possible valid configurations, where a valid configuration is one that is allowed by the rules of the game. You will also have an edge between configurations (or states) $A$ and $B$ if you can go from state $A$ to state $B$. For example, configurations where you have two queens on the same square or where you have a queen in a blocked area will not be represented in your graph as a node (or state).
In the case of the 8 queens problem, you actually have more than one solution, so you will have more than one goal state (or configuration of the board).
When using the BFS, the goal will then be to find a path from the initial configuration of the queens (that is, the initial state) to one of these 18 (or 12) goal states.
In the case of the 8 queens problem, there are 4,426,165,368 possible configurations (so you will likely not be able to draw this graph).

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What are daily life examples of SAT problems?
What are (good) daily life examples of SAT problems?
I've thought about this one. The problem of placing a bunch of different kinds of glasses in a shared cabinet in such a way that some constraints would be satisfied, such as putting the longer ones in the back of the shorter ones, so it will be easier to take them when we need.
Is it a good one, or can you think of any other better one?

SAT problems are decision problems that can be categorised as NPC. This informally means although there has not been any solution that can solve these problem in the polynomial order, the solutions of such problems can be satisfied in $O(n^c)$.
About your question, first, you should see your problem has exponential space and cannot be solved in polynomial order; after that, you have to investigate whether its solutions can be satisfied in polynomial time or not. An easy way to model this is to consider the boolean table and an expression to satisfy. Your problem can be considered as the boolean equation and different possibilities of variables can be possible solutions that you may want to verify.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How do we know if GPT-2 is a better language model?
You may have heard of GPT2, a new language model. It has recently attracted attention from the general public as the foundation that published the paper, OpenAI, ironically refused to share the whole model fearing dangerous implications. Along the paper, they also published a manifesto to justify their choice: "Better Language Models and Their Implications". And soon a lot of media were publishing articles discussing the choice and its effectiveness to actually prevent bad implications. I am not here to discuss the ethical components of this choice but the actual performance of the model.
The model got my attention too and I downloaded the small model to play with. To be honest I am far from impressed by the results. Some times the first paragraph of the produced text appears to make sense, but nine times out of ten it is giberish by the first or the second sentence. Exemples given in the paper seems to be "Lucky" outputs, cherry picked by human hands. Overall, the paper may suffer from a very strong publication bias.
However, most article we can read on the internet seems to take its powerfulness for granted. The MIT technology review wrote:

The language model can write like a human

The Guardian wrote

When used to simply generate new text, GPT2 is capable of writing plausible passages that match what it is given in both style and subject. It rarely shows any of the quirks that mark out previous AI systems, such as forgetting what it is writing about midway through a paragraph, or mangling the syntax of long sentences.

The model appears generally qualified as a "breakthrough". These writings do not match my personal experimentation as produced texts are rarely consistent / syntactically correct.
My question is: without the release of the whole model for ethical reasons, how do we know if the model is really that powerful?

You have experimented with a small model (117M parameters). OpenAI has now released the medium sized model (345M parameters). Note that the full model has 1.5B parameters. Also note that GPT-2 has been trained on a tiny fraction of all available text. It's almost guaranteed that a larger model trained on more text will generate better text.
I have experimented with the 345M model, and I agree with The Guardian assessment. See for yourself:
Prompt:

I want to generate beautiful music using machine learning. Do you
  think a computer will ever be able to compose something as novel as
  creations of Mozart, Bach, or Chopin?

Results (first four samples, not cherry picked):

Yes, computers will probably be able to do it by 2020. I think the key
  for us will be to make music that is interesting and beautiful, so we
  will never really know. My hope would be that computers will become
  far more efficient over time – something that is already happening in
  a lot of ways like in big data – while still being able to do more
  things that they are trained on, like find patterns. I think that's
  where I'd want to use neural networks in the future.

We don't know, but it will probably happen. That's an interesting
  question to ask. Of course we have AI, which can do interesting things
  without human input. It can recognize the world, find objects and
  images, it understands language, we can make games for it. But if you
  imagine an AI being able to write music, will it even be good enough?
  How much better than human players? How much better can someone else
  do? AI will never really be at its best because it will always have to
  come up with creative ways to use and play it. To ask a question like
  it is good enough is to ignore that it is human. You are only allowed
  to ask good questions

The software will help, as it already can do. But it will also allow
  more human creators, which is what I think it should do more of to
  bring music to any genre. In terms of composing, it's a matter of
  creativity in humans in general. But how does it achieve that? Let's
  just say that there are fewer humans composing today than ever before.
  Do you remember the day when, by chance, a computer could write lyrics
  to Mozart's Sonata No. 4, or the day when a human wrote lyrics to
  Mozart's La Résistance du Ciel No. 2? That was back in 1960, and it's
  just not true anymore. As we move forward, we are not going to create
  music that human musicians cannot compose. What is your goal as a
  computer music creator?

At the core of this idea is that music is a collection of sounds – it
  was the very first art form, it was the first music that humans
  evolved, a collection of sounds. Music is not an abstract thing of
  sound, but a collection of sounds. You can't write music the way we
  write and compose books and films, because we write and compose with
  sounds in mind. This is where machine learning comes in. You can read
  some books about how to get better music writers or how to improve the
  quality of their music, but you cannot make a difference with this
  kind of information at all. What does music teach you? To write and
  perform beautifully, to convey emotion accurately, and to create
  lasting and rich relationships with

4 random exemples doesn't constitute a representative metric.

We don't have any good metrics to evaluate the quality of generated text. If you read papers on the topic you will see that human evaluation remains the best indicator of quality.


text:
What are the differences between learning by analogy, inductive learning and explanation based learning?
I have heard of the concepts of learning by analogy (which is quite self-explanatory), inductive learning and explanation-based learning. I tried to learn about inductive learning and explanation-based learning, but I don't understand them.
How would you explain all these three concepts? What are the differences between them?
A link to some explanatory article/notes/blog post are appreciated too.

Classical machine learning seem to fall into  inductive learning. While explanation learning looks like a mixture of the process used in NLP with absurly large data augmentation


text:
How can we create a vector space where word spelling and pronunciation can be easily compared?
In natural language processing, we can convert words to vectors (or word embeddings). In this vector space, we can measure the similarity between these word embeddings.
How can we create a vector space where word spelling and pronunciation can be easily compared? For example, "apple" and "ape", "start" and "startle" are very similar, so they should also be similar in this new vector space.
I am eventually looking for a library that can do this out of the box. I would like to avoid implementing this myself.

If you only need the vector space as a way to obtain a similarity measure, you may want to consider a distance measure instead. Similarity and distance are inversely related: identical words have maximum similarity or zero distance, and as the similarity decreases, the distance increases.
For instance, the Wagner-Fischer algorithm computes the edit distance between two strings of characters. This edit distance takes into acccount insertions and deletions, as in your examples, but also substitutions (for example "gray" vs. "grey").
The article linked above includes pseudocode that should translate easily to actual code.

How is the edit distance related to the actual problem? I think you should spend a few words to argue the usefulness of your suggestion with respect to the actual problem.

Not sure whether you mean relationship between edit distance and similarity or wrt vector space.

How does the edit distance relate to e.g. pronunciation?

The edit distance on raw text measures spelling differences. To measure pronunciation differences we need some phonetic transcription of the text. It seems that distinct measures are needed, to deal properly with homophones ( https://en.wikipedia.org/wiki/Homophone)


text:
Why do we use the word "kernel" in the expression "Gaussian kernel"?
I've heard the expression "Gaussian kernel" in several contexts (e.g. in the kernel trick used in SVM). A Gaussian kernel usually refers to a Gaussian function (that is, a function similar to the probability density function of a Gaussian distribution) that is used to measure the similarity between two vectors (or numbers).
Why is this Gaussian function called a "kernel"? Why not just calling it a Gaussian (similarity) function? Does it have something to do with the kernel of a linear transformation?

I believe we use the word kernel because we are transforming the basis.

The usage of the word "kernel" in the context of support vector machines probably comes from its usage in the context of integral transforms.
See the article Kernel of an integral operator, and the questions What is the difference between a kernel and a function? and Why is the kernel of an integral transform called kernel?.
The word "kernel" has been used in many other contexts, such as in computer vision, to refer to a certain function with a special purpose. See e.g. the paper that introduced SIFT.


text:
Do you know any examples of geometric deep learning used in industry?
I'm interested in the industrial use of GDL (see https://arxiv.org/abs/1611.08097). Is it used in industry? That is, does any company have access to non-Euclidean data and process it directly instead of converting it to a more standard format?

GDL is a relatively new field of study. Apparently, the first graph NN model was introduced in 2005 by Scarselli et al. and only recently GDL models have started to be promising, so I don't think there are a lot of industries that have already adopted it. Maybe the chemical industry.

Recursive artificial networks are a type aligned well with GDL, and it is an increasingly popular one. Using non-orthogonal geometries is natural when using directed graphs as semantic association, data flow, or composition networks.
This more free-form approach enters into industry slowly because most programming languages use orthogonal structures designed around multidimensional arrays and nested loops to iterate through them, made popular in the FORTRAN era. It is common that a student learns to loop through an array before learning to call a function or test and branch. Computer science is, in some ways, entrenched in the orthogonal structure.
To call the two categories Euclidean and non-euclidean is inaccurate. The initial trend in machine learning followed a Cartesian paradigm, one locked into 90° angles in data structures. The free-form graph is non-cartesian but still just as Euclidean, as Euclid worked with angles other than 90° frequently.
Apple and Google are likely candidates for this seepage of recursive networks into industry data centers, for NLP applications. Language associations are free-form even though these structures are serialized for speech and written text.
Cognition is also non-cartesian. Graph based optimization (pre-GDL) has been part of cognitive research since the 1970s, especially in the LISP space at MIT and CMU and the U.S. Naval Research facilities for strategic analysis, Optimal strategies for a class of constrained sequential problems by JB Kadane, HA Simon, The Annals of Statistics, 1977. Algorithm 2 is a convergence strategy, but without layers of cells, yet the searching concepts of gradient are already present in the maximal and minimal operations in the search in steps c and f.
The aeronautics and auto industry is a likely candidates for seepage into the embedded computing for automated driving and flight. Most of this is either company confidential or classified.
This answer to the question about topological sophistication provides some background why non-euclidean space may be a more natural way to map associations, flow, and composition.
These are some practical applications.

Convolutional networks on graphs for learning molecular fingerprints by DK Duvenaud, D Maclaurin, J Iparraguirre, Advances in neural, 2015
CD-aware recursive neural networks for jet physics by Gilles Louppe, Kyunghyun Cho, Cyril Becot, Kyle Cranmer, 2019
Stability Properties of Graph Neural Networks by Fernando Gama, Joan Bruna, Alejandro Ribeiro
Isospectralization, or how to hear shape, style, and correspondence by Luca Cosmo, Mikhail Panine, Arianna Rampini, Maks Ovsjanikov, Michael M. Bronstein, Emanuele Rodolà
Fake News Detection on Social Media using Geometric Deep Learning, by Federico Monti, Fabrizio Frasca, Davide Eynard, Damon Mannion, Michael M. Bronstein

We've used recursive networks to align complex models with data acquired during laboratory experimentation and hope to publish on that approach as part of another publication. It is effective, since the relationships between features in the real world form squares and cubes only by chance, so it is not very common. Research into materials and energy cannot be confined by Cartesian conceptions, even if the graphs published are on Cartesian coordinate axes, so that others with traditional analytic geometry backgrounds can understand the graphs quickly.


text:
Why don't people use nonlinear activation functions after projecting the query key value in attention?
Why don't people use nonlinear activation functions after projecting the query key value in attention?
It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations.
This observation applies to the transformer, additive attention, etc.

I'm not sure if I got your question right, for the attention model where exactly would you place the non-linearity? Looking at Graph Attention Networks by Petar Velickovic, they do apply an activation function in eq. 5.

Can you provide an example of someone not using nonlinear activations in their attention?

I think what he means is that the queries, keys and values are computed as linear projections, i.e. the input is simply multiplied by a matrix, q = x * W_q, k = x * W_k and v = x * W_v respectively. We could use a non-linear function on each of them, q = σ(x * W_q) etc., but it is redundant because later on we use the softmax function and at the end a MLP which also has non-linearities in it.

@AndreasK. Related. Why we apply these projections after the embedding layer which is linear? Aren't these two composed linearities redundant?

It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations.

Attention is broadly defined as a following operation ($\text{softmax}$ is sometimes replaced by $\tanh$) :
$$\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$
Where $Q$, $K$ and $V$ are matrices that are some functions of the inputs.
There are three nonlinear operations there:

The inner projection $QK^T$ is nonlinear. We have multiplication of two functions of the inputs. For example, in case of self-attention $Q=X W_Q$ and $K = XW_K$ are two linear transforms of the same $X$, so $QK^T = X \left(W_Q W_K^T\right) X^T$ is a quadratic function of the inputs.
The $\text{softmax}(x_i) = e^{x_i} /\sum_n e^{x_n} $ function is obviously nonlinear ($\tanh$ as well)
The final $\text{softmax}(\dots) V$ product is also nonlinear for the same reasons as (1)

I would say that it is pretty clear that it is definitely not just a linear transformation - there's quite a lot of nonlinearities in the attention block.

This observation applies to the transformer, additive attention, etc.

Let's see what happens next with the outputs of the attention layers:
In the transformer model, outputs of the multi-head-self-attention are fed into a feed-forward network inside each block:

"Feed-forward" means that the inputs are multiplied by a weight matrix and then a nonlinear activation function is applied.
The additive attention approach, directly applies another $\text{softmax}$ on the outputs of what one would call the attention block:
$$e_{ij} = v_a^T \tanh\left(W_as_{i-1} + U_a h_j\right)$$
$$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}$$

To summarize - I don't think that the premise of the question is correct. Various nonlinearities are present both inside the attention blocks and, typically, are applied after the attention is computed.

Self attention does not meat that $Q = K = X$. It only means that $Q=XW_Q$ and $K=XW_K$ or in other words that K and Q are obtained from the same X, as opposed to cross-attention where keys and queries come from different sequences.

@hans I stand corrected, thank you. Edited the answer to reflect that

here are a couple of reasons why nonlinear activation functions aren't typically used after projecting the query, key, and value vectors in attention mechanisms like those found in transformers:
Redundancy: The attention mechanism itself already introduces non-linearity through the softmax function. Softmax takes the attention scores (which are linear dot products) and squashes them into probabilities, creating a non-linear relationship between the input and output.
Later Layers Handle Non-Linearity:  The transformer architecture addresses the need for non-linearity in later stages.  Following the multi-head attention layer, there's a fully-connected feed-forward network (MLP) with one or more hidden layers. These hidden layers typically use ReLU or similar non-linear activation functions, allowing the network to learn complex relationships between the attention outputs and the final prediction.


text:
Should I use the hyperbolic distance loss in the case of Poincarè Disk Model?
I trained a neural network which makes a regression to a Poincarè Disk Model with radius $r = 1$.
I want to optimize using the hyperbolic distance
$$
\operatorname{arcosh} \left( 1 + \frac{2|pq|^2|r|^2}{(|r|^2 - |op|^2)(|r|^2 - |oq|^2)} \right)
$$
where $|op|$ and $|oq|$ are the distances of $p$ and respectively $q$ to the centre of the disk, $|pq|$ the distance between $p$ and $q$, $|r|$ the radius of the boundary circle of the disk and  $\operatorname{arcosh}$ is the inverse hyperbolic function of hyperbolic cosine.
But there is a problem

In the Poincarè Disk Model with $r = 1$, the distance is defined only for vectors which have norm less than $1$.
A neural network does not know this rule, so it can predict vectors with norm greater than $1$.

So, I tried to use the distance defined in a space with $r = 2$, and it works very well for the learning task, but I'm doubtful because the distance doesn't scale in a linear way.
Will there be unwanted effects, in your opinion?

Hi! Can you please explain a little bit more in detail the "regression to a Poincarè Disk Model" sentence?

I don't understand all of the context of your question, but you could just normalize whatever output you get from your neural net.

Hi, I have points in R^100, points in H^10 and an unknown function f(R^100 -> H^10) to estimate.
H^10 is a Poincarè Disk Model with dimension = 10 and radius = 1 but if I use the hyperbolic distance to evaluate the error on predicted points sometimes is not defined (because predicted points are outside the disk i.e. norm > 1) so I used the hyperbolic distance defined in a space with radius = 2

To make an example, if we think in Euclidean space and instead of mse (mean squared error) I used mse/2, the collateral effect is that (probably) the learning will be slower because gradient will try to 'correct' the half value of error.
But I'm a Computer Scientist and not a Matemathician so I don't know which collateral effects will happens if I don't use the rigth hyperbolic distance in a space, because hyperbolic does not scale in the same way as euclidean.
Thanks a lot for the answers by the way


text:
How to deal with a small amount of labeled samples?
I'm trying to develop skills to deal with very small amounts of labeled samples (250 labeled/20000 total, 200 features) by practicing on Kaggle "Don't Overfit" dataset (Traget_Practice have provided all 20,000 Targets). I've read a ton of papers and articles on this topic, but everything I've tried did not improve simple regularized SVM results (best accuracy was 75 and AUC was 85) or any other algorithm result (LR, K-NN, NaiveBayes, RF, MLP). I believe the result can be better (on Leaderboard they go even over AUC 95)
What I've tried without success:

Remove outliers  I've tried to remove 5%-10% outliers with EllipticEnvelope and with IsolationForest.

Feature Selection I've tried RFE (with or without CV) + L1/L2 regularised LogisticRegression, and SelectKBest (with chi2).

Semi-Supervised techniques I've tried co-training with different combinations of two complementary algorithms and :100-100: split features. I've also tried LabelSpreading, but I don't know how to provide the most uncertain samples (I tried predictions from other algorithms, but there were many mislabeled samples, hence was unsuccessful).

Ensembling Classifiers StackingClassifier with all possible combinations of algorithms and this also didn't improve the result (the best is the same as SVM accuracy 75 and AUC 85).

Can anyone give me advice on what I'm doing wrong or what else to try?

Would it be easy to hand-label each data point? If so, you might try active learning.

I know for that technique (Label Spreading work similar - only asking for most uncertain samples). But no, there is 200 continuous features which we don't know what they mean, so it is impossible to label them by hand.

In the forums on the competition they go into several of the approaches, i reccomend looking there

Just as an idea: Maybe you can try a neural network and initialize it with parameters returned from an autoencoder. This way the autoencoder can learn useful representations and you can start from there

Take a look at seme-supervised learning (with GAN). For example, this paper or that one. Also, here is my results. Let my know, if need more info on it, I'll post then an answer

In that particular competition, you can try using GAN to generate new data or adding noise to existing data. You can also use K-means algorithm. You can try using a smaller network and remove bias. May be you can use logistic regression to compare the result. You can also use a PCA method.

Upsampling (adding noise to labeled samples which results in new samples) can be an effective practice. However, I'm not sure about the rest of the above answer.
For example, I'm afraid 250 labeled samples may not be enough for a GAN as we have 200 feature columns. Or, according to the question, the best result is achieved by SVM (not an NN algorithm) so I'm not sure what "using a smaller network and remove bias" means in this context.
It would be great if you can provide more details about the applications of K-Means and PCA here.

May be add some noise to the data to produce more


text:
Given the generative model $P(A, B, C)$, would it be faster to learn $P(A,B,C,D)$, or compose $P(A, B, C)$ with $P(D \mid A,B,C)$?
If we have a neural network that learns the generative model for $P(A, B, C)$, the joint PDF of the random variables $A$, $B$, and $C$.
Now, we want to learn the generative model for $P(A, B, C, D)$.
Is there any theory that says learning $P(A,B,C$) and then composing it with $P(D \mid A,B,C)$ is faster than learning $P(A,B,C,D)$ from scratch?


text:
What is local consistency in constraint satisfaction problems?
In the Constraint Propagation in CSP, it is often stated that pre-processing can solve the whole problem, so no search is required at all. And the key  idea is local consistency. What does this actually mean?

Have a look at https://en.wikipedia.org/wiki/Local_consistency.

I think the consistency propagation method only works for 2-SAT problems, no?

If we can do some reduction in the search space using CSP (constraint propagation) we can drastically reduce the search space or sometimes completely avoid the need for a search by directly reaching the solutions (for e.g. with variables having their domains reduced to size one). It could also happen that we come to a point when a variable domain size becomes zero, in that case no solution exists, given the constraints, so no need for a search.
Constraint propagation basically involves the concept of enforcing local consistency (this is done by enforcing node-consistency, arc-consistency, path-consistency and also global constraints using Alldiif or Atmost).
The terms: nodes, arc, path, etc. basically reflects a CSP problem represented as a graph with nodes as the variables and the arcs/edges as constraints. The process is simply to remove values from the domains of the variables that are inconsistent. Algorithms such as AC-3, PC-2, etc. precisely are for these purposes.


text:
How is parallelism implemented in RL algorithms like PPO?
There are multiple ways to implement parallelism in reinforcement learning. One is to use parallel workers running in their own environments to collect data in parallel, instead of using replay memory buffers (this is how A3C works, for example).
However, there are methods, like PPO, that use batch training on purpose. How is parallelism usually implemented for algorithms that still use batch training?
Are gradients accumulated over parallel workers and the combined? Is there another way? What are the benefits of doing parallelism one way over another?

OpenAI have a post on that: https://openai.com/blog/openai-five/
They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed for batches of 4096 observations which are then averaged.
PPO is actually designed to allow this kind of parallelisation as it uses trajectory segments with a fixed size of $T$ to collect data, e.g. 60 seconds for OpenAI Five, where $T$ is supposed to be "much less than the episode length" (p.5 of PPO paper).

So does it update each worker agent after each policy update?
And is this at all related to 'batched evironments' like can be seen here

The paper Dota 2 with Large Scale Deep Reinforcement Learning goes into greater detail than the initial blog posts.
They call their distributed training framework Rapid, which is also used in some of their robotics work, such as the paper Learning Dexterous In-Hand Manipulation, where they discuss a smaller scale deployment of Rapid (as compared to Dota2/OpenAI V) in section 4.3.


text:
What are examples of good reference books on unsupervised learning?
I am looking for good introductory and advanced books on unsupervised learning. I have already read books like Probabilistic Graphical Models from D. Kholler and Pattern Recognition and Machine Learning from C. M. Bishop. I am also very familiar with the Ph.D. thesis of K. P. Murphy, on Dynamic Bayesian Networks.
I have read all of the above mostly for the probability aspects, not really the applications to AI and ML. I would like to know what are the good books (or references) for unsupervised learning, that focuses on practical exercises and examples instead of deep and abstract concepts.

I didn't find any freely available good book that focuses on UL, but this paper/tutorial could be useful. The book Unsupervised Learning: Foundations of Neural Computation (1999, edited by Geoffrey Hinton and Terrence J. Sejnowski) focuses on unsupervised learning algorithms for neural networks. However, I have not read it, so I don't know if it meets your needs. There are other books online, but I don't know whether they are good or not.


text:
Do we need to use the experience replay buffer with the A3C algorithm?
I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm.
The approached I used is the following:

I have some threaded agents that play their own copy of an enviroment; they all use a shared NN to predict 'best' move given a current state;
At the end of each episode, they push the episode (in the form of a list of steps) in a shared memory
A separated thread reads from the shared memory and executes the train step on the shared NN, training it episode after episode.

Is this an appropriate approach? More specifically, do I need to sample data to train the NN from the shared memory? Or should I push in the shared memory each step, just when it's done by a single agent? In this last case, I wonder how I could calculate discounted rewards.
I'm afraid that with my current approach I'm doing nothing more that presenting n episodes to the NN, with the hope that each agent explores the enviroment differently from other agents (so that NN is presented a richer variety of states).

A3C is an actor critic reinforcement learning algorithm with asynchronous updates, hence it is not really relevant for a DL book to cover it. You'll want to check the original paper for details (https://arxiv.org/abs/1602.01783). There are many implementations which cover the details of the algo, random eg: https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-8-asynchronous-actor-critic-agents-a3c-c88f72a5e9f2.

Being on-policy AC, it's not using a replay buffer like off-policy methods such as DQN. However, It does collect data in order to compute discounted rewards. After you update your network, you should not reuse old experiences (that would be off-policy).

Thanks vega for your quick answer. I'll try to follow the tutorial you suggested as well as the original paper.

It's good to try to implement the algo, good learning experience. But be warned, it is notoriously difficult. Best to use a known stable version, like ikostrikov, or try a better algo like ppo. Good luck!

Well, from the few experiments done so far, I noticed that RL is really difficult. Let me ask another question: if I understood it right, the example you linked proposes a scenario where each agent has its own copy of the model. In examples I've read until now, there is only a single NN copy, and each agent is, more or less, an "example producer" (i.e provides a list of steps). What am I missing?


text:
How does the TRPO surrogate loss account for the error in the policy?
In the Trust Region Policy Optimization (TRPO) paper, on page 10, it is stated

An informal overview is as follows.  Our proof relies on the notion of  coupling,  where  we  jointly  define  the  policies $\pi$ and $\pi'$so  that  they  choose  the  same  action  with  high  probability $(1−\alpha)$.   Surrogate loss $L_π(\hat\pi)$ accounts for the the advantage of $\hat\pi$ the first time that it disagrees with $\pi$,  but not subsequent disagreements. Hence, the error in $L_\pi$ is due to two or more disagreements between $\pi$ and $\hat\pi$, hence, we get an $O(\alpha^2)$ correction term, where $\alpha$ is the probability of disagreement.

I don't see how this holds? In what way does $L_\pi$ account for one disagreement? Surely when $\hat\pi$ disagrees with $\pi$ you will have different trajectories under expectation for each so then $L_\pi$ immediately is different from $\eta$ ?
I understand the proof given, but I wanted to try and capture this intuition.


text:
What approaches are there to apply AI to global economic processes?
Today, AI is mainly driven by own-profit-oriented companies (e.g. Facebook, Amazon, Google). Admittedly, there's a lot of AI in the health sector (even in the public health sector) and there's a lot of AI in the sustainability sector – but also mostly driven by obviously own-profit-oriented companies (e.g. Tesla, Uber, Google).
On the other side, one often hears from hard-core economists that centrally planned (= public-profit-oriented) economies (or economic principles) are "the work of the devil" - and that they failed all over history (sometimes for understandable reasons).
But intelligently planning global economic processes and applying these plans with the help of state-of-the-art AI - given the huge amounts of really big data available, and given the argument that globalization is finally for the benefit of all - would seem to be a rewarding endeavour, at least for parts of the AI community.
Why isn't this endeavour undertaken more decidedly? (Or is it?)
Where do I find approaches to apply AI to global economic processes? (Not only describing and understanding but mainly planning and executing?)

It's currently just too complex
The different sources of information are too varied, in economics this is often referred to as a local knowledge problem, which hampers many large scale plans. Humans can react to slight differences like respecting local traditions, landscapes, history but an artificial intelligence would (currently at least) struggle not to generalise over such a large scale as a whole country's economy.
The real work in this case (and actually most 'AI' tasks) would be collecting all the necessary data. Here that part job is currently insurmountable.
Currently a human lead planned economy would do a better job.

i feel the data exists to generate a decent simulator, and by extension model, the issue is as you said its complexity-- it would require heavy resources and man-hours for research which the government probably wouldnt want to spend on a non-gaurantee... but they do spend alot on smaller more compact research focuses, so who knows, one day the want, the resources, and the execution can all come together

@mshlis Not saying you're wrong but I would be surprised if the data exists with the level of detail needed to make predictions to a better than human level. Do you know for sure this data exists?

nope not at all, but my speculation arises from the belief that the data exists but is not collected/organized. There are endless case studies, economic datasets, etc available, and if parsed and preprocessed could provide information that is similar to the concept at hand. My comment about generating a 'decent simulator' is probably nieve, especially because validation would be difficult, but i dont think it unfeasible

@mshlis We do have a lot of data but I think the key point here is the level of work needed to make predictions at a level better than we could with a human planned economy. The amount of local knowledge needed for this level of economic planning requires a lots more ground work before we can start building usable models and making predictions.

Are you sure that it is just currently too complex and not in principle unachievable? That means: Is this your personal opinion?

I found this article from James Manyika (MacKinsey) helpful: Applying artificial intelligence for social good

Today I have learned about the Cybersyn project, developed 1973 in Chile. It went into the "right" direction - but could not achieve its goal because of the coup d'état.


text:
Why is there an inconsistency in the definitions of the retrace?
In Section 4.3 of the paper Learning by Playing - Solving Sparse Reward Tasks from Scratch, the authors define Retrace as
$$
Q^{ret}=\sum_{j=i}^\infty\left(\gamma^{j-i}\prod_{k=i}^jc_k\right)[r(s_j,a_j)+\delta_Q(s_i,s_j)],\\
\delta_Q(s_i,s_j)=\mathbb E_{\pi_{\theta'}(a|s)}[Q^\pi(s_i,\cdot;\phi')]-Q^\pi(s_j,a_j;\phi')\\
c_k=\min\left(1,{\pi_{\theta'}(a_k|s_k)\over b(a_k|s_k)}\right)
$$
where I omit $\mathcal T$ for simplicity.
I'm quite confused about the definition of $Q^{ret}$, which seems not consistent with Retrace defined in Safe and efﬁcient off-policy reinforcement learning:
$$
\mathcal RQ(x,a):=Q(x,a)+\mathbb E_\mu[\sum_{t\ge0}\gamma^t\left(\prod_{s=1}^tc_s\right)(r_t+\gamma\mathbb E_\pi Q(x_{t+1},\cdot)-Q(x_t,a_t)]
$$
What should I make of $Q^{ret}$ in the first paper?


text:
Understanding Hebb network
Can anyone help me in understanding Hebb networking and how different function like AND, OR used to solve by this network.
I couldn’t understand properly through the google.

Can you please tell us where you heard of the expression "Hebb network"? Aren't you actually interested in Hebbian learning, as opposed to a particular neural network that learns in an Hebbian fashion?

@nbro I am confused on the same thing if Hebbian learning and Hebb Network are same thing.

There is no such thing as a Hebb Network. Perhaps you mean a Hopfield Network? Or its probabilistic version the Boltzmann Machine? There are the models which are trained by Hebbian Learning.

In machine learning, the idea behind Hebbian learning is to strengthen (or weaken) the connection (the weight) between the neurons that have similar (or, respectively, dissimilar) outputs, where "similar" can be defined in different ways (e.g. it could be based on the sign of the output of the neurons).
Hebbian learning is a more biologically plausible way of learning than back-propagation, because it is a "local learning strategy" (you locally update the connections and not all the connections of the model at the same time), as opposed to back-propagation, which is a "global learning strategy" (where all connections are usually updated at once, given a "global error" of the network).
There are several neural networks (or models) that can learn in a Hebbian fashion: for example, the Hopfield network or Numenta's temporal memory.


text:
Why experience reply memory in DQN instead of a RNN memory?
I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how important is experience reply in DQN and similar RL algorithms? If the model needs to learn from memory, why don't we use a recurrent network, which has inbuilt memory to it? What is the advantage of experience reply over a recurrent memory?

A related question: https://ai.stackexchange.com/q/7721/2444.


text:
What are some examples of tasks in which, currently, neuroevolution outperforms gradient-based approaches?
Note: I am NOT asking for general advantages of neuroevolution over standard approaches (e.g.: architecture search, parallelization), I am asking for examples of tasks in which, currently, neuroevolved networks outperform ANNs trained with gradient-based techniques. Of course, this is not opinion based, as I am asking for examples based on facts.


text:
How does a PDDL solver find a solution for a given problem?
As far as I know, in PDDL, an environment is designed as well as the initial state described. When we describe the target state, the solver creates some sort of a graph. How is the graph built and what are the keys (keywords) in PDDL referring to?
I know that there are many flavours of PDDL, but let's go with the standard or the most common version of PDDL.

The question doesn't really make sense: PDDL is a description language that is used to formulate a problem. This description then is the input to a planner; how the planner arrives at the intended solution is not related to the PDDL description.
There are a number of planning algorithms, and you can implement any of them to make use of a PDDL description. The output of the solver is a plan, which is usually an ordered sequence of actions, and a tree or graph structure might be a good way of capturing this.


text:
Is there an LSTM-based unsupervised learning algorithm to label a dataset of curves?
I have a big amount of light curves (image below).

I am trying to label the points as signal or background (the signal appears usually periodically, several times, for a given light curve).
More precisely, I want to identify the downward spikes (class label = 1) from the background (class label = 0).
However, the data is not labeled. I tried labeling it by hand, and using a bi-directional LSTM succeeds in labeling the data points properly. However, there are thousands of light curves and labeling all of them would take very long.
Is there any good unsupervised approach to do this (unsupervised LSTM maybe, but any other method that might work on time series would do just fine)?

It sounds like you’ve labeled some of the data by hand. Can you use a clustering algorithm trained on the small amount of data you’ve already labeled? If the dataset is quite imbalanced, you may be able to separate the data into two clusters and check some of the results for signal by hand? Or maybe you can set it up as an “anomaly detection” network, which you can find many example implementations of online. That way you could separate signal (which would appear as anomaly) from noise (which seemingly dominates the signal).

You can try to work with Gated Recurrent Units or GRU. This will solve your problem of with too much latency time that LSTM required. LSTM also doesn't give preference on newer data too. For more information, you can follow more on this great article
GRU Cells

How do GRUs help labelling the dataset?


text:
What is the goal of a constraint solver?
What is the goal of a constraint solver? How are constraints propagated in a constraint satisfaction search?
Any references are also appreciated.

You want Chapter 6 of Russell & Norvig's AI: A Modern Approach, for a starting place.
The goal of a constraint solver is to find an assignment of values to variables such that every variable is assigned a value, but none of a list of constraints are violated.
An example problem that a constraint solver can solve is the problem of assigning students to groups for a project. The variables are the names of the students. Each variable has a set of possible values it could take on that correspond to the names of the groups. The constraints prohibit assigning certain students to the same group (perhaps they hate each other), or require certain students to be assigned to the same group (perhaps they like each other). More complex constraints are also possible (e.g. at least 2 of these 5 students need to be in the same group).
Constraint propagation is the idea that several constraints might logically imply a stronger constraint. As a simple example, suppose that student $S$ must be assigned to group $1$. That's one constraint. Suppose further that student $T$ must be assigned to the same group as $S$. That's another constraint. Together, however, they imply that $T$ must be assigned to group $1$.
Most constraint solvers operate by iteratively trying to assign values to variables and then backtracking when a constraint is violated. When assignments are made, different constraint propagation methods make different decisions about how much computational effort to spend deriving the logical consequences of constraints, as opposed to simply trying more assignments of values to variables until the consequences become apparent (one of the existing constraints gets violated).
The simplest form of constraint propagation is called forward checking. After each speculative assignment of a value to a variable, forward checking looks at all the constraints the variable is involved in. It then derives the logical consequences of the assignments. For example, continuing with the student problem from above, if $S$ was assigned value $1$, then forward checking would notice the constraint $S=T$, and would, effectively, add the constraint $T=1$ (it actually does something a little different, but it has this effect). As another example, if the assignment $T=2$ were made, forward checking would notice that $S=T$, and would add the constraint $S=2$. It would also detect the contradiction that $S=1$ and $S=2$ are both constraints, and would declare the assignment $T=2$ invalid.
More advanced constraint propagation methods can reason about these kinds of logical consequences before assignment are made. For example, they could derive that $T=1$ before the program starts. Wikipedia has a good summary of these, starting with arc consistency.


text:
Which libraries can be used for image caption generation?
Which libraries can be used for image caption generation?

I'd consider this "asking for a development tool", which is off-topic.

This seems a bit broad; I assume you’re looking for libraries with “black box” models? I doubt you haven’t heard of TensorFlow / Keras and PyTorch. I’m sure you can find some implementations in these libraries on github. Also look into FastAI; I don’t know if they have a black box model already written (I don’t really use it often), but they are very good so long as you don’t need to know too much about what’s under the hood. FastAI is built on top of PyTorch (but it’s starting to now branch out to other frameworks).

You can try to go to this source: https://www.hindawi.com/journals/cin/2020/3062706/.

Image Caption Generation is an interesting problem to work on. I think your question was to know if there are any open-source libraries with built-in functions for Image Captioning. You can build Image Caption Generation models using Frameworks like Tensorflow, PyTorch, and Trax.
I'd also recommend you to read the following papers:

Show and Tell: A Neural Image Caption Generator. Link
Transfer learning from language models to image caption generators: Better models may not transfer better. Link
Image Captioning with Unseen Objects. Link

Also, here are a couple of blog posts you can read:

How to Develop a Deep Learning Photo Caption Generator from Scratch
Learn to Build Image Caption Generator with CNN & LSTM
Image Captioning with Keras

I've been read from some blogs, for image caption generation you can use concepts of a CNN and LSTM model and build a working model of Image caption generator by implementing CNN with LSTM. I think before you want to make something and using this project, you need to have good knowledge of Deep learning, Python, working on Jupyter notebooks, Keras library, Numpy, and Natural language processing and make sure that you have installed all the following necessary libraries:
-pip install tensorflow
-keras
-pillow
-numpy
-tqdm
-jupyterlab
source : https://data-flair.training/blogs/python-based-project-image-caption-generator-cnn/


text:
Do we train a logistic regression model using a dataset that is 3 times bigger than the validation dataset?
Suppose we have a data set $X$ that is split as $X_{\text{train}}$, $X_{\text{val}}$ and $X_{\text{test}}$ and the outcome variable is binary. Let's say we train three different models (logistic regression, random forest, and a support vector machine) using $X_{\text{train}}$. We then get predictions for $X_{\text{val}}$ using each of the three models.
In stacking, is it correct to say that we train a logistic regression model on a data set of dimension $|X_{\text{val}}| \times 3$ with the predicted values and actual values of the validation set? This logistic regression model is then used to predict outcomes for data in $X_{\text{test}}$?

By stacking, do you mean ensemble learning?

Do you mind if I edit the title of your question to be a bit more general so it is more clear the context of the post?

The predictions of each of your initial models will become a feature to feed the meta learner. If you use $n$ initial models, then for each example you will feed the meta learner $n$ features, each feature being a prediction from one of the initial models.
Note that this doesn’t mean the size of your dataset increases. Instead, each member of $X_{val}$ will be represented by $n$ features, with the $k$th feature being equal to the initial prediction of the $k$th initial predictor.
This medium link recommends a package that can help you build a pipeline.
Note: in a sense if you are caching the intermediate predictions of each initial predictor, you’ll end up with $n$ times as many data points as there are entries in $X_{val}$. But these are simply each going to be a feature of another data point, so the dataset hasn’t really increased in size.


text:
Are there reinforcement learning algorithms that ensure convergence for continuous state space problems?
The Q-learning does not guarantee convergence for continuous state space problems (Why doesn't Q-learning converge when using function approximation?). In that case, is there an algorithm which can guarantee convergence?
I am looking at model-based RL specifically iLQR but all the solutions I find are for the continuous action space problem.


text:
What is the relation between Monte Carlo and model-free algorithms?
Monte Carlo (MC) methods are methods that use some form of randomness or sampling. For example, we can use an MC method to approximate the area of a circle inside a square: we generate random 2D points inside the square and count the number of points inside and outside the circle.
In reinforcement learning, an MC method is a method that "samples" experience (in the form of "returns") from the environment, in order to approximate e.g. a value function.
A temporal-difference algorithm, like $Q$-learning, also performs some form of sampling: it chooses an action using a stochastic policy (e.g. $\epsilon$-greedy) and observes a reward and the next state.  So, couldn't $Q$-learning also be considered a MC method? Can an MC method be model-based?

In Reinforcement Learning (RL), the use of the term Monte Carlo has been slightly adjusted by convention to refer to only a few specific things.
The more general use of "Monte Carlo" is for simulation methods that use random numbers to sample - often as a replacement for an otherwise difficult analysis or exhaustive search.
In RL, Monte Carlo methods are generally taken to be non-bootstrapping sample-based approaches to estimating returns. This is a labelling convention within RL - probably because someone called an initial model-free learner a "Monte Carlo method", and the name stuck whilst many refinements and new ideas have since been published under different names.
The historical use of the term is important, since if you mention you are using "Monte Carlo Control", it usually means a very specific subset of methods within RL, to most readers.

So, couldn't -learning also be considered a MC method?

In the general sense perhaps. The argument is perhaps stronger for it if the environment is being simulated on a computer for the agent to learn from.
However, if you start to unilaterally call Q-learning a MC method, you are probably going to just confuse people who have learned the conventions in RL.

Can an MC method be model-based?

In general, yes, because the model can be sampled for planning, and a policy can also be separately sampled - so it is possible to run an MC method with or without a model - depending on whether you sample from the model taking "virtual actions" (for e.g. planning or refining your agent) or from the environment taking real actions. Many RL techniques blur the line between online learning and planning. For instance, using a simulated environment or historical data can be framed as planning for the real environment.
Monte Carlo Tree Search is an example of a model-based technique using the term "Monte Carlo" within RL frameworks. It is used famously within DeepMind's AlphaZero in order to refine a policy and value estimates during self-play.

What is the relation between Monte Carlo and [other] model-free algorithms?

In the context of RL, Monte Carlo is presented as one way to estimate expected utility (or return) - by sampling from the environment and policy until the a complete trajectory is available:
$$v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{T-t} \gamma^k R_{t+1+k} | S_t = s ]$$
MC is contrasted with Temporal Difference (TD) approaches, such as Q-learning, which sample bootstrap estimates using the Bellman Equation:
$$v_{\pi}(s) = \mathbb{E}_{\pi}[R_{t+1} + \gamma v_{\pi}(S_{t+1}) | S_t = s ]$$
The two approaches can be combined in various ways, including TD($\lambda$) methods. With  TD($\lambda$), if you set $\lambda = 0$ then the algorithm is identical to single-step TD learning, and if you set $\lambda = 1$ then it is very similar to a Monte Carlo method. Often setting it to some intermediate value is more efficient than either extreme.

Can you clarify these points "because the model can be sampled for planning, and a policy can also be separately sampled"? Are you saying that using a MC method we can build a model which we use for planning?

@nbro: Well, you could yes, same as Dyna-Q does it for example - just store transition history that you observed and sample from it. Someone is bound to have done that at some point, but typically those learned models are not very robust or useful in RL, so it's not a mainstream approach. More usually model-based means you have some other access to a model via a simulation or the game engine for a board game etc

You are saying that every MC method that uses e.g. a simulation to gain experiences in the form of returns is a model-based algorithm because the simulation itself is a model? Isn't this a little bit of a stretch? Model-based should refer to an algorithm that uses the transition function (or reward function): e.g. value iteration is model-based because it uses the transition function $p$. The simulation only provides a way of experiencing (sampling) the world.

@nbro: No I am not saying that. I am saying that you can often use an existing simulation as a model for model-based learners. All you usually need added to the API is a way to place it into a specific state, and then you have a sampling model (as opposed to a distribution model). Sampling models are adequate for planning, such as MCTS

Ok, but what difference does it make if we use a simulation or not, in terms of being model-free, in this context? Why did you mention it then?

@nbro: Because if you have a simulation available, you usually have a sampling model available for little or no additional effort. Look-ahead planning using a sampling model is essentially what MCTS is.


text:
Why should we use TD prediction as opposed to TD control algorithms?
Consider a problem where we have a finite number of states and actions. Given a state and an action, we can easily know the reward and the next state (deterministic). The state space is really large and has no final state.
There was a paper that for a problem of this type used TD(0) by filling the value table and chose its actions by:
$$\pi(s) = \text{argmax}_a (r(s,a) + \gamma V(s_{(t+1)}))$$
I've read somewhere that is OK to use prediction algorithms when the model is well-described with the objective of choosing best actions and not only evaluating the policy.
What is the purpose, advantages and disadvantages of using TD prediction here instead of a TD control algorithms (and just saving the $Q(s, a)$ table)? If it was about space, you still have to store a table with all the rewards for each state-action pair, right?
I'm not sure if I was able to explain myself very well as I was trying to keep it short, if some clarification is needed please tell me.

Hi Miguel. I have edited your question to hopefully make it clearer. Have I changed the meaning of the question? If yes, feel free to edit it again or roll it back to the previous version.


text:
After a model has been trained, how do I use it to address the real-world problems?
I understand the way we build and train a model, but all of the online courses I've found end with this. I can't find any course explaining the process of utilizing the trained model to address the problem.
Is there any course out there that explains the whole process from data collection, model building, and utilizing the model to solve the real-world problems?

Using a machine learning or AI-powered model once it has been built and tested, is not directly an AI issue, it is just a development issue. As such, you won't find many machine learning tutorials that focus on this part of the work. But they do exist.
In essence it is the same as integrating any other function, which might be in a third-party library:

Package the new function so that it can be called from your production system (this may be the hardest part)
Decide where in the code of your production system to call the new function. In your example case maybe after form is completed describing an incident you could link the top recommended KB articles from the new ticket.
Design and check the associated user experience/UI as appropriate for your development team (in small projects you may skip this step and just implement)
Change your production code to call the packaged function. In any professional development team, this part will have multiple stages, but not really relevant to the question - if you are the ML specialist and delivering your new model to an existing team, you need to talk to them about both the packaging part and the steps involved here.

Often, machine learning models come with a whole bunch of dependencies that the rest of a system does not have. There are a variety of solutions for that, depending on the libraries that you are using, whether you are using a cloud PaaS service etc. You could just build a Docker image to hold all the AI parts and call it passing the input data.
Deploying ML models to production as a job is often a task for Machine Learning Engineer roles. There are courses and articles covering the practical aspects of these steps, if you search for them. Here are a couple:

Deploying Machine Learning Models from Coursera
A blog article on Medium: There are two very different ways to deploy ML models . . .

I have no affiliation with any of the above, have not read articles or taken the courses, and am not able to make any recommendation, even if you told me the technologies you were using for ML and in production currently.
After deploying to production . . .
Your work is not necessarily done. You will want to monitor behaviour of the system. During integration, you should of added logging, or some way to get feedback of performance in the wild.
For instance, if this is an ML system, does the accuracy seen in testing hold up against real life? Is the target population drifting? If the system makes interventions - by e.g. suggesting links, or showing categories to end users - how well are those working in practice? Is the performance and responsiveness of the system fast enough when the service is under load?
If it is more reactive AI system, that takes actions itself, you will similarly want to monitor what it is supposed to optimise, or sample its output for errors and quality control.
All this feedback can go back into a new iteration of design and integration. How to incorporate that will depend a lot on the nature of the system and what you discover, so could be the subject of further questions.


text:
Estimate distance between points in perspective image
I am trying to estimate the real world distance (in metres) between two points in a perspective image using an uncalibrated camera. However, the dimensions of an object in the image are known.
I thought of using pixel per metre, but, being a perspective image, that did not seem like a viable approach. I believe I need the camera matrix from the image (maybe using the known object) and compute the 3D coordinates of the point, then simply compute the Euclidean distance between the points.
If that is the right approach, how may I do it? If there are any alternatives to this approach, kindly let me know.


text:
How do I integrate Tensorflow with EV3DEV?
I've got a Lego Mindstorms EV3 with EV3DEV and EV3-Python installed. I wanted to try out artificial intelligence with the robot, and my first project is going to be to get the robot to try and recognize some images (using convolutions) and do an action related to the image it has seen. However, I can't find a way to use Tensorflow (or any AI module for that matter) on an EV3. Does anyone know how to incorporate Tensorflow or any other modules into the EV3? Help would be gladly appreciated.

There is a YouTube video LEGO EV3 Raspberry Pi Tensorflow Sorting Machine by ebswift that should help you although you will need a Raspberry Pi.  From the abstract:
This is a sorting machine based on the EV3 45544 education kit sorting machine.  The colour sorting camera is substituted with a Raspberry Pi with the v2 camera.  The EV3 is controlled over wifi via RPyC and the object recognition work is done via Tensorflow.
A viewer asked: Can you share links about how to train a model on the PC and move the trained model to the Pi?
ebswifft replied:
Glad you like it :).  Training the model on the PC is following the guide for Tensorflow for Poets https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/.  All you do is install Tensorflow on the Raspberry Pi, clone the github repository onto it and copy the model you trained from the codelabs article onto the Raspberry Pi to run the classifier as per step 5 of the codelabs article.  Onboard image capture on the Raspberry Pi is just done using picamera.  I used a tensorflow version that a user compiled for the Raspberry Pi from here: https://github.com/samjabrahams/tensorflow-on-raspberry-pi/issues/92.  I might do up more of a general step-by-step sometime on my site, I'll report back if I can get that up and running.
There is also BrickClassifi3r:
This Lego Mindstorms EV3 robot uses a neural network to recognize a cube, a cylinder or a small cube put on a conveyor belt. See the video how it works. Each object on the conveyor belt is scanned by an IR-sensor every 40ms for about a second. The resulting data are 24 distance values representing one of the three objects. This data is fed into the neural network on the robot to classify the object within 180ms. In a test with 300 objects it reaches 95,6% accuracy. The neural network has been trained before by a machine learning algorithm with TensorFlow on a PC using a set of 375 training examples - 125 examples for each object.
The ev3-myo project uses LEGO Mindstorms EV3, TRACK3R, a Myo armband and TensorFlow for gesture recognition.

The first one looks great, but I don't have the source code for it. The second one also looks appealing, however, I don't understand how he's connecting the neural network to the robot, as the tensorflow recognitions aren't visible in the .rtf files. Also, this doesn't really how to implement the code. Any ideas on how to do that?

I added some more information for the first option.  Also check out the ev3-mvo project in my answer.

If you haven't used TensorFlow for object detection I suggest you start there before diving into the EV3.

Get a raspberry with a camera, power it by a battery bank, attach it to the EV3 and run a 2 one program on raspberry and another on ev3 communicating with each other via MQTT


text:
How to fix time dimension in time varying data-sets using deep learning model for classification?
Dataset Description
I am working on the famous ABIDE Autism Datasets. The dataset is very big in the sense that it has more than 1000 subjects containing half of them as autistic and the other half as healthy controls. The dataset is taken from 17 sites across the world and each site used a varying time dimension when recording the subjects fMRI.
My Question
I want to use this dataset for a classification task but the only issue is time-varying subjects as features set are fixed to 200 so you can say that I have subjects dimensions like 150 x200, 75 x 200 , 300 x 200... so on. So what are advanced AI or deep learning techniques that I can use to fix this time dimension for every subjects or can anybody suggest some deep learning framework or model that I could use to fix these varying time dimensions across subjects?
My Effort
Approach 1
I have applied PCA to the time dimension and fixed them to 50 and tried other numbers also but it did not produce good accuracy for classification
Approach 2
I also tried to use only specific time points from every subject like taking only 40 time points from every subject to fix the dimension but again it did not work as definitely filtering some time series data on every subject would lose crucial information.

One way to tackle this problem is to identify the longest sequence in the training dataset and than expand (0 padding) all the other inputs to match that size.

I have done this kind of things but did not get good accuracy. I mean I am in search of some machine/deep learning model or some signal processing that could squeeze the information from all the subjects into some fixed dimension. As I have brain fMRI data and zeroing some time points will have no sense as on each time point there should be some value.

Maybe the models you're experimenting with are not fitted for your type of problem. If you want to encode the inputs into a fixed size vector you can use any neural network architecture as an encoder (fx a 1 layer feed forward neural network will do the trick).

In deep learning, it is very common to use Recurrent Neural Networks (RNNs) to handle time-series data with varying input sequence lengths. Check out the RNN Wikipedia for more detail.

I know RNN and similarly there are many other techniques that one can use but can you give me concrete reference or link where this kind of data like I asked in my question with varying time dimension was first fixed using RNN and used for classification thereof?

Could you be a bit more specific on what you're looking for? Are you looking for code that you could use, a tutorial on how to use RNNs, a good ML framework/tool to use, etc.?

No not at all I am not looking for a code. I am just looking if someone has good architecture or model using deep learning or machine learning that I could deploy by myself.


text:
How do I solve the problem of positioning 11 pieces into a 8x8 puzzle?
I was trying to figure out how to create a solver to the puzzle of putting 11 pieces in a board (8 x 8). I created the game in http://www.xams.com.br/quebra. It is possible to turn the piece 90 degrees each time counterclockwise (Girar) and mirror it vertically (Inverter), and so the piece can assume 8 forms.
When clicking in solver button (Resolver), it tries to put pieces randomly in board in brutal force method (it spends a LOT of time). Using this method, I was not able to achieve the result.
I would like to try something smarter than this and having a machine learning algorithm for this would be great. I don´t know how to formulate the problem. How would you start this please?

EDIT:
I am still building the page and there is so much to improve. You need to click the center of 3x3 matrix where you want to put the piece.

EDIT2:

Are you sure it is actually possible at all to complete?

I bought this puzzle (physically) and hope it is possible. I could put 10 pieces. I think they would not sell this is it was not possible (I hope so)

Can this not be solved with MCTS? Just start a piece in one position and try every combination down the tree. If it’s solvable it should be able to find the solution.

I think the machine learning tag is wrong, as there is nothing to learn. I solved a similar problem once with a recursive backtracking algorithm. With brute force it is basically unsolvable.

Using your app, I was able to find a (spoiler alert!) solution manually. At least now you know your puzzle is solvable and you did not waste your money :)
It seems your app has a bug, though. I was unable to put the last piece, as shown in the picture. I was wondering if your solver, as it stands, will ever find a solution.
Now the idea. It may be useful for your solver.
The board has 8x8=64 squares. Each piece will occupy 5 squares and you want to fit 11 pieces, so the final position will have 9 empty squares. Divide the board in two 8x4 rectangles, left and right. Now, it seems only fair that one rectangle should have 5 empty squares leaving the other with 4; with that in mind I've proceeded to fill the right part first and only then the left. After some trial and error, I got lucky.
I don't know if you can reach all solutions with this method. Notice that, in the solution given above, the bottom rectangle will end up with 6 empty squares.
I don't know how to write an efficient solver either. For starters:

Build up a list of bad configurations: small sets of pieces/positions such that whenever you reach them, you know there is no hope.
At each step, put a piece in such way it minimizes the number of forced empty squares.
A variant of the idea above: divide the board in four 4x4 parts; it seems reasonable each part should have at least one empty square but no so many; so, look for solutions forcing these parts to have (1,2,3,3) or (2,2,2,3) empty squares.
Lookup whether this puzzle is known. Names that come to mind: Martin Gardner, Ian Stewart, Sam Lloyd.

Can't say you will ever be able to see a list of all possible solutions.
Nice puzzle, nice app that one you wrote, I've had a good time. Thank you.

I will try the methods you sugested and will be a huge pleasure to mark your answer as the accepted one. I am ver glad that you solved the puzzle. Thanks a lot and congratulations.

Thanks! By the way, I was thinking, maybe you could write a game: say two players, each one put one piece at a time; the first one that could not put a piece loses. You could play on even larger boards, say 10x10, 13x13, etc. That would be hard. And then definitely some AI programming could come in.


text:
Can Q-learning be used for continuous (state or action) spaces?
Many examples work with a table-based method for Q-learning. This may be suitable for a discrete state (observation) or action space, like a robot in a grid world, but is there a way to use Q-learning for continuous spaces like the control of a pendulum?

Q-learning for continuous state spaces
Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the space to a small number of states. Provided you can sample and update enough times, then a few million states is not a major problem.
However, with large state spaces it is more common to use some form of function approximation for the action value. This is often noted $\hat{q}(s,a,\theta)$ to show that it is both an estimate (the circumflex over $\hat{q}$) and that you are learning some function parameters ($\theta$). There are broadly two popular approaches to Q-learning using function approximation:

Linear function approximation over a processed version of the state into features. A lot of variations to generate features have been proposed and tested, including Fourier series, tile coding, radial basic functions. The advantage of these methods are that they are simple, and more robust than non-linear function approximations. Which one to choose depends on what you state space represents and how the value function is likely to vary depending on location within the state space.
Neural network function approximation. This is essentially what Deep Q Networks (DQN) are. Provided you have a Markov state description, you scale it to work sensibly with neural networks, and you follow other DQN best practices (experience replay table, slow changing target network) this can work well.

Q-learning for continuous action spaces
Unless you discretise the action space, then this becomes very unwieldy.
The problem is that, given $s,a,r,s'$, Q-learning needs to evaluate the TD target:
$$Q_{target}(s,a) = r + \gamma \text{max}_{a'} \hat{q}(s',a',\theta)$$
The process for evaluating the maximum becomes less efficient and less accurate the larger the space that it needs to check.
For somewhat large action spaces, using double Q-learning can help (with two estimates of Q, one to pick the target action, the other to estimate its value, which you alternate between on different steps) - this helps avoid maximisation bias where picking an action because it has the highest value and then using that highest value in calculations leads to over-estimating value.
For very large or continuous action spaces, it is not usually practical to check all values. The alternative to Q-learning in this case is to use a policy gradient method such as Actor-Critic which can cope with very large or continuous action spaces, and does not rely on maximising over all possible actions in order to enact or evaluate a policy.
Controlling a pendulum
For a discrete action space e.g. applying one of a choice of forces on each time step, then this can be done using a DQN approach or any other function approximation. The classic example here might be an environment like Open AI's CartPole-v1 where the state space is continuous, but there are only two possible actions. This can be solved easily using DQN, it is something of a beginner's problem.
Adding continuous action space ends up with something like the Pendulum-v0 environment. This can be solved to some degree using DQN and discretising the action space (to e.g. 9 different actions). However, it is possible to make more optimal solutions using an Actor-Critic algorithm like A3C.

How do you calculate q^(s′,a′,θ) for the training? My guess is to calculate backward from the end state of the pendulum. During the first run of the DQN, do you use one value for all q^(s′,a′,θ) or randomize them per example?

@JoachimRives $\hat{q}(s',a',\theta)$ is the output of the neural network or other parametric estimator during training, and $\theta$ stands for the learning parameters. To calculate what the value should be trained to be - for DQN at least - that's the TD target, and the formula for that is shown in the answer. Wehn using DQN, it is important to recalculate the TD target every time you retrieve an example from replay memory.

Do you mean the DQN TD is calculated backward or based on the expected value of succeeding steps leading to the terminal state?

@JoachimRives The value is calculated as in the answer. TD learning (of which Q learning is an example) does not calculate backwards from the terminal state, instead it bootstraps from existing estimates that can be in the middle of a trajectory. Other algorithms do resolve backwards from terminal states - e.g. Monte Carlo control. There are already questions on this site that explain the difference further

Q-Learning for continuous state space

Reinforcement learning algorithms (e.g Q-Learning) can be applied to both discrete and continuous spaces. If you understand how it works in discrete mode, then you can easily move to continuous mode. That's why in the literature all the introductory material focuses on discrete mode, as it's easier to model (table, grid, etc.)
Supposing you have a discrete number of actions, the only difference in a continuous space is that you will be modeling the state each $X$ amount of time ($X$ being a number you can choose depending on your use case). So, basically, you end up with a discrete space, but probably with an infinite number of states. You apply then the same approach you learned for discrete mode.
Let's take the example of self-driving cars, at each $X$ms (e.g $X=1$), you'll be computing the state of the car which are your input features (e.g direction, orientation, rotation, distance to the pavement, relative position on the lane, etc.) and take a decision of the action to take as in discrete mode. The approach is the same in other use cases, like playing games, walking robots, and so on.

Note (continuous action space):

If you have continuous actions, then in almost all use cases the best approach is to discretize your actions. I can't think of an example where discretizing your actions will lead to a considerable deficiency.

More formally, you are suggesting to discretise the continuous state space. Do you have any references to works/papers that apply a discretization of the state space? I think it might be helpful to cite them, if that's the case.

If action space is continuous you could model it by outputting a mean and variance of a proper distribution and then use the outputs to sample from the distribution parameterized by those outputs (ie. applying voltages).

For references using discretisation you can take a look at  the Arcade Learning Environment paper which uses frame-skipping techniques (also experimented in the well-known Playing Atari with Deep Reinforcement Learning paper). Current RL papers having video as input do the process at the level of each frame thanks to computation power being available.


text:
If the average rewards start high and then decrease, could that indicate that the PPO is stuck at a local maximum?
I'm trying to train a PPO agent in a 3D balance ball environment. My action space is continuous.
In the following graph, each dot shows the average reward from 100 episodes.

Could this graph indicate that it's stuck at a local maximum?  Do I need to promote exploring by increasing the entropy, or does this look like a bug with my implementation?
I am trying to maximize the average rewards.
When I look at the loss, it does seem like it is minimizing the cost.  I thought that the loss function is dependent on the rewards because the advantage is calculated based on them.  The normalized advantage is then factored into the policy loss, which should say which direction for the policy to step toward.
It seems like the actions are a bit noisy but the platform sometimes seems to try to keep the ball from falling off.
Hyperparameters:

Learning rate = 0.01
Entropy Coefficient = 0.01
Value Function Loss Coefficient = 0.5
Gamma/Discount Factor = 0.995
MiniBatch Size = 512
Epochs = 3
Clip Epsilon = 0.1

I had the same problem where the reward kept decreasing and started to search for answers in the forum.
I let the model trained while I search. As the model trained, the reward started to increase. You can see the tensorboard graph for rewards in validation time.
.
The fall continued until around 100k~ steps and did not change a lot for 250k~ steps.
After 350k~ th step, it slowly started to increase. Without knowing your number of steps trained, I would suggest training for more steps.
Also, I read about this (Reward first decreasing and then increasing) in an RL paper, if I find it I will mention it here.


text:
What is the relationship between MLE and naive Bayes?
I have found various references describing Naive Bayes and they all demonstrated that it used MLE for the calculation. However, this is my understanding:
$P(y=c|x)$ $\propto$ $P(x|y=c)P(y=c)$
with $c$ is the class the model may classify $y$ as.
And that's all, we can infer $P(x|y=c)$ and $P(c)$ from the data. I don't see where the MLE shows its role.

And that's all, we can infer P(x|y=c) and P(c) from the data. I don't see where the MLE shows its role.

Maximum likelihood estimate is used for this very purpose, i.e. to estimate the conditional probability $p(x_j \mid y)$ and marginal probability $p(y)$ .

In Naive Bayes Algorithm ,using the properties of conditional probability, we can estimate the joint probability
$$
p(y, x_1, x_2, \dots, x_n) =  p(x_1, x_2, \dots, x_n \mid y) \; p(y)
$$
which is the same as
$$
p(y=c|x)\propto p(x|y=c)p(y=c)
$$
What we need to estimate in here, are the conditional $p(x_j \mid y)$ and marginal $p(y)$ probabilities, and we use maximum likelihood for this.
To be more precise: Assume that the $x$ is a feature vector $(x_1, x_2, ...x_n)$ - which means nothing more than the data point is a vector. In Naive Bayes Classification the assumption is that these features are independent of each other conditional on the class. So, in that case this term
$$p(x|y=c)$$ is replaced by $$\prod_{i=1}^{n} p(x_i|y=c)$$ -- this is because of the independence assumption that lets us simply multiply the probabilities.
The problem of estimating the class to which the data point belongs reduces to the problem of maximizing the  likelihood $$\prod_{i=1}^{n} p(x_i|y=c)$$ -- which means assigning the data $(x_1, x_2, ...x_n)$  to the class $k$ for which this likelihood is highest. This will give the same result as MLE which takes the form  of maximizing this quantity -- $$\prod_{i=1}^{n} p(x_i|\theta)$$ where $\theta$ are the assumed parameters.


text:
How to manage large amounts of image data for training?
Right now, I am trying to synthesize training images for a CNN and due to the nature of the application, there is a finite number of sample images to learn from.
From other research, I expect to be using about 200,000 training images at a resolution of 1280*720, which with 3 channel at 8 bits will take about 550 GB to save uncompressed. This number can and probably will rise in the future, meaning more memory that I will need to provide.
I imagine that there are applications that required even more training data with higher complexity and that there are solutions to handling that such as compression techniques and the like.
My question: Are there solutions for the memory management beyond compressing the images with JPEG and such besides generating and instantly consuming the pictures without saving them to permanent memory?

may I ask you, did you find out the solution? Like you said, if your training data is 550GB, how did you let CNN to train? Thank you so much.

I suggest you fine-tune an existing model. Knowledge transfer models in many image processing tasks are now open sourced and you can build your model on top of them. Also, knowledge transfer models are trained on a large datasets and can quickly converge to your case-study with a little of task-specific extra training.

This way you will use few data to tune the model which leads to less
  memory use and less training time. You will also take advantage from a
  ready-to-use architecture and get accurate results.

Depending on your case-study you can choose from this list of computer vision pre-trained models.

Most of the algorithms seem accessing the training set sequentially so the images need not be loaded into memory all at the same time.
It is technically fully possible to build a workstation with 1 Tb of RAM or more, using a server barebone in a tower form factor (see this, for instance, and would support multiple GPUs) but this only makes sense if the image loading is a bottleneck. Current SSDs are rather fast so you need to measure this before spending money on such a beast.


text:
Should I use leave-one-out cross-validation for testing?
I am currently working with a small dataset of 20x300. Since I have so few data points, I was wondering if I could use an approach similar to leave-one-out cross-validation but for testing.
Here's what I was thinking:

train/test split the data, with only one data point in the test set.

train the model on training data, potentially with grid-search/cross-validation

use the best model from step 2 to make a prediction on the one data point and save the prediction in an array

repeat the previous steps until all the data points have been in the test set

calculate your preferred metric of choice (accuracy, f1-score, auc, etc) using these predictions

The pros of this approach would be to:

You don't have to split the data into train/test so you can train
with more data points.

The cons would be:

This approach suffers from potential(?) data leakage.

You are calculating an accuracy metric from a bunch of predictions that potentially came from different models, due to the grid searches, so I'm not sure how accurate it is going to be.

I have tried the standard approaches of train/test splitting, but since I need to take out at least 5 points for testing, then I don't have enough points for training and the ROC AUC becomes very bad.
I would really appreciate some feedback about whether this approach is actually feasible or not and why.

This is an old question, but it's not clear to me what your doubt really is. LOOCV is typically used for testing and/or validation, so what is your real question here? By the way, I edited the post to put in the title what I think is your question, but, to be honest, I don't really know what it is.

Concerning $k$-fold Cross Validation, I like to think of it by considering two extremes you can do: Leave-One-Out Cross-Validation where you leave one sample each time and train your model on the remaining $n-1$, and 2-fold Cross Validation at which you split your dataset in half and train (and validate) two models on two different halves.
The important aspect when choosing $k$ is a bias-variance tradeoff. Note that in LOOCV you train each model using almost as many samples as there are available ($n-1$), so the validation step should give you an unbiased estimate of the real test error. However, each model in LOOCV is trained each time on almost exactly same dataset!. This has important consequences, since the output of each model is highly correlated with each other. Since the mean of highly correlated variables has big variance, LOOCV will suffer from huge variance.
On the other hand, in 2-fold CV the models share no common samples, so they are not correlated and therefore their outputs have low variance. But since we train each model using only a half of available samples, the procedure will have a high bias (the estimated test error will be off from the true test error).
What to do in this scenario? Choose something in the middle. Usually $k=5$ and $k=10$ should be a good choice.

I would love to do 5 or 10 fold cross validation, but since the dataset only has 20 samples this is simply not feasible. The set will have only 4 or 2 samples respectively, which will yield horrible ROC AUC.


text:
How to solve peg solitaire with a graph search?
Problem
I've been reading research papers on how to solve a peg solitaire using graph search, but all the papers kind of assume you know how to do the reduction(polynomial time conversion) from the peg solitaire to the graph, which I do not, but this is how I assumed it was done.
For those of you unfamiliar, here is a video that illustrates how to play this game, and here's an image of a board.

The goal is to only have one peg on the board and you get rid of pegs by jump one peg over another. A peg can only jump if the it's jumping into an empty space as shown in the picture above.
What I've tried
I had the idea of converting the problem to a tree where each node represents the state after an action is taken and each edge represents the action taken. So the root node would the initial state which is the board shown above then it's children would be the state of the board after any of the possible legal actions that can be taken.
So, for example:

Then the children of each of those nodes would be the possible moves for them and you can find a solution once you've reached depth 31 in the tree because there are 32 pegs and you win the game where there's only 1 left.
Is this the right approach? It feels a little too abstract because I'd have to represent the edges as peg moves, but that's weird cause they're usually numbers or constraint.

Your approach seems reasonable to me. The edges do not necessarily have to be numbers, but, if you wish, you could also encode the actions as numbers. For example, the weight of an edge could represent the "cost" of the corresponding action. If there's no natural cost associated with an action, then you can add a unit cost for each action.


text:
How do I represent a multi-dimensional state using a neural network?
I have a set of 15 unique playing cards from a deck of 52 playing cards. A given state is represented by the respective card values in the set of 15 cards, where the card value is a prime number associated with that card. For example, AH is represented by 3.
How should I represent a single state for the NN? Should it be a list of the 15 prime numbers representing the list of cards? I was hoping that I could represent a single state as the sum of each of all 15 prime numbers and then throw that sum through a sigmoid function. My concern, however, is that the NN will lose information if I reduce the dimension of the state to a single attribute (even if that attribute is unique to that state - the sum of n prime numbers is unique compared to the sum of any other n prime numbers).
How important is the dimensionality of each state for Deep Q Learning? I'd really appreciate even some general direction.

Are there always 15 cards under consideration in your state? For example, do the cards get "played" or split between players to make hands or sets which might vary in size from 15 etc? I can see that as a way that you lose information, unless you also track $n$ for each group of cards being represented.

The size would vary, but I can keep track of $n$.

@NeilSlater Since the size of $n$ is decreasing by 1 (a card is discarded) every round, would it be better to have $n$ neural networks for each hand length or just input an empty card (say a value of 0) to the remaining input nodes?

I don't want to answer in the comments. However, the clarification that n could vary does mean that just the pure sum of numbers is not enough to guarantee uniqueness because e.g. $2 + 3  = 5$ - so you cannot represent the whole state of play in the way that you hope

So, assuming I used some sort of binary or one-hot encoding, what would you suggest I do in regards to my previous question? Thanks for the help btw! I am quite new to NN

In my humble opinion, it seems like it is important to have them separated, if having a certain card can influence the result in some way that is not its prime value, instead of not only using the sum. But it depends on the game and its rules. For example:
If having 5 cards of hearts in the set of 15 cards makes you win the game, then if you only represent the state as the sum of the value of the cards, the DQN will never learn that it was not the whole 15 cards that caused you to win, but only a specific part of the set.

So that implies that my input layer would have 15 nodes (each is fed a number that represents a card in the list). Do I feed each node the prime number? Do I feed each node the "sigmoided" prime number?

@HarryStuart I'm fairly sure that In the input you can put the prime number if you want to represent the cards that way, or just some different number for each card.
What exactly do you mean by using a sigmoided prime number?

I mean squishing the number between 0 and 1 using the sigmoid function.

I don't think that would be necessary. By the way, for a simple representation like your case, sometimes linear function aproximators might work and converge better.

You can also use this kind of approach https://stats.stackexchange.com/questions/147110/card-games-and-neural-network-inputs


text:
Are there tools to help clean a large dataset so that it only contains faces?
I have a fairly large dataset consisting of different images with and without persons that I want to use for a project.
The problem is that I only want the pictures that contain faces, and it is best if there is only a crop of the face.
I already looked at Facenet and Openface, but I thought that there must be a simpler already trained solution just to sort the dataset so I can get started with my own project.

You could try importing them into a photo management app; they often have face recognition built in to help you identify people in your pictures.

@OliverMason Thanks, but i have about 30k pictures which probably takes a long time to import :/

I don't know of a tool but you could write a simple script to detect faces and crop it. It's quite simple with the Haar cascade in openCV to detect faces and use inbuilt functions to crop your image based on the size of the detected face.
Hope that helps !

I have found a script that does what i need:
https://github.com/leblancfg/autocrop

I developed a python script to crop faces using MTCNN. I found this to be the most accurate of all the face cropping algorithms at the expense of being somewhat slower.
The function I developed is on the kaggle website at https://www.kaggle.com/gpiosenka/detect-align-resize-rename-facial-images.
The markup first cell explains how to use it. In a nutshell this function will crop the image files in the input directory and store them in an output directory. If there are multiple faces in an image it selects and crops the largest facial image. It also gives you the option to "align" the cropped images. By align I mean the cropped image is rotated so the eyes are horizontal. You also have the option to resize the cropped image. Finally you also have the option to rename the images in a numerically sequential order and to change the image format to one you specify. This is convenient because if you download images from the internet they have ungodly names and come in a vast variety of image formats. Also the function checks if any images are duplicates of each other. If duplicates are present a listing is provided and you can elect to delete duplicate images. Normally when you build a data set you want to eliminate duplicates so that if you split the data set into train, test and validation sets there is no "leakage" between the sets.
I have used this function on thousands of images  that have faces in them and it works extremely well. The cropped results are just what you want.On rare occasions a cropping error may occur so it is always wise to review the cropped results.


text:
What are the value of the pixels of the convolved image?
I'm studying convolutional neural networks from the following article https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/.
If we take a grayscale image, the value of the pixel will be between 0 and 255. Now, if we apply a filter to our "new image", can we have pixels whose values are not included in this range? In this case, how can we create the convolved image?

The convolved image can be considered a feature map, where each new neuron represents some indication (or lack-there-of) of a feature in some receptive field of the original image, so no it does not need to be a valid image in the output.
If you specifically care for it to be an image as an output, you can do a couple of things:
1) normalize the produced feature map to some set range that youre working in (0-255 or 0-1)
2) make the filter a valid probability distribution, and you know the output will stay in the same range as the input (ex: Gaussian filters)

Thanks for the answer. Another question: in figure 9 it is written:" Black=negative" What has the author done in this case? Black was 0 pixel value before and then he claim that black pixels are negative pixel values. Why?

@MementoMori black was 0 when referring to just drawing an image in RGB space was the context he made the comment earlier, but now youre in a feature space (not 0-1 / 0-255), so when drawing it, for simplicity he mapped all negative values to 0 (black in RBG) and positive to 1 (white in RGB) just to give you a visual. He then uses the RELU (relu(x) = max(0, x)) to eliminate all negative values to show you how the feature map looks afterwards.

I think normalisation into range 0-1 is needed or at least to 5-10 cliping   because the values will become astronomical after many layers. Convolved image has a vector of features for each pixel. Take a RGB for example -> each color is a feature in one pixel, the next map will be like 'horisontal line, vertical line, circle' for one pixel surroundings.