text:
Simple text recognition with neural network
In my attempt at trying to learn neural network and machine learning I'm am trying to create a simple neural network which can be trained to recognise one word from a given string (which contains only one word). So in effect if one where to feed it a string containing the trained word but spelled wrong the network would be able to still recognise the word. Can anybody help me with some pseudo code or a start of a code. Or a general explanation of how to to this because I have read like 6 articles and 8 example projects and still have no clue how to do this

OCR is not perceived as a good example of AI, so your questions is not really about AI.

I know but I didn't find a proper stacked exchange to put this in

Welcome to AI.SE! Unfortunately, implementation questions are off-topic here, but if you have questions about a social or conceptual aspect of AI, we can help! (For example, a part of the articles you read that you didn't understand could be turned into something interesting.) You might be able to ask some form of this question on Stack Overflow after you do some more research and have a more specific problem statement.

If you maybe focus on how ocr recognition works using neural network, I think it would be fine.

An optimal solution for the task as stated, would be some alignment algorithm like Smith-Waterman, with a matrix which encodes typical typo frequencies.
As an exercise in NNs, I would recommend using a RNN. This circumvents the problem that your inputs will be of variable size, because you just feed one letter after another and get an output once you feed the delimiter.
As trainingsdata you'll need a list of random words and possibly a list of random strings, as negative examples and a list of slightly messed up versions of your target word as positive examples.
Here is a minimal character-level RNN, which consists of only a little more than a hundred lines of code, so you might be able to get your head around it or at least get it to run. Here is the excellent blog post by Karpathy to which the code sample belongs.

If I'm reading it correctly, this question has nothing to do with optical character recognition. You want to create a system that takes a digital string of characters as input, then finds the best match from a predetermined list of words. That sounds like a task for if-then-else logic and dictionary lookup. It might be possible to use a neural net, but not easy.
A neural net takes a fixed number of inputs, each of which are a value between zero and one. A major hurdle is that you probably want variable-sized inputs. Another hurdle is that you'll need to code the inputs some way onto numbers.
These hurdles can be overcome but they are tipoffs that neural networks aren't well-suited for the task.


text:
How can I oppose two AI agents with keras / tensoflow?
I am trying to use tensorflow / keras to play a text based game. The game opposes two players that play by answering questions by choosing an answer among the proposed ones.
Game resembles this:

Questions asked from player 1, choose value {0, 1, 2}
Player 1 chooses answer 1
Questions asked from player 2, choose value {0, 1}
Player 2 chooses answer 0
( and so on )

The issue is that I do not have any data to use for training the agents and it not possible to evaluate each actions of the agent individually.
My idea is to get 2 agents to play against each other and evaluate them depending on who won / lost ( the games are very short with about 20 to 30 decisions made for each player ).
The issue I have is that I do not know where to start.
I normalized my input, but I do not know how to get the 2 agents to compete, as I do not have any training data as shown in the tutorials and the agents have to complete a full game in order to evaluate their performance.

Keras/Tensorflow are mostly used of developing/training/deploying neural networks. For descision making problems, if you want to use machine learning, reinforcement learning is in most cases applied. Some reinforcement learning methods use neural networks (and therfore tensorflow) internally. You can check baseline implementations of different methods here.
When using reinfocement learnig, for most of the reinforcement learning methods, you do not need and preexisting data, but you need and environment (esentially a simulations of your game). If you interface your game to one of the baseline agents you can start the agents playing ant training ocurs during they play. You can find a tutorial to start an agent here.
If you only have a feedback (a reward) after a full game of 20 decisions, it will be really hard to train the agents, as you are facing a good example for what is called the problem of spare reward.
There are some ways to deal with sparse reward, like Hindsight Experience Replay and maybe Curiosity, but improving the density of the reward would be very beneficial.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Reinforcement Learning with Adaptive Action Magnitude
How to select an action in a state if the action does not necesarily cause the environment to change state?
Given 10 states ($S_0$ to $S_9$) and in each state $i$ there are two actions defined $(1,-1)$. $1$ increases a parameter of the environment and $-1$ decreases it.
For example, if the parameter is speed and it is currently 1 rad/s, corresponding to ${S_i}$, an action can be either increase or decrease this and therefore transition to the next state, ideally $S_{i-1}$ or $S_{i+1}$.
It is unclear how to formulate a reinforcement learning problem in this case. The problem is the same magnitude of the parameter change through an action.
The range of the parameter is (3, 25) and step size is 1. The problem is thath the responce of the environment is not the same in every satate. In some states a parameter change with magnitude 1 results in a state transition in some state the magintude proves to be to small to provoke a transition change.
For example, if the envitonment is instate $S_1$ and action 1 is applied, how can the magintude of the paremter be adopted in a way which assures that the environment will transition to state $S_2$?  How can the step in the paremeter change be made adaptive? Actually my environment is uncertain that is why I don't know exactly whether this action will take me to next state or not.
For your information, I am using Q learning off policy algorithm. Suppose state 9 is the goal state and  my Q table is $10 \times 2$.

There are more possible approaches to tackle this problem:

Use a reinforcement learning method which can cope with a continous state space. This would eliminate the need for discretizing the state space which in turn, if I understood correctly, leads to problems in transitioning between states. You can also consider selecting a reinforcement learning method that can cope with a continous state space and a continous action space.
Actions do not have to be immediate. If your action 1 is transitioning form $S_i$ to $S_{i+1}$ you can have a function wich continously increases the spead until the next state is reached and only then you consider the execution of the action complete. Furthermore, the magnitude of the actions can be dependent on the current state, if it assures state transition.
You can add more actions. Use RL for your advantage and define not just $+1$ and $-1$ but $+0.1$, $0.3$, $+0.5$, $+0.7$, $+1$ (and same for deacreasing the speed) etc. Add a negative reward for all actions which do not cause a state transition or which cause higher jump then needed. However, care must be taken to make sure you have the same speed fro each sampling. Eg. if you are in $S_1$ and the action +0.1 does not cause a state transition to $S_2$ you have to reset the state (if possible) or fail the epoch, since the state you are in is not only dependent on the action and the previous state. In other words the speed of e.g. $1 rad/s$ (with a tolarance band) defined as $S_1$ will be slightly higher and ending up in $S_2$ after applying a slight increase in the speed takes less change in the speed than from $S_1$ without the "unnoticed" speed change.

Ok thank you very much for a very nice explanation. Actually in my problem, I have finite states available (say 10). There are total 20 actions (say discrete value from 3 to 23). I know state 1 is bad and state 10 is good. In each state, i can only try few actions (not all 20). The problem actually lies here. When I do simulation, I know if I increase action by some number relative to previous action I will transition to next state. But In real implementation, it does not. Can I do like binary decisions (1 means keep increasing action and -1 keep decreasing until I transition to next state.

So you are training in a virtual environment (simulation) which does not reflect your real setup used in the expoitation?

Yes, I still did not implement it in real but i am worried before implementation. My real environment is patient and so I can only try when I am confident about my algorithm. The virtual environment is somehow relating with real but not 100 percent.

Can I make Q table like states * decision actions and decision actions are +1,-1. 1 means increase the previous action by some number and -1 decrease it by some number until i reach to next state. and I will update Q table only when  i transition to next state.

As stated in point 2, the action does not have to be immediate and if it brings your environment to the same state in reality as in simulation it does not have to be the same implementation or it does not have to act in the same way.

You can also have a different parameter in the action execution which links action magnitude to the current state. e.g for a slow speed state apply less magnitude. This does not necessarily have to be in the Q table.

Thank you for the perfect explanation. I got your point.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What trait of a planning problem makes reinforcement learning a well suited solution?
Planning problems have been the first problems studied at the dawn of AI (Shakey the robot). Graph search (e.g. A*) and planning (e.g. GraphPlan) algorithms can be very efficient at generating a plan. As for problem formulation, for planning problems PDDL is preferred. Although planning problems in most cases only have discrete states and actions, the PDDL+ extention covers continuos dimensions of the planning problem also.
If a planning problem has non-deterministic state transitions, classical planning algorithms are not a well suited solution method, (some) reinforcement learning is considered to be a well suited solution in this case. From this point of view, if a planning problem has a state transition probability less than 1, classical planning methods (A*, GraphPlan, FF planner, etc.) are not the right tool to solve these problems.
Looking at reinforcement learning examples, in some cases environments are used to showcase reinforcement learning algorithms which could be very well solved by search/planning algorithms. They are fully deterministic, fully observable and sometimes they even have discrete action and state spaces.
Given an arbitrary fully deterministic planning problem, what is/are the chartacteristic(s) which make reinforcement learning, and not "classical planning" methods better suited to solve the problem?

Note that, "pure" reinforcement learning algorithms, like Q-learning, are not planning algorithms. In RL, planning algorithms are e.g. value iteration or policy iteration, which require a model of the environment to solve the planning problem. See this answer. So, you may want to clarify that in your post.

I inverted the statement to make it less general. Not all rl algo can solve planning, but non-deterministic planning can be solved by rl algo

If by planning you mean finding a policy, yes, RL can be used for that. However, typically, in planning, you assume to have a model (i.e. the transition system or function), but not all RL algorithms use a model of the environment (so not all RL algorithms are "planning" algorithms in this sense), so I would clarify this: i.e. algorithms like Q-learning don't "plan" before executing, they execute (i.e. immediately take actions in the environment and observe the outcomes). I think you're referring to algorithms like value iteration. Anyway, this is an interesting question.

planing is generating a sequence of decisions which leads to the goal. If I train a policy, I can use the policy to generate a plan, right? If I have a planning problem I have a model. If want to solve it with model free rl, can I just use the model as an environment? It might be very inefficient, but it could be used that way, right?

Model-free algorithms, like Q-learning, don't use the transition function, i.e. the model, which is often not available. Q-learning is not considered a planning algorithm because it actually doesn't plan anything (i.e. it doesn't try to find the plan before executing the actions: it actually executes the actions, which can be catastrophic, and learns from trial-and-error, i.e. Q-learning is a learning algorithm, because it learns from experience).

That's why I'm saying that you should edit your post to clarify the difference between "RL algorithm" (like Q-learning) and "dynamic programming algorithm" (like value iteration). Both these algorithms are used to solve MDPs, i.e. find a policy, but they are not both planning algorithms. If you read this answer, you should understand better what I'm talking about.

I am not sure I agree with you, but that would be a different question... Maybe I will post that question also. I am really interested generally when to use rl vs. classical planning and/or control theory and/or classical optimization. I added the (some) qualifier to make sure to exclude the rl algo.s which would not be usable.


text:
How to choose method for solving planning problems?
There are many methods and algorithms dealing with planning problems.
If I understand correctly, according to Wikipedia, there are classical planning problems, with:

a unique known initial state,
duration-less actions,
deterministic actions,
which can be taken only one at a time,
and a single agent.

Classical planning problems can be solved using classical planning algorithms. The STRIPS framework for problem description and solution, using backward chaining) of the GraphPlan algorithm can be mentioned here.
If actions are non-deterministic, according to Wikipedia, we have a Markow Decision Process (MDP), with:

duration-less actions,
nondeterministic actions with probabilities,
full observability, or partial observability for POMDP
maximization of a reward function,
and a single agent.

MDPs are mostly solved by Reinforcement Learning.
Obviously, classical planning problems can also be formulated as MDPs (with state transition probabilities of 1, i.e. deterministic actions), and there are many examples (e.g. some OpenAI Gyms), where these are successfully solved by RL methods.
Two questions:

Are there some characteristics of a classical planning problem, which makes MDP formulation and Reinforcement Learning a better suiting solution method? Better suiting in the sense that it finds a solution faster or it finds the (near)optimal solution faster.
How do graph search methods like A* perform with classical planning problems? Does STRIPS with backward chaining or GraphPlan always outperform A*? Outperform in the sense of finding the optimal solution faster.

This is post contains interesting and useful information (in fact, I was one of the people that had upvoted it), but you're asking at least 3 distinct questions here (the title's question and the other 2 questions at the end of the post), each of which should be asked in its own separate post, which makes this post too broad. I would suggest that, if you're again/still interested in these topics, you try to ask more specific questions, each of which in its separate post.

I will reformulate. In the meantime I made significant progress in better understanding/answering the question.

I would suggest that you create a new post and delete this one. A new post can become hot, while this old post no.


text:
Deep NN architecture for predicting a matrix from two matrices
Recently my friend asked me a question: having two input matrices X and Y (each size NxD) where D >> N, and ground truth matrix Z of size DxD, what deep architecture shall I use to learn a deep model of this representation?

N ~ is in the order of tens
D ~ is in the order of tens of thousands

The problem is located in the domain of bioinformatics, however, this is more of an architectural problem. All matrices contain floats.
I tried first a simple model based on a CNN model in keras. I've stacked input X and Y into an Input Matrix of size (number of training examples, N, D, 2). Outputs are of size (number of training examples, D, D, 1)

Conv2D layer

leaky ReLU

Conv2D layer

leaky ReLU

Dropout
Flattening layer
Dense (fully connected) of size D

leaky ReLU
droout

Dense (fully connected) of size D**2 (D squared)

leaky ReLU
droout

Reshaping output into (D,D,1) (for single training set)

However, this model is untrainable. It has over billion parameters for emulated data.
(Exactly 1,321,005,944 for my randomly emulated dataset)
Do you find this problem solvable? What other architectures I might try to solve this problem?
Best.

Is there any relation between X and Y through Z? Perhaps X1*Z = Y1?

There is no known relationship between X and Y. We only know that the output matrix Z is symmetrical, therefore we could predict the "upper/lower triangle" of it.

Is there any other relationships between the matrices? If Z isnt a transformation is it part of a decomposition? Although the relationship between X and Y isnt know, is the goal to find this relationship? What should Z be representing?

Just to be clear, you have an output (and ground truth) of at least 10,000 x 10,000, or 100 million elements, and you would like to generate it to match an observed distribution over 20 x 10,000, or 200 thousand elements? How many training examples do you have?

Are there any simplifying assumptions that are reasonable to make in the problem domain? The big ones that may help would be sparsity (only some small percentage of inputs and outputs are populated, or different from some mean distribution), or smoothness (the 10,000 x 10,000 output could be approximated reasonably by e.g. a 1,000 x 1,000 output, and maybe has similar properties to a real-world image)

@NeilSlater thanks for your questions. My friend responded, and said that he has millions of training examples. In addition, the output/truth is 49,995,000 values and it is symmetrical matrix. It could be approximated, as he states, perhaps by a factor of 1000 and that would be still reasonable

I understand from that the matrix is smooth, perhaps representing a grid of evaluations over a complicated but smooth function of the inputs? That's a lot of data that your friend has. Millions of records, each of which contains ~ 25M numerical values. We're taking about a dataset size getting up to a Petabyte here . . . not something I have any experience of. If an approximation with downsampled outputs (and maybe inputs) might have value, I suggest start with that as your model. Reduce the grid sizes as much as you can to start, see if that model would be useful.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Literature on computational modelling involving neuronal ensemblies
Straying from the current trends in deep learning, there is an, arguably, interesting idea of neuronal ensembles possibly providing an alternative to the current "layered feature detectors" framework for neural network construction by being considered a basic computational unit instead of one feature detecting neuron. This idea certainly has at least some presence in neuroscience circles, but I found it hard to find studies, attempting to obtain a working computational ensemble based model, which would try to solve any of the existing computer vision/NLP tasks or anything of the sort. This may be just due to me looking in the wrong places, but in any case, I would appreciate any references to papers, exploring building neural network architectures with neuronal ensemblies involvement.
Just to be clear, I would be interested in any papers on computational modelling of ensemblies even if they are not trying to solve any particular ML task, but it would be better, if the topic of the research is more closely aligned with computer science instead of neurobiology even if CS connection is of a more exotic kind; for example, paper trying to see, whether you can store different concepts and their relations in the ensemble based network is more desirable than paper, trying to accurately model individual neuron and synaptic plasticity dynamics and see, that ensembles emerge, if you scale the system. But again, I would be glad to get references to research in both of these example topics and many more.


text:
Can programs like AlphaGo be said to be means of dealing with computational intractability?
I mean this in the sense that Go is unsolvable but AlphaGo seems able to make choices that are consistently more optimal than a human player's choices.
It is my understanding that Game Theory turned out to have limited applications in real world scenarios because of the profound complexity of such scenarios and degree of hidden information.  Is it fair to say that there is now a method for dealing with this?
I fully understand that Go is a game of complete information, which has a very specific meaning, but it occurs to me that the inability to generate a complete game tree (computational intractability) could be seen as form of incomplete information, even if it is not traditionally thought of in those terms.

I should probably note that my perspective is one of a "serious" game designer, where complexity serves the same function as chance and hidden information, which is to say as a balancing factor that "levels the playing field".

I think that the technique AlphaGo used to solve the computational intractability problem of the search space are not new, it uses the Monte Carlo Tree Search. The real innovation in AlphaGo was to figure out how to compute the evaluation function of a move, that was the really tricky part. For this they used combinations Deep and Reinforcement learning techniques.

So basically a big step in a greater overall movement, with Go as a convenient proving ground.  Am I wrong in thinking there are profound implications to where we're getting in this field?

You are not wrong but we need make sure that we don't get into the mindset of "When you have hammer everything looks like a nail". History have shown again and again that once we develop some new approach to a problem, people start to apply that approach to various problems and there is excitement but in the end the approach runs out of fuel and we need to look for something else

In some way the followup question was a reaction to a Business Insider article which touched on an initial application with the conclusion that the DeepMind acquisition looks like it will pay off, but didn't really take the next step to the idea that there are a myriad of technological systems where this could apply.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Feasibility of sorting a basket of clothes in the real world
I'm wondering how feasible it is to create a machine that can separate clothing from a basket.
At the most basic level it would distinguish between tops, pants, button downs and socks
Programmatically, I'd image this would require training a neural network to recognize these items, but in real time it becomes exponentially difficult to do this in a small space at a fast rate:

pick up an item
lay it in such a way that is recognizable
deduce whether it is a top, button down, etc.
sort it accordingly

If this sounds ridiculous please let me know...
If it is possible :
would this be based on some sort of computer vision?
or only a well trained neural network?
Any insight is much appreciated!

Peter Abbeel does work in deep learning for robotics, and one of the projects they've tackled is manipulating clothes. Here's a video from 2011 of their robot folding laundry (one piece at a time).
There are also companies attempting to market this; seven dreamers makes the Laundroid and Foldimate claims that it will start taking pre-orders in 2017.


text:
Can I limit the possible choices for a computer vision framework to recognize?
To risk giving away too much info, im building a piece of hardware with the job of  identifying the object in front of it.
If it can only be one of three different items, how can I tell the computer with simplecv?
Basically, I've found a way to limit the choices down to just a handful of potential objects, which should increase the probability of it recognizing the object correctly. Is there a way to limit the choices for the algorithm?
Me: hey raspberry pi - you see that thing in front of you?
Raspberry Pi: That thing? Ohh you mean that piece of food that might be a ham and cheese sandwich, but also kinda looks like a fish, with a slight twist of pe-
Me: - whoa okay, hold on! It's either a grilled cheese sandwich, or an apple
RPI: ohhh well that's easy! it's clearly (with 98% confidence) a grilled cheese sandwich
Any thought are appreciated!

I would recommend using a convolutional neural network. CNNs are the bleeding edge for image recognition currently. They should be able to solve your problem with a high accuracy, just maybe without the sort of dialogue that you provided in your question. In order to train a convolutional neural network, you would have to use a different computer than your desktop and "deploy" it to your raspberry pi. This means that your desktop computer would find the optimal wight configurations for your neural network and the images you want to detect, and then you would be able to send the neural network to your raspberry pi in order to be used. If you want to, you could train your entire convolutional neural network from scratch, using random weights to begin, or you could use a process known as transfer learning where you take an existing neural network, usually deep, and train it a little bit further according to your existing data set. In order to train your neural network from scratch, you will need a very large volume of sample images to train it. In order to do transfer learning, you will only need about 1000 images for each of your classes in order to achieve very accurate results for your specific task. For specific libraries that you can use, I would recommend using Matlab if you have access to it as it is extremely easy to train and run deep neural networks, and it is easy to deploy the model to your raspberry pi. You can implement transfer learning in Matlab with 10 lines of code. Next, if you do not have access to Matlab, I would recommend if you have some python experience, using tensorflow. Tensorflow supports a lot of deep learning algorithms and should be able to for your problem although it will not be as easy as using MATLAB. Finally, Microsoft has a toolkit called CNTK which is fairly similar to tensorflow, but instead you can use it in C# or other .net languages.


text:
Is this a good way to represent Connect 4 to a Neural Network?
I'm attempting to make a bot for the Connect 4 competition on http://riddles.io
My bot isn't horrible, like it's getting up the ladder, but it cannot compete with the winning bots.
I'm using a Neural Network which is fully connected with one hidden layer. Internally it uses the sigmoid function as the activator in each Neuron. I've trained it over 500,000 games with TD-Lambda back propagation. The alpha and beta values (i.e. learning rates) are set to 0.1 each, and the lambda for the eligibility trace is set to 0.7. There are 2 outputs nodes, one to give the value of this position for Player 1, the other to give the value of the position for Player 2. Upon a win, these are back-propagated with -1 for a loss and 1 for a win, on a draw they are both back-propagated with a 0.
There is a bias input and weight for every neuron as well.
All of the weights are initialised to a random value between +-4*sqrt(6/totalNumberWeights).
The board state is represented to the network as:

For each space in the board, 2 values:

For the first value if player 1 occupies this space it's a 1, otherwise it's a 0
For the second value if player 2 occupies this space it's a 1, otherwise it's a 0
If both are 0 it would mean it's a free spot

For each space on the board another 2 values:

For the first value, if placing a token here would result in a Connect 4 for player 1, it's a 1, otherwise a 0
For the second value, if placing a token would get player 2 a Connect 4, then it's a 1 otherwise a 0
So if no one would win by placing a token here, it's two 0 values for these two inputs

Two final values, the first indicating whether it's player 1's turn or not, then second indicating if it's player 2's turn or not

What I see when my bot is playing is that it makes what I assume are somewhat clever moves, like it's preparing for both horizontal and diagonal moves.
But when the other bot will get a Connect 4 on the next move, my bot fails to place a token there. The best bot in the comp seems capable of setting itself up to get three in a row with a free space on both sides, so that it will definitely get a Connect 4. Again, my bot does not seem to be able to see this coming.
What I think the issue is, other than potentially my learning rates, is that I've represented the board in a bad way. Is there a better way to represent it to the Network so that it can more accurately estimate the value of a board state, and so that it doesn't fail to identify an immediate Connect 4 threat?

When you say "back propagate" you mean via the eligibility trace or lambda return, in order to set the TD target? Or do you literally mean that you back propagate reward signals to the NN for the end position? Are you also using some kind of forward/planning search with alpha/beta pruning?

Hi @neil I'm back propagating after every move, NN plays both sides. I adjust the weights and eligibility traces for the current state using the previous states values according to https://web.archive.org/web/20100625030049/http://www.cse.unr.edu/robotics/bekris/cs482_f09/sites/cse.unr.edu.robotics.bekris.cs482_f09/files/backgammon.pdf At the terminal state I change 'y1' in the above equations to be either 0 for a draw or +-1 for win/lose. I use a 2-ply search to find the best move to make based upon the average value of all the opponents possible response move.

Oh and I reset eligibility traces after every game.


text:
TF Keras: How to turn this probability-based classifier into single-output-neuron label-based classifier
Here's a simple image classifier implemented in TensorFlow Keras (right click to open in new tab): https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/quickstart/advanced.ipynb
I altered it a bit to fit with my 2-class dataset. And the output layer is:
Dense(2, activation=tf.nn.softmax);

The loss function and optimiser are still the same as in the example in the link above.
loss_fn   = tf.losses.SparseCategoricalCrossentropy();
optimizer = tf.optimizers.Adam();

I wish to turn it into a classifier with single output neuron as I have only 2 classes in dataset, and sigmoid does the 2 classes good. Tried some combinations of output activation functions + loss functions + optimisers, but the network doesn't work any more (ie. it doesn't converge).
For example, this doesn't work:
//output layer
Dense(1, activation=tf.sigmoid);

//loss and optim
loss_fn   = tf.losses.mse;
optimizer = tf.optimizers.Adagrad(1e-1);

Which combination of output activation + loss + optimiser should work for the single-output-neuron model? And generically, which loss functions and optimisers should pair well?

Advice from Neil, yes, output targeting class labels is still classification.
Output range is in contiguous range:

This is regression
For example: Linear activation, it has full numeric range of outputs.

Output targeting class labels is classification:

Single output neuron with sigmoid-like functions. This will classify to 2 classes, although Y data can be normalised to classify more classes.
Multiple output neurons (probabilities of classes) with sigmoid-like functions (softmax is mainly used). This will classify to 2 or more classes.

Multiple combinations of loss functions and optimisers can make the single-neuron output layer work, with different configs for them. Note that learning rates of different optimisers are different, some take 1e-1, some need 1e-3 for good training.
For example, this combination should work:
loss_fn   = tf.losses.LogCosh();
optimizer = tf.optimizers.RMSprop(1e-3);

From my trying out, these other combinations also work for single output neuron with my data (Adam, Adamax, Nadam, RMSprop work when learning_rate=1e-3 instead of 1e-1):
                         Adadelta  Adagrad  Adam  Adamax  Ftrl  Nadam  RMSprop  SGD
BinaryCrossentropy       Yes       Yes      --    --      --    --     --       Yes
CategoricalCrossentropy  Yes       --       --    --      --    --     --       --
CategoricalHinge         --        --       --    --      --    --     --       --
CosineSimilarity         --        --       --    --      --    --     --       --
Hinge                    Yes       Yes      --    --      --    --     --       Yes
Huber                    Yes       Yes      --    --      --    --     --       Yes
LogCosh                  Yes       Yes      --    --      --    --     --       Yes
Poisson                  Yes       Yes      --    --      --    --     --       Yes
SquaredHinge             Yes       Yes      --    --      --    --     --       Yes

KLD: lambda a,b: KLD(a,b)
MAE,MAPE,MSE,MSLE: lambda a,b: Mxxx(a,b)
The above lambdas are direct functions, not classes like SGD, Adadelta, etc.
SparseCategorialCrossentropy: Seems not working with single output neuron.


text:
Backpropagation: Chain Rule to the Third Last Layer
I'm trying to solve dLoss/dW1. The network is as in picture below with identity activation at all neurons:

Solving dLoss/dW7 is simple as there's only 1 way to output:
$Delta = Out-Y$
$Loss = abs(Delta)$
The case when Delta>=0, partial derivative of Loss over W7 is:
$\dfrac{dLoss}{dW_7} = \dfrac{dLoss}{dOut} \times \dfrac{dOut}{dH_4} \times \dfrac{dH_4}{dW_7} \\
= \dfrac{d(Out-Y)}{dOut} \times \dfrac{d(H_4W_{13} + H_5W_{14})}{dH_4} \times \dfrac{d(H_1W_7 + H_2W_8 + H_3W_9)}{dW_7} \\
= 1 \times W_{13} \times H_1$
However, when solving dLoss/dW1, the situation is very different, there are 2 chains to W1 through W7 and W10, and now, how should the chain for $\dfrac{dLoss}{dW_1}$ be?
Furthermore, at an arbitrary layer, with all outputs of all layers already calculated plus all gradients of weights on the right side also calculated, what should a formula for $\dfrac{dLoss}{dW}$ be?

I finally solved it out but it's long, it's not only chain rule, it includes quotient rule too. And, this is only the third last layer, once a DNN has more layers then it's more complex.
$\dfrac{dLoss}{dW_1} = \dfrac{d}{dW_1}(Out-Y) = \dfrac{d}{dW_1}Out = \dfrac{d}{dW_1}(H_4W_{13} + H_5W_{14}) \\= \dfrac{d}{dW_1}H_4W_{13} + \dfrac{d}{dW_1}H_5W_{14} \\= W_{13} \times \dfrac{d}{dW_1}H_4 + W_{14} \times \dfrac{d}{dW_1}H_5 \\= W_{13} \times \dfrac{d}{dW_1}(H_1W_7 + H_2W_8 + H_3W_9) + W_{14} \times \dfrac{d}{dW_1}(H_1W_{10} + H_2W_{11} + H_3W_{12}) \\= W_{13} \times \dfrac{d}{dW_1}H_1W_7 + W_{14} \times \dfrac{d}{dW_1}H_1W_{10} \\= W_{13}W_7 \times \dfrac{d}{dW_1}H_1 + W_{14}W_{10} \times \dfrac{d}{dW_1}H_1 \\= (W_{13}W_7 + W_{14}W_{10}) \times \dfrac{d}{dW_1}(X_1W_1 + X_2W_2) \\= (W_{13}W_7 + W_{14}W_{10}) \times X_1$


text:
How to plot Loss Landscape with more than 2 weights in the network
For a single neuron with 2 weights, I can plot the loss landscape and it looks like this (OR data, sigmoid activation, MAE loss):

But, when the neuron accepts more inputs, which means more than 2 weights required, or when there are more neurons, more layers in the network; how should the 3D loss landscape be plotted?

I don't think you can, as the weights increase so does the number of dimensions. You could perhaps plot 3 weights using color in the graph as the 4th dimension, but I don't think you really could for more

@Recessive oh, but i can see some very complex loss landscape this way: https://www.pyimagesearch.com/wp-content/uploads/2019/10/train_val_loss_landscape.png

the image in the comment right above is from this article: https://www.pyimagesearch.com/2019/10/14/why-is-my-validation-loss-lower-than-my-training-loss/

Nope, that's actually just a mislabelled graph by medium. The original is about halfway down this page: http://firsttimeprogrammer.blogspot.com/search/label/Machine%20Learning. You'll see this is a graph for f(x,y) = x**2 + y**2 –2*x*y. You can see the (falsely) modified image halfway down this page: https://medium.com/@RosieCampbell/demystifying-deep-neural-nets-efb726eae941. EDIT: The medium post actually specifies this graph is for only 2 weights. It's just the pyimagesearch that loses this in translation.

It seems not possible to plot loss values (z) against all combinations of weights in all layers, especially when the network is big with thousands or millions of params; in that case, the number of points to plot is too too big.
And also, the 3D space can't be used to plot more than 3 dimensions.
However, with a deep network with lots of weights, these can be plotted:

Loss value against any pair of 2 weights
Turn the layer right before output layer (single neuron) into a layer of 2 neurons, and loss can be plotted against these 2 weights (but doesn't make much sense as the meaning of loss value depends on all other weights also)

Example plot when there are 2 neurons in the layer right before output layer (of 1 neuron):


text:
How many hidden layers are needed for this training data set
I'm trying to separate classes in 3D space, the data are as in the sketch below:

There are 3 classes: 0,1,2; and with the look into the sketch, it seems that I need 3 planes to separate the classes, thus how many hidden layers should be in the DNN? Any roughly how many neurons in each layer?
Some say the number of hidden layers is roughly the number of separation times, so I put 3 hidden layers and it worked! But any reasons behind that simple measure?

You need to perform Hyperparameter Tuning to identify -

Number of hidden layers.
Number of neurons in each of the hidden layers.
Dropout
The activation function you use in each of your hidden layers.

There parameters are only related to how you build your model. There are others that relate to training like batch size, number of epochs and so on. Your model's performance ultimately depends on how well you tune your hyperparameters.
Also note that hyperparameter tuning is a trial and error task because it depends on several factors that may not be obvious to us. With experience, experts do build certain thumb rules about what may be the right choice, but there is no way to generalize it. "Some say the number of hidden layers is roughly the number of separation times" - is just another thumb rule. You simply need to find out what best suits your scenario.


text:
LSTM network doesn't converge, what should be changed?
I'm testing out TensorFlow LSTM layer text generation task, not classification task; but something is wrong with my code, it doesn't converge. What changes should be done?
Source code:
import tensorflow as tf;

# t=0  t=1    t=2  t=3
#[the, brown, fox, is,     quick]
#   0  1      2    3       4
#[the, red,   fox, jumps,  high]
#   0  5      2    6       7

#t0 x=[[the],  [the]]
#   y=[[brown],[red]]
#t1 ...
#t2
#t3
bsize = 2;
times = 4;

#data
x = [];
y = [];
#t0        the:     the:
x.append([[0/6],   [0/6]]); #normalise: x divided by 6 (max x)
#          brown:   red:
y.append([[1/7],   [5/7]]); #normalise: y divided by 7 (max y)
#t1
x.append([[1/6],   [5/6]]);
y.append([[2/7],   [2/7]]);
#t2
x.append([[2/6],   [2/6]]);
y.append([[3/7],   [6/7]]);
#t3
x.append([[3/6],   [6/6]]);
y.append([[4/7],   [7/7]]);

#model
inputs  = tf.placeholder(tf.float32,[times,bsize,1]) #4,2,1
exps    = tf.placeholder(tf.float32,[times,bsize,1]);

layer1  = tf.keras.layers.LSTMCell(20)
hids1,_ = tf.nn.static_rnn(layer1,tf.split(inputs,times),dtype=tf.float32);

w2      = tf.Variable(tf.random_uniform([20,1],-1,1));
b2      = tf.Variable(tf.random_uniform([   1],-1,1));
outs    = tf.sigmoid(tf.matmul(hids1,w2) + b2);

loss  = tf.reduce_sum(tf.square(exps-outs))
optim = tf.train.GradientDescentOptimizer(1e-1)
train = optim.minimize(loss)

#train
s    = tf.Session();
init = tf.global_variables_initializer();
s.run(init)

feed = {inputs:x, exps:y}
for i in range(10000):
  if i%1000==0:
    lossval = s.run(loss,feed)
    print("loss:",lossval)
  #end if
  s.run(train,feed)
#end for

lastloss = s.run(loss,feed)
print("loss:",lastloss,"(last)");
#eof

Output showing loss values (a little different every run):
loss: 3.020703
loss: 1.8259083
loss: 1.812584
loss: 1.8101325
loss: 1.8081319
loss: 1.8070083
loss: 1.8065354
loss: 1.8063282
loss: 1.8062303
loss: 1.8061805
loss: 1.8061543 (last)

Colab link:
https://colab.research.google.com/drive/1TsHjmucuynCPOgKuo4a0hiM8B8UaOWQo

Apologies!  This is out-of-scope for SE:AI.  (We deal mostly with theory, as opposed to troubleshooting.)  You might want to consider asking on Overflow.

writing here my suggestion, because i haven't earned the right to comment yet.
Your main "problem" could be your loss function. It converges, this is why your loss value is decreasing. So I suggest to let it maybe train longer.
Alternatively you could change the loss function to fit your need. For example you could use:
loss  = tf.reduce_mean(tf.square(exps-outs))

You will get a smaller loss value which decreases clearly after every output.
I hope this helps :)

yes Fabian is right, if you increase the range, you get convergence:

seems it won't converge as there's a conflict, illogical: x=0 --> y=1 together with x=0 --> y=5

reduce_mean or reduce_sum is kinda the same

I'm still working on how to make the code work for text generation, but the following converges and work for text classification:
import tensorflow as tf;
tf.reset_default_graph();

#data
'''
t0      t1      t2
british gray    is => cat (y=0)
0       1       2
white   samoyed is => dog (y=1)
3       4       2
'''
Bsize = 2;
Times = 3;
Max_X = 4;
Max_Y = 1;

X = [[[0],[1],[2]], [[3],[4],[2]]];
Y = [[0],           [1]          ];

#normalise
for I in range(len(X)):
  for J in range(len(X[I])):
    X[I][J][0] /= Max_X;

for I in range(len(Y)):
  Y[I][0] /= Max_Y;

#model
Inputs   = tf.placeholder(tf.float32, [Bsize,Times,1]);
Expected = tf.placeholder(tf.float32, [Bsize,      1]);

#single LSTM layer
#'''
Layer1   = tf.keras.layers.LSTM(20);
Hidden1  = Layer1(Inputs);
#'''

#multi LSTM layers
'''
Layers = tf.keras.layers.RNN([
  tf.keras.layers.LSTMCell(30), #hidden 1
  tf.keras.layers.LSTMCell(20)  #hidden 2
]);
Hidden2 = Layers(Inputs);
'''

Weight3  = tf.Variable(tf.random_uniform([20,1], -1,1));
Bias3    = tf.Variable(tf.random_uniform([   1], -1,1));
Output   = tf.sigmoid(tf.matmul(Hidden1,Weight3) + Bias3);

Loss     = tf.reduce_sum(tf.square(Expected-Output));
Optim    = tf.train.GradientDescentOptimizer(1e-1);
Training = Optim.minimize(Loss);

#train
Sess = tf.Session();
Init = tf.global_variables_initializer();
Sess.run(Init);

Feed = {Inputs:X, Expected:Y};
for I in range(1000): #number of feeds, 1 feed = 1 batch
  if I%100==0:
    Lossvalue = Sess.run(Loss,Feed);
    print("Loss:",Lossvalue);
  #end if

  Sess.run(Training,Feed);
#end for

Lastloss = Sess.run(Loss,Feed);
print("Loss:",Lastloss,"(Last)");

#eval
Results = Sess.run(Output,Feed);
print("\nEval:");
print(Results);

print("\nDone.");
#eof


text:
Why is embedding important in NLP, and how does autoencoder work?
People say embedding is necessary in NLP because if using just the word indices, the efficiency is not high as similar words are supposed to be related to each other. However, I still don't truly get it why.
The subword-based embedding (aka syllable-based embedding) is understandable, for example:
biology   --> bio-lo-gy
biologist --> bio-lo-gist

For the 2 words above, when turning them into syllable-based embeddings, it's good because the 2 words will be related to each other due to the sharing syllables: bio, and lo.
However, it's hard to understand the autoencoder, it turns an index value into vector, then feed these vectors to DNN. Autoencoder can turn vectors back to words too.
How does autoencoder make words related to each other?

The information you are probably missing is that word embeddings are learned on the basis of context. For example, you might try to predict a vector for a word from the wordvectors of the other words in the same sentence.
This way word vectors of words that occur in similar contexts will turn out to be similar. You can think of it as word vectors not encoding the word themselves but the contexts in which they are used. Of course ultimately that is the same.

The subword-based embedding is rather visual and easily understandable. However, the autoencoder embedding is what machines understand the componential meaning of words.
1) An autoencoder embedding layer can be trained together with other layers to fit with the relation of data in dataset.
2) Or the embedding layer can be kept unchanged as used as a function, required that the embedding layer must have been trained on a similar task like said in 1)
And as stated in the question, embedding is important in NLP as word indices don't show the full meanings of words, they must be separated into embedding values for better efficiency.
TensorFlow random uniform trainable embedding: https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding
TensorFlow utility class for creating subword-based encoder: https://www.tensorflow.org/datasets/api_docs/python/tfds/features/text/SubwordTextEncoder


text:
How to map X to Y for TensorFlow RNN training data
Usually for DNN, I have the training data of matching X (2D) to Y (2D), for example, XOR data:
X = [[0,0],[0,1],[1,0],[1,1]];
Y = [[0],  [1],  [1],  [0]  ];

However, RNN seems strange, I don't get it how to match X to Y, input of RNN layer is 3D and output is 2D (rightclick to open in new tab): https://colab.research.google.com/drive/17IgFuxOYgN5fNO9LKwDijEBkIeWNPas6
import tensorflow as tf;

x = [[[1],[2],[3]], [[4],[5],[6]]];
bsize = 2;
times = 3;

#3d input
input = tf.placeholder(tf.float32, [bsize,times,1]);

cell  = tf.keras.layers.LSTMCell(20);
rnn   = tf.keras.layers.RNN(cell);
hid   = rnn(input);

sess = tf.Session();
init = tf.global_variables_initializer();
sess.run(init);

#results in 2d
print(sess.run(hid, {input:x}));

The example data seen on https://www.tensorflow.org/tutorials/sequences/recurrent are:
 t=0  t=1    t=2  t=3     t=4
[the, brown, fox, is,     quick]
[the, red,   fox, jumped, high]

How to map these data from X (3D input for RNN layer) to Y (2D)? (Y is 2D because RNN layer output is 2D).

I'm not 100% sure I underststood the question but I'll try to answer it. By default keras is returning a tensor of shape: (batch_size, state_size) which corresponds to the last hidden step for each batch. If you want to return all the hidden states you need to set return_sequences to True, and it will return a 3d tensor with shape: (batch_size, timesteps, output_size).

@razvanc92 actually 2d or 3d is the side matter, the main thing i would like to ask is how to create the training data X, Y for the brown-fox-red-fox example data on  https://www.tensorflow.org/tutorials/sequences/recurrent

The very first step you need to do, is to embed each work into a fixed size vector. Afterwards from my understanding, at time t=0 the input will be X[:, 0] or [the, the] in your case. The target will be the next time stamp t=1 X[:,1] or [brown, red]. Following this logic you got X already, and for Y it will be X[:, 1:] (all the words from X, pushed by one plus probably a special character to mark the end of the sentence.).

@razvanc92 i know about embedding, but i'm just trying out with class index first

I found out how to get 3D output from LSTMCell so that I can matmul with output weights + biases and subtract with expected values:

Inputs & expecteds should be: placeholder(,[times,batch_size,num_inp]) instead of batch_size first then times. However, tf.keras.layers.LSTM will ask for [batch_size,times,num_inp]
Use tf.nn.static_rnn with a list of inputs, instead of 1 input

Source code:
import tensorflow as tf;

x = [[[1],[2],[3]],[[4],[5],[6]]];
times = 2;
bsize = 3;

#3d input
inputs = tf.placeholder(tf.float32, [times,bsize,1]);

cell   = tf.nn.rnn_cell.BasicRNNCell(20);
hids,_ = tf.nn.static_rnn(cell,tf.unstack(inputs,times),dtype=tf.float32);

sess = tf.Session();
init = tf.global_variables_initializer();
sess.run(init);

#results in 2d
print(sess.run(hids, {inputs:x}));


text:
Is it still called linear separation with a layer of more than 1 neuron
A single neuron will be able to do linear separation. For example, XOR simulator network:
x1 --- n1.1
   \  /    \
    \/      \
             n2.1
    /\      /
   /  \    /
x2 --- n1.2

Where x1, x2 are the 2 inputs, n1.1 and n1.2 are the 2 neurons in hidden layer, and n2.1 is the output neuron.
The output neuron n2.1 does a linear separation. How about the 2 neurons in hidden layer?
Is it still called linear separation (at 2 nodes and join the 2 separation lines)? or polynomial separation of degree 2?
I'm confused about how it's called because there are curvy lines in this wiki article: https://en.wikipedia.org/wiki/Overfitting

I found out the curvy zigzag green line is not polynomial as if it were polynomial, a vertical line won't cut that curvy line more than 1 time.
It's the combination of straight lines (linear separation) of multiple neurons in the same layer. So it's linear separation ('linear' by previous_layer_output*weight, 'separation' by activation function), at multiple nodes.

What you have depicted is a nonlinear classificator. Although each stage does a linear separation, the sequential composition of linear separations is nonlinear. The nonlinearity of the neuron is key in this regard, as otherwise it would all be equivalent to a matrix multiplication, which is linear. You were right about the degree, although it's rarely called like that. People usually describe just the number of layers, and I guess the main reason is that it's not directly equivalent to such polynomials, as that depends on other factors (e.g. the activation function).


text:
Is it necessary to standardise the expected output
Normalisation transform data into a range:
$$X_i = \dfrac{X_i - Min}{Max-Min}$$
Practically, I found out that the model doesn't generalise well when using normalisation of input data, instead of standardisation (another formula shown below).
Before training a neural net, data are usually standardised or normalised. Standardising seems good as it makes the model generalise better, while normalisation may make the model not working with values out of training data range.
So I'm using standardisation for input data (X), however, I'm confusing whether I should standardise the expected output values too?
For a column in input data:
$$X_i = \dfrac{(X_i - Mean)}{Standard\ Deviation\ of\ the\ Column}$$
Should I apply this formula to the expected output values (labels) too?

What is the difference between "standardization" and "normalization"? I think these terms would refer to the same concept and, from your equation, it seems like you're referring to some form of "normalization". Can you clarify that? Also, maybe you should describe the type of task you're trying to solve: is it classification or regression? How do your labels look like? Are they integers or floating-point numbers? Do they have a range?

normalisation: values are scaled into a range [a,b] that is: x = (x-min)/(max-min); and standardisation is x = (x-mean)/(sqrt(sum((x-mean)^2)/n)); they r different concepts

It depends, as mentioned in the comments, on your model and labels. For example, how would you use standardization on a multi-classification problem?
Generally, standardization is more favorable for input data as its mean is around 0.
I assume you have a regression model and in that case, using standardization could be better than normalization.


text:
Reinforcement learning simple problem: agent not learning, wrong action
I am pretty new to RL and I am trying to code a simple RL task with pytorch.
The goal/task is the following:
The initial state is $t_o$ and the agent takes an action $\Delta_t$: $t_o +\Delta_t = t_1$.
If $t_1$ equals 450 or 475 then it gets a reward, else he does not get a reward.
I am training the agent with DQN algorithm on a NN ( with: 2 Linear layes: fist layer n_in=1 n_out 128 and second layer n_in=128 and n_out=5):

observation space($t_i$) is 700 --> $t_i \in [0,700[$
action space ($\Delta_t$) is 5 --> ($\Delta_t \in [-50,-25,0,25,50]$)

epsilon_start=0.9#e-greedy threshold start value
epsilon_end=0.01#e-greedy threshold end value
epsilon_decay=200#e-greedy threshold decay learning_rate=0.001# NN
optimizer learning rate batch_size=64#Q-learning batch size

Unfortunately it does not seem to converge to the values $t_i=$ 450 or 475. I doesn't seem to care about getting a reward.
How can I improve my code so that the agent learns what I am trying to teach him?
I put my code below in case the explanations were not clear enough:

import gym
from gym import spaces

class RL_env(gym.Env):
    metadata = {'render.modes': ['human']}

    def __init__(self):
        super(RL_env, self).__init__()

        n_actions_delta = 1 #delta_t
        self.action_space = spaces.Discrete(5)

        n_observations = 1 #time

        self.observation_space = spaces.Discrete(700)

        #initial time
        self.time = 0

        self.done = 0
        self.reward = 0

    def reset(self):
        self.reward = 0
        self.done = False
        return self.reward

    def step(self,delta_t):
        print('self time',self.time)
        d_t = np.arange(-50,70,25)

        self.time = (self.time + d_t[delta_t])%700
        print('delta time',d_t[delta_t],'-->','self time',self.time)

        if self.time == 475 or self.time == 450:
            self.reward = 1

        else:
            self.reward += 0

        info = {}
        print('total reward',self.reward)
        print('\n')
        return self.time,self.reward, self.done, info

    def render(self, mode='human', close=False):
        print()

import gym
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.autograd import Variable
from torch.distributions import Categorical
dtype = torch.float
device = torch.device("cpu")
import random
import math
import sys
if not sys.warnoptions:#igrnore warnings
    import warnings
    warnings.simplefilter("ignore")

#hyper parameters
epsilon_start=0.9
#e-greedy threshold start value
epsilon_end=0.01#e-greedy threshold end value
epsilon_decay=200#e-greedy threshold decay
learning_rate=0.001# NN optimizer learning rate
batch_size=64#Q-learning batch size

env = RL_env()

#use replay memory (-> to stabilize and improve our algorithm)for training: store transitions observed by agent,
#then reuse this data later
#sample from it randomly (batch built by transitions are decorrelated)
class ReplayMemory:#allowes the agent to learn from earlier memories (speed up learning and break undesirable temporal correlations)
    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
    def push(self, transition):#saves transition
        self.memory.append(transition)
        if len(self.memory)>self.capacity:#if length of memory arra is larger than capacity (fixed)
            del self.memory[0]#remove 0th element

    def sample(self, batch_number):#samples randomly a transition to build batch
        return random.sample(self.memory, batch_number)

    def __len__(self):
        return len(self.memory)

#Dqn NN (we want to maximize the discounted, cumulative reward)
#idea of Q-learning: we want to approximate with NN maximal Q-function (gives max return of action in given state)
#training update rule: use the fact that every Q-function for some policy obeys the Bellman equation
#difference between the two sides of the equality is known as the temporal difference error (we want to min -> Huber loss)
#calculate over batch of transitions sampled from the replay memory
class DqnNet(nn.Module):
    def __init__(self):
        super(DqnNet, self).__init__()

        state_space = 1
        action_space = env.action_space.n
        num_hid = 128
        self.fc1 = nn.Linear(state_space, num_hid)
        self.fc2 = nn.Linear(num_hid, action_space)
        self.gamma=0.5 #Q-learning discount factor (ensures that reward sum converges,
                        #makes actions from far future less important)
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.sigmoid(self.fc2(x))
        return x

#select action accordingly to epsilon greedy policy
#sometimes we use model for choosing action, other times sample uniformly
#probability of choosing a random action will start at epsilon_start and will decay (epsilon_decay) exponentially
#towards epsilon_end
steps_done=0
def predict_action(state):
    global steps_done
    sample=random.random()#random number
    eps_threshold=epsilon_end+(epsilon_start-epsilon_end)*math.exp(-1.*steps_done/epsilon_decay)
    steps_done += 1
    if sample>eps_threshold:
        x  = eps_threshold,model(Variable(state,).type(torch.FloatTensor)).data.max(0)[1].view(1, 1)
        return x#chose action from model

    else:
        x = eps_threshold,torch.tensor([[random.randrange(env.action_space.n)]])
        return x#choose random action uniformly

#wtih the update_policy function we perform a single step of the optimization
#first sample a batch, concatenate all the tensors into a single one, compute Q-value and max Q-value,
#and combine them into loss
def update_policy():
    if len(memory)<batch_size:#we want to sample a batch of size 64
        return
    transitions = memory.sample(batch_size)#take random transition batch from experience replay memory
    batch_state, batch_action, batch_next_state, batch_reward = zip(*transitions)#convert batch-array of Transitions
                                                                              #to Transition of batch-arrays
    #-->zip(*) takes iterables as arguments and return iterator

    batch_state = Variable(torch.cat(batch_state))#concatenate given sequence tensors in the given dimension
    batch_state = batch_state.resize(batch_size,1)
    batch_action = Variable(torch.cat(batch_action))
    batch_next_state = Variable(torch.cat(batch_next_state))
    batch_next_state = batch_next_state.reshape(batch_size,1)
    batch_reward = Variable(torch.cat(batch_reward))

    #print('model batch state',model(Variable(batch_state[0])))
    current_q_values = model(batch_state).gather(1, batch_action)#current Q-values estimated for all actions,
                                                                 #compute Q, then select the columns of actions taken,
                                                                 #these are the actions which would've been taken
                                                                 #for each batch state according to policy_net
    max_next_q_values = model(batch_next_state).detach().max(1)[0]#predicted Q-values for non-final-next-states
                                                                  #(-> gives max Q)
    expected_q_values = batch_reward + (model.gamma * max_next_q_values)

    #loss is measured from error between current and newly expected Q values (Huber Loss)
    loss = F.smooth_l1_loss(current_q_values, expected_q_values)

    # backpropagation of loss to NN --> optimize model
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    return loss, np.sum(expected_q_values.numpy())

def train(episodes):
    scores = []
    Losses = []
    Bellman = []
    Epsilon = []
    Times = []
    Deltas = []

    for episode in range(episodes):
        state=env.reset()#reset environment
        print('\n')
        print('episode',episode)

        epsilon_action = predict_action(torch.FloatTensor([state]))

        action = epsilon_action[1] #after each time step predict action

        next_state, reward, done,info = env.step(action.item())#step through environment using chosen action

        epsilon = epsilon_action[0]
        Epsilon.append(epsilon)
        print(reward,'reward')

        state=next_state
        Times.append(state)
        scores.append(reward)

        memory.push((torch.FloatTensor([state]),action,torch.FloatTensor([next_state]),
                         torch.FloatTensor([reward])))#action is already a tensor
        up = update_policy()#update_policy()#update policy

        if up != None:
            Losses.append(Variable(up[0]))
            print('loss',Variable(up[0]))
            Bellman.append(up[1])

        #calculate score to determine when the environment has been solved
        mean_score=np.mean(scores[-50:])#mean of score of last 50 episodes
        #every 50th episode print score
        if episode%50 == 0:
            print('Episode {}\tScore: {}\tAverage score(last 50 episodes): {:.2f}'.format(episode,scores[-50:],mean_score))

    #print('Losses',Losses)
    Losses = torch.stack(Losses).numpy()
    #print('Losses',Losses)
    plt.plot(np.arange(len(Losses)),Losses)
    plt.xlabel('Training iterations')
    plt.ylabel('Loss')
    plt.show()

    Bellman = np.array(Bellman)
    #print('Bellman',Bellman,'\n')
    plt.plot(np.arange(len(Bellman)),Bellman)
    plt.xlabel('Training iterations')
    plt.ylabel('Bellman target')
    plt.show()

    #print('scores',scores)
    plt.plot(np.arange(len(scores)),scores)
    plt.xlabel('Training iterations')
    plt.ylabel('Reward')
    plt.show()

    #print('epsilon',Epsilon)
    plt.plot(np.arange(len(Epsilon)),Epsilon)
    plt.xlabel('Training iterations')
    plt.ylabel('Epsilon')
    plt.show()

    print('Times',Times[-25:])
    print('Deltas',Deltas[-25:])

    Times = np.array(Times)
    print('Times',Times)
    #plt.figure(figsize=(31,20))
    plt.figure(figsize=(9,7))
    plt.plot(np.arange(len(Times)),(np.array(Times)))
    plt.xlabel('Training iterations')
    plt.ylabel('t')
    plt.show()

    Times_1 = np.array(Times[-300:])
    print('t',Times)
    plt.figure(figsize=(9,7))
    plt.plot(np.arange(len(Times_1)),(np.array(Times_1)))
    plt.xlabel('Last 300 Training iterations')
    plt.ylabel('t')
    plt.ylim(0,1000)
    plt.show()

model = DqnNet()#policy
memory = ReplayMemory(20000)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

train(10000)

I see some issues in your code of the environment. Firstly, and probably most importantly, you should not be incrementing the reward. In your code, every time the agent hits $t=475$ for example, the reward given by the environment increases by 1. So if the agent oscillates between $t=450$ and $t=475$, at each timestep the environment gives a greater and greater reward. Your Q values have no reason to converge. Instead of incrementing the reward, you should just return a reward of $1$ at those two desired states and return $0$ otherwise.
Secondly, it appears that you don't reset the state during reset(), I think you should have self.time = 0 in there (unless this was intentional).
Edit : I also question why you're using DQN for this problem. This problem should be very easy to solve with standard (tabular) Q Learning.
Edit 2 : Your training loop looks wrong. You reset the environment at each iteration in your loop, so your episodes are one timestep long each. What I think you want instead is to have another loop under reset() something like
for episode in range(episodes):
  s = env.reset()
  episode_score = 0
  for step in range(max_timesteps):
    a = agent.predict_action(s)
    ns, r, d, _ = env.step(a)
    episode_score += r
   ...
    agent.update_policy()
   ...
  scores.append(episode_score)
 ...

thanks, I just corrected the typo in the reset function and changed the reward scheme.. unfortunately, still no convergence towards t = 475 or 450 ... and to answer your question: i am still new to reinforcement learning, and wanted to learn how to implement DQN  on a simple problem, just for the sake of learning and understanding

I also suggest you increase gamma. Gamma=0.5 is fairly low, and it's very possible that the rewards are just not propagating back to other states due to the exponential decay. Try with gamma=0.99 or something. Not sure if this will be the definitive solution, but it can't hurt.

Also, can you describe the results you get? As in, what do you mean by "no convergence towards t=..."? Can you talk about what you expect vs what you observe?

thank you, so let me explain: in the end i want the agent to do the following 475 +0 = 475 --> 475 + 0 = 475 --> 475 +0 = 475.... OR something like 475 -25 = 450 --> 450 +25 = 475 --> 475 -25 = 450 .... (i want the sum to always be 475 or 450)... what I observe: it just takes the same action for any state ( for example 450 +25 = 475 --> 475 + 25 = 500 -->  525 + 25 = 550 --> 550 + 25 = 575....).. so it just learns to take always the same action over and over again without caring about the result

Noticed another issue, updated my original answer.

Thank you very much for your comments and suggestions.. unfortunately i still don't get what I expect the agent to learn ( i changed the hyperparameters too, but no success so far)... probably DQN is not very well suited for this type of task as you mentioned

If you're just doing this as a learning exercise, why don't you try an openai gym environment? DQN definitely can solve cartpole, so I suggest you try that to figure out if there's something wrong with your DQN implementation.


text:
Is there a way to beat AlphaGo Zero with different method?
As I read the research from
https://deepmind.com/research
It seem AlphagoZero use zero knowledge and use Reinforcement learning to improve the ai skill of playing.
Is there a way to beat AlphagoZero? can anyone share an idea.
My friend said we can find a specific move to beat AlphagoZero.
From my point of view I think the only way to beat AlphagoZero is computational power.
But I can't find any other way to beat AlphagoZero, I hope there is many other way to beat it.
Also wanna keep some related topic below:
Would AlphaGo Zero become perfect with enough training time?


text:
Trading off "Memory" vs "Optimization"
I've been researching the following topic. Or rather, I would like to but I can't find anything because I'm not sure what to look for.
I am interested weather there are some concepts or models that explain how humans (or cognitive systems in general) trade off remembering what to do in a particular situation versus thinking about the board state and devising a new solution. Consider the following example:
A person plays chess. On their turn they must decide what to do. They could act according to one of the following extreme strategies:

Remembering all possible board configurations and how the game ended (we'll call that Memory)
Use the known set of rules to simulate the game in their mind and choose the optimal path (Optimization)

I'd say the best choice is a trade off between these two. But how? When to choose one of the strategies?
Is there a research field that is working on these kind of questions? What would I look for?

It is called memoization and is not specific to AI

The The Oxford Companion to Chess has entries on only 700 named openings, and lists another 1327 opening variations in the index, and I wouldn't be surprised if someone out there had them all memorized.  For an algorithm, however, storing that number of openings is trivial, and Chess algorithms traditionally made use of high-quality "game books" which are move-by-move records of chess games.
The problem with intractable models, like chess is that they are unsolvable, so brute-force is not an option, and any decision on a move, whether "memory" or "optimality" based, at least in the context of current advances, are probabilistic.  (The exception is endgames where the the problem becomes tractable, and perfect play can be achieved.)
Essentially, the memory of a board state is only a small piece of the puzzle because the unsolvability of the game means that particular board state from a recorded game, where that state led to a successful outcome, will not guarantee a similar result in a new game, because new, more optimal strategies can always emerge.
Secondly, because of the extreme variation engendered by non-trivial games, it is extremely unlikely any current game would mirror the previous "memorized" game.
It is my understanding that "memory" as you define has been utilized in Chess engines, accessing a database of previous games to evaluate a position and derive a good move, but the intractability of Chess in general makes this method sub-optimal, and it has been definitively supplanted by newer methods (machine learning, neural networks).
In the famous Giraffe Chess paper, Matthew Lai states:

The job of the evaluation functions is to assign scores to positions statically (without looking ahead)

But the method is highly nuanced and probabilistic.  Nevertheless, I think you'll be interested because the problem presented in the paper is rooted in the human vs. AI play method issue:

...the way computers play chess is very different from how humans play. While both humans and computers search ahead to predict how the game will go on,
  humans are much more selective in which branches of the game tree to explore. Computers, on the other hand, rely on brute force to explore as many continuations as possible, even ones that will be
  immediately thrown out by any skilled human... How can a human searching 3-5 positions per second be as strong as a computer searching 200
  million positions per second? And is it possible to build even stronger chess computers than what
  we have today, by making them more computationally efficient? Those are the questions this
  project investigates.

There is no simple answer to the trade-off question due to the complexity of the problem.  But this is an area where tremendous advances are being made.
I'd strongly recommend reading Giraffe Chess (September, 2015) as a primer.
 It produced very exciting results, synopsized in the MIT Tech Review article "Deep Learning Machine Teaches Itself Chess in 72 Hours, Plays at International Master Level".  (Giraffe's creator currently works as a Research Engineer in Machine Learning at DeepMind.)

If you can remember everything and there's no randomisation in your outcome like chess, there is absolutely no reason not to do that. Anybody who can remember all the possible board configurations in chess, by definition plays perfect chess. A perfect player would never lose.
Unfortunately, most practical problems can't be solved by brute-force, and that includes chess. All chess players learn general chess principles. Nobody could memorize all possible opening variations.
If you can somehow memorize every possibility, why not? But can you?

Of course you can not do that. But you can, let's say, memorize 10 prototypic configurations. And then starting from them you can go on and optimize it. The question is now: how much should you memorize optimally.
I'm just looking for a field that's working on these kind of questions.


text:
Neural network for detecting individual human face
In past few weeks, I have learned a lot about Neural Networks. Now, I am looking forward to create a Neural Network program that can recognize individual human faces. I tried searching it online but was able to find only small pieces of information.
What are the steps for implementing such a program from scratch?

What exactly do you want? A program that tracks a human face OR a program that can detect a human face and the person that face belongs to?

I want exactly the second kind of program.

I am assuming that you are new to all this. You can start with making a basic human face detection. Train the program to detect a human face with very good accuracy. This will help you to get familiar with the coding ground related to image processing and basic machine learning.
After that train your program to identify faces of only 2-3 people. Trying for too many in the beginning won't be a good idea. Test your program's accuracy in different situation, with a different number of crowd, etc.
If it's working fine then you can train your program for more individuals. In addition to this leave room for learning from experience in your code. Some codes only learn once and they implement the same thing for their whole life. A nice example of this kind is OCR. If a face is detected wrongly OR your program detects a new face about which it doesn't know anything. Then you should be able to tell the program and it should include that in its database. I think some form of reinforcement learning will help. Not much sure about it though.
Now the Implementation
I will highly recommend you to learn python and get familiar with OpenCV. You can think of OpenCV as a collection of libraries. I find them very helpful for image processing and machine learning. Another good thing about it is that you can import OpenCV in python, Java or C++ according to your need.
OpenCV has an inbuilt function that allows it train a neural network for positive and negative images. The success of your program depend highly on your choice of positive and negative images, so choose them wisely. The result of the training is stored as a haar cascade file. This cascade file can be used in your program to use the trained data and function accordingly.
For basic human face detection, you can find the cascade file online and implement a code like this

face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml');

For detecting individual faces you will need to make different classes. The number of classes will represent the number of individual faces you want to detect in the beginning.
You can find OpenCV tutorials here.

DLIB has AA-class face detector based on ResNet model.
Here is C++ example.
The accuracy is much better compared to HAAR/LBP but performance is worse (but much depends on parameters passed to HAAR/LBP detector)
The latest version of OpenCV also contains face detector based on ResNet model, but I did not try this.

If you want to implement recognition you've just to train a convnet or CNN on a lot of images in which there are faces, and then you classify it 1 if there are faces and 0 if there aren't. If you want to do detection you have to use different approaches like a cascade classifier whit CNN or an object detection network like YOLO or SSD.

Thanks for the information, my purpose is to find a particular face in images. Let's say i have a particular person's face image, i would like to achieve all the images with that face. What's the best way to face up the problem?


text:
Use of machine learning for analyzing companies enlisted in stock market
Can current trends and tools, in the field of machine learning, replicate the complexity of financial market? If yes, then what are the tools available in this domain.
Q. I am trying to build a model to infer results from stock market using the concept to create a graph on the companies enlisted. Can anyone suggest me approaches to do so?

In one way the answer is NO. You can't incorporate all the necessary details in an AI program to correctly predict the financial market, at least not in currently. A particular event is a result of many actions and events from the past. Everything is like a chain.
You might have heard of the ButterFly Effect. It simply states that small things can have very large effects. Like flap of butterfly's wings leading to a hurricane. You can read more about it here.
Now back to your question. I believe you are somewhat familiar with the financial system and stocks. Close your eyes and try to imagine all the data that is being churned into machines. Apart from that, there are many decisions that lead to changes in the financial market directly. Like policies made by the government deals bagged by the company related to imports and exports, death of the main employee (like CEO, founder), and other many things. Apart from that, there are uncountable things that indirectly affect the market. I want you to think about it for a while. Now consider doing this for all the stocks present. After even stock values of one are dependent on the others in one way or another.
This amount of data, in other ways considering all these things, is not possible for current generation computers to process. You might have a hit with quantum computers coming though.
Now what you can do, is try to make a Deep Network (with Reinforcement Learning) that can predict the behavior and shift in the direction of some stocks. For example, whether it will go up for down or try to keep a constant line on the graph.
Selection of Features is always an important key to the success of Machine Learning algorithms. Another problem that stock market-related AI programs face is that Economics is itself an emerging and novice field of science. It's not as mature as physics, chemistry or biology. Thus finding out and deciding over the choice of features is another difficult job.
I found this paper online. Haven't given a look to it but, I believe this might help you a bit.

Thanks @Shivam , i found the paper to be helpful

the processing of such large data is definitely an issue in this type of program though.

Huge data is always an issue, on top of that not able to place all of it in the main memory at a single time is another issue. That's where algorithms like, reservoir sampling comes into play.

with regards to processing large data, can we build our own server for processing data in parallel ? Big data technologies along with introduction of nosql databases, will these make large scale processing possible ?

Yes, you can do that but, the data that you will need to process to predict stocks with around 100% accuracy is way more. But, to some extent, you can do that.

Reviewing the Question
There are multiple questions contained within this posted question.  (One of the sentences end with a period, but it is clearly intended to be a question.)  All are good questions and fairly easy to answer, assuming that the word 'replicate' can be replaced with 'model' or 'simulate'.  (Because the financial world is chaotic, any meaningful replication would likely require a quantum level reproduction of earth and everyone and everything on it.)
The kind of modelling, analysis, and visualization of results is done all the time in the research we do.  Approaches and proof of concepts have been provided to insurance, banking, and health organizations along these lines and can be discussed here in general terms within the constraints of any confidentiality agreements.
Restating the Questions
It is best if I restate the questions the way I understand them from the information available in the original post to ensure I understand what the questioner wishes.

What are the current trends in modelling and the production of predictive results from historical market data?
To what degree can the complexity of financial landscape be simulated?
What are the best approaches to create a graph that represent aspects of the financial market given a list of tradable securities?

Please indicate if my restatement distorts any of the intent contained in the original three questions.
A graph comprised of vertices and edges commonly represents relationships between legal entities.  Applying this visualization to represent probable relationships between legal entities from matrices of historical market data is quite possible and may have many uses for financial analysis.  Such a graph can be visualized using GraphViz, Mathematica, Matlab, or various libraries available for use from programming environments of Python, C++, Java, LISP, JavaScript or other languages.
Vertices
Instead of vertices representing legal entities registered as tax entities, as in many of the web services that display graphs from public records and purchased aggregated corporate data, the vertices in the graph presumably envisioned by the questioner would represent tradable securities.  The attributes of such vertices might be.

Exchange
Unique exchange ID (symbol)
Name
An array of vectors of historical trade metrics (with each vector containing a UTC time stamp)

Edges
Edges represent the probable strength of financial connectedness between any two vertices representing two tradable securities.
Because the nature of relationships and the associated details between the corporations offering tradable securities and the mindsets of all the trading agents are obscured, relationships must be inferred naively (without factual knowledge of causality), perhaps using the probability relations of Rev. Thomas Bayes (1701 – 1761) or other more sophisticated methods (some of which cannot, for legal reasons, be detailed here).
Relational models must be created (likely more than one) to capitalize on identifiable features in the trading metrics of one of two selected vertices and match that feature with the same or another feature in the other of the two vertices.  The correlation must be statistical and designed in such a way as to be resistant to effects outside the relationship between the two legal entities associated with the two tradable securities.
Naive Bayesian classification, other statistical approaches, FFTs, or neural nets may apply to assist with achieving a functional correlation value.  Windowing the data in a loop will be necessary to implement sensitivity to single events sparsely spaced in the time domain of the historical data.
To attempt to guess causality, you will need to apply different temporal shifts to see if the feature of one preceded the feature of the other and by how much.  (If event A in security B preceded event C in security D, and this pattern repeats over a range of months or years, then there is a probability greater than zero that event C was caused by event A.
The science (and perhaps the art) of creating a set of potential mathematical models of how various corporate and trading relationships may have impacted historical trade metrics between any two securities is the first hurdle in this proposed best approach.
Using various known methods, a probability distribution of the single dimensional or multi-dimensional strength of the relationship, for each of the proposed models, can be calculated from the historical data of the two entities between which one of the many inter-vertex analyses is occurring.
Statistics of these distributions would then be the attributes of the edge shown between two vertices.  For more intuitive usability, the following attributes would need to be available via point and click drill down for each edge and each model tried between the two tradable securities connected by the edge.

Median relationship strength
Mean relationship strength
Standard deviation of relationship
Direction of causality
Median delay in causality

Measures to Make Computation Time Practical
To accomplish the above each vertex would ideally be compared with each other vertex, for each model, iterating through temporal parameters of the model to determine relationships involving time delays.
If there were a hundred tradables to consider, ten probabilistic relationship models, a thousand temporal permutations that must be tried to converge on a good fit between each model and the historical data, a hundred iterations to converge for each temporal window, a window of a thousand temporal observations, ten thousand windows to cover the entire range of historical data, and a thousand cycles for each test of fit, the primary computations would be 100 x 99 x 10 x 1000 x 100 x 1000 x 10,000 x 1000 = 99 x 10^18 CPU cycles.
(The number 99 comes from the fact that, without some permutation elimination scheme, the histories of each of 100 vertices must be compared with those of the other 99.)
Several methods may be applied together to reduce this set of expanded permutations to permit batch process completion after the close of the market in NY or Hong Kong and before the time zone dependent dawn.

Filtering and then decimating (removing redundancy) the historical data
Truncating the historical data to analyze only the recent (and therefore the most relevant) historical data
Widening the error margin to only what is displayed (such as two significant figures)
Optimizations of algorithms, the mathematics behind them, the machine instruction representation of computations, or the mapping of values to data types
Distributing analysis processes to take advantage of parallel computing
Limiting the list of tradable securities using a narrowing set of inclusion criteria
Early elimination of possible edges between unlikely relational candidates using heuristics, model simplifications, neural nets, or fuzzy logic

Prediction
Once models are generated and functional and some of the visualized constructs can be verified, then the relational models can be used to predict probable events before they occur.  This may seem like science fiction to some, but we predict physical, social, and economic events all the time.
In the case of the profitability in relation to markets, if such predictive tools were to be distributed to eleven other traders, the ability to use the tool to generate profit would almost immediately deteriorate to one twelfth in monetary value.
In fact, this is probably the state of the market today.  Only those with automated tools are probable winners, funneling money from those without tools.
The AI Research Perspective
Although the above does not seem like AI the way it is described, often what is conceived as an intelligent agent and appears intelligent in behavior after deployed, refined, and tuned, appears like straight software engineering when one gets into the details of implementation.
Furthermore, if the method for interfacing with the models used to match features in the history of two tradables is generalized so that arbitrary models can be added or modified at will without damaging the effectiveness of job execution, one can build some sort of analogy of a genetic algorithm to search for models that exhibit higher correlations and therefore progressively enhance predictive capabilities.
Meta Modelling
At this point in development, model development is still largely up to the researcher.  However, once a model interface, perhaps employing the bridge and facade design patterns, is developed, it is possible to generalize the concept of historical feature correlation between two tradables as models with a set of mutation operations and develop concurrent processes that employ an automated experimental test fixture to develop new models without programmer intervention.
Although the details of such meta-modelling cannot, for legal reasons, be detailed here, the meta-model design options naturally become apparent after some experience is gained after implementing and deploying the above approach in a real scenario with actual tradable historical data.
Using Off the Shelf Code, Libraries, and Frameworks
Obviously, there is appreciable monetary value to this type of development, therefore it is unlikely that anyone will post (or even sell) code specific to this domain.  However using super-computing platforms, basic analysis algorithms such as FFT functions, and statistics packages with correlation coefficient routines, naive Bayesian capabilities, and convergence detection support will certainly assist in reducing the development effort required to implement and test this approach or others like it.

approach using graph structure is good alternative for handling data here... :)

You can actually build a statistical model (not exactly and AI) to predict the stock market trend. But the prediction accuracy will be very bad.
The accuracy of your model depends on the following things.

No of variables which affect the model's output directly or indirectly.
The model's reaction time (How fast can your model generate an output corresponding to a change in the input).
The maximum lifespan of your model (Describes the the time the model takes to mature + the span of time the model stays relative to the context of it's variables)

NOTE: This is not in anyway a correct answer to your question. I just wanted to state that anything and everything in this universe can be modeled given that your are aware of all the variables which affect it and how they interact with each other.

...If it's in no way the correct answer to the question, why did you post it?


text:
Which methods or algorithms to develop a learning application?
I am creating a game application that will generate a new level based on the performance of the user in the previous level.
The application is regarding language improvement, to be precise. Suppose the user performed well in grammar-related questions and weak on vocabulary in a particular level. Then the new level generated will be more focused on improving the vocabulary of the user.
All the questions will be present in a database with tags related to sections or category that they belong to. What AI concepts can I use to develop an application mentioned above?

You don't really need an AI to do all those things. Can you please elaborate on certain reasons why you will be needing AI algorithms instead of normal ones. Your application will only require certain degree of programming and data base skills.

@Ugnes ,Any application on earth can have AI capabilities.

I don't really understand what you're asking.   You've identified a number of candidate  approaches --> "machine learning , planning systems , reinforcement learning and case-based-reasoning", and one or more of those might be useful for the system you're building.  But what exactly is it you want to know?

@mindcrime The OP has a concept that needs to be narrowed down.And this question has received vague answers.Where were we?

@Ugnes This question needs revision.

even though that's not really AI, the easiest way to do that would probably be to put coefficients on each question
e.g. your question would have something like
grammar=0.55
vocabulary=0.45
if(won){
  success=-1
}else{
  success=1
}

the lower the level, the less questions of this type will appear
at the end of the level, you sum each question times how much did the player succeed
player.level is a real number array
for each question{
   player.level["grammar"]=player.level["grammar"]+question.grammar*question.success
   player.level["vocabulary"]=player.level["vocabulary"]+question.vocabulary*question.success
}

then when you initialize the level, generate the questions according to the player's level
choice is a random real number between 0 and sum of all player.level (offseted to be above 0)
for each player.level{
   if choice>player.level{
      choice=choice-player.level
   }else{
      return question_type=player.level.type
   }
}

though a "true" AI can make the same or better choices of questions, it will probably be slower, and require more data than a simple algorithm like this one.

You can implement a Reinforcement Learning agent with the following aspects:
Action Space
An action of adding or removing a question of a certain category, e.g.
Add a grammar question, remove a grammar question, add vocabulary question, etc.

State Space
Using the Flow Model, you can build a vector of flow features per topic. such that each element in that vector is either challenging, normal, boring

such that

where -1 is boring, 0 is just right, and +1 is challenging, or the other way around
then you flatten this vector into a scalar integer

where n is number of subjects, and Z is the set of integer numbers
Reward Signal
Use a combination of:

participation time
player score
distance from equilibrium

to define a reward signal. and make the model learn the optimal flow for this user.

such that  is user score in each subject
and

is a distance function defining how far this user is from equilibrium in skills to learn
Algorithm
using the above model, you can build a simple Q-Learning agent, trying to maximize the reward signal by picking the optimal action, in terms of adding and removing questions, or challenges of a certain subject.
and of course, the reward signal is a function of how long the player spends playing, how challenged or bored the player is, and how far the player is from the optimal learning goal

where v is the value of a certain state, Q is the value of a state/action pair
Good luck

A simple decision tree would be suitable, and represents one of the easiest forms of AI.  Don't over-complicate simple problems.
The simplest approach would likely run faster, and require less system resources, so, unless this project is designed as an entree into machine learning, that approach is way overkill for this simple problem.

could you please revise your answer(opinion based)

See my edits.  We appreciate you contributing, but a little more elaboration would make your answers more useful.

You could build a MLC(machine learning classifier)  based on her/his answers and apply that to a large database of questions to see which type of questions will probably yield an incorrect answer.
And then provide those questions (Assuming one would learn more from working at her/his weaknesses) to provide a steeper learning curve, continuously adapting the MLC.
http://www.deeplearningbook.org/ Chapter 5, also findable on github, has a very clear and relatively simple explanation on how to build a basic MLC, and what the problems/critical issues/parameters in application could be/are.

Could you please expand the acronym MLC at its first use?  Using acronyms without expanding them can confuse people who aren't already familiar with the term, especially if the term is overloaded with multiple expansions.

@mindcrime Thank you! I did. I'm fairly new with it myself so I should've known.