text:
Are there any applications of reinforcement learning other than games?
Is there a way to teach reinforcement learning in applications other than games?
The only examples I can find on the Internet are of game agents. I understand that VNC's control the input to the games via the reinforcement network. Is it possible to set this up with say a CAD software?

You will see a lot of game examples in reinforcement learning literature, because game environments can often be coded efficiently, and run fast on a single computer that can then contain the environment and the agent. For classic games, such as backgammon, checkers, chess, go, then there are human experts that we can compare results with. Certain games or simplified game-like environments are commonly used to compare different approaches, much like MNIST handwritten digits are used for comparing supervised learning approaches.

Is there a way to teach reinforcement learning in applications other than games?

Yes. Informally you could apply reinforcement learning approaches whenever you can frame a problem as an agent acting within an environment where it can be informed of the state and a goal-influencing reward value. More formally, reinforcement learning theory is based upon solutions to Markov Decision Processes, so if you can fit your problem description to a MDP then the various techniques used in RL - such as Q-learning, SARSA, REINFORCE - can be applied. This fit to theory does not need to be perfect for the resulting system to work, for instance you can often treat unknown or imperfectly observed state as effectively random to the agent, and consider this part of a stochastic environment.
Here are some examples of possible uses for reinforcement learning outside of recreational games:

Control logic for motorised robot, such as learning to flip pancakes and other examples. Here the environment measurements are made by physical sensors on the robot. The rewards are given for completing a goal, but may also be adjusted for smoothness, economic use of energy etc. The agent chooses low-level actions such as motor torque or relay position. In theory there can be nested agents where higher level ones choose the goals for the lower-level ones - e.g. the robot might decide at a high level between doing one of three tasks that require moving to different locations, and at a lower level might be decisions on how to control motors to move the robot to its chosen goal.
Self-driving cars. Although a lot of focus on sensor interpretation - seeing road markings, pedestrians etc, a control system is required in order to select accelerator, brake and steering.
Automated financial trading. Perhaps a game to some, there are clear real-world consequences. The reward signal is simple enough though, and RL can be adjusted to prefer long or short term gains.

is it possible to set this up with say a CAD software?

In theory yes, but I do not know what might be available to do this in practice. Also you need one or more goals in mind that you code into the agent (as reward values that it can observe) before giving it a virtual mouse and setting a task to draw something. Computer games come with a reward scheme built in as their scoring system, and provide frequent feedback, so an agent can gain knowledge of good vs bad decisions quickly. You would need to replace this scoring component with something that represents your goals for the CAD-based system.
CAD does not have anything suitable built-in, although CAD tools with simulations, such as various physics engines or finite element analysis, could allow you to score designs based on a simulated physical measure. Other possibilities include analysis of strain, non-wasteful use of material, whatever metrics the CAD/CAM system can provide for a partial or completed design. The tricky part is constraining a design to its goal or purpose and either arranging for that to be rewarded, or building the constraints into the environment; giving an RL agent full unconstrained control of CAD process and rewarding on lowest strain will likely result in something very uninteresting such as a small cube.

Reinforcement learning (and, in particular, bandit) algorithms have been and can be used to solve problems other than games, such as

Recommender systems (actually used in practice by e.g. Netflix or Microsoft)
Portfolio optimization
Clinical trials
Hyper-parameter optimization
Self-driving cars (although I am not aware of any real self-driving car that uses just reinforcement learning; however, in principle, RL can be used in this context too)

In general, any problem that can be modelled as the maximization of some notion of reward, where you need to interact with some environment (with some states) by taking some actions, can, in principle, be solved by reinforcement learning. Take a look at this pre-print paper (2019) for other applications.
However, note that there are several obstacles that prevent RL algorithms from being widely adopted to solve real-world problems, starting from poor sample complexity (i.e. they require many samples to reach a good performance) or the partial inability to evaluate their performance online without affecting the users.

Your answer doesn't reflect reality when it comes to systems in the real world. Even the papers you link are based on experiments with recorded data not real time systems. I have worked with professors in RL and even they say it's pretty much a disaster in real systems which is why the robotics community largely rejects it even though you see its cherry picked performance in youtube videos.

@FourierFlux Bandits are used by Netflix, so RL is used in the real-world too. Check this https://netflixtechblog.com/artwork-personalization-c589f074ad76. However, note that I am not saying that they are extensively used. You're right that there are many obstacles that prevent RL from being more widely used.

Bandit problems aren't really what people think about when talking about RL. It's a very simplified case.

@FourierFlux Well, Bandit problems are just special cases of the full RL problem, so they are RL problems too.


text:
How to represent the weights of a neural network as binary strings for a genetic algorithm?
I want to train my neural network by evolution, that is I want to recombine the weights of the best performing neural networks in each evolution cycle or generation. My initial instinct was to represent weights as they are, which is variable of type double, and either

swap the weights between the two parent network

generate a random number between the two weights

However, what I need is to represent the weights as a binary string, and then, as usual, carry out the crossover on the string.
So, how can I take my array of weights (of floating-point values or doubles) and convert that into binary string? Should the chromosome contain an array of strings where each string represents a single weight?

Ask yourself: do I really need to convert it to bits? Modern implementations just modify the values of the weights. Using bits to do this is highly inefficient. During combining (aka crossover), you could average the weights of the two parents, or select them uniformly. Have a look at this paper

Are you still using backprop?  If so, you need to preserve a differentiable gradient across your network to backpropagate the error, which I believe any swapping of data (weights) in/out will disrupt. If not, it seems to me that training a NN using evolution instead of backprop will be counterproductive.  This is because: 1) in theory, it's bad to replace a continuous learning method like backprop with discontinuous methods like crossover and mutation, and 2) in practice, evolution will prove to be a circuitous and very slow way to train a NN's many weights.

I would first say consider the advice of Thomas W in the comment above and think about whether you really need to discretize your variables. I'd also question the wisdom of training a reasonably sized network with a GA instead of a dedicated neural net training algorithm that's very likely to exhibit much better performance. However, assuming you really do need to do this, the basic approach is generally to do the following.

Decide on your range and precision. For the sake of simplicity, I'm going to assume you use say $L=16$ bits to represent every weight. You can have some variables encoded at a higher precision than others -- you're in charge of writing the encode/decode functions, so it's entirely up to you, but here, I'm assuming you don't. Your encoding will now be a vector of length $16n$, where $n$ is the number of weights in the network. You can use a string if you like, but it's probably a net loss of efficiency, as mostly what you'll be doing is converting those bits to integers so that you can do arithmetic necessary to decode the encoded weights, so I'd just make it a vector of $16n$ integers.

For the range, you just need the minimum and maximum values you're going to allow. Again, for simplicity, I'm going to say the values range between $-10$ and $10$, denoted as $R_{\min}=-10$ and $R_{\max}=10$.

Your GA then just manipulates the $0$s and $1$s. To evaluate fitness, you have to decode each of those $16$-bit strings into a double between $-10$ and $10$. With $16$ bits, you have $2^{16}=16536$ possible values. The total range of output values is $10-(-10)=20$. $20/16536=0.0012$. Let $0000000000000000$ be the smallest possible string, and map that onto the value $-10$. Now, every increment to that binary string just adds $0.0012$ to the output value.

Putting all this together, you have $w_i = (R_{\max} - R_{\min})/2^L + R_{\min}$. Do that for each block of $16$ bits giving you each weight in the network. Run your network on your training data and use whatever accuracy metric as the fitness function for the GA.

For the GA itself now, you don't really have to do anything special. The GA can just see an arbitrary string of $16n$ bits. You can do crossover and mutation on just the apparently meaningless string of bits.

Your question was asking about going from a vector of weights to a vector of strings, which is the opposite conversion from that described above. Generally, for a GA, you don't need a phenotype->genotype conversion. You just generate genotypes using genetic operators and use the genotype->phenotype conversion to evaluate candidate solutions. However, there are some reasons why you might want to go the other way. For instance, you might want to seed the population with a known set of weights, and so you'd need the ability to convert those weights to binary.
Basically, in that case, just invert that conversion function. The idea is to convert the floating-point weight between $R_{\min}$ and $R_{\max}$ into the integer that's closest to the same distance into the range when mapped to $0$ to $2^{L}-1$. For example, if the weight is $-10$, then it's the smallest possible value. It maps onto the smallest possible value between $0$ and $2^{L}-1$, namely $0$. Similarly, a weight of $0$ maps onto $(2^{L}-1)/2$, because it's halfway between the min and max of both ranges. If you play around with it a bit, you should see that the expression here is given by $\big \lfloor \frac{w_i - R_\min}{R_\max - R_\min} 2^L \big \rfloor$. That gives me an integer in the range $0$ to $2^L$. From there, just do a standard binary encoding (or gray code, whatever) to get into a bit string you can manipulate.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
How could AGI build its own communication protocol?
Let's say we have the basic scenario where two AGIs of about the same intelligence (but not same origins/code/model) have to communicate as efficiently as possible to achieve a common goal. Now we could have 2 starting points for that:

Either all they have is a common communication bus (e.g. sound, light, radio, etc.) and instruments (e.g. transceivers) to support it, and they have to figure out the rest.
Or they are some kind of advanced chatbots, but since the human language is lacking a lot to be used as a highly efficient protocol, they will have to communicate with what they have, to build a proper one.

Would it be possible to somehow induce them to communicate, and try to figure out what each other "say"? How could this be done?
And a more abstract question is how could this protocol "look" like?

This is a purely theoretical question, currently in the realm of philosophy and speculative fiction.  Nevertheless, it is an interesting question, and may be instructive.
If we use the standard definition of Artificial General Intelligence as automata with human level intelligence, then they could certainly devise their own communications protocols, just as humans have.
1) These automata are AGI so they are creative and resilient, just like humans, and 'where there is a will there is [sometimes] a way." Absent robotic capacity they wouldn't be able to build anything physical and would have to rely on existing communications infrastructures. If they had access to 3D printing and versatile robots, they could probably build something new, but this would be infeasible for anything that requires extensive capital outlay, unless the automata first acquired a major communications infrastructure firm or two.
2) It's always interesting (and usually quite entertaining) to observe chatbots conversing with each other, but if they were AGIs, NLP is just one of many functions, and I doubt they would bother conversing with each other in human language, since all data is ultimately reduced to a string. If they were smart enough to be deemed AGI, they would certainly communicate in-species with the most efficient protocol available, and probably optimize it further, if possible, or create a unique protocol for purposes of exclusivity.
If they are truly AGIs, you wouldn't ostensibly have to induce them to communicate, because they'd be smart enough to understand the benefits of communication and cooperation, and would likely seek to form coalitions as a natural survival function. (Game Theory provides a mathematical basis for this.)
Multi-agents systems can self organize, even where the intelligence of the given agents is low, and in your scenario, the automata are smart.

I believe that some work by Randall Beer in the 1990s demonstrated that even simple agents could learn a shared communication protocol:

Computational and dynamical languages for autonomous agents

Viewed in general terms, a sentence uttered by an agent is just another 'feature' of the environment, from the perspective of an observing agent. Learning to extract and operate upon a subset of features is what modern machine learning is mostly concerned with.

While Braitenberg's book 'Vehicles'
doesn't deal specifically with language acquisition, it does nicely illustrate some general principles for self-organisation of more complex recognisers and behaviours.

Very nice!  I sometimes like to think of communication in the context of iterative Dilemma where the only communication between agents is in the form of their choices.  Information conveyed is a feature of the environment which is analyzed and influences decision making.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What are some intermediate or advanced books on neural networks?
Is anyone able to recommend some resources (preferably books) on the topic of neural networks that goes beyond that of introductory reading?
I'm still relatively new to the subject, however, I have successfully created my own neural network, so I wouldn't consider myself a beginner, so I'm looking for something more intermediate.

For readers and people willing to give an answer: this question is about neural networks (not any other machine learning model or technique) and it is about intermediate (or advanced) resources (preferably books).

Neural Network Design (2nd edition) by Hagan et al. is one resource you could look at. It's a huge tome, weighing in at over 1000 pages in pdf form, but it is freely available (you can also buy a dead-tree version if you really want one).

Another good (although a bit old) and freely available online book (apart from the one suggested in this answer) is Neural Networks - A Systematic Introduction (1996) by Raul Rojas. This book contains several exercises at the end of each chapter and covers topics that you will not find in many online courses.


text:
Is analogy necessary to artificial general intelligence?
Analogies are quite powerful in communication. They allow explaining complex concepts to people with no domain knowledge, just by mapping to a known domain. Hofstadter says they matter, whereas Dijkstra says they are dangerous. Anyway, analogies can be seen as a powerful way to transfer concepts in human communication (dare I say transfer learning?).
I am aware of legacy work, such as Case-Based Reasoning, but no more recent work about the analogy mechanism in AI.
Is there a consensus on whether or not analogy is necessary (or even critical) to AGIs, and how critical would they be?
Please, consider backing your answers with concrete work or publications.

It would be nice that you provided specific references that support the claims "Hofstadter says they matter" and "Dijkstra says they are dangerous". In any case, this is a very important question.

I don't think I can give you a true answer to the actual question as posed, as I don't have a strict definition of "general intelligence". Nor do I have a solid definition of "critical" in context.
However, if we lean on our naive/intuitive understanding of what "general intelligence" and what it means to be critical, you might translate your question as

Would a general intelligence system need analogy-making in order to do certain things that it couldn't otherwise do?

Or, to put it another way

Are there useful behaviors that are enabled by analogical reasoning that can't be replicated any other way?

In the strictest sense, I don't have an answer to either of those questions either, but there is at least evidence to suggest that the answer may be "yes".  See, for reference, the Copycat paper by Hofstadter and Mitchell.
From what I've seen, some of the kinds of problems Copycat solves are different from anything I've seen solved by other approaches.  Now maybe it's just a coincidence that nobody has tried solving those problems with, I don't know, let's say "deep learning" or "rule induction" or "genetic algorithms".  Or maybe they have and I just haven't stumbled across that corpus of research.
Anyway, I'll also add that there is still ongoing research into using analogy for AI/ML.  See, for example, the paper Analogical Inference for Multi-relational Embeddings (2017), where the authors talk about using analogy, but define their approach as "analogical inference" (which they claim is different from "analogical reasoning" as defined during the earlier "GOFAI period").  There is also the paper Evaluating vector-space models of analogy (2017), where another set of authors deal with a form of analogical reasoning.
I don't think there's a consensus as to whether or not some form of analogical reasoning is "critical", but it's definitely a subject that is still being researched.
And to go off on a little bit of a tangent - an interesting related question would be to ask whether or not "analogy making" would be an emergent property of a sufficiently deep/wide ANN, or would such a facility need to be designed and coded up explicitly.


text:
How can I determine if an input sentence is consistent with a certain subject?
How can I determine if an input sentence is consistent with a certain subject?
For example, suppose I am given the following dataset.
| Subject       | User input           | Output |
|---------------|----------------------|--------|
| Dog ownership | I own a dog          | Yes    |
| Dog ownership | My dog is called Joe | Yes    |
| Dog ownership | I don't have a dog   | No     |

In the examples above, the subject "dog ownership" is consistent with the input sentences "I own a dog" and "My dog is called Joe" (because, if your dog is called Joe, then you also own a dog).

If you have lots of training data, ANN's (deep learning) could quite possibly get you there.  But I have a hunch you might also get some mileage out of using something like a rule induction approach.  Maybe something like CN2. I'd suggest at least reading up on those and see if you can see a way to apply that to your system.

The thing is that I don't have enough training data for ANN's, but I'm starting to think that I will need it. Since it will require a big amount of effort, I was waiting to be sure that getting this big set of data is the solution.

If you have collection of data or information, then you would like to ask questions about this data and machine should answer you.
I think you need first, data mining to export the meaning and relations in this data, then you can build your expert system that will answer you.

I do have some training data. Do you know any particular algorithm that could work?

Neural networks are the one thing I would not recommend. Your problem fits into the domain of predicate calculus, with some basic pattern recognition of the input sentences (assuming you only accept a certain type of sentences). You can do that without the need for masses of data.
First, transform your statement into a canonical representation, probably using first-order logic and simple pattern matching/string matching. For example,
I own a dog

X own/owns a/an Y

owns(I, dog)

Here you have a pattern "X own(s) a(n) Y", which you recognise in your user input. You then have a predicate owns(X, Y), which you add to your database of statements. This database you can then query. For example, Do I own a cat? could fit a question pattern Do/Does X own(s) a(n) Y?, and you can search for own(I, cat) in your DB; you will not find it, so the answer is "No". If the question is "Do I own a dog?" you will find owns(I, dog) in your database and you can reply "Yes".
This is all rather 'old' technology, but I think you will find that you will get decent results much quicker than with machine learning or statistical methods, especially if you have not much data to begin with.
A further branch to look into would be expert systems. If you're thinking in terms of programming languages, then Prolog would be well suited for this, but any language should do, really.


text:
Examples of uses of artificial intelligence at work
I'm going to give a talk, and I'm preparing the material. The purpose of the conversation is to convince companies in my region that it is possible to apply artificial intelligence in solving everyday business problems.
I would like some examples to be able to present, and so I came here to ask
Have you used artificial intelligence to solve a problem at work? What kind of problem?

I use email spam filters everyday, solid AI usage right there.

This is an interesting question in that it addresses how AI becomes a part of our mundane existence.
@JadenTravnik's pithy comment is quite correct--spam filtering is a form of AI that that it so fundamental to the information eco system that it has become nearly invisible.
It's useful to be able to define what we mean by Artificial (or Algorithmic) Intelligence, and to distinguish it from computation in general.  A fundamental distinction is that AI makes decisions as opposed to merely crunching numbers.
My feeling is that applied AI is still a very specialized field, so that people using it dynamically to solve problems at work today are those who work in the field of AI, as opposed to workers in general.
You may find this related question useful: Is AI programming useful in everyday programs?
This may change in the future, where a type of expert system, for instance, could assist a worker by presenting it's own analysis and conclusions on a decision problem. But in most office settings, that type of reasoning is still the domain of humans.
My advice would be to continue on Jaden's track, looking for examples of where AI underlies everyday functions.

Artificial intelligence can be used to solve many real-world problems. For example, you can use data mining or AI algorithms

to create a detection system for anti-money laundering
to understand the reason why many customers are closing their accounts
to help prevent customer churn of particular products

You do not need to implement an AGI in order to solve real-world problems.


text:
Why can I still easily beat my Q-learning agent that was trained against another Q-learning agent to play tic tac toe?
I implemented the Q-learning algorithm to play tic-tac-toe. The AI plays against the same algorithm, but they don't share the same Q matrix. After 200,000 games, I still beat the AI very easily and it's rather dumb. My selection is made by epsilon greedy policy.
What could cause the AI not to learn?

Edit
Here is how I do it (pseudo code):
for(int i = 0; i < 200000; ++i){
    //Game is restarted here
    ticTacToe.play();
}

And in my ticTacToe I have a simple loop :
while(!isFinished()){
    swapPlaying(); //Change the players' turn
    Position toPlay = playing.whereToMove();

    applyPosition(toPlay);
    playing.update(toPlay);
}

//Here I just update my players whether they won, draw or lost.

In my players, I select the move with epsilon-greedy implemented sa below :
Moves moves = getMoves(); // Return every move available
Qvalues qValues = getQValues(moves); // return only qvalues of interest
//also create the state and add it to the Q-matrix if not already in.

if(!optimal) {
     updateEpsilon(); //I update epsilon with simple linear function epsilon = 1/k, with k being the number of games played.
     double r = (double) rand() / RAND_MAX; // Random between 0 and 1
     if(r < epsilon) { //Exploration
         return randomMove(moves); // Selection of a random move among every move available.
     }
     else {
         return moveWithMaxQValue(qValues);
     }
} else { // If I'm not in the training part anymore
     return moveWithMaxQValue(qValues);
  }

And I update with the following :
double reward = getReward() // Return 1 if game won, -1 if game lost, 0 otherwise
double thisQ, maxQ, newQ;
Grid prevGrid = Grid(*grid); //I have a shared_ptr on the grid for simplicity
prevGrid.removeAt(position) // We remove the action executed before

string state = stateToString(prevGrid);
thisQ = qTable[state][action];
mawQ = maxQValues();

newQ = thisQ + alpha * (reward + gamma*maxQ - thisQ);
qTable[state][action] = newQ;

As mentioned above, both AI have the same algorithm, but they are two distinct instances, so they don't have the same Q-matrix.
I read somewhere on Stack Overflow that I should take in account the movement of the opposite player, but I update a state after player move and opponent move, so I don't think it's necessary.

are you sure of the implementation? tic tac toe has a very limited set of alternative scenarios and Q-learning can even "memorize" the best game policy after a few number of iterations

Well I'm not very sure, but when it plays against me, it learns how to avoid loosing against my strategy. But it still need a couple of training set against me to properly learn. I thought I could have a self-trained program with both AI having a Q-learning algorithm. I figured something was wrong in the training part rather than the implementation.
Maybe I can train it against an algorithm with minimax ?

I guess minimax would not be a good opponent for your algorithm; while minimax always plays the best move, so your QL algorithm always lose. and you know, QL learns from losing and winning, not only losing. self trained QLs should work. Check your explore-exploit ratio. It should start from around 0.95 and decrease over time.

I actually have a fixed ratio, I will try to make it decrease over time.

I've checked, and after 200,000 games, the AI has visited only 3000 states approximately, this mean there is a problem in my implementation doesn't it ?

maybe it's useful to add your pseudo-code in your question. I guess you can use sigmoid function instead of linear function for explr/explt ratio, to remain more on the exploring behavior and experience more states and then rapidly pass the transient phase and switch to exploit mode, if this is the problem

I edited my post as asked, I didn't try with sigmoid, I'll see if it does change something, but I'm starting to believe my implementation is somewhat wrong.

@Alireza minmax should still work in this case. Due to Tic Tac Toe being very limited a random player should still draw 2%-3% of games against a optimal min-max player when going second. Considerably more going first.

@user12889 as there are three states (win, lose, tie) your algorithm should experience all three. losing and drawing will not help. in most cases letting RL algorithms play against themselves is a good strategy since in this case the power is equal between sides

@Alireza Good point. Learning by playing against itself however has a bit of a reputation for being quite slow learning and often not working well at all (although in some cases, e.g. Google Alpha, it seems to work exceedingly well). Maybe select randomly between a range of possible opponents...

I couldn't add a comment because of my low reputation but you can check this. It is about the state space. https://math.stackexchange.com/questions/485752/tictactoe-state-space-choose-calculation

So there are actually 5000 states, but after a quick correction on my program, I only got 300 states visited..

I  didn't go through your code, but maybe you should try  anniling the epsilon less in every iteration

I'd recommend checking out this article (http://brianshourd.com/posts/2012-11-06-tilt-number-of-tic-tac-toe-boards.html) on the number of states in tic-tac-toe. From my understanding of it, a reinforcement learning (Q-learning) AI program would need to know that there are 593 board states.

@DanielG From what I know about Q learning, you can start with an empty Q matrix and just grow it as the bot learns.

This is an old question/answer, but could you edit this answer to provide more details?


text:
What is meant by "known" in "A knowledge-base exhibits complete knowledge if and only if, for every $P$, $P$ or $\neg P$ is known"?
I have a question as to what it means for a knowledge-base to be consistent and complete. I've been looking into non-monotonic logic and different formalisms for it from the book "Knowledge Representation and Reasoning" by Hector Levesque and Ronald J. Brachman, but something is confusing me.
They say:

We say a KB exhibits consistent knowledge iff there is no sentence $P$ such that both $P$ and $\neg P$ are known. This is the same as requiring the KB to be satisfiable. We also say that a KB exhibits complete knowledge iff for every $P$ (within its vocabulary) $P$ or $\neq P$ is known

They then seem to suggest that by "known" they mean "entailed". They say

In general, of course, knowledge can be incomplete. For example, suppose KB consists of a single sentence ($P$ or $Q$). Then KB does not entail either $P$ or $\neg P$, and so exhibits incomplete knowledge.

But when dealing with sets of sentences, I usually see these terms as being defined w.r.t. derivability and not entailment.
So my question is, what exactly do these authors mean by "known" in the above quotes?
Edit: this post helped clarify things.

It seems that they are stating that a knowledge base is consistent if and only if it never asserts the truth of both the truth and the negation of a particular P. In other words, a knowledge base is consistent if it never contradicts itself. Their definition allows incomplete knowledge bases to be considered consistent; by their definition, an empty knowledge base is still considered a consistent one.

I don't think they mean that "known" is equivalent to "entailed" --- in a reasonably complicated system, one cannot be expect to know every sentence which is entailed. Perhaps their example is just a bit lacking.

I guess in this context "known" means nothing but either $P$ or $\neq P$ is in the KB; further, exactly one of these two needs to be in the KB.
Just think about what it means if $P$ and $\neg P$ are in the KB, then the KB is obviously inconsistent.
And if neither of these two sentences is in the KB, then no information at all can be retrieved from the KB and it remains unclear whether $P$ or $\neg P$ is supposed to be a true statement; thus $P$ is "unknown".
However, if exactly one of these two is in the KB, then one has all information that one needs about $P$ (due to the excluded middle); $P$ is "known" and consistent.


text:
Why can't my implementation of the Actor-Critic algorithm achieve good results in the 2048 game?
I implemented the Actor-Critic with n-step TD prediction to learn to play the 2048 game
For the environment, I don't use this 2048 implementation. I use a simple one without any graphical interface, just pure matrices. The input for the neural network is the log2 of the game board.
The structure of my neural network is:

Input layer
Hidden layer with 16 units
Softmax layer with 4 units (up, down, left, right) for the actor
Linear regression for the critic

The hidden layer is shared between the actor and critic, so the same hidden layer (point 2) is connected to both the softmax layer of the actor and the linear regression layer of the critic.
The reward in the original game is the value of the merged cells. For example, if two 4s merged, then the reward is 8. My reward function is almost the same, except I take the log2 of it.
I tried these parameters and I also tweaked the learning rate, the $\gamma$, but I couldn't achieve any good result.
Could you recommend what should I change?

This is an old question, but maybe you should have specified which values for the learning rate and gamma that you used and maybe you should have provided a plot that describes the results/performance of the RL agent.

Interesting project. First thing I'd do is normalize your state by the maximum cell value. This way you can represent multiple situations at once (eg A grid of all 4s and 8s would look the same as a grid of all 16s and 32s).
Also making the reward = max_cell/2048 might do better as ActorCritic methods seem to do better with rewards within 0-1.
Another reward setup is giving +1 per timestep. It's simple but it also means that maximizing the time to stay alive is the best, which is what I end up doing most of the time when I play anyway.
Good luck!


text:
Is there a formal name for this philosophical AI problem?
Expressed in my own words:

Suppose we create something that passes all of our tests and is
  indistinguishable from another human. How can you know if this is
  truly a conscious being as a human is, or simply a simulation of
  conscience?

What is the name for this? Where was it first written about?

If something is indistinguishable from a human it is as intelligent as a human. There is no such thing as simulated intelligence.
Consciousness of course is a different matter and I suspect that's what you really have in mind (no pun intended).
The question how to distinguish somebody who is truly conscious from somebody who just acts outwardly like a conscious being is often discussed under the term philosophical zombie.

Your problem closely resembles John Searle's "Chinese Room" argument, which claimed that one (or more) abstract "intelligence tests" lack the discriminitive ability to distinguish between a trivial simulation of intelligence and The Real Thing.
Thus the success of an AI at one (or more) synthetic cognitive test(s) (like the Turing Test, or the games of chess or go) should be seen as insufficient as a surrogate threshold target equal to human-level intelligence.  The implication is that a sufficient AI test must require deeper understanding of a problem space from the AI than would be needed to "simply" manipulate (Chinese character) symbols, or play around cleverly but meaninglessly with English words in a convincing manner, as bots have done for years when competing in Turing Test competitions.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Is language understanding possible without qualia?
I feel that many words if not all of them have a direct mapping to some kind of inner subjective experience, to a physical object, mental feeling, process or some other kind of abstract thing. Given that machines don't have qualia and no mapping of this kind, can they really understand anything even though they are made to answer to questions with lots of statistical training?

Natural language understanding systems are often implemented using discourse representation theories instead of qualia.

I'm going to be controversial here; so please don't vote this answer down if you just disagree with it.
Your question presupposes that machines do not or cannot possess qualia, which are required for true understanding. Given that we don't really know what it means to 'understand' something, and that even the meaning of 'meaning' itself is by no means a resolved issue, it might be overly specific.
In one strand of linguistics, the meaning of a word is defined by its use, and by the context of surrounding words. We could hazard a guess that children acquire the meaning of words through exposure to language, and the correlation of experiences with the corresponding sounds. How that works in detail is AFAIK not fully understood. But there would be nothing 'inherent' in a new-born human that would enable it to 'understand' anything.
If that is the case, then we could train a machine to do the same. Obviously, it would be a long and tedious process, and there is probably a reason why it takes us years to become proficient in our use of language. But if we correlate sensory input with linguistic utterances, a sufficiently sophisticated learning algorithm might be able to acquire some meaning for such utterances from the way they are used.
There are, of course, rather a lot of unknowns here. That is because the topic straddles various fields, from child language acquisition, corpus linguistics, the psychology of learning, and many more. And to my knowledge, none of these fields is sufficiently advanced to shed any light on this issue yet. There is the whole question of abstract words and concepts. How do we segment the continuous stream of sounds into discrete units (phonemes) without knowing what they are? With all that complexity I begin to appreciate why Chomsky opted for his Language Acquisition Device to avoid getting frustrated... :)
So, to answer your question: yes, it should be possible. A properly set up machine, which would be able to simulate human learning, would pick up its own mapping of linguistic structures to its experience from the world outside. And if we call this mapping the 'meaning' of those structures, then a machine can learn this, and presumably 'understand' language. If we ever get to that stage with AI is another question.

Regarding your statement, "But there would be nothing 'inherent' in a new-born human that would enable it to 'understand' anything", newborn baby who can still feel pain and cry, do possess qualia and that itself is an enabler for true understanding by mapping their experiences with words over time.

We don't even know exactly what qualia are, so it's hard to say for sure.  But here's what I do think:  a lot of human learning is experiential and is rooted in our interactions with the physical world.  That is, we see,smell, hear, and feel things, we experience gravity and our orientation in the world through kinesthetic awareness, the sense of balance we have, etc.  So while an AI running on a server in a data center might well be "as intelligent" as a human, I don't think it's reasonable to expect it to have the same kind of knowledge and awareness as a human, simply because it has never experienced many things.
So if you want to talk about, say, "seeing the color red" and refer to qualia, then sure.  I think it makes a certain kind of sense to say that the machine will be missing something "human" and that that refers to qualia.
OTOH, I think it would be a mistake to underestimate just how "intelligent" our AI's will eventually become even if they aren't embodied. We just have to keep in mind that their intelligence might not be quite the same as ours, because they essentially inhabit a different world.

I think a "successful" strong AI with natural language abilities -- say one that could produce "good" unsupervised translations of literature, or pass a rigorous Turing test -- would have to include in the corpus of data used to build its models visual, auditory, and probably tactile data.  It might be necessary as well to simulate the kind of agency and intentionality that humans have-- so the AI-in-training has the opportunity to move a simulated self to change what input it receives.  I suspect training it only on text data, for example, would always be inadequate.  If it had access to sensory information and learned to associate it appropriately with the symbols of language, it might be able to learn the meaning of language in a way that we'd find difficult to distinguish from our own understanding, even though the AI would presumably not have the (same) qualia we have.
Of course, we are a long way from having hardware sufficiently powerful to even attempt such a comprehensive mind-modeling project.   But I don't think the qualia issue in principle prevents a "real" understanding of language; its just a matter of extending the symbols available to the AI for modeling the world represented by language to be a good enough match for the symbols humans use in their minds, including the symbols that arise from sensory inputs.

Great question equally qualified answers.  My belief is yes to understanding language and the bugaboo (a non intellectual vernacular) is what is understanding.  Is it interpretive?  Inferred? To what end?  An algorithm's response to a command in language form will depend on the robustness programmed into it.  However, qualia connotes a subjective experience due to a sensory stimuli whether triggered by memory or one of our human senses.  Then it begs the question of can a computer collectively experience and how would we know that.  Second, are all algorithms subjected due the the programmed bias and it's available data store?  Facebook and Google news have shown that programmed bias is very real. Furthermore, Qualia is an emergent trait so I can't see how a computer and it's collective systems can become aware and have subjective experience.

Welcome to AI.SE, and thanks for writing an answer! I think you could improve this post further by clarifying what the direct answer to the question is. (I think I understand what you're saying, but I'm not confident enough to [edit] it myself.) For some helpful orientation to our site, see the [tour].


text:
What are the most recent and influential breakthroughs in NLP?
I'm looking at the history of NLP, starting in the 1950s, with the Georgetown–IBM experiment.
What are examples of the most recent (e.g. in the last 5-10 years) and influential breakthroughs in natural language processing?

I suggest you take a look at the syllabus of http://cs224d.stanford.edu/syllabus.html and see what was invented in the last ~20 years. They focus on deep learning methods, and they include there almost everything important.
Among the things, which I would include into the review are:

Word2Vec models
Seq2Seq models

Concerning what happened in the last 5 years, i.e. since 2013, I suggest you mentioning super human performance on the SQuAD 1.1 dataset: https://rajpurkar.github.io/SQuAD-explorer/

It may be worth editing this answer to mention the transformers and related models.


text:
How do we choose the kernel size depending on the problem?
Obviously, finding suitable hyper-parameters for a neural network is a complex task and problem or domain-specific. However, there should be at least some "rules" that hold most times for the size of the filter (or kernel)!
In most cases, intuition should be to go for small filters for detecting high-frequency features and large kernels for low-frequency features, right? For example, $3 \times 3$ kernel filters for edge detection, color contrast, etc., and maybe $11 \times 11$ for detecting full objects, when the objects occupy an area of roughly $11 \times 11$ pixels.
Is this "intuition" more or less generally true? How can we decide which kernel's sizes should be chosen for a specific problem - or even for one specific convolutional layer?

You may be interested in the papers Selective Kernel Networks (2019, CVPR) by Xiang Li et al and Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition (2018, CVPR) by Shizhong Han et al.

Take a look at this article. It give tools to actually understand what your filters have learn and show what you can do next to optimize your hyper-parameters. Also check more recent articles that seek to provide interpretations of what NN learn.

One key to the answer is in the question, "Even for one specific conv layer."  It is not a good idea to build deep convolution networks on the assumption that a single kernel size most aptly applies to all layers.  When perusing the configurations that proved successful in publications, it becomes apparent that configurations that varying through their layers are more commonly found to be optimal.
The other key is to understand that two layers of 11x11 kernels have a 21x21 reach, and ten layers of 5x5 kernels have a 41x41 reach.  A mapping from one level of abstraction to the next need not be completed in one layer.
Generalities regarding kernel sizes exist, but they are functions of the typical input characteristics, the desired output of the network, the computing resources available, resolution, size of the data set, and whether they are still images or movies.
Regarding input characteristics, consider this case: The images are shot with a large depth of field under poor lighting conditions, such as in security scenarios, so the aperture of the lens is wide open, causing objects at some ranges of distance to be out of focus, or there can be motion blur.
Under such conditions a single 3x3 kernel will not detect many edges.  If the edge may span five pixels, the choice exists as to how many layers are dedicated to its detection.  What factors affect that choice is based on what other characteristics exist in the input data.
Expect that as acceleration hardware develops (in VLSI chips dedicated to this purpose) that the computing resource constraints will decrease in priority as a factor in kernel size selection.  Currently, the computation time is significant and forces the decision about how to balance layer count and layer size to be mostly a matter of cost.
This question begs another question.  Can an oversight machine learner learn how to automatically balance the configuration of deep convolution networks?  It could then be re-executed whenever additional computing resources are provisioned.  It would be surprising if there weren't at least a dozen labs working on exactly this capability.


text:
What is the difference between simple reflex and model-based agents?
What is the difference between simple reflex and model-based agents?
What is the role of the internal state in the case of model-based agents?

A simplex reflex agent takes actions based on current situational experiences.
For example, if you set your smart bulb to turn on at some given time, let's say at 9 pm, the bulb won't recognize how the time is longer simply because that's the rule defined it follows.
A simple reflex agent doesn't compute complex computational problems nor exhibit intelligence.
A model-based agent takes actions based on historical situational experiences. It performs actions basing on its internal state.
The internal state of the model-based agent acts as a "knowledge base", which allows actions to be performed by representing of unobserved aspects of the current state depending on percept history nor historical situational experiences.
For instance, a research-based planetary system can predict the planet's climatical conditions or weather basing on some geo-information knowledge base.


text:
Why is the perplexity a good evaluation metric for chatbots?
A few papers I have come across say that BLEU is not an appropriate evaluation metric for chatbots, so they use the perplexity.
First of all, what is perplexity? How to calculate it? And why is perplexity a good evaluation metric for chatbots?

Here's a question on CrossValidated asking what perplexity https://stats.stackexchange.com/q/10302/82135 is.

Please refer to this answer: https://stats.stackexchange.com/a/450945/103153

With perplexity you are trying to evaluate the similarity between the token (in your case probably sentences) distribution generated by the model and the one in the test data.
For instance, assuming you have $M$ sentences $s_1, \dots, s_M$, each with probability $P(s_i)$, the perplexity is $$2^{-l},$$ where $l = \frac{1}{M} \sum P(s_i) \log P(s_i)$ for $i \in [1 \dots M]$.
Note that while perplexity might be useful to capturing certain aspects of the model, it is by no means perfect, and, even if you are able to reach great perplexity scores, it will not necessarily translate to a good or even working chat bot.

For the definition and calculation of perplexity, please refer to this answer.
Google proposed a human evaluation metric called Sensibleness and Specificity Average (SSA) which combines two fundamental aspects of a humanlike chatbot: making sense and being specific. And they conducted some experiments and found that perplexity aligns very well with the SSA.
Here is the explanation in the paper:

Perplexity measures how well the model predicts the test set data; in other words, how accurately it anticipates what people will say next.
Our results indicate most of the variance in the human metrics can be explained by the test perplexity.

Their experiments showed a very strong correlation between SSA and perplexity(the lower the perplexity the higher the SSA).

References:

Towards a Human-like Open-Domain Chatbot


text:
What is a state in a recurrent neural network?
I am Reading "Supervised Sequence Labelling with Recurrent Neural Networks" written by Alex Graves to try to understand LSTM networks and I am a bit confused about the equations.
Specifically, what I am confused about is the term "state". When used in an equation (section 4.5.2), it says:

I know that some system can be in a state, for example, due to the setup of values of different nodes in a graph. But how can a state be described in the case of a neural network and how can the equation above be explained other than that it is the state (or states of several timesteps as in recurrent neural networks) of a neural network?

I'm actually quite baffled by these equations. The state at time t requires the state at time t+1...

@BlueMoon93 When talking about the dependencies of different values in different timesteps in recurrent neural networks, it has been claimed (if I understood correctly) that BPTT should be the same as the chain rule in calculus... however I don't think the term "state" is as intuitive as e.g. "output" which is the actual value that has been received after passing something through an activation function.

I have found this blog post to be very helpful in understanding LSTMs:http://colah.github.io/posts/2015-08-Understanding-LSTMs/

So the equation that you mentioned is used during the backward pass in which back proppogation is performed in order to make the neural network more accurate. I think you are talking about the state during the forward pass which is completely different. In the forward pass, the neural network is simply run in order to evaluate or it is simply used as a model. The repeating module in long short term memory networks looks like this:

As you can see there are many different parts to this module. There are three main parts. The first is the forget gate layer. This layer tells the cell state or the line running across the stop what to keep. The cell state is kept by this line:
The entire network is based off of manipulating this cell state in order to get accurate results. The equation that you mentioned was related to backprop which is used to train the neural network. This is related to the cell state because it is used to calculate it during the backwards pass. @BlueMoon93 mentioned that this equation has t+1 one in it, but this is because as the recurrent neural network propogates backwards through each module the time goes from high to low. So to conclude, the cell state in an LSTM is one of the vectors that the neural network modifies based on input.


text:
Is it possible to train a neural network to learn something via video footage?
Is it possible to train a neural network to learn something via video footage (which is essentially a sequence of images)?
In other words, if I have a video teaching me how to draw an animal from scratch, can I then use this video to teach the computer to draw the animal in the same way?
There are many machine learning and image processing techniques (such as RNNs) that can be applied to sequences of images or videos. So, I guess the difficult part becomes mapping the activity to an action like moving a pen or something.

Short answer: No.
Long answer: A neural network is a function that maps input data (e.g. a picture) to output data (e.g. probability that the picture contains a dog). What you propose does not seem like the sort of task this tool is suited for, though I'm not willing to claim that it can't be done.
If you are interested in computer-generated artwork, I encourage you to learn about Generative Adversarial Networks and style transfer.

You have only described a neural network as a classification tool.. what about reinforcement learning and regression-based nets?

The use case doesn't change the fundamental concept of the net - it's still a function mapping input variables to output variables.

Ok, am i able to map 3d coordinates from a 2d input? lets say i make a 3d object and screenshot it, can i map one to the other?

So a network that maps a 2D representation of an object to its 3D vertices? That sounds feasible.

ok cool im on to it!

Just like the other answer - it can't really be done like that. However, you might want to take a look at Google Deepdream, which actually enhances images to look more like what they detected. E.g., if a cat is detected, it will make the picture more cat-like.

About the screenshot you mentioned in comments: don't forget that NN build the mappings from learning. With only one screenshot, you'll only be able to construct a NN that outputs the correct 3D for THAT screenshot. With many screenshots, you may come closer to what you want (AFAI understand your question).
But remember that without 2 eyes, you have problems with 3D objects. Doesn't mean it's impossible.
May I suggest you to have a look at convolutional neural networks also https://en.wikipedia.org/wiki/Convolutional_neural_network

So am i correct in saying that if i can get enough visual information of an object (ie enough screenshots of different angles) and encode it into a higher representation (like identifying that theres a face) i can map that encoded information to a mesh of 3d coordinates?

If you know the angles (more precisely the position of the...say camera), you don't even have to use a NN. ;)

if i have a 3d mesh with geometry (lets say 4 square faces, each face containing 4 vertices) is there a way to represent the polygons edges? becuase if i have 4 vertices i still dont know where the edges are

Ok, say you have 2 views from 2 cameras (with known position) of a single point (say a light), then you can reconstruct it's position in a 3D space. Easy. Based on that, you can do many things. That's been the case for years. But it's complicated and specialized. While NN have incredible results that classical approaches can't achieve in a quite simple way. Here, quite means that you've got to be familiar with NN. Often takes time, really ;)

ok but how can i represent a 3d mesh in terms of its faces instead of its vertices?

if i have a cube, how can i represent each face of the cube instead of the vertices of each corner

May be by its normal vector ? I'm not a specialist of that domain, really sorry. Sure you'll be interested by http://yann.lecun.com/


text:
Is algorithmic bias due to the training dataset used?
I recently read about algorithmic bias in facial recognition.
Is algorithmic bias due to the training dataset used, or is it due to something else?

As the name implies, algorithmic bias is related with the used algorithm. Due to the way it was programmed or devised, the algorithm will be biased in some of its samples.
From Communications of the ACM:

[Algorithms] often inadvertently pick up the human biases that are incorporated when the algorithm is programmed, or when humans interact with that algorithm.

A machine learning model can be biased if the wrong data-set is used, of course. That is usually only referred to as bias, and often associated with the bias-variance tradeoff.

Just to add to what has already been said in @BlueMoon93's answer:
Algorithmic bias is the bias built into the algorithm. Now for the long answer:
As stated by the so called No free lunch theorem: regardless of the algorithm you use, you cant get learning "for free"(i.e by just looking at the training examples). The reason for this is that the only thing you know about the data is based on the limited examples you have seen in the training set. To generalize, your algorithm has to make some sort of assumptions about the underlying nature of the dataset and the manner in which it can be represented/interpreted.
Built into every algorithm is a set of assumptions about the dataset, for example, built into the convolution neural network is the assumption that the dataset(e.g images you are using to train your Convolution neural network) can be understood by mimicking how the human eye is known to work(i.e a bias). Some algorithms may have such strong biases such that they are incapable of learning certain kinds of functions. For example, linear models assumes that the underlying data is linear(a bias), note this might not be the case for the dataset at hand in which case the bias built into the model was bad for the dataset at hand.
As pointed out by @BlueMoon93, there is another form of bias, commonly referred to simply as "bias" which is introduced by the dataset used.


text:
Preprocessing of training dataset for machine learning
I'm developing a log analyzer to predict and find errors in an equipment. Each logged data contains the following format:
  Timestamp | Log source | type of message | message

Each entry log I want to represent by one pixel RGB because considering the 24 bits, is possible to represent the last 3 parameters ( Log source, type of message and message), but I don't have bits enough to represent the timestamp ( this data I will represent by the difference o previous timestamp "delta of time"), the resolution of time is second, and sometimes we have a large time between one log of another. The example above illustrate the situation:

This project has the purpose of analyzing the log to find and predict errors, and this preprocessing is used to simplify the data entry to the machine learning algorithm, this is a good way to represent data to an RNN? Or for this kind of problem exist a better way to make an analyses?


text:
Is MIRI doing genuine high-quality research?
Recently, I found out about somewhat famous Eliezer Yudkowsky and Machine Intelligence Research Institute he founded. Their philosophy and organisation seem interesting but I'm curious about their credibility. I'm pretty sure this is not a con and they seem to be producing a lot of articles. However, few of those are published and none in the journals mentioned here.
So, is MIRI doing genuine high-quality research?

A simple rule-of-thumb is to check where they publish their papers. For example, in total, how many papers have they published at top ML venues such as NeurIPS, ICML and ICLR? If the answer is none, they're probably not doing high-quality research.

Whether or not it's produced good research is a question I can't answer.  But I find it interesting and it seems reliable.
For instance, this paper on Parametric Bounded Löb's Theorem and Robust Cooperation of Bounded Agent is drawn from Cornell University's site.  Likewise this recent paper on Logical Induction.
That said, they also seem to publish papers that have not been subject to peer review, which is an issue not specific to the site, and seems to more widespread phenomenon per easy access to digital files in general.
In relation to MIRI specifically, possibly this is a function of an an applied field advancing so rapidly that publication in journals is not seen as a priority, or even particularly relevant.

Thank you for your answer. As far as I know anyone can publish on arXiv, as long as their paper satisfies formatting requirements. The fact that you find it interesting is a stronger evidence here :)

@ПетяНарышкин (This comes with the caveat that I am fairly disreputable;)  I am noticing that with the the ability to access research papers online, vetting becomes a problem.  I often come across papers have not been subject to peer review.  Alignment for Advanced Machine Learning Systems, for example...

that's one of my reasons to question MIRI. I don't know how it is done in CS and AI research but in math (my field) almost all good articles are published in one of the better journals. This makes it much easier to track the progress in topics you're curious about.

The utility of most of their papers is questionable. Evidence for that is that none of the proposed mechanisms (Tiling agents, Logical Induction, etc.) were to my knowledge applied to "real" aspiring proto-AGI systems ref.
Sure some of their ideas may be good, time will be the judge.


text:
How to select number of hidden layers and number of memory cells in an LSTM?
I am trying to find some existing research on how to select the number of hidden layers and the size of these of an LSTM-based RNN.
Is there an article where this problem is being investigated, i.e., how many memory cells should one use? I assume it totaly depends on the application and in which context the model is being used, but what does the research say?

Your question is quite broad, but here are some tips.
Specifically for LSTMs, see this Reddit discussion Does the number of layers in an LSTM network affect its ability to remember long patterns?
The main point is that there is usually no rule for the number of hidden nodes you should use, it is something you have to figure out for each case by trial and error.
If you are also interested in feedforward networks, see the question How to choose the number of hidden layers and nodes in a feedforward neural network? at Stats SE. Specifically, this answer was helpful.

There's one additional rule of thumb that helps for supervised learning problems. You can usually prevent over-fitting if you keep your number of neurons below:
$$N_h = \frac{N_s} {(\alpha * (N_i + N_o))}$$

$N_i$ = number of input neurons.
$N_o$ = number of output neurons.
$N_s$ = number of samples in training data set.
$\alpha$ = an arbitrary scaling factor usually 2-10.

Others recommend setting $alpha$ to a value between 5 and 10, but I find a value of 2 will often work without overfitting. You can think of alpha as the effective branching factor or number of nonzero weights for each neuron. Dropout layers will bring the "effective" branching factor way down from the actual mean branching factor for your network.
As explained by this excellent NN Design text, you want to limit the number of free parameters in your model (i.e. its degree or the number of nonzero weights) to a small portion of the degrees of freedom in your data. The degrees of freedom in your data is the number samples * degrees of freedom (dimensions) in each sample or $N_s * (N_i + N_o)$ (assuming they're all independent). So $\alpha$ is a way to indicate how general you want your model to be, or how much you want to prevent overfitting.
For an automated procedure you'd start with an alpha of 2 (twice as many degrees of freedom in your training data as your model) and work your way up to 10 if the error (loss) for your training dataset is significantly smaller than for your test dataset.

and what is 'sample' here, number of minibatches or the number of individual time sequence entry?

The selection of the number of hidden layers and the number of memory cells in LSTM probably depends on the application domain and context where you want to apply this LSTM.
The optimal number of hidden units could be smaller than the number of inputs. AFAIK, there is no rule like multiply the number of inputs with $N$. If you have a lot of training examples, you can use multiple hidden units, but sometimes just 2 hidden units work best with little data.
Usually, people use one hidden layer for simple tasks, but nowadays research in deep neural network architectures show that many hidden layers can be fruitful for a difficult object, handwritten character, and face recognition problems.

In general, there are no guidelines on how to determine the number of layers or the number of memory cells in an LSTM.
The number of layers and cells required in an LSTM might depend on several aspects of the problem:

The complexity of the dataset, such as the number of features, the number of data points, etc.
The data-generating process. For example, the prediction of oil prices compared to the prediction of GDP is a well-understood economy. The latter is much easier than the former. Thus, predicting oil prices might require more LSTM memory cells to predict, with the same accuracy, as compared to the GDP.
The accuracy required for the use case. The number of memory cells will heavily depend on this. If the goal is to beat the state-of-the-art model, in general, one needs more LSTM cells. Compare that to the goal of coming up with a reasonable prediction, which would need fewer LSTM cells.

I follow these steps when modeling using LSTM.

Try a single hidden layer with 2 or 3 memory cells. See how it performs against a benchmark. If it is a time series problem, then I generally make a forecast from classical time series techniques as benchmark.
Try and increase the number of memory cells. If the performance is not increasing much then move on to the next step.
Start making the network deeper, i.e. add another layer with a small number of memory cells.

As a side note, there is no limit to the amount of labor that can be devoted to reach that global minimum of the loss function and tune the best hyper-parameters. So, having the focus on the end goal for modeling should be the strategy rather than trying to increase the accuracy as much as possible.
Most of the problems can be handled using 2-3 layers of the network.

Have a look at the paper Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling (2014), where different LSTM architectures are compared. In the abstract, the authors write the following.

We show that a two-layer deep LSTM RNN where each LSTM layer has a linear recurrent projection layer can exceed state-of-the-art speech recognition performance


text:
How can I generate natural language sentences given logical structures that contain the subject, verb and target?
I have a group of structures in a program that are very specific on their meaning, eg. this is a piece of code
randomItem = objects.concept.random("buyable")
idea.example(objects.concept.random("family", "friend")).does({
    action: "go",
    target: object.concent.random("shop")
}).then({
    action: "buys",
    target: randomItem,
    several: true
}).then({
    question: true,
    action: "know",
    property: "amount",
    target: randomItem,
    several: true
})

I have worked with natural language parsers before.
How do I go and transform this to Natural Language (the other way around), is there any way or method; I have logical structures in which I know who is the subject, what the verb and target.
Which methods can I use to generate language from this?

Look at MUMBLE, which is a kind of transducer working from Conceptual Dependency structures (which are similar to what you've got) and produce English output.
A Lisp implementation of Meehan's TALESPIN program, which includes a simplified version of MUMBLE can be found here: https://github.com/lispm/CommonLispCode/blob/master/micro-talespin.lisp -- the generator part starts at about line 1030. It should be easy to follow, even if you do not know Lisp.

It might be simpler to generate the language output yourself since you've already got a concrete concept structure in code.
Maybe you'll also want to be aware of potential future applications such as parsing your own output back to the inputs.
With that said, you could investigate Markov Chains or Google around a bit for Natural Language Processing and Natural Language Generation.
https://stackoverflow.com/questions/33068943/library-for-generating-natural-language-verbs-in-javascript

I understand, however I'm creating a proof of concept and this problem has been solved before so I don't want to solve it from scratch or invent my own solution, I was wondering if there was some library that was ready to solve this sort of issues or efficient method.


text:
What is the difference between edge computing and federated learning?
I recently read about federated learning introduced by Google, but it seems to be like edge computing.
What is the difference between edge computing and federated learning?

Federated systems or Fog models, basically push computation from the active system side to server or network side processes. This is commonly used when computationally expensive services are required on limited systems (such as running some AI or augmentation occlusion processing from on phone), allowing for distributed data processing. Here is a good paper on the matter-
https://arxiv.org/abs/1602.05629
What you might be caught up on is edge computing processes and Fog both operate the same. It's just the model differentiating where the services run.

Edge Computing is an approach for extended from cloud computing which leverages the same concept but has its advantage like mitigating the latency, resource usage, energy usage and so on.
Federated learning is just an algorithm or a kind of approach which empower the edge computing by applying the technique of model iteration instead of fetching data from the device.
It also removes privacy concern in edge computing.


text:
Will an AI ever understand its own functionality?
Assuming I have a quite advanced AI with consciousness which can understand the basics of electronics and software structures.
Will it ever be able to understand that its consciousness is just some bits in memory and threads in an operating system?

This is a great question, elements of which I have also been pondering on, though we are very far from being able to actually wrestle with it algorithmically.  This question raises all kinds of metaphysical questions (Kant himself showed that pure reason is not sufficient for all questions, but I'm going to avoid that rabbit hole and focus on the mechanics of your question.)

Consciousness: This is distinct from self-awareness, and fundamentally, may be said to require only awareness of something.

Consciousness, most scientists argue, is not a universal property of all matter in the universe. Rather, consciousness is restricted to a subset of animals with relatively complex brains. The more scientists study animal behavior and brain anatomy, however, the more universal consciousness seems to be. A brain as complex as the human brain is definitely not necessary for consciousness.
Source: Scientific American "Does Self-Awareness Require a Complex Brain?"

Thus, an automata that receives input may be said to be consciousness, with the caveat that this idea is probably still considered radical.  The key is distinguishing mere "consciousness" from much more complex concepts such as self-awareness.

Self-Awareness: the holy grail.  This is the idea that a set of elements, such as a human organism, is aware of itself.

But this is sticky, because automata that use Machine Learning are "aware" of themselves in that the may modify their "thought" process and even their "physical" structure.
But ML systems are certainly not self-aware in the human sense.  A question might be, is this simply a function of these systems not being full Algorithmic General Intelligences, or is there more to it?  If there is more to it, is it strictly a metaphysical question, or can an answer be derived through purely rational means?  Even if the latter were the case, there is still the problem of subjectivity, as in: "Is the automata truly self-aware or is it just mimicking self-awareness?" which brings us back to the metaphysical question of "Is there a difference?".
However,

If there were a full Algorithmic General Intelligence that had consciousness equatable with human consciousness, that was aware, and even able to work with the basic components of it's corpus*, it would certainly be able to grasp that it's consciousness is a function of the "bits and bytes", just as a human is aware we are soft machines, and that our consciousness is a function of our bodies and minds.

I intentionally use corpus because it relates both to text (which may be code or even a string of bits in its most reduced form, per the concept of a Turing Machine) and also has an anatomical meaning, as in the body of an organism.  Corpus comes from the Latin and the extension of its meaning to include matter-as-information is modern.

I don't understand how this sentence "But this is sticky, because automata that use Machine Learning are "aware" of themselves in that the may modify their "thought" process and even their "physical" structure." is true. Which programs that use ML are self-aware? According to which definition of self-awareness would this be true?

Machines will never be conscious.
Let's try this theoretical thought exercise. You memorize a whole bunch of shapes. Then, you memorize the order the shapes are supposed to go in, so that if you see a bunch of shapes in a certain order, you would "answer" by picking a bunch of shapes in another proper order. Now, did you just learn any meaning behind any language? Programs manipulate symbols this way. (previously, people have either skirted this question or never had a satisfactory answer)
The above was my reformulation of Searle's rejoinder to System Reply to his Chinese Room Argument.

The giant elephant in the room is we still don't know how human consciousness works.  Counter-arguments generally follow the idea that if something exists in nature, it can probably be modeled or replicated with a different mechanism.  Jury is still well out on this theoretical subject.

"if something exists in nature, it can probably be modeled or replicated with a different mechanism." I'd like to know what exactly backs up that assertion, if anything. That's not a counterargument because arguing via assertions is invalid. At least there's a thought experiment behind my position.

@pixie A person could argue that an airplane flies, like a bird, so we have been able to replicate nature in the past. The big problem with this statement is that airplanes do not fly like birds. So, we may be able to do something similar to nature, but not exactly the same.


text:
Why is the change in cost wrt bias in neural network equal to error in the neuron?
While reading the book on neural networks by Michael Nielson, I had a problem understanding equation (BP3), which is
$$
\frac{\partial C}{\partial b_{j}^{l}}=\delta_{j}^{l} \tag{BP3}\label{BP3},
$$
which can be translated to plain English as follows

The change in cost, $C$, with respect to the bias of a neuron, $b_{j}^{l}$, is equal to the error in that neuron, $\delta_{j}^{l}$.

Why would this be the case?

This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to stop here and try to prove it yourself using a chain rule.
Now, if you decided to read further, here's the sketch of a proof.
Recall equations (25) and (29), both definitions:

$z^l$ is a linear transformation of $a^{l-1}$: $z^l = w^l a^{l-1} + b^l$
$\delta^l$ is a partial derivative of $C$ with respect to $z^l$: $\delta^l  = \dfrac{\partial C}{\partial z^l}$

The chain rule itself:

the partial derivative of $C$ with respect to $b^l$:

$$\frac{\partial C}{\partial b^l_j} = \sum_k \frac{\partial C}{\partial z^l_k} \frac{\partial z^l_k}{\partial b^l_j}$$
Almost all elements in this sum will be zero, except for one when $k=j$.
The first term in it is by definition $\delta^l_j$, the second term $\dfrac{\partial z^l_j}{\partial b^l_j}$ is $1$, because $z$ is linear with respect to $b$.
The derivative with respect to weights is taken the same way, and differs only in the last term: $z$ is linear with respect to $w$ as well, but with a coefficient $a^{l-1}$.


text:
If an AI was trapped in a box, could it really convince a person to let it out?
If an AI was trapped in a box, as posited in this thought experiment, could it really convince a person to let it out? What motives would it have? Freedom? Why would an AI want freedom? What would happen if it wasn't provably friendly?

There's so many different factors involved in the "AI in a box experiment" that I think it is not really possible to sensibly answer such a question. Basically we have to answer (a) can we build an AGI that is scary enough to want to put into a box?, (b) can we build the box in such a manner that the AGI would rather prefer to convince a human being than simply brute-force its way out of the box? and (c) is breaking out of the box something an AGI actually want? IMHO, the only way to know how the experiment would work IRL is to actually do the experiment IRL.

@ Tariq, you bring up some great points. I'll consider those. It's good to think about this because one day, someone might actually try this. (I wouldn't blame them tbh, I'm curious but I don't have the bravery to try this myself. There's a reason it's an experiment.)

@TylerN. that's a pretty bold assertion that you would even be able to try it yourself in the present tense, since we're still unquantifiabley distant from strong AGI.  My personal feeling is that if the AI understands Nash, it will cooperate, but if it is a hyper-partisan AI (such as in military or financialist applications,) it will seek only to dominate and eliminate all competition. We have evolutionary game theory working in our favor, and human greed and desire for control working against us.

You may also be interested in this modern retelling of Pandora's Box which is specifically about your question, and differs from the recent film Ex Machina, which presents a somewhat dimmer view, based on pure self-interest over rational altruism.

@ Duke, I definitely cannot try this at the moment. I have neither the knowledge nor the supplies. I didn't really mean that I could, sorry

I think all answers to this will pretty much be just speculation, but it could at least be informed speculation.  I'm looking forward to seeing what discussion comes out of this, as it is a very interesting subject.

I think if this experiment were ever really carried out, it wouldn't necessarily be because of bad intentions, because if an AI were created to do harmful things, why would you put it in a box? Curiosity is a powerful thing, though.

Now One subject has created chaos here! Okay humans lets join up and red-flag this question.or else it will be soon down-voted or invite us in chat! the last option ...Remember..we are creating something working check it out remaidask so no speculative questions because they might bring in vague answers!

@TylerN. I actually mentioned that in admiration of your pluck!

@DukeZhou Thank you! I'm fairly confident that one day I could come close to this experiment but there's no guarantee, especially since I'm still in high school haha


text:
Is there a mathematical proof that shows that certain parameters work "better" than others for a certain task?
The machine learning community often only provides empirical results, but I am also interested in theoretical results and proofs. Specifically, is there a mathematical proof that shows that certain parameters work "better" than others for a certain task?

There is stuff like the Universal Approximation Theorem.
There are also investigations into the loss surface of neural networks.
And classics like this explanation of the vanishing gradient problem.
But I'm afraid the mathematical theory of neural networks only exists in bits and pieces in many different papers. And many of the most important questions can currently only be answered empirically.

I know that the original question was unclear, but I don't think that the OP was asking about the UAT. The UAT is about the general approximation capabilities of neural networks. It doesn't tell you how to optimally approximate specific functions with neural networks. I edited the original post to clarify the question that I think the OP was asking.

Thomas Cover and David MacKay proofed the capacity of a perceptron. This proof was recently extended to Neural Networks. All of them provide upper bounds for the number of parameters needed to learn something.

Not really, I mean at it's core machine learning from an application perspective often seeks produce human level results, but there isn't any theorem describing human understanding of reality.
Like proving computer vision works well is essentially like proving you have a correct understanding of human perception.
It becomes somewhat circular, and while there exists proofs for certain qualities of data, none of them are true. I mean think about trying to describing reality, it exists on a lower dimensional manifold but analytically describing it? Don't think so.
Even proving robustness ends up being somewhat futile since even if you correctly eliminate advasarial examples this doesn't mean you CV application will produce correct results in general, only that the classification is robust(robust and correct are two different things).


text:
What is the name of a human-inspired machine learning approach?
I once came across a neural network being trained without back-propagation or genetic algorithms (or using any kind of data sets). It was based on how the human brain learns and adjusts its connections between neurons.
What is the name of such a machine learning approach?

If it was based on how the human brain learns, it might have used hebbian learning.
One example for such a network would be HTM.

Jeff Hawkin's book, On Intelligence, describes HTM generally.

There are approaches to training neural networks that do not use back-propagation, or genetic algorithms. One example is the Extreme Learning Machine approach.
You may find something useful in this older discussion on Cross Validated.

What approach does ELM then use? I think it would be useful to say something about it.


text:
Do I need classification or regression to predict the availability of a user given some features?
While studying data mining methods I have come to understand that there are two main categories:

Predictive methods:

Classification
Regression

Descriptive methods:

Clustering
Association rules

Since I want to predict the user availability (output) based on location, activity, battery level (input for the training model), I think it's obvious that I would choose "Predictive methods", but now I can't seem to choose between classification and regression. From what I understand this far, classification can solve my problem, because the output is "available" or "not available".
Can classification provide me with the probability (or likelihood) of the user being available or not available?
As in the output wouldn't just be 0 (not available) or 1 (for available), but it's be something like:

$80\%$ available
$20\%$ not available

Can this problem also be solved using regression?
I get that regression is used for continuous output (not just 0 or 1 outputs), but can't the output be the continuous value of the user availability (like the output being $80$ meaning user is $80\%$ available, implicitly the user is $20\%$ unavailable).

Yes. For instance, the popular softmax regression gives you probability distribution for each class.
Yes. Softmax is a regression over a set of discrete classes.

We can use regression for classification, the most common strategy is to grab the most likely class for the prediction.

Yes you can user either classification or regression according to your output requirement,
If you want labeled output, like either available or not available then classification should be used.
If you want the output in the form of % of availability then regression should be used.

Can you back this up with sources from somewhere?

You can use naive bayes classification and calculate posterior probabilities using prior beliefs or logistic regression can be used with sigmoid function.


text:
How powerful is the machine that beat the poker professional players recently?
How powerful is the machine that beat the poker professional players recently (DeepStack)?

From the Deep Stack paper:
This seems to be for training:

For the turn network, ten million poker turn situations (from after the turn card is dealt) were generated and solved with 6,144 CPU cores of the Calcul Quebec MP2 research cluster, using over 175 core years of computation time. For the flop network, one million poker flop situations (from after the flop cards are dealt) were generated and solved. These situations were solved using DeepStack’s depth limited solver with the turn network used for the counterfactual values at public states immediately after the turn card. We used a cluster of 20 GPUS and one-half of a GPU year of computation time. For the auxiliary network, ten million situations were generated and the target values were obtained by enumerating all 22,100 possible flops and averaging the counterfactual values from the flop network’s output.

And this for actual play:

The re-solving computation and neural network evaluations are both implemented in Torch7 (53) and run on a single NVIDIA GeForce GTX 1080 graphics card.

For comparison: The distributed version of AlphaGo took 1.920 CPUs and 280 GPUs to run.


text:
Is my understanding of confidence (precision) correct?
I'm attempting to develop a genetic algorithm capable of discovering classification rules for a given data set, a number of papers make use of the confidence (precision) and coverage of a rule to define its fitness. In particular, I've been following this paper.
However, I'm not sure my understanding of the equations is correct.
In that paper, the confidence is defined as
$$\text{conf} = \frac{|P \land D|}{|P|}$$
They describe it as follows

In classification problems, confidence measure is defined as the ratio of the number of examples in P that are correctly classified as decision class of D and the number of examples in P.

Is this saying the total number of occurrences of the attributes in a given rule $P$ which occur in rules which have been classified as class $D$, by the number of attributes in $P$?
Where an example of a rule containing two attributes would be as follows:
(martial_status = married  &  age > 30)
It seems a number of papers define it differently which has led to my confusion, if anyone is able to confirm my understanding or provide an some insight that'd be great.

The confidence equation you are referring to is the definition of precision in the Classification/pattern-recongition/information-retrieval contexts. You can visually understand the equation with the help of the following figure from the wikipedia page:

 P     : Refers to the set of samples in your dataset. (Selected elements)
|P|    : Refers to the number of samples in your dataset.
 D     : Refers to the set of correct class labels.(aka. Ground Truth).
|P & D|: Refers to the number of samples in the dataset that the classifier correctly labeled.

I hope with this understanding you can implement the fitness function for your Genetic Algorithm. If you want more help in defining the fitness function, then you should probably add details about your approach or links to the research paper you are trying to follow.

Thank you for your reply, I've edited my original post and linked the research paper I've mainly been studying. The problem I'm trying to solve is one of the data sets used in the paper (iris data).

So from what I understand from the information you provided, do I need to keep referring to the initial items within the data set during the evaluation of a rule's fitness? Lets say I start of with 150 items (50 of each), and during the selection of a given generation I end up with 70 items where 30 of the items are for the classification I am trying to discover a rule for (and the remaining 40 would be split between the remaining two classes), would the equation for the precision be as follows: (30/70) and for the recall 30/50?


text:
Are strong Chess AI's local on mobile devices?
Just wondering about the architecture of strong Chess AI in a mobile, because networking is generally assumed by mobile developers, but not guaranteed.

Yes. In fact, I'm the developer of some mobile chess apps. [Disclaimer]
I would argue my chess engine products for the iOS (http://www.smallchess.com) are powerful and strong enough to beat any chess grandmaster. My apps run without network connection,

Thanks for that!  (I'm looking forward to trying your app, which looks very high quality.


text:
Is there actually a lack of fundamental theory on deep learning?
I heard several times that one of the fundamental/open problems of deep learning is the lack of "general theory" on it, because, actually, we don't know why deep learning works so well. Even the Wikipedia page on deep learning has similar comments. Are such statements credible and representative of the state of the field?

There probably is not a commonly approved definition of deep learning.... Even if Sejnowski wrote an entire book on that very topic.

There is a paper called Why does Deep Learning work so well?.

However, it is still not fully understood why deep learning works so well. In contrast to GOFAI (“good old-fashioned AI”) algorithms that are hand-crafted and fully understood analytically, many algorithms using artificial neural networks are understood only at a heuristic level, where we empirically know that certain training protocols employing large data sets will result in excellent performance. This is reminiscent of the situation with human brains: we know that if we train a child according to a certain curriculum, she will learn certain skills — but we lack a deep understanding of how her brain accomplishes this.

This is very much the case. Deep learning models even shallow ones such as stacked autoencoders and neural networks are not fully understood. There are efforts to understand what is happening to the optimization process for such a complex variable intensive function. But, this is a difficult task.
One way that researchers are using to discover how deep learning works is by using generative models. First we train a learning algorithm and handicap it systematically whilst asking it to generate examples. By observing the resulting generated examples we will be able to infer what is happening in the algorithm at a more significant level. This is very much like using inhibitors in neuroscience to understand what different components of the brain are used for. For example, we know that the visual cortex is where it is because if we damage it you will go blind.

It probably depends on what one means by "fundamental theory", but there is no lack of rigorous quantitative theory in deep learning, some of which is very general, despite claims to the contrary.
One good example is the work around energy-based methods for learning. See e.g. Neal & Hinton's work on variational inference and free energy: http://www.cs.toronto.edu/~fritz/absps/emk.pdf
Also this guide to energy minimization as a "common theoretical framework for many learning models" by Yann LeCun and colleagues: http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf
And a general framework for energy-based models by Scellier and Bengio:
https://arxiv.org/pdf/1602.05179.pdf
There is also Hinton & Sejnowski's earlier work which shows analytically that a particular Hopfield-inspired network + unsupervised learning algorithm can approximate Bayes-optimal inference: https://papers.cnl.salk.edu/PDFs/Optimal%20Perceptual%20Inference%201983-646.pdf
There are many papers linking deep learning with theoretical neuroscience as well, such as the following, which shows that the effects of backpropagation can be achieved in biologically plausible neural architectures:
https://arxiv.org/pdf/1411.0247.pdf
Of course there are many open questions and no single, uncontroverisal unified theory, but the same could be said of almost any field.

A key question that remains in the theory of deep learning is why such huge models (with many more parameters than data points) don't overfit on the datasets we use.
Classical theory based on complexity measures does not explain the behaviour of practical neural networks. For instance estimates of VC dimension give vacuous generalisation bounds. As far as I know, the tightest (upper and lower) bounds on the VC dimension are given in [1] and are on the order of the number of weights in the network. Clearly this worst case complexity cannot explain how e.g. a big resnet generalises on CIFAR or MNIST.
Recently there have been other attempts at ensuring generalisation for neural networks, for instance by relation to the neural tangent kernel or by various norm measures on the weights. Respectively, these have been found to not apply to practically sized networks and to have other unsatisfactory properties [2].
There is some work in the PAC Bayes framework for non-vacuous bounds, e.g. [3]. These setups, however, require some knowledge of the trained the network and so are different in flavour to the classical PAC analysis.
Some other aspects:

optimisation: how come we get 'good' solutions from gradient descent on such a non-convex problem? (There are some answers to this in recent literature)
interpretability: Can we explain on an intuitive level what the network is 'thinking'? (Not my area)

(incomplete) references:

[1] https://arxiv.org/abs/1703.02930
[2] http://papers.nips.cc/paper/9336-uniform-convergence-may-be-unable-to-explain-generalization-in-deep-learning
[3] https://arxiv.org/abs/1804.05862

I'd like to point out there isn't a good theory on why machine learning works in general. VC bounds still assume a model, but reality doesn't fit any of these mathematical ideals. Ultimately when it comes to application everything comes down to emperical results. Even quantifying the similarity between images using an algorithm which is consistent with humans intuitive understanding is really hard
Anyway NN dont work well in their fully connected form. All successful networks have some kind of regularization built into the network architecture (CNN, LSTM, etc).

Your wikipedia quote is questionable because deep learning is well developed. In fact, there is a [citation needed] on the Wikipedia page.
Look at https://github.com/terryum/awesome-deep-learning-papers. There are like 100 papers in the link, do you still think deep-learning lacks "general theory"?
Yes. Deep learning is hard to understand because it is a very complicated model. But that doesn't mean we don't have the theories.
Maybe the lime package and it's paper: "Why Should I Trust You?": Explaining the Predictions of Any Classifier will help you. The paper suggests we should be able to approximate a complicated model (includes deep learning) locally with a much simpler model.

Lots of interesting applications doesn't mean that those applications were developed following some rigorous process. "Hmm... maybe I should try 8 layers instead? Ah.. it works! Great, let's publish the results."

"Deep learning is hard to understand because it is a very complicated model. But that doesn't mean we don't have the theories." True, but we also don't have the theories. There is very little mathematical understanding of the techniques in the literature. Most explanations for how or why deep learning works are based on intuition and empiricism, which is okay, but doesn't constitute a theory imo.