text:
Why does Stephen Hawking say "Artificial Intelligence will kill us all"?
This quote by Stephen Hawking has been in headlines for quite some time:

Artificial Intelligence could wipe out humanity when it gets too clever as humans will be like ants.

Why does he say this? To put it simply: what are the possible threats from AI (that Stephen Hawking is worried about)? If we know that AI is so dangerous, why are we still promoting it? Why is it not banned?
What are the adverse consequences of the so-called Technological Singularity?

Well, if we see hierarchy of evolution, lesser intelligent species are consumed one way or another by higher intelligent species, so grass consume nutrients, insects consume grass, birds consume insects and human consume birds (you get the point even if the sequence is vague). But imagine if human themselve evolve better then they are now, dont you think they would want to press for dominance?

It's not just Hawking, you hear variations on this refrain from a lot of people.  And given that they're mostly very smart, well educated, well informed people (Elon Musk is another, for example), it probably shouldn't be dismissed out of hand.
Anyway, the basic idea seems to be this: If we create "real" artificial intelligence, at some point, it will be able to improve itself, which improves it's ability to improve itself, which means it can improve it's ability to improve itself even more, and so on... a runaway cascade leading to "superhuman intelligence".  That is to say, leading to something that more intelligent than we area.
So what happens if there is an entity on this planet which is literally more intelligent than us (humans)? Would it be a threat to us?  Well, it certainly seems reasonable to speculate that it could be so.   OTOH, we have no particular reason, right now, to think that it will be so.
So it seems that Hawking, Musk, etc. are just coming down on the more cautious / fearful side of things.  Since we don't know if a superhuman AI will be dangerous or not, and given that it could be unstoppable if it were to become malicious (remember, it's smarter than we are!), it's a reasonable thing to take under consideration.
Eliezer Yudkowsky has also written quite a bit on this subject, including come up with the famous "AI Box" experiment.  I think anybody interested in this topic should read some of his material.
http://www.yudkowsky.net/singularity/aibox/

I think the AI Box is actually a bad introduction to AI risk; it targets a specific small part of the explanation, which is "how could anyone ever be convinced to let an AI out of the box?". To someone who thinks that AI is worth using, they've already decided it's worth letting out of the box. The Orthogonality Thesis is probably the best place to start, I think.

Fair enough.  I just happen to be more familiar with Yudkowsky's writings.  I certainly don't mean to diminish the importance of other works on this topic.

Because he did not yet know how far away current AI is... Working in an media AI lab, I get this question a lot. But really... we are still a long way from this. The robots still do everything we detailledly describe them to do. Instead of seeing the robot as intelligent, I would look to the human programmer for where the creativity really happens.

As Andrew Ng said, worrying about such threat from AI is like worrying about of overpopulation on Mars. It is science fiction.

That being said, given the rise of (much weaker) robots and other (semi-)autonomous agents, the fields of the law and ethics are increasingly incorporating them, e.g. see Roboethics.

To put it simply in layman terms, what are the possible threats from AI?

Currently, there are no threat.
The threat comes if humans create a so-called ultraintelligent machine, a machine that can surpass all intellectual activities by any human. This would be the last invention man would need to do, since this machine is better in inventing machines than humans are (since that is an intellectual activity).  However, this could cause the machine to invent machines that can destruct humans, and we can't stop them because they are so much smarter than we are.
This is all hypothetical, no one has even a clue of what an ultraintelligent machine looks like.

If we know that AI is so dangerous why are we still promoting it? Why is it not banned?

As I said before, the existence of a ultraintelligent machine is hypothetical. Artificial Intelligence has lots of useful applications (more than this answer can contain), and if we develop it, we get even more useful applications. We just have to be careful that the machines won't overtake us.

There are a number of long resources to answer this sort of question: consider Stuart Armstrong's book Smarter Than Us, Nick Bostrom's book Superintelligence, which grew out of this edge.org answer, Tim Urban's explanation, or Michael Cohen's explanation.
But here's my (somewhat shorter) answer: intelligence is all about decision-making, and we don't have any reason to believe that humans are anywhere near close to being the best possible at decision-making. Once we are able to build an AI AI researcher (that is, a computer that knows how to make computers better at thinking), the economic and military relevance of humans will rapidly disappear as any decision that could be made by a human could be made better by a computer. (Why have human generals instead of robot generals, human engineers instead of robot engineers, and so on.)
This isn't necessarily a catastrophe. If the Vulcans showed up tomorrow and brought better decision-making to Earth, we could avoid a lot of misery. The hard part is making sure that what we get are Vulcans who want us around and happy, instead of something that doesn't share our values.

He says this because it can happen. If something becomes smarter than us, why would it continue to serve us? The worst case scenario is that it takes over all manufacturing processes and consumes all matter to convert it into material capable of computation, extending outward infinitely until all matter is consumed.
We know that AI is dangerous but it doesn't matter because most people don't believe in it. It goes against every comfort religion has to offer. Man is the end-all-be-all of the universe and if that fact is disputed, people will feel out of place and purposeless.
The fact is most people just don't acknowledge it's possible, or that it will happen in our lifetimes, even though many reputable AI experts put the occurrence of the singularity within two decades. If people truly acknowledged that AI that was smarter than them was possible, wouldn't they be living differently? Wouldn't they be looking to do things that they enjoy, knowing that whatever it is they do that they dread will be automated? Wouldn't everyone be calling for a universal basic income?
The other reason we don't ban it is because its promise is so great. One researcher could be augmented by 1,000 digital research assistants. All manual labor could be automated. For the first time, technology offers us real freedom to do whatever we please.
But even in this best case scenario where it doesn't overtake us, humans still have to adapt and alter their economic system to one where labor isn't necessary. Otherwise, those who aren't technically-trained will starve and revolt.

Do you have a sauce for this -> "AI experts put the occurrence of the singularity within two decades", please? ;)

I can't seem to find the original survey, but these sites come close:
http://aiimpacts.org/predictions-of-human-level-ai-timelines/
– http://aiimpacts.org/ai-timeline-surveys/  –
http://techemergence.com/artificial-intelligence-risk/


text:
What are the techniques for detecting and preventing overfitting?
I'm worrying that my neural network has become too complex. I don't want to end up with half of the neural network doing nothing but just take up space and resources.
So, what are the techniques for detecting and preventing overfitting, to avoid such problems?

This question seems to be a superset of this.

Usually you keep track of training loss and validation loss and apply proper regularization technique (such as L1, L2, dropout, DropConnect, etc.).

The more interesting technique is to observe your validation loss with respect to the number of parameters in the network (often controlled by the number of layers/feature maps). If the validation starts dropping with raising your model's complexity, then your optimization might be bad or simply the model remembers all of the training samples and overfits badly.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What evolutionary algorithms are there that model epigenetics?
What evolutionary algorithms are there that model or incorporate some notion of epigenetics? What are the pros/cons of those approaches? Are there vast insufficiencies or wide-open questions about their usefulness?

Over the last few years, evolutionary computation research has shown increasing interest in including some aspect of epigenetics. For example:

A 2008 paper by Tanev and Yuta
Work from Lee Spector's genetic programming group
A recent paper by Ricalde and Banzhaf

It would really be nice if you could also (at least briefly) address the other question "What are the pros/cons of those approaches?".

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is there a strong argument that survival instinct is a prerequisite for creating an AGI?
This question stems from quite a few "informal" sources. Movies like 2001, A Space Odyssey and Ex Machina; books like Destination Void (Frank Herbert), and others suggest that general intelligence wants to survive, and even learn the importance for it.
There may be several arguments for survival. What would be the most prominent?

@DukeZhou Would you have public documents on the topic?

Not as yet. (It's going to be a long process as our development cycle is very slow.  I also probably shouldn't have said mentioned AGI in this context because what I'm really looking at is more of a generalized utility function that mitigates risk, not a super-intelligence!)   Wanted to update you, but probably best if I delete my original comment for being overly-optimistic/misleading in this regard.

Essentially, my thought was that a general economic utility function would form a good basis for an AGI by providing a prime motive.  (The idea is that general intelligence is just a degree of capability re: an infinitely expanding set of problems.)  It would relate to survival instinct in the sense that the automata is driven toward optimal decisionmaking in the context of maintaining and bettering it's condition, (though not to the point of mindless expansion which could be harmful to its "ecosystem), where every decision is ultimately regarded as economic.

(if you have an "online venue" where those topics can be discussed, that'd be great---I think we are getting kicked off this site)

Steve Omohudro wrote a paper called Basic AI Drives that steps through why we would expect an AI with narrow goals to find some basic, general concepts as instrumentally useful for their narrow goals.
For example, an AI designed to maximize stock market returns but whose design is silent on the importance of continuing to survive would realize that its continued survival is a key component of maximizing stock market returns, and thus take actions to keep itself operational.
In general, we should be skeptical of 'anthropomorphizing' AI and other code, but it seems like there are reasons to expect this beyond "well, humans behave this way, so it must be how all intelligence behaves."

I assume Omohudru uses "maximize returns" in the context of minimax, where mitigation of downside is part of the equation.  From my perspective, "win else survive" is a basic extension of minimax in cyclic games.

The concept of 'survival instinct' probably falls in the category of what Marvin Minsky would call a 'suitcase word', i.e. it packages together a number of related phenomena into what at first appears to be a singular notion.
So it's quite possible that we can construct mechanisms that have the appearance of some kind of 'hard-coded' survival instinct, without that ever featuring as an explicit rule(s) in the design.
See the beautiful little book 'Vehicles' by the neuroanatomist Valentino Braitenberg for a compelling narrative of how such 'top down' concepts as 'survival instinct' might evolve 'from the bottom up'.
Also, trying to ensure that intelligent artefacts place too high a priority on their survival might easily lead to a Killbot Hellscape.

I will answer by quoting the book The Myth of Sisyphus:

There is only one really serious philosophical question, and that is suicide.

So, probably, we need some degree of a survival instinct if we don't want our AGIs to "terminate themselves" (whatever that means), whenever they get existential.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Could AIs self-develop in the future?
Is it possible that, at some time in the future, AIs will be able to initiatively develop themselves, rather than passively being developed by humanity?

This is known as the intelligence explosion hypothesis or recursive self-improvement.

The question is about self-development not self-improvement, so this answer is wrong, given that self-development does not necessarily imply self-improvement.

Humans might create somewhere in the future a so-called ultraintelligent machine, a machine that can surpass all intellectual activities by any human. This would be the last invention man would need to do, since this machine is better in inventing machines than humans are (since that is an intellectual activity). Also, since humans can create machines as good as the ultraintelligent machine, this machine can create better machines, which in turn can create better machines, etcetera. This is known as the Intelligence explosion, and it is also called recursive self-improvement.
The existence, let alone the development, if an ultraintelligent machine is still hypothetical. We are nowhere close to creating an ultraintelligent machine.

The question is about self-development not self-improvement, so this answer is inopportune, given that self-development does not necessarily imply self-improvement.

It depends what you mean by 'develop themselves' - in a rather limited sense, an online machine learning approach such as Genetic Algorithms 'develops itself' to provide better solutions.
There is already a theoretical model that represents the ultimate concept of development: Juergen Schmidhuber's Goedel Machine is constructed so as to self-modify when it can prove that this modification is optimal.

The question is about self-development and not self-improvement or modification of the own source code. I suppose that the OP meant AIs that develop other AIs, not necessarily improving them.


text:
How can the theory of multiple intelligences be incorporated into AI?
I have been wondering since a while ago about the theory of multiple intelligences and how they could fit in the field of Artificial Intelligence as a whole.
We hear from time to time about Leonardo Da Vinci being a genius or Bach's musical intelligence. These persons are commonly said to be (have been) more intelligent. But the multiple intelligences speak about cooking or dancing or chatting as well, i.e. coping with everyday tasks (at least that's my interpretation).
Are there some approaches on incorporating multiple intelligences into AI?
Here's a related question: How could emotional intelligence be implemented?.

If anything, multiple intelligences are much more obvious in AI than in other fields, because we haven't yet unlocked how to do transfer between domains.
As an example, AlphaGo is very, very good at playing Go, but it's got basically nothing in the way of bodily-kinesthetic intelligence. But other teams have built software to control robots that does have bodily-kinesthetic intelligence, while not being good at the tasks that AlphaGo excels at.
This sort of modular intelligence is typically referred to as 'narrow AI,' whereas we use the term 'general AI' (or AGI, for Artificial General Intelligence) to refer to intelligence that we've built that can do roughly as many different kinds of things as people can do.

On possibility is a blackboard architecture.  Envision each different "kind" of intelligence as a discrete agent, and let the agents collaborate using the blackboard model.  Now you have an AI with multiple intelligences.
This is something I've actually been experimenting with, and while I don't have any particularly impressive results to share or anything, I hold a strong belief that an approach that is at least somewhat like this will be crucial to developing an AGI.  And that is rooted in my belief that the human mind does have "multiple intelligences" and that they collaborate something like this.


text:
What are some examples of Classical AI applications?
I believe that Classical AI uses deductive thought processes. For example, given as a set of constraints, deduce a conclusion.
What are some examples of successfully applying Classical AI to real-world problems?

The term classical AI refers to the concept of intelligence that was broadly accepted after the Dartmouth Conference and basically refers to a kind of intelligence that is strongly symbolic and oriented to logic and language processing. One basic point is the duality body vs. mind. It's in this period that the mind starts to be compared with computer software.
Two classical historical examples of this conception of intelligence

Deep Blue, whose aim in life was to be the master of chess, ruling over the (not-so) intelligent mankind

Eliza a computer-based therapist that turned out to trigger a critic to the classical AI

Two technical examples of classical AI

Expert systems, which are computer programs that strongly rely on the type of constraints and conclusions that you refer to, in order to accomplish feats of apparent intelligence

Fuzzy logic, which is an extension of multivalued logic, but with continuous values instead of discrete ones

Note that in all cases the hardware (once compared with the body) does not play any role: Intelligence is abstract and independent from the material world.


text:
Can autoencoders be used for supervised learning?
Can autoencoders be used for supervised learning without adding an output layer? Can we simply feed it with a concatenated input-output vector for training, and reconstruct the output part from the input part when doing inference? The output part would be treated as missing values during inference and some imputation would be applied.

I don't quite understand. If you train it with input-output vectors, you'll also need output vectors while doing inference to feed it to the network. What are you going to do about that?

No, they would be treated as missing values and imputed in some way. The autoencoder would then try to reconstruct it (multiple iterations may be necessary,). The question is precisely about the feasibility of this idea. I'll edit to clarify.

One such paper I know of and which I implemented is Semi-Supervised Learning using Ladder Networks . I quote here their description of the model:

Our approach follows Valpola (2015), who proposed a Ladder network where the auxiliary task is
  to denoise representations at every level of the model.  The model structure is an autoencoder with
  skip connections from the encoder to decoder and the learning task is similar to that in denoising
  autoencoders but applied to every layer, not just the inputs. The skip connections relieve the pressure
  to represent details in the higher layers of the model because,  through the skip connections, the decoder can recover any details discarded by the encoder.

For further explanations on the architecture check Deconstructing the Ladder Network Architecture by Yoshua Bengio.

I recall reading papers about such systems, if I understand you correctly, but can't recall the titles at the moment.
The idea was to use character-based generative RNNs, train them on sequences encoded like "datadatadatadata|answer", and then when feeding in "otherdatadata|" then it would continue to generate some kind of expected answer.
But, as far as I recall, that was just a neat illustration since if you have the data to do something supervised, then you'd get better results by conventional methods.


text:
Does the Chinese Room argument hold against AI?
Back in college, I had a Complexity Theory teacher who stated that artificial intelligence was a contradiction in terms. If it could be calculated mechanically, he argued, it wasn't intelligence, it was math.
This seems to be a variant of the Chinese Room argument. This argument is a metaphor, where a person is put in a room full of Chinese books. This person doesn't understand a word of Chinese but is slipped messages in Chinese under the door. The person has to use the books, which contain transformation rules, to answer these messages. The person can apply the transformation rules but does not understand what (s)he is communicating.
Does the Chinese room argument hold? Can we argue that artificial intelligence is merely clever algorithmics?

It depends on the definition of (artificial) intelligence.
The position that Searle originally tried to refute with the Chinese room experiment was the so-called position of strong AI: An appropriately programmed computer would have a mind in the exact same sense as humans have minds.
Alan Turing tried to give a definition of artificial intelligence with the Turing Test, stating that a machine is intelligent if it can pass the test. The Turing Test is introduced here. I won't explain it in detail because it is not really relevant to the answer. If you define (artificial) intelligence as Turing did, then the Chinese room experiment is not valid.
So the point of the Chinese room experiment is to show that an appropriately programmed computer is not the same as a human mind, and therefore that Turing's Test is not a good one.

"Strong AI" is typically used to refer to AI that is intelligent enough to improve itself, not that it has a "mind" in the sense that humans have "minds."

There are two broad types of responses to philosophical queries like this.
The first is to make analogies and refer to intuition; one could, for example, actually calculate the necessary size for such a Chinese room, and suggest that it exists outside the realm of intuition and thus any analogies using it are suspect.
The second is to try to define the terms more precisely. If by "intelligence" we mean not "the magic thing that humans do" but "information processing," then we can say "yes, obviously the Chinese Room involves successful information processing."
I tend to prefer the second because it forces conversations towards observable outcomes, and puts the difficulty of defining a term like "intelligence" on the person who wants to make claims about it. If "understanding" is allowed to have an amorphous definition, then any system could be said to have or not have understanding. But if "understand" is itself understood in terms of observable behavior, then it becomes increasingly difficult to construct an example of a system that "is not intelligent" and yet shares all the observable consequences of intelligence.

First of all, for a detailed view of the argument, check out the SEP entry on the Chinese Room.
I consider the CRA as an indicator of you definition of intelligence. If the argument holds, yes, the person in the room understands Chinese. However, let's sum up the three replies discussed in the SEP entry:

The man himself doesn't understand Chinese (he wouldn't be able to understand it when outside the room), but the system man+room understands it. Accepting that reply suggests that there can exist an intelligent system which parts aren't themselves intelligent (which can be argued of the human body itself).
The system doesn't understand Chinese, as it cannot interact with the world in the same way a robot or a human could (i.e. it cannot learn, is limited in the set of questions it can answer)
The system doesn't understand Chinese (depending on your definition of understanding), and you couldn't say a human performing the same feats as the Chinese room understands Chinese either.

So whether the argument, or a variant of it holds, depends on your definitions of intelligent, understanding, on how you define the system, etc. The point being that the thought experiment is a nice way to differentiate between the definitions (and many, many debates have been held about them), in order to avoid talking past each other endlessly.

Depends on who you ask! John Searle, who proposed this argument, would say "yes", but others would say it is irrelevant. The Turing Test does not stipulate that a machine must actually "understand" what it is doing, as long as it seems that way to a human. You could argue that our "thinking" is only a more sophisticated form of clever algorithmics.

What excellent responses!  When Searle published his paper, I was still in college, so my own understanding was limited.  ...And I attempted to resolve "my version" of what he was trying to say. It was a hugely rewarding effort. Since then a lot has happened.  In one world, I simply dropped my pursuit of working through the CR problem, and went on to other problems.  Still, as recently as March 2023, I received an email from my brother-in-law with a link to Jeffrey Kaplan's "most famous thought experiment..." video.  Reluctantly, I watched.  If you haven't seen this <30 minute lesson, you really owe it to yourself.  Here's the link: https://www.youtube.com/watch?v=tBE06SdgzwM
Since then, I've realized that I wasn't the only one who didn't understand what Searle was talking about.  Searle himself, though a profound and astute philosopher, did not understand programming enough to weigh in on AI.
Try this: write some code.
(I'll show you my answer)
I wrote a loop in C
int i;
for (i=0;i<10;i++) {
  printf("%f\n",3.1415);
}
Now answer, "why did you <placeholder>?"
Why did you write a loop?
Why only 10 iterations?
Why not more? Less?
Why print 3.1415?  You could have done anything in that space.
It doesn't matter what the questions are (assuming they relate, in spirit).
It doesn't matter what the answers are.

All substance, no matter how trivial, is not syntax; it is semantics.
Almost every line.  Of every program.  Ever written.  Is already loaded with semantics.  To limit computer programs to being syntax-only is a world-class oversight.  I accepted the premise.  Lots of people did.  The Chinese Room is built on this casually-delivered error.  So if you found yourself disagreeing with Searle's premise but not really being clear as to why...
Searle paints an arid ghost town, devoid of semantics, and challenges us to spot the spirit of AI on his canvas.  No wonder there was an AI winter!  Turn away from that erroneous artwork, and see the colors and hear the sounds of the real world.  The spirit of AI is in that rich world.
also, when's the last time you wrote a program that was just a big case statement?


text:
What is the relevance of AIXI on current artificial intelligence research?
From Wikipedia:

AIXI ['ai̯k͡siː] is a theoretical mathematical formalism for artificial general intelligence. It combines Solomonoff induction with sequential decision theory. AIXI was first proposed by Marcus Hutter in 2000[1] and the results below are proved in Hutter's 2005 book Universal Artificial Intelligence.[2]

Albeit non-computable, approximations are possible, such as AIXItl. Finding approximations to AIXI could be an objective way for solving AI.
Is AIXI really a big deal in artificial general intelligence research? Can it be thought as a central concept for the field? If so, why don't we have more publications on this subject (or maybe we have and I'm not aware of them)?

"Current artificial intelligence research" is a pretty broad field. From where I sit, in a mostly CS realm, people are focused on narrow intelligence that can do economically relevant work on narrow tasks. (That is, predicting when components will fail, predicting which ads a user will click on, and so on.)
For those sorts of tools, the generality of a formalism like AIXI is a weakness instead of a strength. You don't need to take an AI that could in theory compute anything, and then slowly train it to focus on what you want, when you could just directly shape a tool that is the mirror of your task.
I'm not as familiar with AGI research itself, but my impression is that AIXI is, to some extent, the simplest idea that could work--it takes all the hard part and pushes it into computation, so it's 'just an engineering challenge.' (This is the bit about 'finding approximations to AIXI.') The question then becomes, is starting at AIXI and trying to approximate down a more or less fruitful research path than starting at something small and functional, and trying to build up?
My impression is the latter is much more common, but again, I only see a small corner of this space.

You're not actually addressing the questions in the current post. The first question is "is AIXI really a big deal in artificial general intelligence research?". The question strictly asks about the importance of AIXI in AGI research, it does not ask about if you think other specific tools are better for the corresponding tasks instead of narrowing approximations of AGI models to the same specific tasks. In the post, another question is: "why don't we have more publications on this subject?" No answer to this question in your post.

Is AIXI really a big deal in artificial general intelligence research?

Yes, it is a great theoretical contribution to AGI. AFAIK, it is the most serious attempt to build a theoretical framework or foundation for AGI. Similar works are Schmidhuber's Gödel Machines and SOAR architecture.
AIXI is an abstract and non-anthropomorphic framework for AGI which builds on top of the reinforcement learning field, without a few usual assumptions (e.g., without the Markov and ergodicity assumptions, which guarantees that the agent can easily recover from any mistakes it made in the past). Even though some optimality properties of AIXI have been proved, it is (Turing) uncomputable (it cannot be run on a computer), and so it is of very limited practical usefulness. Nonetheless, in the Hutter's book Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability (2005), where several properties of AIXI are rigorously proved, a computable but intractable version of AIXI, AIXItl, is also described. Furthermore, in the paper A Monte Carlo AIXI Approximation (2009), by Joel Veness et al., a computable and tractable approximation of AIXI is introduced. So, there have been some attempts to make AIXI practically useful.
The article What is AIXI? — An Introduction to General Reinforcement Learning (2015), by Jan Leike, which is one of the contributors to the development and evolution of the AIXI framework, gives a gentle introduction to the AIXI agent. See also The AIXI Architecture at the Stanford Encyclopedia of Philosophy for a possibly gentler introduction to AIXI.

Can it be thought of as a central concept for the field?

Yes, the introduction of AIXI and related research has contributed to the evolution of the AGI field. There have been several discussions and published papers, after its introduction in 2000 by Hutter in the paper A Theory of Universal Artificial Intelligence based on Algorithmic Complexity.
See e.g. section 7, "Examples of Superintelligences", of the paper Artificial General Intelligence and the Human Mental Model (2012), by Roman V. Yampolskiy and Joshua Fox. See also https://wiki.lesswrong.com/wiki/AIXI which contains a discussion regarding a few problems related to AIXI, which need to be solved or possibly avoided in future AGI frameworks. Furthermore, see also this and this articles.

If so, why don't we have more publications on this subject (or maybe we have and I'm not aware of them)?

There have been several publications, mainly by Marcus Hutter and associated researchers. You can see Marcus Hutter's publications on the following web page: http://www.hutter1.net/official/publ.htm.
If you are interested in contributing to this theory, there are several ways. If you are mathematically well educated, you can attempt to solve some of the problems described here (which are also mentioned in the Hutter's 2005 book mentioned above). Furthermore, you can also contribute to new approximations or improvements of existing approximations of the AIXI agent. Finally, you can build your new AGI framework by avoiding the problems associated with the AIXI framework. See also projects promoted by Hutter. It may be a good idea to also take into account e.g. Gödel Machines and related work, before attempting to introduce a new framework (provided you are capable of it).
I think that this theory has not attracted more people probably because it is highly technical and mathematical (so it is not very easy to understand unless you have a very solid background in reinforcement learning, probability theory, etc.). I also think that most people (in the AI community) are not interested in theories, but they are mainly guided by practical and useful results.

AIXI is really a conceptual framework. All the hard work of actually compressing the environment still remains.
To further discuss the question raised in Matthew Graves answer: given our current limited level of ability to represent complex environments, it seems to me that it doesn't make a lot of practical difference whether you start with AIXI as defining the 'top' of the system and working down (e.g. via supposedly generalized compression methods) or start at the 'bottom' and try solve problems in a single domain via domain-specific methods that (you hope) can subsequently be abstracted to provide cross-domain compression.

The second paragraph is the result of your only opinion. You give zero argumentations/explanations why you think like that. For me,  "given our current limited level of ability to represent complex environments" is definitely not a sufficient explanation or argumentation.

@nbro To quote a famous AI researcher: "We have yet to represent even a single concept on a computer", certainly not with the kind of malleability that comes naturally to humans. Thus in practice, its difficult to determine the utility of AIXI because we don't have a strong notion of the kinds of representations it needs to manipulate, or how it could usefully manipulate them.

AIXI is important,  the reinforcement learning we already see is a smaller classical version of the full intractable model,  that one would require quantum computation to see it come to fruition.


text:
In what ways can connectionist AI be integrated with GOFAI?
In what ways can connectionist artificial intelligence (neural networks) be integrated with Good Old-Fashioned A.I. (GOFAI)? For instance, how could deep neural networks be integrated with knowledge bases or logical inference? One such example seems to be the OpenCog + Destin integration.

A neural net with even a single hidden layer is capable of Universal function approximation - it can approximate any continuous function 'as closely as you like'.
Hence, one option would be to look for GOFAI applications that would benefit from this property - for example, in state-space search approaches where the utility of a state is not readily defined in advance, and could instead be learned.

This has actually been done, so you could edit this answer to provide a reference or link to an example of such a work.


text:
Selecting the right technique to predict disease from symptoms
I'm trying to come up with the right algorithm for a system in which the user enters a few symptoms and the system has to predict or determine the likelihood that a few selected symptoms are associated with those existing in the system. Then, after associating them, the result or output should be a specific disease for the symptoms.
The system is comprised of a series of diseases with each assigned to specific symptoms, which also exist in the system.
Let's assume that the user entered the following input:
A, B, C, and D

The first thing the system should do is check and associate each symptom (in this case represented by alphabetical letters) individually against a data table of symptoms that already exist. And in cases where the input doesn't exist, the system should report or send feedback about it.
And also, let's say that A and B were in the data table, so we are 100% sure that they're valid or exist and the system is able to give out the disease based on the input. Then let's say that the input now is C and D, where C doesn't exist in the data table, but there is a possibility that D exists.
We don't give D a score of 100%, but maybe something lower (let's say 90%). Then C just doesn't exist at all in the data table. So, C gets a score of 0%.
Therefore, the system should have some kind of association and prediction techniques or rules to output the result by judging the user's input.
Summary of generating the output:
If A and B were entered and exist, then output = 100%
If D was entered and existed but C was not, then output = 90%
If all entered don't exist, then output = 0%

What techniques would be used to produce this system?

I think you're coming at your problem slightly wrong... what you're essentially talking about is a belief network.
You may want to look into existing Bayesian Learning techniques to get your head around this, but belief networks commonly use the exact scenario you're talking about; using a set of known (or uncertain facts) statements to produce some inferred probability of a particular output.
Even more, they often express this through disease-symptom based examples in tutorials! Try here.
My point is that it would be better to use a belief network as the theoretical groundwork is all already there for you, instead of an ANN.

If you want to implement a bayesian network from scratch, you're going to have to understand the raw mathematics that drive them. There are a couple of suites out there to run bayesian networks on without having to understand all of the (somewhat confusing at times) maths, such as Netica (https://www.norsys.com/netica.html)


text:
How do artificial neural networks store data compared to biological neural networks?
Do scientists know by what mechanism biological brains/biological neural networks store data?
I only have a vague notion that we store data in our brains by altering synapses.
I also would love a decent description of how a vanilla artificial neural network stores data.
So, my 2 questions are

How is data stored in a biological neural network?

How is data stored in an artificial neural network?

Second question first: Data is stored in an ANN in the form of weights in the adjacency matrix between neurons. During training, these weights are updated by a learning algorithm (such as backpropagation).
First question: according to award-winning neuroscientist Tim Bliss:

“It’s been accepted really since the turn of the 20th century, since the time of the Spanish neuroscientist Ramón y Cajal, that really the only place where memories can be stored is at synapses, the junctions between nerve cells.

A protein called the NMDA receptor plays a key roll in the strengthening of synaptic connections (which is more broadly achieved by a form of Hebbian Learning).


text:
Is it possible to implement Asimov's Three Laws of Robotics?
Would it be possible to put Asimov's three Laws of Robotics into an AI?
The three laws are:

A robot (or, more accurately, an AI) cannot harm a human being, or through inaction allow a human being to be harmed1

A robot must listen to instructions given to it by a human, as long as that does not conflict with the first law.

A robot must protect its own existence, if that does not conflict with the first two laws.

The most challenging part is this section of the first law:

or through inaction allow a human being to be harmed

Humans manage to injure themselves unintentionally in all kinds of ways all the time. A robot strictly following that law would have to spend all its time saving people from their own clumsiness and would probably never get any useful work done. An AI unable to physically move wouldn't have to run around, but it would still have to think of ways to stop all accidents it could imagine.
Anyway, fully implementing those laws would require very advanced recognition and cognition. (How do you know that industrial machine over there is about to let off a cloud of burning hot steam onto that child who wandered into the factory?) Figuring out whether a human would end up harmed after a given action through some sequence of events becomes an exceptionally challenging problem very quickly.

Defining "harm" and in particular, "allowing harm via inaction" in any meaningful way would be difficult. For example, should robots spend all their time flying around attempting to prevent humans from inhaling passive smoke or petrol fumes?
In addition, the interpretation of 'conflict' (in either rule 2 or 3) is completely open-ended. Resolving such conflicts seems to me to be "AI complete" in general.
Humans have quite good mechanisms (both behavioral and social) for interacting in a complex world (mostly) without harming one another, but these are perhaps not so easily codified. The complex set of legal rules that sit on top of this (polution regulations etc) are the ones that we could most easily program, but they are really quite specialised relative to the underlying physiological and social 'rules'.
EDIT: From other comments, it seems worth distinguishing between 'all possible harm' and 'all the kinds of harm that humans routinely anticipate'. There seems to be consensus that 'all possible harm' is a non-starter, which still leaves the hard (IMO, AI-complete) task of equaling human ability to predict harm.
Even if we can do that, if we are to treat as actual laws, then we would still need a formal mechanism for conflict resolution (e.g. "Robot, I will commit suicide unless you punch that man").

In that situation, they would punch the man, because the punch wouldn't kill them. However, committing suicide will kill the guy.

That already requires more knowledge than human beings often exhibit (people are often killed by a single punch) and one can easily construct arbitrarily complex variations.

I think this is almost a trick question in a sense. Let me explain:
For law 1, any AI would abide by the first rule unless it was deliberately created to be malevolent, in that the AI it would understand harm was imminent but do nothing about or would actively attempt to harm. Any 'reasonable' AI would (try its best to) prevent any harm it understood, but couldn't react to imminent harm 'outside it's knowledge', thus satisfying law 1. Any AI that 'tries its best' to prevent harm works here.
For law 2, it is simply a matter of design. If one can design an AI capable of parsing and understanding the entirety of human language (beyond just speech), just program it to act accordingly, mindful of the first law. Thus, I think we can develop an AI that will obey every command it understands but getting it to understand anything and everything I believe is impossible.
For law 3, it rides in the same vein as law 1.
In conclusion, I think there is no philosophical problem with implementing such an AI, but that the actual design of such an AI is fundamentally impossible (understanding all possible harms, and all possible commands).

The paper The First Law of Robotics (a call to arms) (AAAI-94), by Weld and Etzioni, discusses the first Asimov's law, some technical issues it gives rise to (some of them are already mentioned in the other answers), and how they could be addressed (they propose a simplistic way to formalize the first law, but they don't claim it is the right way to do it). You should read it for more details.


text:
What technologies are needed for a self-driving car?
Google, Tesla or Apple have all built or are building their own self-driving cars. As an expert in a related area, I am interested in knowing at a high level, the systems and techniques that go into self-driving cars.
How easy is it for me to make a tabletop prototype (large enough to accommodate the needed computing power needs)?

You're going to need some way to 'see' the area around the car, and to track the speed of nearby objects. Google uses a combination of LIDAR, radar, conventional cameras, and occasionally sonar (see  here for a high-level overview). This technology is quite expensive, and can easily cost thousands of US dollars.
However, a bigger obstacle than the expense of the hardware (which would be smaller for a table-top prototype) is the software complexity. Like many major projects, the software for self-driving cars is the result of years of work from AI research teams, and thus extremely difficult to duplicate on your own.
That said, you're not trying to make a state-of-the-art self-driving car.  Assuming you're an expert in image processing and robotics, you can probably create a basic prototype, (like something that drive in a limited table-top environment). However, it's still going to take a lot of time and money.

The radar is used to detect and track objects under bright ambient lighting, or what is it used for? Thanks!

Is this radar the same type of radar that is used for detecting the locations of planes and boats?

This answer really needs some references to improve its reliability.  Right now, you're saying that self-driving cars use AI techniques, but you don't really say 1. for what and 2. which specific techniques. It would be nice to have some examples.


text:
Is there any research work on the usefulness of the concept of "personality" for an AGI?
Humans have been endowed with personalities by nature, and it is not clear (to me at least) if this is a feature or a bug. This has been explored in science fiction by various notions of Borg-like entities. It is my belief that, for narrative reasons, such stories usually end with the humans with their flawed personalities winning in the end.
Are there experts who have analyzed, perhaps mathematically, design criteria for an AI agent with weakly enforced goals (e.g. to maximize reproduction in the human case) in an uncertain environment, and ended up with the answer that a notion of personality is useful?
If there are AI researchers, philosophers (or maybe even science fiction) writers who have examined this question in their work, I would be happy to know about those too.

First, a note on the question itself.

Humans have been endowed with personalities by nature, and it is not clear (to me at least) if this is a feature or a bug.

In my opinion, this is a statement that constrains the question, since it assumes that the personality is given. To me, it feels a bit like playing god: Artificial (given) Intelligence would hence imply Artificial (given) Personality. This approach to the problem seems to be supported by the next fragment:

a notion of personality is useful

I point to the above because I don't think that artificial intelligence... Intelligence itself, actually, need to be given or assigned, or even have a use in the sense of a purpose.

The previous note was about emergence, which is a topic that user217281728 briefly addressed in their answer. In this second approach, the particular traits just happen, or develop. The interaction between the (so-called) agents and their environment, as well as fellow agents can give place to new behaviour patterns, not designed beforehand.
In an evolutionary approach, if the personality would happen to have an advantage (or at least not represent a disadvantage), then it could just appear. Of course, I am making a number of assumptions and demarcations here as well:

I am thinking about embodied intelligence
I speak of evolutionary robotics
I think on social issues being of importance
I assume that personality could emerge

Now, an example that I find extremely interesting is that of the little mobile robots which could move around and end-up in a pool of food or a pool of poison. And they, somehow, by some odd chance, recognised or made a relation between signals sent by other robots, and the presence of food. Or not. That was more or less the thing: Some robots (kind of) learned to conceal information and thus had more time to eat themselves. Well, I would have a couple of personality adjectives for such guys.
Here you find the article and here you find some videos and related stuff.
And with that, we land at my last point: We humans put the adjectives, according to our social conditioning. We call Marvin depressive and R2D2 lovely and charming.
If they perceive their personalities as constructive or damaging, will always depend on our own judgment. In the end, it is quite common under humans to disagree on personality issues, too.

Bonus
Remember when HAL got emotional, on the face of death?
It gets human when it loses its cool, before the flawed-personality human astronaut :)

'Personality' is something of a 'suitcase word' (Minsky) for quite a large collection of (presumably reasonably consistent) observable traits.
It seems clear that there is a certain collective advantage in having a consistent personality - specifically that it affords observers some learning gradient in an otherwise uncertain environment. This is of particular importance because those consistencies might have been arrived at using different learning mechanisms than the ones a given observer has.
Hence, in any non-trivial coevolutionary system, other organisms will inevitably make use of any such consistencies. Consider a simple robot, called Alice, say, that has the trait of 'quickly flashing red when it sees a blue robot'. It makes sense for all observers to exploit everything that they perceive as correlating with Alice's behavior, in particular, the prediction that a blue robot is likely to be present.
The best reference I can recommend on this (which shows that we tend to ascribe 'personality' to even very simple mechanisms) is 'Vehicles' by Valentino Braitenberg.

"Usefulness" can only be measured against some purpose. Once you pass AGI - which really means "generally animal-like AI, because it seems general to us" - then you've passed into a world of potentially undefined behavior.
Part of what makes a human free and sets us apart from the other animals is the fact that our purposes, capabilities and possibilities aren't fully defined. We're open ended.
To clarify terms, I interpret "Strong AGI" as "potentially super intelligence, but at least human level."
When we say "Strong AGI" vs just "AGI," we're not saying that one is more open ended than the other. We are saying that the stronger one is simply smarter on some axis.
So to ask whether a particular trait would be "more useful" to a Strong AGI - that would depend on the purpose of the AGI. But here's the catch: if a thing had just one purpose, then the most efficient solution to fulfilling that one purpose will always be a narrow solution, not a general one. When the purpose of the object is known before hand, giving that object more general capability than is necessary for that purpose is counterproductive.
That's why it's impossible to make declarative prescriptions about what a free, open-ended AGI should or shouldn't need. Such prescriptions would nullify the open-ended freedom of utility that its generality implies. We can speak declaratively about lesser robots and animals.
But for any given problem, the solution we will want to find is the most well-defined, narrow, efficient one available - not the most general one.
In other words, sure, personalities could be useful for a Strong AGI, assuming the problems in question involved personalities.


text:
Are there any results on how difficult certain languages are to learn for chatbots?
Has there any research been done on how difficult certain languages are to learn for chatbots?
For example, CleverBot knows a bit of Dutch, German, Finnish and French, so there are clearly chatbots that speak other languages than English. (English is still her best language, but that is because she speaks that most often)
I would imagine that a logical constructed language, like lobjan, would be easier to learn than a natural language, like English, for example.

The easiest language to learn by a bot is the language that has the most training data available. Sure, language structure matters, but when there are multiple order of magnitude differences in the size of training data, then that overwhelms everything else.
One should expect English, French or Chinese to get much better results than any constructed language even if (especially if?) the system includes no language-specific tuning.

It's not the language itself but rather the structure and for the language's ambiguity, for example in English: person a says "John and Bob (his fish)" person B says "He died!", posed question, to whom does person B refer to by he died.  More than the language, but the application. You can write a chat bot in Assembly, C, C++, C#, Java or Python. The all work a bit differently but can accomplish the same result, but one language might have more pros or cons to the other. So it will boil down to not language but the understanding of what is being said, research of language in the brain has come to confirm we associate a meaning/feeling/and other inputs with a given language.
So to conclude: English is by far the most chaotic for a chat bot but Japanese is actually the best due to the way the language itself is written/spoken. There is more structure to it and less ambiguity.
I'm a Software Engineer and An AI Researcher for the past 7 years.


text:
Could curiosity improve artificial intelligence?
While thinking about AI, this question came into my mind. Could curiosity help in developing a true AI? According to this website (for testing creativity):

Curiosity in this context refers to persistent desire to learn and discover new things and ideas. A curious person

always looks for new and original ways of thinking,
likes to learn,
searches for alternative solutions even when traditional solutions are present and available,
enjoys reading books and watching documentaries,
wants to know how things work inside out.

Let's take Clarifai, an image/video classification startup, which can classify images and video with the best accuracy (according to them). If I understand correctly, they trained their deep learning system using millions of images with supervised learning. In the same algorithm, what would happen if we somehow added a "curiosity factor" when the AI has difficulty in classifying an image or its objects? It would ask a human for help, just like a curious child.
Curiosity makes a human being learn new things and also helps to generate new original ideas. Could the addition of curiosity change Clarifai into a true AI?

It could be possible to add a simulation of "curiosity" to an AI, but doing so will probably require some hard-coded rules that would themselves probably not be creative. And since there is "no right answer to this question", then it's probably not likely that curiosity is the missing factor to AI. For an example where curiosity can be added to an AI, see the Mario Lives! video, where someone built a video game AI that could become "curious".

Does this addition of curosity changes clarifai into a true AI?

As per my answer to this question, we don't know what the ingredients for a 'true AI' are. Via the Turing Test and its variants, the best we can do is "know one when we see one".
Curiosity certainly appears necessary for intelligence, though it doesn't seem sufficient - a lemming-like creature curious to see what's at the bottom of a steep cliff might not survive long enough to learn caution, even if it had the learning mechanisms to do so.
Here is some work by Schmidhuber on Artificial Curiousity.
Pierre-Yves Oudeyer has also done quite a lot of work on this and Active Learning/Intrinsic motivation.

when the AI has difficulty in classifying a image or its objects it should ask a human for help just like a curious child

It's called active learning, it's already used quite often.

It's a well known concept that's already used
What we call "curiosity" in humans and animals is in effect the chosen level of the "exploit vs explore" tradeoff for any active system. For example, the field of reinforcement learning is one approach that studies implementations of what essentially is the equivalent of curiosity; and we have research on how much curiosity is best e.g. multi-armed bandit concept.
So "using curiosity" is something that we already do as much as we can/should/are able to, but it would usually be called in some other, more specific term to specify the exact meaning instead of the vague word of "curiosity".

Curiosity by itself does not improve intelligence.
It increases the chances of better understanding a given subject, given that curiosity is coupled with actions in that direction.
For example:
I am curious about how to make pancakes and decide to find a recipe but stop at the first instance of an answer with steps to follow.
Curiosity needs to be coupled with the desire to improve a given understanding and be followed by a review of current knowledge with the aim of updating the previousely reached conclusions. Provided that the used logic that judges improvements is correct.
Curiosity will not necessarily be benefitial for an improved intelligence.  But to allow for an improved intelligence, curiosity is a mandatory requisite.
Curiosity is a symptom of an evolving intelligence.

Curiosity is used successfully with Random Network Distillation (RND). OpenAI has published a detailed article about their approach using this method, which was especially successful with previously unsolved games like Montezuma’s Revenge.
While this does not fully answer your question about curiosity being required to build a true AI, it shows that previously unsolved problems became solvable introducing curiosity in the reward system.


text:
How do Google cars recognize the traffic signs?
The paper Multi-column Deep Neural Networks for Image Classification (pages 7-8) shows an attempt at recognizing the traffic signs, with lower error rates, by using multi-column deep neural networks.
Are Google cars using similar techniques of predicting signs using DNN, or are they using some other method?

I'm not sure what Google is using to perform that task, but most companies use region based convolutional neural nets to locate traffic signs and other objects.
But other companies use a deep neural network + bag of words approach to find objects.
See: Bag-of-Words Based Deep Neural Network for Image Retrieval which shows a general approach, to get the exact location you can use Feature Matching or Random Boxes.

https://www.researchgate.net/publication/284505205_Bag-of-Words_Based_Deep_Neural_Network_for_Image_Retrieval this link shows a general approach, to get the exact  location you can use Feature Matching or Random Boxes


text:
What is the current state-of-the-art in Reinforcement Learning regarding data efficiency?
In other words, which existing reinforcement method learns with fewest episodes? R-Max comes to mind, but it's very old and I'd like to know if there is something better now.

There isn't really one specific method which makes any RL agent have faster learning. Rather there is a long list of methods which have shown to increase the speed of learning and they can sometimes play nicely with each other.
Some examples:

Options
True Online Methods
Asynchronous Methods

These are the most influential and promising methods I can think of but the list of techniques is not limited to these 3.

There is a very interesting learning agent. They call it Neural-Episodic-Control. Here is the link for the paper: https://arxiv.org/abs/1703.01988 . Their experiments show that NEC requires an order of magnitude fewer interactions with the environment than agents previously proposed.


text:
What kind of education is required for researchers in AI?
Suppose my goal is to collaborate and create an advanced AI, for instance, one that resembles a human being and the project would be on the frontier of AI research. What kind of skills would I need?
I am talking about specific things, like what university program should I complete to enter and be competent in the field.
Here are some of the things that I thought about, just to exemplify what I mean:

Computer sciences: obviously, the AI is built on computers, it wouldn't hurt to know how computers work, but some low-level stuff and machine-specific things do not seem essential, I may be wrong of course.

Psychology: if AI resembles human beings, knowledge of human cognition would probably be useful, although I do not imagine neurology on a cellular level or complicated psychological quirks typical to human beings like the Oedipus complex would be relevant, but again, I may be wrong.

This question seems to be very similar to this, although I am not sure if you focused on AGI or not, but you certainly mentioned "human-like AI".

As a full-time AI researcher myself, I'd say that a PhD in machine learning would certainly be one useful option.
However, in order make much-needed progress, AI needs avoid falling into the trap of thinking that currently fashionable methods are any kind of 'silver bullet'. There's some danger that a PhD that heads straight into (say) some sub-sub-sub area of DL would end up imposing too much bias on the student's subsequent perspective.
AI research is an essentially multi-disciplinary activity. Other possible backgrounds therefore include:

Mathematics or physics (to first degree or PhD level). A strong background in either of these never did anyone any harm. People who are competent in these fields tend to be able to turn their abilities to new domains relatively easily.
Software Engineering. One of the things that AI needs are integrative architectures for knowledge engineering. Here's why. I believe that one of the reasons that we haven't yet managed to do OCR at the level of a 5 year old is that we've yet to accept that we have to 'build a sledgehammer to crack a nut'. Software architects are used to managing large-scale complexity, so they may be able to help.
Cognitive Science, Psychology, Cognitive Linguistics. The reasons here are obvious.

Above all, I personally think that a good AI researcher should be creative, inquisitive and prepared to question received wisdom, all of which are more important in practice than specifics of their background.

Research on AI seems to be getting wider these days (2016). First, "obvious" few departments (no order):

Computer Science (e.g. computation theory, algorithms): AI researchers there assume that intelligence is a kind of computation, under various forms (e.g. a neural network, a logic system).
Software Engineering: Assuming we find a good model for AI, how do you make it? This is what the engineer will want to figure out. And it can be hard to map mathematical models to an engineered piece.
Statistics and Probabilities (more specific than just Mathematics, which is also close to Computer Science): This is about Data Science, notably as a foundation to Machine Learning, the most active branch in AI---which "just" covers the learning part.
Physics: This is particularly relevant now for hardware (see below).
Neuro Science: Understand how the brain works, as an inspiration to create an artificial one, is the home for Connectionists. Recently, Hassabis and his team at Google Deepmind made several breakthroughs related to reinforcement learning, memory, attention, etc.

Recently Electric Engineering is getting a lot of light, together with the related branches of Physics. Several public and private laboratories focus on "brain chips". To name a few: IBM (who's working on that for some time already), Nvidia, and Facebook. Circa 2010, it became clear that techniques like deep learning require horsepower, thus an increasing focus on creating more powerful, smaller, more energy efficient chips. And on top of that, there is all the work in Quantum Computing.
But the thing is, there seems to be many more fields that are getting involved in AI research. We should mention Chemistry and Biology, as both inspiration and tools to make new models or hardware (e.g. chips that do not use silicon, so they can get smaller).
As for 2016, the above fields are the most active, and promise to remain very active for quite some time. Pick your own depending on your interest, skills, or mere intuition!
To finish, we may be surprised in a few years when we look back at where AI has come from. I believe that if we manage to build an AGI, it will leverage all these fields anyway. I guess the thrill is to be part of the story.


text:
Why do zero-sum perfect information games satisfy the conditions of Von Neumann's theorem?
The Von Neumann's Minimax theorem gives the conditions that make the max-min inequality an equality.
I understand the max-min inequality, basically min(max(f))>=max(min(f)).
The Von Neumann's theorem states that, for the inequality to become an equality f(.,y) should always be convex for given y and f(x,.) should always be concave for given x, which also makes sense.
This video says that for a zero-sum perfect information game, the Von Neumann's theorem always holds, so that minimax always equal to maximin, which I did not quite follow.
Questions

Why do zero-sum perfect information games satisfy the conditions of Von Neumann's theorem?

If we relax the rules to be non-zero-sum or non-perfect information, how would the conditions change?

Here is one of many web pages devoted to the application of Minimax in Tic-Tac-Toe solving: Minimax Algorithm in Game Theory | Set 3 (Tic-Tac-Toe AI – Finding optimal move)
Google search will yield many such results. (I will return to this answer when I have more time, and post more links with more context.)
Here's a high-level explanation on how minimax may be used in the non-chance, perfect information, sequential, partisan, zero-sum set of games [M]:

A non minimax evaluation function would look to place integers on the gameboard in positions that grant maximum value (power & influence)
A minimax evaluation function would look to place integers on the gameboard in positions that maximize value while minimizing the max value of the opponents next placement

This is partly due to the utilization of "resource stealing strategies" in [M] games.

Note: I need to do some more thinking on the sub questions regarding application beyond zero-sum, perfect-information games, but I don't see any issue with it.   Minimax/maximin has wide and fundamental utility in an array of contexts, not strictly related to games except in the sense that every problem can be understood as a type of game or puzzle.


text:
How to reinvent jobs replaced by AI?
In general, what possibilities are there for reinventing job descriptions that could be replaced by an automated AI solution?
My initial ideas include:

Monitoring the AI and flagging its incorrect actions.
Possibly taking over the control in very challenging scenarios.
Creating/gathering more training/testing data to improve the accuracy of the AI.

Given that some people have given answers both considering AGIs and narrow AIs (although you were apparently only interested in jobs replaced by narrow AIs, as you say in a comment but you don't say in this post), I've tagged this post with both [tag:agi] and [tag:weak-ai], which makes it more general.

With AI technology at its current stage (or at least reasonably close to this stage), the jobs you proposed may very well be openings created by AI automation. However, sufficiently advanced AI technology--- the kind that can function as general purpose labor replacement--- will make even these jobs obsolete. This is because such an AI would be able to improve itself and as a result would surpass human level intelligence---meaning that it would take actions which we would not necessarily be able to understand or justify.
This is all assuming that the intelligence of humans is not some kind of upper limit on the possibilities of intelligence in general. As far as I know there is no reason to think we represent such an upper limit.
To summarize...will such jobs exist?
Sub-human artificial intelligence: sure.
Beyond-human artificial intelligence: no reason to think they would be necessary as the AI can do them itself.

Thanks. Your answer is valid since I didn't define the type of AI in my question. I didn't actually mean general AI, since it wouldn't necessarily need any help from humans as you said. I'm more after solutions that apply for an AI that is greater or equal to weak AI but less than general AI.

AI production overseers - People who will command AI to build and control mines and factories. It's like a strategic game, but in the real world.
Explanation: AI even with the intellect of a bee and ability to understand/execute commands in combination with appropriate technologies will be enough to create robots, which will be able to build mines and factories to build new robots or things. Nevertheless, this is can be not enough to build an efficient production. Therefore, an overseer of each production will command it in order to establish or maintain production at the necessary level.
This job can also become obsolete if the AI reaches a high enough level to substitute overseers. However, even if AI become smarter than people, this doesn't guarantee that it will understand people better than overseer human. Therefore, even in this case, it's not a good idea left AI without control by humans.

AI Gatekeepers
Their job would be to make sure they (AI) don't accidentally become our overlords.
AI Tax
Each robot that replaces a human worker is taxed. This line of thought was influenced by Bill Gates's recommendation on adding taxes for robots and an article on universal basic income.
If the gold rush for AI puts many of us out of work (which is quite possible), we should find a way to minimize its impact on the society. One way would be to tax each robot a reasonable amount (not to exceed the cost of human labor) so that it can go to a super-fund for universal basic income. That fund can be used to give everyone in that community a basic salary, so it can prevent hunger, homelessness, and other problems associated with poverty. In my humble opinion, it will be very difficult to implement. Some countries and cities are testing this as a proof-of-concept for now. We will see how that works out.
AI Traders
People that sell AI robots.

AI Diplomat
This may be a bit far-fetched. Let's say, in the next few centuries, the AI becomes advanced enough to earn their own civil rights through the supreme court or legislative branch. We should have experts with the good relationship with AIs. We should give them enough knowledge and tools to make sure AIs see our existence as beneficial to them.
AI Counter Task Force
It could be part of law enforcement or the military with experts in various fields of AI/technology, who are also physically fit to go into battles when needed.
Entrepreneurs
People who encourage the building of jobs and careers that are always hard to automate. As a result, the human being will always have value. We should inspire our entrepreneurs to have a good balance of automation and manual jobs. What if all of your AIs fail in one day and your competition wins because they had more human labor.


text:
Which predictive algorithm can be used to predict a number given other numbers?
I am currently searching for a supervised learning algorithm that can be used to predict the output given a large enough training set.
Here's a simple example. Suppose the training dataset is {[A=1, B=330, C=1358.238902], result=234.244378} and the test dataset {[A=893, B=34, C=293], result=?}
My intention is to predict ? using the input values and result given in the training dataset.
What algorithm would be effective for this problem given the wide range of my input/output values? Would this require some sort of regression algorithm?

You need more training data, ideally 10000+. 2 From my experience xgboost shows good results for such kind of data

@StepanNovikov thank you for the recommendation - I do have a fairly large training set already (roughly 4000+). I will also check-out xgboost. Thanks

@DukeZhou I think textual classification algorithm..can work in that scenario.

Impossible to solve until you define an error measurement ( by example $|R-R'|$ or $(R-R')^2$ ) and how this error changes when A, B and C change.
Extreme example: assume $R()$ is random (unrelated to A, B, C values) but static (always same $R(A,B,C)$ for same values of A,B,C). Given some values of A, B, C, you can only answer the value of $R(A,B,C)$ when A,B,C was in the training set. $R(A,B,C)$ is undefined and no predictable when A,B,C was not in the training set.
Moreover, improvements can be done if $R()$ has some properties, by example, if it is possible to state that $R(A,B,C)=R(B,A,C)$ or that $R(A1,B1,C1)=R(A2,B2,C2)$ if $A1+B1+C1=A2+B2+C2$.

Yes, you are trying to predict a real number output, so this is a regression problem. To know what kind of algorithm would be best I think you have to ask how much data you have and what you know already about the relationships of the numbers. If you try simple linear regression, what kind of error will you get?
If you were to try linear regression and you get an error that is acceptable, then it may be a very simple problem. Beyond linear regression you can look to more advanced things such as Gaussian processes and neural networks which will all make the kinds of predictions you are seeking.

Without seeing more data it's hard to say for sure.  Superficially, this looks like a regression type problem.  As you mention, there is a lot of spread on the input values, but that doesn't necessarily mean that something like linear regression wouldn't work. Try it and see what kind of coefficient of correlation you get.  If it's really low, you probably need a different approach, OR the data might actually not have any (or much) predictive power in this scenario.
Beyond linear regression, you might find that there's a more complicated mathematical relationship between the inputs and outputs, that could be determined using symbolic regression.  Another possibility, if there is a complex non-linear relationship in play, is that an artificial neural network approach might work well.


text:
Is the mean-squared error always convex in the context of neural networks?
Multiple resources I referred to mention that MSE is great because it's convex. But I don't get how, especially in the context of neural networks.
Let's say we have the following:

$X$: training dataset
$Y$: targets
$\Theta$: the set of parameters of the model $f_\Theta$ (a neural network model with non-linearities)

Then:
$$\operatorname{MSE}(\Theta) = (f_\Theta(X) - Y)^2$$
Why would this loss function always be convex? Does this depend on $f_\Theta(X)$?

Answer in short:  MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation functions. Source for my answer is here.

Convexity

A function $f(x)$ with $x ∈ Χ$ is convex, if, for any $x_1 ∈ Χ$, $x_2 ∈ Χ$ and for any $0 ≤ λ ≤ 1$,
  $$f(λ x_1 + (1 − λ) x_2) ≤ λf(x_1) + (1 − λ) f (x_2).$$

It can be proven that such convex $f(x)$ has one global minimum.  A unique global minimum eliminates traps created by local minima that can occur in algorithms that attempt to achieve convergence on a global minimum, such as the minimization of an error function.
Although an error function may be 100% reliable in all continuous, linear contexts and many non-linear contexts, it does not mean the convergence on a global minimum for all possible non-linear contexts.
Mean Square Error
Given a function $s(x)$ describing ideal system behavior and a model of the system $a(x, p)$ (where $p$ is the parameter vector, matrix, cube, or hypercube and $1 ≤ n ≤ N$), created rationally or via convergence (as in neural net training), the mean square error (MSE) function can be represented as follows.
$$e(β) := N^{-1} \sum_{n} [a(x_n) − s(x_n)]^2$$
The material you are reading is probably not claiming that $a(x, p)$ or $s(x)$ are convex with respect to $x$, but that $e(β)$ is convex with respect to $a(x, p)$ and $s(x)$ no matter what they are.  This later statement can be proven for any continuous $a(x, p)$ and $s(x)$.
Confounding the Convergence Algorithm
If the question is whether a specific $a(x, p)$ and method of achieving an $s(x)$ that approximates the $a(x, p)$ within a reasonable MSE convergence margin can be confounded, the answer is, "Yes."  That is why MSE is not the only error model.
Summary
The best way summarize is that $e(β)$ should be defined or chosen from a set of stock convex error models based on the following knowledge.

Known properties of the system $s(x)$
The definition of the approximation model $a(x, p)$
Tensor used to generate the next state in the convergent sequence

The set of stock convex error models certainly includes the MSE model because of its simplicity and computational thrift.


text:
Are these statements about the performance of neural networks as a function of the number of hidden layers contradictory?
In this note Justin Domke says that

In practice, neural networks seem to usually find a reasonable solution when the number of layers is not too large, but find poor solutions when using more than, say, 2 hidden layers.

But in Bengio's remark, he says

Very simple. Just keep adding layers until the test error does not improve anymore.

There seems to be a conflict. Can anyone explain why they suggest differently? Or am I missing something?

There are many problems requiring more than two hidden layers. Randomly select a recent Google journal paper on deep learning, you'll see their network could have something like 5 (or more) hidden layers.
Justin Domke wrote his notes for students, so he probably tried to make his points as simple as possible. For a "typical" machine learning problem that students would most likely work on, two hidden layers should be sufficient. But that doesn't add up for a real practical problem. "Deep" learning usually mean more than two hidden layers.
Number of hidden layers is network design that nobody knows for sure. Yoshua Bengio's suggestion is common and simple. It's not a mathmatic proof, but simply a guideline if you don't know what to  do. You just repeat and repeat, until you see the test error no longer improve.

Your first link is from 2011, which essentially predates the current deep learning explosion. In the many years that have since passed (AlexNet 2012, ResNet 2015) we have since found that if you keep adding layers, we generally do see improved performance.

This is due to improved training techniques and optimization breakthroughs (residual connections, ReLU, dropout etc.). But do note that the result can be diminishing. In particular, take a look at Deep Equillibrium Models, which essentially allow us to train (in the limit equivalence) infinite depth neural networks.

In fact they are telling the same thing: Plot the x-axis as the number of hidden layers, and the y-axis as the performance (e.g. classification accuracy), then this curve will have an upside-down U shape.
Justin's note is clearly saying the same thing as what I wrote above, with added note that the maximum of the curve will happen when x = 2, and Bengio's note is saying the same thing without telling you where the peak could be.


text:
Can a neural network be used to predict the next pseudo random number?
Is it possible to feed a neural network the output from a random number generator and expect it learn the hashing (or generator) function, so that it can predict what will be the next generated pseudo-random number?
Does something like this already exist? If research is already done on this or something related (to the prediction of pseudo-random numbers), can anyone point me to the right resources?
Currently, I am looking at this library and its related links.
https://github.com/Vict0rSch/deep_learning/tree/master/keras/recurrent

See Can machine learning analyze random number generator? at Cryptography SE.

If we are talking about a perfect RNG, the answer is a clear no. It is impossible to predict a truly random number, otherwise it wouldn't be truly random.
When we talk about pseudo RNG, things change a little. Depending on the quality of the PRNG, the problem ranges from easy to almost impossible. A very weak PRNG like the one XKCD published could of course be easily predicted by a neural network with little training. But in the real world things look different.
The neural network could be trained to find certain patterns in the history of random numbers generated by a PRNG to predict the next bit. The stronger the PRNG gets, the more input neurons are required, assuming you are using one neuron for each bit of prior randomness generated by the PRNG. The less predictable the PRNG gets, the more data will be required to find some kind of pattern. For strong PRNGs this is not feasable.
On a positive note, it is helpful that you can generate an arbitrary amount of training patterns for the neural network, assuming that you have control over the PRNG and can produce as many random numbers as you want.
Because modern PRNGs are a key component for cryptography, extensive research has been conducted to verify that they are "random enough" to withstand such prediction attacks. Therefore I am pretty sure that it is not possible with currently available computational resources to build a neural network to successfully attack a PRNG that's considered secure for cryptography.
It is also worth noting that it is not necessary to exactly predict the output of a PRNG to break cryptography - it might be enough to predict the next bit with a certainty of a little more than 50% to weaken an implementation significantly. So if you are able to build a neural network that predicts the next bit of a PRNG (considered secure for cryptography) with a 55% success rate, you'll probably make the security news headlines for quite a while.

Wow thanks for the explanation behind this. I am trying to analyse the pattern and predict the next bit and it is not a perfect RNG, but somewhat solid PRNG. But it is not state of the art either. I think with a little computational power and proper implementation I cnould predict it with 60-70% if not more. If possible, can you point any resources where I can read more about this. I am not from a research background and more of a developer.

Old question, but I thought it's worth one practical answer. I happened to stumble upon it right after looking at a guide of how to build such neural network, demonstrating echo of python's randint as an example. Here is the final code without detailed explanation, still quite simple and useful in case the link goes offline:
from random import randint
from numpy import array
from numpy import argmax
from pandas import concat
from pandas import DataFrame
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense

# generate a sequence of random numbers in [0, 99]
def generate_sequence(length=25):
    return [randint(0, 99) for _ in range(length)]

# one hot encode sequence
def one_hot_encode(sequence, n_unique=100):
    encoding = list()
    for value in sequence:
        vector = [0 for _ in range(n_unique)]
        vector[value] = 1
        encoding.append(vector)
    return array(encoding)

# decode a one hot encoded string
def one_hot_decode(encoded_seq):
    return [argmax(vector) for vector in encoded_seq]

# generate data for the lstm
def generate_data():
    # generate sequence
    sequence = generate_sequence()
    # one hot encode
    encoded = one_hot_encode(sequence)
    # create lag inputs
    df = DataFrame(encoded)
    df = concat([df.shift(4), df.shift(3), df.shift(2), df.shift(1), df], axis=1)
    # remove non-viable rows
    values = df.values
    values = values[5:,:]
    # convert to 3d for input
    X = values.reshape(len(values), 5, 100)
    # drop last value from y
    y = encoded[4:-1,:]
    return X, y

# define model
model = Sequential()
model.add(LSTM(50, batch_input_shape=(5, 5, 100), stateful=True))
model.add(Dense(100, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc'])
# fit model
for i in range(2000):
    X, y = generate_data()
    model.fit(X, y, epochs=1, batch_size=5, verbose=2, shuffle=False)
    model.reset_states()
# evaluate model on new data
X, y = generate_data()
yhat = model.predict(X, batch_size=5)
print('Expected:  %s' % one_hot_decode(y))
print('Predicted: %s' % one_hot_decode(yhat))

I've just tried and it indeed works quite well! Took just a couple of minutes on my old slow netbook. Here's my very own output, different from the link above and you can see match isn't perfect, so I suppose exit criteria is a bit too permissive:
...
 - 0s - loss: 0.2545 - acc: 1.0000
Epoch 1/1
 - 0s - loss: 0.1845 - acc: 1.0000
Epoch 1/1
 - 0s - loss: 0.3113 - acc: 0.9500
Expected:  [14, 37, 0, 65, 30, 7, 11, 6, 16, 19, 68, 4, 25, 2, 79, 45, 95, 92, 32, 33]
Predicted: [14, 37, 0, 65, 30, 7, 11, 6, 16, 19, 68, 4, 25, 2, 95, 45, 95, 92, 32, 33]

This is not learning to predict the random sequence -- it is learning to echo it.  Concretely, the training samples, X, consists of 5 random integers, and the output, y, is the 4th integer of the 5. For example, if X = [15, 33, 44, 30, 3], y = 30. The LSTM is learning to echo the 4th sample.

Yes, good point. I still find it to be a very interesting practical example of LSTM usage. If you know how to learn something like Mersenne Twister from seed only given as an input, please post it here as I'd be really interested to see. Seems possible with enough samples, but I might be completely wrong.

Being a  complete newbie in machine learning, I did this experiment (using Scikit-learn ):

Generated a large number (N) of pseudo-random extractions, using python random.choices function to select N numbers out of 90.
Trained a MLP classifier with training data composed as follow:

ith sample : X <- lotteryResults[i:i+100], Y <- lotteryResults[i]

In practice, I aimed to a function that given N numbers, coud predict the next one.
Asked the trained classificator to predict the remaining numbers.

Results:

of course, the classificator obtained a winning score comparable with the one of  random guessing or of other techniques not based on neural networks (I compared results with several classifiers available in scikit-learn libraries )
however, if I generate the pseudo-random lottery extractions with a specific distribution function, then the numbers predicted by the neural network are roughly generated with the same distribution curve ( if you plot the occurrences of the random numbers and of the neural network predictions, you can see that that the two have the same trend, even if in the predicytions curve there are many spikes. So maybe the neural network is able to learn about pseudo-random number distributions ?
If I reduce the size of the training set under a certain limit, I see that the classifier starts to predict always the same few numbers, which are among the most frequent in the pseudo-random generation. Strangely enough ( or maybe not ) this behaviour seem to slightly increase the winning score.

Adding to what Demento said, the extent of randomness in the Random Number Generation Algorithm is the key issue. Following are some designs that can make the RNG weak:
Concealed Sequences
Suppose this is the previous few sequences of characters generated: (Just an example, for the practical use larger range, is used)
lwjVJA
Ls3Ajg
xpKr+A
XleXYg
9hyCzA
jeFuNg
JaZZoA

Initially, you can't observe any pattern in the generations but changing them to Base64 encoding and then to hex, we get the following:
9708D524
2ECDC08E
C692ABF8
5E579762
F61C82CC
8DE16E36
25A659A0

Now if we subtract each number form the previous one, we get this:
FF97C4EB6A
97C4EB6A
FF97C4EB6A
97C4EB6A
FF97C4EB6A
FF97C4EB6A

This indicates that the algorithm just adds 0x97C4EB6A to the previous value, truncates the result to a 32-bit number, and Base64-encodes the data.
The above is a basic example. Today's ML algorithms and systems are capable enough to learn and predict more complex patterns.
Time Dependency
Some RNG algorithms use time as the major input for generating random numbers, especially the ones created by developers themselves to be used within their application.
Whenever weak RNG algorithms is implemented that appear to be stochastic, they can be extrapolated forwards or backwards with perfect accuracy in case sufficient dataset is available.


text:
What is $USV^T$ in the context of word embeddings?
Here is an excerpt from the notes of the first lecture of the course CS224n: Natural Language Processing with Deep Learning.

3 SVD Based Methods
For this class of methods to find word embeddings (otherwise known
as word vectors), we first loop over a massive data set and accumulate word co-occurrence counts in some form of a matrix $X$ and then perform Singular Value Decomposition on $X$ to get a $USV^T$ decomposition. We then use the rows of $U$ as the word embeddings for all words in our dictionary. Let us discuss a few choices of $X$.

What does $USV^T$ refer to in this context?

$USV^T$ refers to the result of the singular value decomposition (SVD).
An $m \times n$ matrix $X$ can be written with the help of three matrices
$$X = USV^T,$$
where $U$ is an $m \times m$ unitary matrix, $S$ is a diagonal $m \times n$ matrix with real entries called singular values, and $V$ is a unitary $n \times n$ matrix. The $^T$ is the Hermitian transpose. SVD has applications, e.g, in optimization problems, principal component analysis, etc. Wikipedia has quite a long article.


text:
What is the difference between logic-based and rule-based AI?
I always thought rule-based was synonymous with logic-based AI. Logic has axioms and rules of inference, whereas rule-based AI has a knowledge base (essentially, axioms) and if-then rules to create new knowledge (essentially inference rules).
But in their famous article What is a Knowledge Representation?, Davis, Shrobe and Szolovits seem to imply that they are not:

Logic, rules, frames, and so on, embody a viewpoint on the kinds of things that are important in the world. Logic, for example, involves a (fairly minimal) commitment to viewing the world in terms of individual entities and relations between them. Rule-based systems view the world in terms of attribute-object-value triples and the rules of plausible inference that connect them, while frames have us thinking in terms of prototypical objects.

Is this only saying that rule-based are propositional, whereas logic-based is usually meant to mean predicate logic? Or is there more to it than this?

Rule-based systems cover a wide range of systems. Some make use of boolean if/then/else rules, others may use weighting or even probabilistic inference. Some operate on frames, some on java objects, some on propositions that can be formulated in predicate logic. An example of a popular rule system is Drools.
Some rule systems can be expressed as a subset of predicate logic. For example, SWRL is a W3C standard rule language that extends OWL Description Logic (DL) with horn rules. Both OWL-DL and SWRL are expressible in first-order predicate logic. However, not all rule languages are directly expressible in this way, as rule languages encompass such a range of semantics. Even in cases like Prolog, there are subtleties. Pure Prolog is a subset of FOL, but actual existing Prolog implementations are not FOL subsets (e.g. order of precedence matters).
The W3C Rules Interchange Format (RIF) working group has done a lot of work attempting to unify these different perspectives. I'm not sure the best place to start to see a summary, but you can see some useful material on the group charter page.


text:
How can I calculate the number of matrix additions, multiplications and divisions in AlexNet?
I'm a first-year student in machine learning and I really recently started to immerse myself.
I need to calculate number of:

matrix additions
matrix multiplications
matrix divisions

which are processed in the well known convolutional neural network - AlexNet.
I found some materials about it, but I'm really confused about where to start.
So, the overall structure might look like:

But, how can I calculate operations for each type distinctly?


text:
Can a neural network with linear activation functions produce a connection of linear functions?
From a lecture in machine learning, I know that a linear activation function can only produce a linear function, but I don't know if it can produce a connected linear function, like the one in the image below. This function consists of multiple concatenated lines.

I don't think so... that would make the output non-linear

A linear activation would not be able to separate the data like you have shown, no matter how many layers you throw into the network.
If we had multiple linearly activated layers, each feeding into each other, the neurons in the previous layer would calculate some weighted sum of the input and send it to the next layer as input, where the next layer's neurons would also calculate a weighted sum on that input, and, in turn, it fires based on another linear activation function.
No matter how many layers and neurons there are, if all are linear in nature, the final activation function of the last layer is also a linear function of the input of the first layer.
That means that any or all of these layers can be replaced by one layer. This completely loses the advantage of stacking layers because any multilayer network is equivalent to a single layer with linear activation, given that a combination of linear functions in a linear manner is still another linear function.
A good way to visualize this messing around with Tensorflow playground, which has a spiral data set similar to your data.
In contrast to the failure of linear activation functions for this data in the previous link, check out this much smaller network using a Tanh activation function which can separate the data within 100 or so iterations.
For further reference on activation functions, checkout this blog post.

how does this answer fit with the Universal Approximation Theorem? The OP's space in bounded, and the curve is continuous, so a single-hidden-layer network of sufficient width should be capable of learning the curve even with linear activation.

The condition on the activation function $\varphi$ for the UAT to hold is that $\varphi$ is bounded, continuous and nonconstant. With linear $\varphi$ it's either not bounded or a constant--it doesn't matter that the space is bounded.

Since I can't comment, there are a few caveats to previous answers.
For instance, if you knew beforehand what the expected boundary function for that variable was, then you could transform it first.
For instance, if you knew one feature was expected to be sinusoidal, you could transform your data (theta) using $f(x) = a*sin(\theta)$ first then expect the variable to be linear. At that point it would fit a linear function and solve for the amplitude instead. Of course, in large problems, this may be impractical, but still possible.
Likewise with quadratic data. I could add 1 new feature for every continuous variable that looks like $x^2$ and capture any quadratic dependence.
Another solution in your shown example would be if you knew the data was piece-wise, you could train 3 small, easily trained, NNs for instance with different boundaries. During each new training, you set new boundaries for your functions and see which produces the best results. In your case, this would be something close to $[-\infty,-0.5], [-0.5,0.5]$, and $[0.5,\infty]$.
You could argue that this is a single piece-wise neural net just like any traditional piece-wise function.
While this last approach seems very simplistic/impractical, this is actually done in practice. For instance in autonomous driving, a separate NN could be trained for object detection (human, lights, signs etc.), another for steering wheel positioning for different speeds,conditions, and turning radius, another for how to accelerate, etc. and then the system runs them all and combines their outputs into a full driving car. Each output is then its boundary of the puzzle on what region of "driving space" it handles.
So while I would say the simple answer is no, there are still ways around it if you have some prior knowledge.
As a response for the Universal Approximation Theorem, see this wiki, which says

One of the first versions of the theorem was proved by George Cybenko in 1989 for sigmoid activation functions. It was later shown that the class of deep neural networks is a universal approximator if and only if the activation function is not polynomial."