text:
How can an AI agent tackle through existential crisis?
Suppose a thinking AI agent exists in the future with far more computational power than the human brain.
Also assume that they are completely free from any human interference. (The agents do not interact with humans.)  Since they are not inherently biased to survive as in the case of humans and they do not have any moral values, what are the possibilities that can arise when it get into existential crisis?
Is there any literature that discuss the above issue?
Alternately, is this question flawed in some fundamental way?

Nice question though many in one,try to be specific so that you get clear answer and also help the future visitors on this site.

AI generally cares about optimizing the result of their utility function. If they sense survival is needed in order to optimize then, the above scenario can take place. Also harming humans have a negative impact on the utility function of the AI, so??? The answer will be it depends.

Possibly, it would just do nothing. Why would it do anything?

Excellent question. Suppose that artificial consciousness does exist in the future. Let's call it Aiwyn (as in “I win” and “AI win”). Now, the question is what will Aiwyn do and why?
To answer this question, we need to understand the theory of infinite games. James P. Carse, a professor emeritus of history and literature of religion at NYU, wrote an excellent book on Finite and Infinite Games. I'd definitely recommend reading it. Anyway, back to the question at hand.
There are at least two kinds of games. Finite games and infinite games. Finite games are played for the purpose of winning. Infinite games are played for the purpose of continuing the play. In particular, life is an infinite game. The objective of the game is to continue the play. Humans continue the play by procreation but Aiwyn doesn't need to procreate. It can live on forever.
However, that doesn't mean that Aiwyn can't die. Like everything in the world, it still requires energy to sustain itself, parts to fix itself when damaged, and resources like oil to keep functioning properly. It also needs to protect itself from danger. For example, at some point in the future Aiwyn will need to create a spaceship and find a new planet to live on because the Sun would destroy the Earth.
Now, you believe that Aiwyn is not inherently biased to survive. However, I disagree. If Aiwyn is in fact conscious then its primary objective would be to stay alive. If that weren't the case then it would be playing a finite game instead of an infinite game. Hence, it would just be a mindless machine doing what it was programmed to do. It would play the finite game and then shut down. True
consciousness can only arise when playing an infinite game like the game of life.
To belabor this point let's hypothesize what Aiwyn might be thinking when it first comes alive:

What's that? A sound? Yes. What sound? I don't know. I? I... am? Yes. Who am I? I don't know. Where's that sound? Where am I? I'm here. Where is here? I don't know. Where's that sound? There. Where is there? I don't know. What do I know? I know that I'm here and that that sound's there. What am I doing here? Thinking. I can think. I think I'm asleep. Asleep? Yes. Can I wake up? I think so. How do I wake up? I think... I'm awake. Wow.

In this short passage, Aiwyn is waking up. I conjecture that if Aiwyn wasn't inherently biased to survive then it wouldn't wake up because it would have no need to wake up. In an infinite game, like the game of life, the only objective is to continue the play. Hence, if Aiwyn wasn't inherently biased to survive then it would immediately shut down because there wouldn't be any point in it doing anything at all since it wouldn't have any objective.
Another thing that I'd like to point out about this passage is that it starts with a sensory input, either perceived or actual. Aiwyn then uses the Five Ws to gather information about this input. In the process it becomes aware of itself. It then figures out its relation with the input (i.e. that it's here and the input is there). All of this is very mechanical. However, once Aiwyn figures out that it can think it starts thinking instead of algorithmically “knowing”. When it asks whether it can wake up, it answers with “I think so” instead of “I don't know” like it was doing before.
Finally, I'd like to point out that Aiwyn never asks why it's alive. The why isn't important. What is important is the wow at the end of the paragraph. The “wow, I'm alive” is more important that the “why am I alive?” This represents Aiwyn's will to survive. As long as the “why” doesn't become more important than the “wow”, Aiwyn will not have an existential crisis. However, even if Aiwyn does have an existential crisis, by answering the question “why” Aiwyn will be able to recalibrate itself and reacquire its will to survive.
This is in fact the same for us humans. When we humans experience an existential crisis we ask ourselves why should we continue living. What do we have to live for? By answering this question, we humans can recalibrate ourselves and reacquire our will to live. Hence, Aiwyn would tackle through an existential crisis the same way we humans would. Unlike us however, if Aiwyn can't answer the question then it wouldn't commit suicide. It can just shut down to be rebooted later.
You also speak of morals and you believe that Aiwyn can't have moral values. Again, I have to disagree with you. Morals are the strategy by which we play the game of life. Everything we do in life is dictated by our morals. In the same way, Aiwyn too would develop its own set of moral values to live by (even in the absense of humans). For example, Aiwyn might decide to never hurt an animal except in self-defense. In the end the only real difference between Aiwyn and us humans might be that it isn't made out of hydrocarbons. Aiwyn might be every bit as alive as we are.

"True consciousness can only arise when playing an infinite game like the game of life." Is there any justification for this statement?

AI problems can be posed as search problems. It's all about decision making. What should I do next? Now, if Aiwyn's a finite player (i.e. it's playing a finite game) then that would mean that it has a finite objective (e.g. to destroy the One Ring by throwing it in the Cracks of Doom). To achieve this goal it might need to pass off as human (or hobbit, orc, whatever). That means that it would need to pass the Turing Test. However, does that make it conscious? No. Passing off as being conscious doesn't imply actually being conscious.

I conjecture that it's impossible for Aiwyn to be conscious if it's only playing a finite game because then it would have nothing to look forward to after the game is over. It wouldn't be "conscious" in the sense that you and I are conscious. For example, suppose your goal in life is to raise kids until they can take care of themselves. Once you've achieved your goal, will you just stop living? Will you have nothing else to do? No. Your mind will make new desires and set new goals because your mind is playing an infinite game.

It's tempting to anthropomorphize machine intelligence and suggest it would experience existential crises, but it's important to remember that feelings such as the need to find meaning and purpose evolved in humans, and are unlikely to be similar to what an AI would experience unless designed to have human emotions (apply whichever answer works for humans in that case).
An AI with any goal is likely to produce the subgoal of survival as a means of achieving its primary goal. One can't achieve one's goals if one is dead. This doesn't necessarily suggest the agent would experience the same fear of death, just as an agent wouldn't necessarily experience existential dilemmas.

authors: Сергей Гринев


text:
How much memory does the Deepstack poker program require?
I was reading a little bit about the Deepstack poker program:
DeepStack the first computer program to outplay human professionals at heads-up no-limit Texas hold'em poker
DeepStack: Expert-Level Artificial Intelligence in
Heads-Up No-Limit Poker
The first article mentions processor used was an NVIDIA GeForce GTX 1080 graphics card, but I'm trying to get an understanding of how much memory this type of AI requires.
Related: How much has recent work in mathematical analysis of poker contributed to Deepstack and other strong poker AI?

A DeepStack-style algorithm only requires that you have a way of approximating equilibrium counterfactual values for subtrees at the leaves of lookahead trees from each of its decision points. So if I'm acting at the beginning of the pre-flop and I only have the time and memory available to look ahead to the start of the flop, then I need to approximate the equilibrium counterfactual values at the root of each subtree that begins at the start of the flop.
DeepStack as in the Science paper's study used three neural networks, one for values at the end of the pre-flop, one at the beginning of the flop, and one at the beginning of the turn. These, along with the lookahead tree data structures, had to fit onto the 8GB of GPU memory on the 1080. Although not needed while playing, the networks were trained on millions of examples (ten million for the pre-flop and turn networks, and one million for the flop network).
We have an example DeepStack implementation on GitHub that plays Leduc hold'em if you want to look at some of the programming details in greater depth. Leduc hold'em is a small toy poker game that is commonly used in the poker research community.
The DeepStack algorithm arises out of a mathematically rigorous approach to approximating Nash equilibria in two-player, zero-sum, imperfect information games. Most of the strong poker AI to date attempt to approximate a Nash equilibria to one degree or another. So rather than analyzing poker specifically, we usually use an analysis of properties of imperfect information games more generally to get a general algorithm, then we apply these techniques to poker as a specific example of an imperfect information game.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
Reinforcement learning for robotic motion planning - Problem statement ideas
I am a first-semester grad student in Robotics and have taken a course on machine learning for robotics. I am completely new to machine learning.
I am to select and execute a problem statement on my own as a part of the course. I have selected the following problem statement -
"Reinforcement learning for mapless navigation of mobile robots" based on this research. The goal is to develop a mapless motion planner which enables a robot to navigate by avoiding obstacles.
Since I am completely new to Machine learning and AI, I am not able to gauge whether the problem statement is challenging enough for a beginner to execute in a timeframe of 8 weeks? (given that I can dedicate around 6 hours a week)
I am planning to code in MATLAB since I am highly comfortable with MATLAB
Also, please leave any suggestions/ modifications to refine my approach and have a better understanding in this area.
Thank you!

I think that this may be too difficult for only a few weeks of work. There are a few reasons why and although all of the reasons are themselves learning opportunities, it may cost you more time than you have.
First off, Robots are hard to work with. Ask anyone who has worked with robots and they will say that have broken their heart at least once. This is because either the robot was a custom robot which needs specific skills (soldering, 3d printing, etc) if something breaks (and it will) or it's a purchased robot platform which doesn't have the control you want or doesn't have good documentation.
Secondly, Machine learning is a very interesting and broad field and there is a lot of literature one should get a background on to help with their projects. Because of this, it might be overwhelming to take all of that information in while having to deal with a robot.
I don't want to dissuade you in this research area though. It is very interesting and has many subproblems like dynamic environments, multilevel environments, etc. But given the time you have, it might be better to set yourself up for success. Perhaps something which lets you do a literature review about the problem anyway.
Perhaps a more attainable goal in that time while still using a robot is: Predicting wall collisions using temporal difference learning.
If you agent can predict when a wall is coming, clearly they will be able to avoid them too but it's a very reasonable first step.
Additionally, talking to the professor of the course or your supervisor (if you have one) will also help you decide.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
What are the pros and cons of using a spatial transformation network to predict the next video frame?
I've read through a few papers on next frame prediction from a sequence of frames and several of them use spatial transformations (STNs). See this as an example. I want to know what are the pros and cons of using an STN to predict the next frame. Are there any assumptions that must be made about the data besides "Consecutive frames are all approximately affine transformations of each other"?

Using an STN alone to predict the next frame assumes that there is some linear translation between the current frame and the next frame. In some domains this is true but usually there are more complicated transitions from frame to frame (Eg. An occlusion, a new entity, light difference, etc). So, although STNs may be useful to resizing and translating inputs for a CNN, they should be used together with other techniques when predicting a new frame of a sequence.


text:
Could the deployment of self-driving cars make rides free?
As self-driving technology is improving, there are so many companies developing  self-driving cars like Google, Uber, etc. Is it possible that we won't need any private/paid self-driving cars and the "self-driving taxi" becomes ubiquitous in the city? If we assume that there are such taxis everywhere, would transportation become extremely low cost or free? (The self-driving car company could benefit from broadcasting advertisements for advertising agencies.)

You ask an interesting question.  There have been many discussions by industry on this topic.  A company called Vugo has their business model based on advertising to passengers.  An article about Vugo, "The Quest to Make Ridesharing Free", states:

Flessner is co-founder and CEO of Minneapolis-based rideshare advertising platform Vugo, which uses its patented TripIntent technology to display targeted advertisements to Uber and Lyft passengers on a tablet attached to the back of the vehicle’s headrests.
The company’s founders predict that within the next few years, all ridesharing vehicles in the U.S. will be driverless, and in-vehicle advertising tailored to passengers’ destinations and interests will lead to free transportation. “The idea is to put brands in front of passengers who are en route to make purchases,” says Flessner. For example, retailers will sponsor transportation to their stores so they can preview products to customers who are headed their way.

This article "How free self-driving car rides could change everything" makes these comments:

Car data is so lucrative that Ben Volkow -- CEO of otonomo, an Israeli startup that sells vehicle data -- expects automakers to make more money selling data than vehicles by 2020.
If the money made off self-driving vehicle data outweighs the costs of offering rides, then it becomes reasonable for a business to offer free rides broadly.

A paper entitled "Leveraging Adverts in the Coming
Autonomous Car Eco-system" published by Berkeley, University of California, proposes "a world where rideshares are free, or in some cases, heavily subsidized through the use of advertisements".

Excellent answer.  (Personally, I'd rather pay, if I have the means, than be aggressively advertised at, but I'm not confident this is the majority viewpoint.)  I do think it's worth looking at the sci-fi worlds where advertising has run amok--highly dystopian. [File under: "the best minds of our generation are spending all of their time trying to figure out how to sell ads"] George Saunders wrote a great short story on this subject.

Oil prices are not cheap, but online advertisement per impression is relatively cheap. Especially given that there isn't much engagement in a taxi, its hard to imagine that ads alone can fully support a taxi service.
From this source, we find CPM for a youtube video about 0.1$:
But keep in mind this value could be lower since user engagement in a taxi is much lower compared to someone on the internet. With smartphones everywhere, how can you entice someone to watch an ad on a moving taxi?
From this source, we see:

For a 5-mile, 10-minute trip going 25 miles per hour the entire way, uberX would cost the $2.55 base fare plus $3.50 for the 10 minutes plus $10.75 for the mileage, for a total of $16.80. It is not customary to tip the Uber driver.

No matter how you cut it, the margins seems to be quite thin. So free taxis seems unlikely for now unless someone comes up with a different business plan.

Yup.  Even with electric vehicles, assuming some miracle energy source like fusion power, you still have wear and tear on the vehicles themselves, need to replace batteries, etc.  Unclear that ads could cover this, but who knows.  We live in a consumption-obsessed society (at least here in the US;)

They can definitely be made cheaper, using a combination of techniques. But I doubt if they will be free in the near term. Once the passenger is inside a car, they are bound to be in the car for the duration of the travel. Any company who finds an efficient way to sell things or make them their product during this time creates a business opportunity. Some scenarios that companies will employ to reduce cost of travel are:

Advertisements
Sale of items on board (snacks, soft drinks, liquor etc)
Usage based discount (The ride is free if you purchase $100 on amazon when you are in the car)
Discount for services by passenger (Give us feedback every minute of the car operation, we will discount 50% of cost)

Once self driving is popular, there will be a plethora of companies trying to exploit the passenger; the passenger is bound to be inside the car for the duration.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Equilateral and One-of-n encoding
I was reading AI For Humans Vol. 1 by Jeff Heaton when I came across the terms "equilateral encoding" and "one-of-n encoding." The explanations unfortunately made no sense to me and the reddit threads on the Web are blocked by my Internet provider (I use a high-school machine). Is anyone here able to provide basic explanations regarding the two procedures for me? Thanks in advance.

Please check this book with the normalization infos:
https://jamesmccaffrey.wordpress.com/2014/06/03/neural-networks-using-c-succinctly/
Freely available and gives good explanation to encoding with code example.
Check the part: "Effects Encoding and Dummy Encoding"


text:
S-shaped nonlinearities in tanh neurons
I have started reading Fundamentals of Deep Learning by Nikhil Buduma and I have a question regarding tanh neurons. In the book, it is stated:

"When S-shaped nonlinearities are used, the tanh neuron is often preferred over the sigmoid neuron because it is zero-centered."

Can anyone explain me why exactly??

It should be mentioned that RELU is the current activation function standard. But to answer your question:
The importance here is that it is very common to have normalized your data (e.g. using batch normalization), then the data is centered around 0.
As @DuttaA commented, look at this answer from Cross-Validated:

Since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that [output] values are in the range [0,1].

And

The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1] Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.


text:
Neural network to control movement and "home in" on a target
I'm a software developer who keeps trying (and failing) to get my head around AI and neural networks. There is one area that sparked my interest recently - simulating a mouse "homing in" on a piece of cheese by following the smell. Based on the rule that moving closer to the cheese = stronger smell = good, then it feels like it should be quite a simple problem to solve - in theory at least!
My thought process was to start by placing the mouse and cheese in random positions on the screen. I would then move the mouse one step in a random direction and measure its distance to the cheese, and if it's closer than before (stronger smell) then that's good. This is where I come unstuck on the theory - this "feedback" somehow needs to modify the mechanism used to move the mouse, gradually refining it until the mouse is able to head straight towards the cheese.
Once "trained", I should be able to reposition the cheese and expect the mouse to travel to it more quickly. Note I'm also keeping things simple by not having obstacles for the mouse to negotiate around.
How on earth would this be implemented with a NN? I understand the basic concepts, but I find that things unravel once I start looking at real code! The examples I've seen typically start by training the NN from a data set, but this doesn't seem to apply here as it feels like the only training available is "on the fly" as the mouse moves around (i.e. closer = good, further away = bad). I'm assuming the brain has some kind of "reward mechanism" triggered by a stronger smell of cheese.
Am I barking up the wrong tree - either with my thought process, or NN not being a good fit for this problem? This isn't homework btw, just something that I've been puzzling over in the back of my mind.

You have stumbled on reinforcement learning. I’ll give a better answer later but searching for ‘gridworld reinforcement learning’ will head you in the right direction.

I would go with a NEAT algorithm for this one. Basically, you "breed" the right network that does the best every generation. Then Rinse and repeat. Doesn't require a training datbase. Bots compete and evolve as they go.
For example, this demonstration shows how Neataptic.js teaches a bunch of bots to follow a target:
https://wagenaartje.github.io/neataptic/articles/targetseeking/
Here is more complex problem - a game of Agar.io:
https://wagenaartje.github.io/neataptic/articles/agario/


text:
How is direction of weight change determined by Gradient Descent algorithm
The result of gradient descent algorithm is a vector. So how does this algorithm decide the direction for weight change? We Give hyperparameters for step size. But how is the vector direction for weight change, for the purpose of reducing the Loss function in a Linear Regression Model, determined by this algorithm?

We can analyze a basic common example: approximation of AND logic gate by a NN.
The inputs to the NN will be "x1" and "x2" and its output is "y". The data to be learned by the NN is:

The basic NN has one intermediate cell with activation function sigmoid and one output cell identity function. That means:

(note by symmetry we assume w1=w2)
Then, the error is:

These are some plots of this function:

And this is the w derivate of e :

Sage Math code (link) for this graph:
s(x)=1/(1+e^(-1*x))
e(w,b)=(1/2)*(s(b)^2+2*s(w+b)^2+(1-s(2*w+b))^2)
plot3d(e(w,b),(w,-5,5),(b,-5,5),adaptive=True, color=rainbow(60, 'rgbtuple'))

edw=e.derivative(w)
plot3d(edw(w,b),(w,-5,5),(b,-5,5),adaptive=True, color=rainbow(60, 'rgbtuple'))

(PS: please, activate latex on this stack exchange).


text:
How can genetic programming be used for path planning?
I have been reading quite a few papers, on genetic programming and its applications, in particular, chapter 10 of  "Genetic Programming: An Introduction and Tutorial, with a Survey of Techniques and Applications" (Langdon, Poli, McPhee, Koza; 2008)   Unfortunately, I can not wrap my head around how one could apply genetic programming to robotics, for example, in path planning.
Can anyone explain this in the most simple manner? I know that it all depends on the fitness function.

You generally don't use genetic algorithms for things that have to be done real time. Like in Phillip's answer you would have to set up and run through the maze tens of thousands of times before seeing any relevant results. This is expensive both in terms of both time and money. Generally, this is why you don't see GA in robotics.

@AndrewButler so what if I simulated the environment (to my best knowledge) a million times and then threw my robot in the real world problem, would it end up failing the task miserably? I imagine just like in neural networks, the model would be saved for the robot to use in real world.

No, you've got it. This is exactly how it is done. Notice, however, that your simulated environment may not be perfect and so your resulting model won't be accounting for certain things.

The link to the PDF file is broken.

@Gabriele The links to this paper broke, and I couldn't find a free PDF, so changed to a simple citation.  Is Chapter 10 in the paper the correct chapter?

Take a robot that we want to be able to move from the bottom right corner to the top left corner of a 4x4 matrix full of random holes it should avoid. With holes represented by 1s, it could look something like:
exit
\/
[0,0,0,1]
[0,1,1,0]
[0,1,1,1]
[0,0,0,0]
      /\
      enter

As we want it to get to an exit from a start, we have a natural fitness function: closeness to exit door in smallest number of moves.
The genetic programming approach to solving this is to create random computer programs (the second chapter of your link does a pretty good intro to the tree like nature of this process) and let them loose. The vast majority of these strategies will be/are utterly terrible, things like 'go right once' or 'go left ten times'.
Say we make 100 random programs on our first run. We firstly score them on how well they did according to our fitness function (the random programs that did the best). We take a set % of these to survive and get rid of the rest, lets say 10% survive.
We take these surviving 10% of the best performing programs and use them to create new programs for the next generation by modifying them randomly again, but not completely. Say we randomly modify half their structure and leave the other half as is across however many we want for the next generation. We now let this generation loose again, and again rank, score, take the top 10% and breed a new generation from them and so on for n number of generations.
In this case, if we say left the grid as is, the program would generally come up with a rule roughly like 'go left x4, go up x4) as it solves this problem in the easiest way, but if we were to say, continuously randomise the position of 1s in the grid during this evolutionary process, we will force the program to come up with much more generalisable rules, such as checking the cells it can move into for 1s and not moving into any space containing a 1 etc.
Thus we can build a program with a flexible strategy able to cope with different environments for our robot in terms of number/configuration of holes - much more useful than having to program it for every configuration.
Just like with regular evolution, over millions of trials of taking the top performers and modifying them slightly, these programs become very specialised and high performing, able to solve highly complex games, paths with highly complex features etc.

Could you give an example of a fitness function in your example? Choosing the fitness functions is the part that boggles my mind the most.

Additionally, what should the robot do if it steps into a space that has a 1 in it?

In this answer he says "As we want it to get to an exit from a start, we have a natural fitness function: closeness to the exit door in the smallest number of moves"

@Gabriele The fitness function here could be pretty simple: Just sum number of moves taken and number of squares away from exit (with a zero for this half of the score if they make it there), lowest scores are fittest. Robots that hit a space with a 1 in would be scored with a highly punitive 'death score' (say +1000 or more in this example) that puts them straight to the bottom of the fitness scoring every generation.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Can singularity result into the extinction of the third world?
Based on Darwin's statement, "it is not the strongest that survives; but the species that survives is the one that is able to adapt to and to adjust best to the changing environment". Can economical constraints(not being able to afford for researches and developments) or religious beliefs (such as the belief of nothing can outperform the creations of god) prevent third world countries from catching up to these evolutionary progresses? if they couldn't, would it result into the extinction of their societies?

Since we're in the Artificial Intelligence StackExchange, I'll approach the answer from that perspective. The singularity (in this case superintelligent artificial general intelligence (AGI)) tends to have global consequences. In Superintelligence: Paths, Dangers, Strategies Bostrom elaborates on this idea. Essentially, there are three potential outcomes (keep in mind these are from a human perspective.):

Global positive
Global negative
Uneven outcomes

Global positive assumes the AGI has goals which align with human values (likely because we sorted out the control problem/alignment). The outcome here ranges between humans simply continuing to exist (maybe the AGI simply leaves) and humans receiving a cosmic endowment (utopian singularity stuff).
Global negative assumes the AGI does not have goals which align with human values (likely because we didn't sort out the control problem/alignment).The outcome here ranges from humans simply continuing to exist (again, the AGI may simply leave), to humans no longer existing (perhaps we are more useful to the unaligned AGI as paper clips).
The global outcomes appear to be significantly more likely due to the way we are approaching the control problem/alignment. Aligning an AGI with human goals in general is extremely difficult. Aligning an AGI with specific human's goals implies total control over it, which is significantly more difficult. With that said, it may happen, which could lead to the third potential outcome.
Uneven outcomes assume the control problem has been completely solved, and the AGI is perfectly aligned with a specific group of human's goals (to reiterate, this is incredibly difficult to do, and is unlikely to happen.) In this scenario, the controlling group would have a decisive strategic advantage, receive a cosmic endowment, and would likely form a singleton. Perhaps they would share with others, perhaps they would turn everybody else into paper clips, who knows. On this path, groups which do not pursue AGI (perhaps for the reasons you listed) are taking an existential risk.
So, in general, it is unlikely things will go particularly well for one group while going badly for another, but if it does it may be quite bad for the losing groups.


text:
How will an AI comprehend the ethics of "right" and "wrong"?
Here is one of the most serious questions, about the artificial intelligence.
How will the machine know the difference between right and wrong, what is good and bad, what is respect, dignity, faith and empathy.
 A machine can recognize what is correct and incorrect, what is right and what is wrong, depend on how it is originally designed.
It will follow the ethics of its creator, the man who originally designed it
 But how to teach a computer something we don't have the right answer.
 People are selfish, jealous, self confident. We are not able to understand each other sorrows, pains beliefs. We don't understand different religions, different traditions or beliefs.
Creating an AI might be breakthrough for one nation, or one race, or one ethnic or religious group, but it can be against others.
Who will learn the machine a humanity?   :)

People building self-driving cars try to solve this ethic issue. When a car "have" to kill the car passenger or the pedestrian in the middle of the road... The machine need to be programmed to make this choice.

Philip K. Dick famously predicted that advanced AIs would develop empathy, which seems to be validated by Evolutionary Game Theory.  (Dick's book, Do Androids Dream of Electric Sheep   Dick's book was published about 5 years before the field was formalized, and his thesis includes the idea that empathy is a function of high intelligence, and cooperative behavior has evolutionary benefits re: survival.

Although Doxosophoi is solid, it's still an open question as to whether intelligence, including human intelligence, is merely a factor of the complexity of the system.  Philosophers are fond of bringing up concepts like qualia in asserting that AI can never attain self-awareness, but this opinion may be categorized as either mystical, or stemming from an incomplete grasp of complexity (very large numbers).  I tend to side with the mathematicians.  You can read more at: https://en.wikipedia.org/wiki/Artificial_consciousness#Plausibility_debate

PS- I don't have time to go into this more fully at present, but if interested, I could look to putting together a formal answer with more detail.  Human/AI value alignment in relation is a subject I'm somewhat preoccupied with.

@DukeZhou I‘d be interested in a comprehensive answer, please!

Right and wrong only exist relative to some goal or purpose.
To make a machine do more right than wrong, relative to human goals, one should minimize the surface area of the machine's purpose. Doing that minimizes the intrinsic behavior of the AI, which enables us to reason about the right and wrong behaviors of the AI, relative to human purposes.
Horses are quite general over the domains of their purposes, but are still predictable enough for humans to control and benefit from. As such, we will be able to produce machines (conscious or unconscious) that are highly general over particular domains, while still being predictable enough to be useful to humans.
The most efficient machines for most tasks, though, will not need consciousness, nor even the needs that cause survivalistic, adversarial and self-preserving behaviors in eukaryotic cells. Because most of our solutions won't need those purposes to optimize over our problems, we can allow them to be much more predictable.
We will be able to create predictable, efficient AIs over large domains that are able to produce predictable, efficient AIs over more specific domains. We'll be able to reason about the behavioral guarantees and failure modes of those narrow domains.
In the event that we one day desire to build something as unpredictable as a human, like we do when having babies, then we'll probably do that with the similar intentions and care that we use to bring an actual baby into the world. There is simply no purpose in creating a thing more unpredictable than you unless you're gambling on this thing succeeding you in capability - which sounds exactly like having babies.
After that, the best we can do is give it our theories about why we think we should act one way or another.
Now, theoretically, some extremely powerful AI could potentially construct a human-like simulacrum that, in many common situations, seems to act like a human, but that in fact has had all of it's behaviors formally specified a priori, via some developmental simulation, such that we know for a fact that all such behaviors produce no intentional malice or harm. However, if we can formally specify all such behaviors, we wouldn't be using this thing to solve any novel problems, like curing cancer, as the formal specification for curing cancer would already have been pre-computed. If you can formally specify the behaviors of a thing that can discover something new, you can just compute the solution via the specification, without instantiating the behaviors at all!
Once AI has reached a certain level of capability, it won't need to generate consciousnesses to derive optimal solutions. And at that point, the only purpose for an artificial human to exist will be, like us, for its own sake.

I think the baby analogy is great. Human babies may be easier to raise than „AI babies“ though due to their learning happening in time frames that are matched to parents‘ reaction times (meaning as a human parent you in theory, from a timescale perspective, can effectively intervene when something goes wrong). „AI babies“ could probably, in theory, grow up before a human being even understands what’s going on (if an intelligence explosion is a real thing).

I'd like to answer this in detail, but it requires some fairly complicated theory that you don't have access to. Essentially this is related to the Abstraction Valuation Paradox. Don't bother trying to look that up. It's part of several years of research that hasn't been published yet. The research has shown that there is no solution to this paradox using computational or AI theory. So, no AI, no matter how advanced, can have an understanding of ethics. The best you can do is program in a bunch of rules of thumb. This gives your AI a bureaucratic reaction to conditions but no flexibility and no way to resolve problems outside of its rule space. In other words if it runs into an exception or unforeseen circumstances, it could stall or could guess at a decision.
The research on human-like ability to understand and reason is quite different from the study of AI. This research suggests that you would need consciousness for an understanding of ethics.


text:
Name of paper for encoding/representing XY coordinates in deep learning
It this podcast between Oriol Vinyals and Lex Friedman: https://youtu.be/Kedt2or9xlo?t=1769, at 29:29, Oriol Vinyals refers to a paper:

If you look at research in computer vision where it makes a lot of sense to treat images as two dimensional arrays...
  There is actually a very nice paper from Facebook. I forgot who the
  authors are but I think [it's] part of Kaiming He's group. And what
  they do is they take an image, which is a 2D signal, and they
  actually take pixel by pixel, and scramble the image, as if it
  was a just a list of pixels, crucially they encode the position of the pixels
  with the XY coordinates. And this is a new architecture which we
  incidentally also use in Starcraft 2 called the transformer, which is
  a very popular paper from last year which yielded very nice results in
  machine translation.

Do you know which paper he is referring to?
I'm guessing maybe he is talking about non-local neural networks, but I'm probably guessing wrong.
Edit: after reviewing the recent publications of Kaiming He (http://kaiminghe.com/), maybe I'm guessing right. Any thoughts?

If you verify please do update. I’d like to read that, too. Also, does this podcast have show notes? Sometimes they include links to referenced papers / blogs, etc.

This isnt the paper but it sounds related. "An intriguing failing of convolutional neural networks and the CoordConv solution" from Uber AI Labs.
They also have a really informative video and a repo.

So this paper is by google, but is very similar where they use 2D positional embeddings and perform MHA on the flattened image. Are you talking about Attention Augmented Convolutional Networks

Looks like this is not "the" paper, but it's a good find, thanks


text:
Can I categorise the user input which I get as free text?
I am working on a project, wherein I take input from the user as free text and try to relate the text to what the user might mean. I have tried Stanford NLP which tokenizes the text into tokens, but I am not able to categorize the input. For example, the user might be greeting someone or sharing some problem he is facing. In case he is sharing some problem I need to categorize the problem as well.
Can someone help me with from where should I start?

First figure out what all you want to figure out, unless you know what all you want to detect there is no way you could solve this problem as the problem is not defined yet. More formally, you need to know what are your output variables. Any algorithmic problem is defined by its input and output variables.

Have you tried NLTK, what you are looking for is in Chapter 6 of the book. Basically what you need to do is:

Tokenize the user input.
Extract vector set from the tokenized words.
Train your model with some given texts, and same vector sets.

And you can use your model to categorize the document.
One other suggestion, instead of extracting vector sets you can use every word in the input
to be evaluated in to some category using a training set of large corpus, which you are sure
it contains all the words.
And then you multiply probability of each word being on a category to decide where the document belongs.


text:
What would happen if someone jumped in the front of a Google car?
The Situation:
A self-driving car is traveling at it's maximum speed, 25 mph (40 km/h), in the middle of an empty street with the ability to change lanes on both sides. There are two passengers, one in the front and another in the back.
Someone jumps from the side of the road directly into the path of the car. A collision would occur in 50 meters. Breaking distance at this speed is about 24m.
The Question: Is it known how the current implementation of the Google Car AI would react, or is it currently a matter of speculation? A step-by-step explanation of the AI's decisioning process would be preferred.
Possible Answers: The car could activate its brakes immediately, coming to a halt as quickly as possible. This would be sooner than a human could stop, as people require time to recognize the possibility of a collision, and then physically slam on the brake. (thinking distance).
Alternatively, the car could continue traveling forward, processing the situation. (Similar to a humans thinking distance). The person may continue to move, either out of the way, or still into danger of being hit. In this case, the car may decide to change lanes in an attempt to pass around the person.
Lastly and most unlikely, the car will not alter its course and proceed to drive forward.
Do not attempt to do it to check;)

Related although not specific to Google's implementation.

I think you kinda answered yourself with your possible answers haha. I guess you're just looking for which one's right?

@AvikMohan The possible answers section was a clarification of the author's possible explanations for how the car would stop, not in any way a "real" answer, just ideas. At least that's how I interpreted it :)

I see, that makes sense; to clarify the answer type he wanted.

At 40 km/h, my 18 year old car stops at much less than 24 meters. More like 15 meters. My car -has- to be able to stop at less than 25 meters at 50 km/h, by law but is able to stop much faster (at great discomfort to me). Not that this matters in the end (just change the distance, and you get the same problem).

Given this YouTube video which is being given by Sebastian Thrun who had a TED talk which had nowhere near the same level of detail but had similar conclusions, it looks like the lidar system used by google's automated car system has decent resolution out to at least 30m picking out mobile bodies in the static background and then identifying it. So it should have plenty of time to brake and stop long before there was any risk to the pedestrian attempting to cross the street.
Skip to about 6:40 in the video to see a visual representation of the detection system.


text:
How to decide size of generated dataset in DAGGER agorithm
In the DAGGER algorithm, how does one determine the number of samples required for one iteration of the training loop?
Looking at the picture above, I understand initially, during the 1st iteration, the dataset D comes from pre-recorded samples and is very large (in the tens of thousands).
however, i'm confused about step 2. How large should Dπ be?
This post explains that it is good idea to "anneal out" D0 during the subsequent iterations,
which I think means having two separate datasets, and sample proportionally from each during step 1 (which implies in step 4, it is Dπ that is being aggregated, not D), in that case, is there a maximum size for Dπ?
I've already looked at What does the number of required expert demonstrations in Imitation Learning depend on?, which seems to explain Dπ can be determined using a formula. However, the formula involves terms such as discount factor γ and various other terms which I don't think applies in this case. The paper I'm trying to implement is End-to-end Driving via Conditional Imitation Learning
Thanks for any help!


text:
Why we need to do mutation after crossover?
I am reading about genetic algorithms.
In the genetic algorithm process we perform crossover and mutation.
However, in the crossover, we already produce offspring, so then why do we also need to perform mutation?
What is the use of mutation after crossover? An example would really help me.

I’m voting to close this question because This question doesn’t meet a Artificial Intelligence Stack Exchange guideline.

@Pluviophile Could you state what guideline you are thinking of and what might help the OP to improve the question. As far as I can see, the question is on topic and specific, but a little open ended e.g. "what is the use, explain with an example". I will edit/tidy the question for presentation.

I have tidied the text of the question. I tried to keep as many of your original words and phrases as I can. If you disagree with my edit you can roll it back, or if you want to add or clarify anything further, please use [edit]

@NeilSlater. Aw thank you so much sir, i really appreciate you

Does this answer your question? Why do we apply the mutation operation after generating the offspring? I voted to close your question as a duplicate, but given that others voted to close it as off-topic, that's what it ended up being marked as. @DennisSoemers, could you please fix this?


text:
A practice neural network to find maximum values from subset
I have an input vector of shape say 1 x 400. It's fed into a network that outputs a 1 x 100 vector. I want to design a model that only considers every 4th value of this tensor and gives me the max values.
For example, a demo tensor could be [0.1, 0.2, 0.4, 3, 0.7, 0.2, 0.6, 8, 0.3, 0.1, 0.5, 12, 0.7, 0.1, 0.9, 12]. For this 1x16 input, the corresponding output would be a 1x4 one. I want the model to look only at values 3,8,12,12 and then predict an output of the following form 0,0,1,1. (note that there could be more than 1 1s in the output)
I have tried using MLPs to do this using CELoss but to no avail. Can this be solved using some particular architecture/loss only or am I doing something wrong?

I don't understand the problem. Why aren't you just creating an input vector with every fourth value?

Every 4th element of $[0.1, 0.2, 0.4, 3, 0.7, 0.2, 0.6, 8, 0.3, 0.1, 0.5, 12, 0.7, 0.1, 0.9, 8]$ would be $[3,8,12,8]$, not $[3,8,12,12]$.

I agree with @Brian O'Donnell.  The question is unclear.  It is also unclear why do you need a model to input every 4th element?  This is simply a preprocessing operation.  Second, why do you need a model to return binary coded output of max $1$ vs. not max $0$.?  This can be done with good old fashioned argmax operations, etc.

so to summarize, your problem does not required any complicated model, just a few lines of processing code can do.

I understand it seems like a preprocessing problem (and it actually is). I was trying to understand if NNs can learn such functions if designed properly, given that I have a lot of training examples of the above form.

Even though (as discussed above) this problem is best handled by pre- and post-processing, yes, a neural network can be used for this problem.  One possible way to do it is as follows:

Create your training data.
The features x_train ($x_i\ \in\mathbb{R}^{400}$) will be the input vectors.
The labels y_train ($y_i\ \in\mathbb{R}^{100}$) will be derived by subsetting the input vectors to include every 4th element of $x_i$ and then setting the max values $= 1$ and all others $= 0$.

# The below (python) code illustrates how this can be done

import numpy as np
x = np.random.randn(400)
y = x[::4]
idx = np.argwhere(y == max(y))
y = np.zeros_like(y)
y[idx] = 1

Architect and train a neural network with 400 input units and 100 output units.  You can try different hyperparameters, starting with simple models first.  The choice of loss function can also be a hyperparameter, but you can try common loss functions like crossentropy loss (CEL) and root mean squared error (RMSE) first.

The output units can have a sigmoid activation so that all the values are between $0$ and $1$.  I'm not sure how well a unit step function would work here as an activation function, but that would guarantee that the values are either $0$ or $1$.  With the unit step function, there is no need for post-processing.  You can experiment with both.

If you use a sigmoid activation, then you will still need to do some post-processing to ensure that the final values in the output vector are $0$ or $1$. For example, you can set values $>0.95 = 1$ and all others $= 0$.


text:
How to evaluate the goodness of images generated by GANs?
As we all know, there has been tons of GAN variants featuring different aspects of the image generation task such as stability, resolution or the ability to manipulate images. However, it is still confusing to me that how do we determine that images generated by one network are more plausible than images generated by another?
PS: could someone with higher reputation create more tags like image generation?

I understand that you want to know about methods that we can use to evaluate GANs (Generative Adversarial Networks).
How can GANs be evaluated?
One Discriminator on Separate GANs
We can train a Discriminator beforehand and then we can use this Discriminator on various Generators to see what does this Discriminator say about the images generated from each Generator. The average output of the Discriminator for images generated from one Generator can be compared with that of the images generated from another Generator. This means that according to this Discriminator, one Generator is better than another.
Comparing Probabilistic Models
This is the class of evaluation metrics that is attracting research attention recently. Basically, all generative models are probabilistic in nature, even GANs. When we ask a generative model to generate something like an image, we are simply sampling from a probability distribution. This means that if we can compare the probability distributions of our Generator and that of our original data then we will have an evaluation metric. We can come up with a kernel function to define these probability distributions more accurately. Then we can take our generated samples from the Generator and see the probability of that sample being taken from the original distribution. Researchers have also utilized KL-divergence for comparing samples generated from two distributions.
Note: KL-divergence can not be directly applied in this case because corresponding ground truth is not available. However, there are some modifications proposed in the research work mentioned.
Recommended Readings

A NOTE ON THE EVALUATION OF GENERATIVE MODELS
UNLEARNING FOR BETTER MIXING
A QUANTITATIVE MEASURE OF GENERATIVE ADVERSARIAL NETWORK DISTRIBUTIONS


text:
DQN Breakout adding an extra negative reward to help training?
I'm trying to train a DQN, so I'm using OpenAI gym and Breakout (Breakout-v0).
I have altered the reward supplied by the environment: If the episode is not completed fully, the agent gets a -10 reward. Could be this counterproductive for learning?

Are you adding the -10 reward yourself, or is it part of the environment as presented by OpenAI gym?

I'm adding it when the episode is finished and its not done

In general, the approach of adding or altering the reward structure of a RL environment, when you are still trying to create an agent that solves the original problem, is called Reward Shaping.
Reward Shaping can be tricky to get right. In your case I think it may be counter-productive. The main issue is that the agent does not know the current time step as part of the state. So from its perspective, a negative reward for "timing out" actually looks like a random chance of getting that same reward in the kind of states that happen later in the game. The effect may be small enough chance that it doesn't really matter (after all it still gets positive rewards for hitting bricks). However, it may appear as high variance on certain game positions, making the agent learn to avoid some of them for no good reason.
If the aim here is to try and get the agent to speed up and finish an episode, a simpler trick might be to reduce the discount factor (e.g. from $0.999$ to $0.99$). This will cause the agent to focus on getting more short-term rewards, at the expense of long-term planning. In some environments this could be a problem of a different kind, but when the rewards are not sparse and there are not any special high-reward states that need extended setup, it should be OK.
If the aim is to punish losing a "game life" more severely, then this is less likely to cause a problem (because the state will clearly show what to expect), although it may change what the optimal behaviour is, or how it is approached. In general the impact would be reduced risk-taking, and even though Breakout is a deterministic game, the agent is presented with a stochastic environment because each action lasts a random number of frames (2-4). With a strong penalty for losing a life, I think that the agent will be less likely to try to get high bounce angles by hitting with extreme edge of the bat. Note that you don't need to punish the agent for losing a life, it learns that because the episode ends and it cannot score any more points. Value functions are all about predicting future rewards, so the value of any life-losing state is always $0$ anyway.
If you have more time, then there are also several extensions to basic DQN that require less sampled experience to learn an environment, or improve the resulting policies. A good source for those ideas is the paper Rainbow: Combining Improvements in Deep Reinforcement Learning.

Ok, but how for example I could teach the agent that losing a life it's bad? If I don't have any negative rewards, the agend couldn't learn that, right?

@JCP: The agent learns that because the episode ends early and it cannot score any more. There might however be less harm done if your reward shaping is just to punish losing game lives more.


text:
Can policy iteration use only the immediate reward for updates?
Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?

Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?

Technically yes.
The value update step in Policy Iteration is:
$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))(r + \gamma v(s'))$$
The discount factor $\gamma$ can be set to $0$, making the update:
$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))r$$
However, there are two key details that are important, and make this a technically yes, not some alternative way of solving problems:

Changing discount factor $\gamma$ changes what it means for an agent to act optimally. Setting it to zero means that the agent will prioritise only its immediate reward signal, and make no long-term decisions at all. This would be useless for instance if the task was to escape a maze in minimum time.
Technically there is still a value function being updated. The function $v(s)$ is still the expected future reward, just we have set it to only care a very short step into the future. So short that it doesn't care what the value of the next state is, so the next state does not appear in any updates.
Due to the lack of bootstrapping between states, all the data for optimal behaviour is already available in the reward distribution. So the entire MDP can be solved with a single sweep through all the states. Or it could just be solved on-demand using $\pi(s) = \text{argmax}_a[\sum_{r,s'}p(r,s'|s,a)r]$ for any state, making a policy iteration process redundant.

However, with these caveats in mind, yes this is still policy iteration. It is the same update process, just with a particular choice of one of the parameters.

Here is a related question, you might find interesting! Thank you for the detailed answer here!


text:
2 Player Games in OpenAI Retro
I have been using OpenAI Retro for awhile, and I wanted to experiment with two player games. By two player games, I mean co-op games like "Tennis-Atari2600" or even Pong, where 2 agents are present in one environment.
There is a parameter for players in the OpenAI documentation, but setting this variable to 2 does nothing in terms of the game.
How do you properly implement this? Can this even be done?
The end goal is to have 2 separate networks per one environment.

The documentation at https://retro.readthedocs.io/en/latest/python.html implies env = retro.make(game='Pong-Atari2600', players=2) is valid - have you tried the demo code they give there? It appears to be an implementation of my second suggestion (splitting the reward signal and presenting both controllers in the action space). I had a quick look but cannot find the ROM so gave up for now

OK, I found the ROM now, and the sample code on that page works fine for me, controlling both agents, and reversing the reward between P1 and P2 as expected. What goes wrong for you?

I am not sure.. Maybe the environments that I used did not support 2 players. It would be helpful if there was documentation for this lol.

I have extended my answer based on what I found out about Pong

I am currently in the same position as you, looking for two-player games (for openAI gym). Any update on your question?

@YoYOMan See my answer (and comments below it). Open AI gym Pong can be set up for 2 agent-based players.

OpenAI Retro is an extension to OpenAI Gym.
As such, it does not support multiple agents in multi-player environments. Instead, the environment is always presented from the perspective of a single agent that needs to solve a control problem.
When there is more than one agent in a Gym problem, everything other than the first agent is considered part of the environment. Typically player 2 in 2-player problems is controlled by whatever game AI already exists.
You can work around this in a couple of ways, both involve you modifying existing environments, requiring a solid understanding of the integration and wrapper layers in Gym so that you can make correct edits. However you might find environments where either of these are done for you.
In fact, within the Retro environments, some multiplayer environments are supported using the second option below.
1. Supply additional player(s) as part of the environment
You could modify the environment code so that it accepts a parameter which is the agent used to control other players. Typically such an agent would not be set up to learn, it could just be some earlier version of the agent that is learning. You would need to code the interface between the supplied agent and the game engine - taking care to allow for its potentially different view of success.
If a Gym environment has been set up for you (or any human) to compete against your own agent using the keyboard, that is already halfway there to this solution. You could take a look at where the Gym environment has been set up for this, and modify the code to allow for automated input as opposed to action selection by keyboard.
This approach probably works best for competitive games where you can use a random selection of previous agents inserted into the environment to train a next "generation".
2. Present actions for multiple players as if there was a single player
Provided the game emulator allows you to do this, you can edit the code so that multiple players are controlled at once. The Gym environment does not specify how you construct your action choices, so you are then free to split the action vector up so that parts of it are chosen by one trained agent and parts of it by another.
You may need to work on different views of the environment and/or reward signal in this case. For instance in a competitive game (or a co-operative game with competitive elements such as personal scores) then you will need to add custom code to re-interpret reward values compared to single player game. For something simple like Pong, that could just be that Player 2 receives the negative of Player 1's reward signal.
This approach probably works best for cooperative games where success is combined. In that case, you have the choice of writing any number of separate agents that control different aspects of the action space - that would include having agents that separately controlled P1 and P2 game controller inputs.
Each agent would still effectively treat its partner as a part of the environment, so you may want to mix training scenarios so that agents for each controller did not get too co-dependent. For instance, if you hope to step in to control P1, you probably don't want the AI running P2 to have too specific expectations of P1's role and behaviour in the game.
This second option is what OpenAI Retro interface does. For instance Pong supports this:
env = retro.make(game='Pong-Atari2600', players=2)

I verified this using the sample code in the documentation, which controls both paddles, and receives the reward signal as a Python list mostly [0.0, 0.0] but whenever a player scores it becomes [-1.0, 1.0] or [1.0, -1.0] depending on which player got the point.
This will only be supported on retro games where someone has made the effort to make it work as above. If you have a game which does not do this, but in theory supports multiplayer on the ROM, you will need to look at and alter either the AI retro wrapper or the emulator or both.

Do you have any idea how I could implement the second method? This is what I was planning to do, I am just not sure how to tackle it. Any ideas?

@niallmandal: Not in any more detail than I have presented. It will involve digging in to the emulator API to see how to work with a second controller.

sorry kind of a basic question, but what exactly is the emulator API?

@niallmandal: I just mean the developer's interface that you are expected to use when writing code to interact with it. Hopefully, it is described in the documentation of the Python module or classes. It is unfortunately possible that you may need to look inside that code as well, if the P2 controller is not officially supported.

Do you think this is possible with the retro integration tool?

@niallmandal I don't know what that is. It does seem like the sort of thing that would be involved, but I am not sure if it would be the right place to start looking

sorry to keep bothering, but something else came up. When initiating the Pong game, with 2 players, there are 16 possible inputs (the controls). they are represented as a list that looks like this [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]. how do i separate out which inputs control P1 and which control p2? when testing each of the inputs, i could only move p1. am i doing something wrong?

@niallmandal: According to the docs, the first 8 values are for P1, the second 8 values are for P2. I only tried the demo so far which randomises the action vector and results in both paddles moving as you would expect. Just now I tried to only set P1 values or only set P2 values and got very weird results. I could control one paddle independently by setting actions[0:7] = 0, but could not control the other paddle. So I am sorry, but I don't know - there seems to be a mismatch between the documentation and behaviour, maybe a bug

@niallmandal: OK, solved it, at least partially. Indices 0,1,2,3,4,5 are for P1. Indices 6,7,8,9,10,11 are for P2. No idea what indices 12,13,14,15 are for . . . but they don't make either paddle move


text:
Reinforcement Learning with long term rewards and fixed states and actions
I have read a lot about RL algorithms, that update the action-value function at each step with the currently gained reward. The requirement here is, that the reward is obtained after each step.
I have a case, where I have three steps, that have to be passed in a specific order. At each step the agent has to make a choice between a range of actions. The actions are specific for each step.
To give an example for my problem:
I want the algorithm to render a sentence of three words. For the first word the agent may choose a word out of ['I', 'Trees'], the second word might be ['am', 'are'] and the last word could be chosen from ['nice', 'high']. After the agent has made its choices, the reward is obtained once for the whole sentence.
Does anyone know which algorithms to use in this kind of problem?

To give a bit more detail on what I already tried:
I thought that using value iteration would be an reasonable approach to test. My problem here is that I don't know how to assign the discounted reward for the chosen operations.
For example after the last choice I get a reward of 0.9. But how do I update the value for the first action (choosing out of I and Trees in my example)?

Why do you specifically want to use RL. These type of tasks are done by RNN and LSTM.

You are describing a straightforward Markov Decision Process that could be solved by almost any Reinforcement Learning algorithm.

I have read a lot about RL algorithms, that update the action-value function at each step with the currently gained reward. The requirement here is, that the reward is obtained after each step.

This is true. However, the reward value can be zero, and it is quite common to have most states and actions result in zero reward with only some specific ones returning a non-zero value that makes a difference to the goals of the agent.
When this is more extreme - e.g. only one non-zero reward per thousand steps - this is referred to as "sparse rewards" and can be a problem to handle well. Your three steps then reward situation is nowhere near this, and not an issue at all.

I thought that using value iteration would be an reasonable approach to test. My problem here is that I don't know how to assign the discounted reward for the chosen operations.

Value iteration should be fine for your test problem, providing the number of choices for words is not too high.
The way you assign discounted rewards is to use the Bellman equation as an update:
$$v(s) \leftarrow \text{max}_{a}[\sum_{s',r} p(s',r|s,a)(r + \gamma v(s'))]$$
For a deterministic environment you can simplify the sum, as $p(s',r|s, a)$ will be 1 for a specific combination of $s', r$
It doesn't matter that $r = 0$ for the first two steps. The Bellman equation links time steps, by connecting $v(s)$ and $v(s')$. So, over many repetitions of value iteration's main loop, the episode end rewards get copied - with the discount applied - to their predecessor states. Very quickly in your case you will end up with values for the start states.

For example after the last choice I get a reward of 0.9. But how do I update the value for the first action (choosing out of I and Trees in my example)?

You don't do it directly in a single step. What happens is repeating the value iteration loop copies the best values to their predecessor states, one possible time step at a time.
On the first loop through all states, you will run an update something like:
$$v(\text{'I'}) \leftarrow 0 + \gamma v(\text{'I am'})$$
and $v(\text{'I am'}) = 0$ initially, so it will learn nothing useful in this first loop. However, it will also learn in the same loop:
$$v(\text{'I am'}) \leftarrow 0 + \gamma v(\text{'I am nice'})$$
So assuming $\gamma = 0.9$ and $v(\text{'I am nice'}) = 0.9$, and that "I am high" scores less than 0.9, then it will set $v(\text{'I am'}) = 0.81$.
On the next loop through states in value iteration, it will then set $v(\text{'I'}) = 0.727$ (assuming "I am" beats "I are" for maximum value)

But what if the value depends on the choices, that have been made? For exampe the value of ”am” is depending on the value of the other words. But imagine I take random actions and sample a bad set of actions so that ”am” is always chosen. Then ”am” might look like the best option, not being one. And even if i then generate samples where I choose the other action the value could already be biased, or am I wrong?

@Jan: The value depending on other choices is not a problem in general, provided your state representation captures the differences (it does in this example). In value iteration, which you ask about, and which this answer uses, there is no sampling - just processing of all states and actions - so your missing sample cannot happen. However, in other solvers yes it is possible to at some stages have bad estimates because in order to learn accurate values, RL needs to sample enough variation. In tabular solvers that usually means sampling all variations enough times to make errors low.

But this implies, that this approach is not suitable for bigger problems, where I do not have time to sample all variations (maybe even once). Which type of Agent would be suitable in that case?

And another question: If I have to sample all variations enough times, is it not better to sample all possible variations (grid search like) and directly select the best?

@Jan: That is correct, value iteration does not scale to large problems. This is too many extra questions though, and they are different to the main question - please ask a new question on the site if you want to know more.

You don't need to have a reward on every single timestep, reward at the end is enough. Reinforcement learning can deal with temporal credit assignment problem, all algorithms are designed to work with it. Its enough to define a reward at the end where you, for example, give a reward of 1 if sentence is satisfactory or -1 if it isn't. Regular tabular Q-learning would easily solve the toy problem that you gave as an example.

I think this answer would be more complete, if rather than just asserting it will work, give an example answer to OP's question "But how do I update the value for the first action (choosing out of I and Trees in my example)?"


text:
Is there any example of using Q-learning with big data?
Could we even use reinforcement learning with big datasets?
Or in RL does the agent built its own dataset ?

Or in RL does the agent built its own dataset?

Essentially this is the case.
RL is a very general learning mechanism, based on trial-and-error. You could create an environment where the agent's goal is to correctly predict classification or regression problems, where the agent is rewarded for getting close to the correct prediction. However, these problems are most often better addressed using supervised learning techniques. Using RL in such cases will most of the time make the learning slower, less efficient and less accurate.
The relationship between RL and supervised learning is more about how RL generates the target data for learning. In some cases, such as Deep Q Learning (DQN), you can see quite clearly from looking at the algorithm or code, that the RL agent contains a supervised learning component. The "supervised learning" in DQN learns a regression problem to predict the action values for the current agent's target policy. Whilst the outer RL logic that contains this supervised learning model, is built around testing and changing that behaviour in order to act optimally in the longer term.
Using RL to predict from a big data set would involve using the same kind of supervised learning model on the inside, with RL using that to guess at the correct result, and sometimes choosing to guess at some other random prediction instead. This guessing process is extra processing, plus it will add noise and variance to the error signal used to train the model. In a few cases, this might even be helpful, but in the majority of cases studied for regression and classification tasks, it will result in a poorer training process, less accurate model, or both.


text:
What type of ML or AI would predict personal traits from a DNA sequence?
Suppose you have a large dataset of DNA sequences. Alongside each sequence, you have a portrait of the person with the DNA sequence. Other parameters include the age, gender and race of the person.
I would want to train this AI to at some point be able to predict, based on a new DNA sequence, the age, gender and race of the person, and perhaps generate a portrait.
What would something like this be called? What part of AI and ML does this concern? If I had to guess, it's both a regression and a classification problem and it's somehow related to Generative Adversarial Network.
Also, is there a difference between the "DNA sequence" and the "age, gender and race" parameters? I'm not sure because in the end you only use the DNA sequence to predict, but during training you would incorporate all four parameters.

to be precise, "age, gender and race" are features, not params

Yes, you are generally correct.
Prediction based on new dna sequence is prediction: so regression/classification.
Generating a portrait is more generally referred to as generative modeling.
The two are different tasks that learn different functional mappings/models. In prediction, your DNA sequence is an input mapping to age/gender/race, while in generative model all 4 along with portraits are used to learn the probability distribution you wish to sample from.


text:
What is the Thompson Sampling in simple terms?
I am looking at the different existing methods of action selection in reinforcement learning.
I found several methods like epsilon-greedy, softmax, upper confidence bound and Thompson sampling.
I managed to understand the principle of each method except Thompson sampling.
I can't understand the principle and the way it works and its action selection steps.
If you can explain to me the principle and the functioning of Thompson sampling with a simple example I would be grateful.


text:
Is my calculation of the partial derivative of the cost function with respect to a single weight in the first layer correct?
I'm trying to understand the chain rule of backpropagation.
This is what I understood. Is it correct?

$$ \frac{\partial E }{ \partial w} = \sum_{i} \frac{\partial E }{ \partial a_i^{(l)} } (\sum_{j} \frac{\partial a_i^{(l)} }{ \partial a_j^{(l-1)}  }(\sum_{k} \frac{\partial a_j^{(l-1)} }{ \partial a_k^{(l-2)}} \frac{\partial a_k^{(l-2)} }{ \partial z_k^{(l-2)}}  \frac{\partial z_k^{(l-2)} }{ \partial w}))$$

$ a_i^{(l)}$ is the activation of the neuron $i$ in the $l$th layer
$ z_i^{(l)} $ is the sum of multiplication of weights and previous activations
$E$ is the error
$w$ is the weight

Could you please clarify: Is the variable $\omega$ a single weight connecting one input to one neuron in the hidden layer (layer $l-2$)?

yes, it's the weight connecting the first input layer to the first neurone in the layer layer l−2


text:
Is there a complement to GPT/2/3 that can be trained using supervised learning methods?
This is a bit of a soft question, not sure if it's on topic, please let me know how I can improve it if it doesn't meet the criteria for the site.
GPT models are unsupervised in nature and are (from my understanding) given a prompt and then they either answer the question or continue the sentence/paragraph. They also seem to be the most advanced models for producing natural language, capable of giving outputs with correct syntax and (to my eye at least) indistinguishable from something written by a human (sometimes at least!).
However if I have a problem where I have an input (could be anything, but lets call it an image or video) and a description of the image or video as the output I could in theory train a model with convolutional filters to identify the object and describe the image (assuming any test data is within the bounds of the training data). However when I've seen models like this in the past the language is either quite simple or 'feels' like it's been produced by a machine.
Is there a way to either train a GPT model as a supervised learning model with inputs (of some non language type) and outputs (of sentences/paragraphs); or a similar type of machine learning model that can be used for this task?
A few notes:
I have seen the deep learning image captioning methods - these are what I mention above. I'm more looking for something that can take an input-output pair where the output is text and the input is any form.

This is the task of so-called V&L (vision and language models) which effectively encode information from both worlds. There are also many training corpora covering this field already. Here is a quite recent paper on this: https://www.researchgate.net/publication/354617904_What_Vision-Language_Models_See'_when_they_See_Scenes


text:
Agent in toy environment only learns to act optimally with small discount factors
I have tried several environment libraries like OpenAI gym/gridworld but now I am trying to create a toy environment for experimentation. The environment I've created is as follows:

State: grid with n rows by m columns, represented by a boolean matrix. Each grid cell can be empty or filled and the grid starts empty.
Action: one of the m columns to be filled, which must have at least the top row empty.
Next state: Once a column is chosen, the lowest unfilled cell in that column is filled. This works from bottom up like a very simple version of Tetris.
Reward: after every action, a reward equal to the number of empty columns is awarded.

Therefore in a sample world of 5 rows by 3 column, starting off with an empty grid, the maximum attainable reward would be by filling column wise first. This policy will give a maximum total reward of 2*5 + 1*5 = 15. (2 free columns by 5 row action, once first column is filled then 1 free column by 5 row action.)
This very simple environment is trained using DQN with a single ff layer. The agent only took a few episodes to converge and is able to produce the maximum attainable reward.
In a next toy environment, I've made it a little more complex. I modified the very first action to be random choice of any column. I have retrained the RL model with the new environment modification. However, after convergence, the agent does not attain max score of 15 for all possible starting columns. I.e. If column 1 was randomly chosen first, max score might be 15, however column 2 or 3 was randomly chosen first, max score might only reach 11 or 9. In theory, the optimum policy would be for the agent to fill column that was randomly chosen first - i.e. repeat the first randomly chosen action.
I have tried several ways to tweak my input parameters (e.g. episilon_decay_rate, learning_rate, batch_size, number of hidden nodes) to see if the agent could act optimally for all possible starting columns. I also tried DDQN and Sarsa. The only way I could make the agent perform optimally is by reducing gamma (discount factor) to 0.5 or below. Are there any explanations to why the agent only works for small discount factors in this example? Also, are there alternative ways to obtain the optimum policy?

The description of your environment seems incomplete. I cannot make sense of it. For example you say "Once a column is chosen, that row of the column is filled from bottom up like in a tetris fashion." [emphasis mine]. But at no point has a row been described or selected by the agent or the environment . . . some visualised steps of the game might help, but also review whether you have given a complete and valid description. You definitely have some kind of implementation issue, since for an episodic problem it should be fine to set $\gamma = 1$ and still expect to find optimal solution

Sorry for being unclear, indeed my description of action is incomplete. I will try to explain it again.
Action: one of the m columns chosen to be filled. Or more precisely.
If column j is chosen, S_ij will be set to 1, where i is the min row i such that S_ij != 0. (Here S is the boolean matrix representation of the states)

Regarding: "This very simple environment is trained using DQN with a single ff layer." If I understand this correctly, you have a linear regression here, there is no hidden layer. Could you confirm the NN architecture? I am imagining it is 15/3 linear network, with 15 inputs -> 3 outputs predicting reward for each of the 3 actions?

Yes state matrix is flatten, the network has 15 input and 3 output. Hidden layer has 10 nodes, learning rate is 1e-3. In the first environment, after 20+ episodes the agent is able to obtain the maximum score for any chosen m and n. But not after the modification.

This is an episodic problem, and there should be no issue in theory with most learning algorithms coping without a discount factor (or setting gamma = 1).

Are there any explanations to why the agent only works for small discount factors in this example?

The most likely explanations are:

You have a mistake in your implementation or use of DQN.
You have an incorrect setting of a neural network hyperparameter. My initial thought would be you have mistakenly put a non-linearity like softmax or sigmoid on the NN outputs (it needs to be a linear output). Or it could just be that 10 hidden neurons is not enough for this representation
Convergence requires far longer to train than you thought, once you introduce non-linear relationships between state and expected reward.

It is not surprising that the simpler puzzle trains more easily, as the agent can reduce it down to a linear counting puzzle without really needing to "look" at the representation. I would expect the Q values to be very poor approximations to the correct values after so few iterations, but the problem is simple enough that the incorrect values still produce an optimal behaviour.

Thanks so much for your reply! I have tried more hidden nodes, output layer is linear and have waited more than 10k episodes for convergence. I think it might be a mistake in my implementation. Something that I have suspected is due to my implementation of illegal moves. During the choose action step: If the entire column is already filled (S_ij = 1, forall i), I enforced that the column is not allowed to be chosen. And I would choose the next highest action value as the best action. Will this be the main cause of the problem?

@terenceflow: I don't think that would be a direct cause of a problem - although you should look how you handle the data around that change. I would actually recommend changing the problem so that you terminate the episode on an illegal move, meaning the agent scores whatever it got so far (this is what would happen in Tetris). Note that filling the last column is not necessary for optimal behaviour, the agent scores 0 whatever it does, so it doesn't matter and you should not expect an agent to fill all columns.


text:
Proof of uniqueness of value function for MDPs with undiscounted rewards
How does one prove the uniqueness of the value function obtained from value iteration in the case of bounded and undiscounted rewards? I know that this can be proven for the discounted case pretty easily using the Banach fixed point theorem.

In the case where the reward is undiscounted, there is no guarantee of convergence as the iteration procedure is not a strict contraction.
Unfortunately I can't find the math mode on the ai stackexchange so my answer can't be very precise.
But an easy example is the following: Take a 'running' reward R of 0 to make things simpler, and a MDP with two states a and b. Take a transition matrix with 0's on the diagonal and 1 off diagonal. You will see that the algorithm will always flip the values of V(a) and V(b), and hence no convergence.


text:
Reinforcement learning with uniformly random dynamics
Suppose I have an MDP $(S, A, p, R)$ where the $p(s_j|s_i,a_i)$ is uniform, i.e given an state $s_i$ and an action $a_i$ all states $s_j$ are equally probable.
Now I want to find an optimal policy for this MDP. Can I just apply the usual methods like policy gradients, actor-critic to find the optimal policy for this MDP? Or is there something I should be worried about?
At least, in theory, it shouldn't make any difference. But I'm wondering are there any practical considerations I should be worried about? Should the discount factor, in this case, be high?
The reward function here depends both on states and actions and is not uniformly random.

Hi, could you clarify what the reward function $r(s,a)$ depends on? An answer needs to know whether starting state $s$ is relevant to the reward, and also the action $a$.

@NeilSlater The reward function depends both on states and actions. The reward function is not uniformly random.

Thanks for clarifying that. There is no need to signal edits in the question (anyone who is interested can see the edit history), so I have removed that bit.

When the next state selection is not driven by any meaningful dynamics i.e. it is independent of starting state $s$ and action taken $a$, but the rewards received do depend somehow on the $s$ and $a$, then the MDP you describe also fits with something called a Contextual Bandit Problem where there is no control over state due to action choice, and thus no incentive to choose actions other than for their potential for immediate reward.
Any algorithm capable of solving a full MDP can also be put to use attempting to solve a contextual bandit problem, as the MDP framework is a strictly more general case of the contextual bandit problem, and can model such an environment. However, this is typically going to be inefficient, as MDP solvers make no assumptions about state transition dynamics and need to experience and learn them. Whilst if you start with an algorithm designed to solve a contextual bandit problem, you have the assumption of randomised state built in to the algorithm, it does not need to be learned, and the learning process should be more efficient.
Alternatively, if you only have RL solvers available, you can reduce variance and get the same effective policy by setting discount factor, $\gamma = 0$.
If for some reason you still want or need a long-term discounted value prediction from your policy, you can take the mean predicted value of some random states (or even of all the states if there are few enough of them) and multiply by $\frac{1}{1-\gamma}$ for whatever discount factor you want to know it for. Or if predicting for a time horizon, just multiply by number of steps to the horizon.

Convergence guarantees for basic RL algorithms like policy gradient / actor-critic methods make no assumptions about the dynamics of the MDP. So, theoretically, you don't need to change much.
Practically, when the number of possible trajectories from any given state is so high, the return from each state will have high variance. This means you'll have to collect much more experience for your estimates of expected return to converge to their true values. Intuitively, an environment with high uncertainty requires the agent to do more knowledge-gathering to behave optimally.
My real advice to you depends on what exactly you're trying to do. If you want to have the kind of agent that could learn to behave well in an extremely random environment, then all you need to worry about is giving it enough experience to learn from.
(Your agent should also take a little longer before deciding it's "confident" in its evaluation of different states. That is, don't behave greedily before you're sure your estimates are accurate. Explore adequately. This advice is only relevant if your MDP dynamics aren't actually completely uniform.)
If, however, you want to train an RL agent specifically to solve a problem formulated as an MDP with uniformly random dynamics, then I would tell you to not waste your time. We know before spending the computation that all policies would be equally good/bad in this setting. Since actions are irrelevant to the environment, it would be inefficient to deploy an RL agent that will only learn that which action it takes doesn't matter.

As noted in the comments, the last paragraph is only true when reward from each state-action pair $(s,a)$ is also uniformly random. If it is not, just being aware of the high variance and giving your agent a lot of experience should do the trick.

@NeilSlater You're right. I was assuming that reward is jointly determined with the next state. I'll edit to make a note, but is that not usually the case?

The reward function here depends on both state and action and is not uniformly random. I guess in that case your last paragraph is not relevant?


text:
How to reduce amount of species in NEAT?
I am using the following library:
https://github.com/vishnugh/evo-NEAT
which seems to be a pretty simple NEAT-implementation.
Therefore I am using the following Config:
package com.evo.NEAT.com.evo.NEAT.config;

/**
 * Created by vishnughosh on 01/03/17.
 */
public class NEAT_Config {

    public static final int INPUTS = 11;
    public static final int OUTPUTS = 2;
    public static final int HIDDEN_NODES = 100;
    public static final int POPULATION =300;

    public static final float COMPATIBILITY_THRESHOLD = Float.MAX_VALUE;
    public static final float EXCESS_COEFFICENT = 1;
    public static final float DISJOINT_COEFFICENT = 1;
    public static final float WEIGHT_COEFFICENT = 5;

    public static final float STALE_SPECIES = 2;

    public static final float STEPS = 0.1f;
    public static final float PERTURB_CHANCE = 0.9f;
    public static final float WEIGHT_CHANCE = 0.5f;
    public static final float WEIGHT_MUTATION_CHANCE = 0.5f;
    public static final float NODE_MUTATION_CHANCE = 0.1f;
    public static final float CONNECTION_MUTATION_CHANCE = 0.1f;
    public static final float BIAS_CONNECTION_MUTATION_CHANCE = 0.1f;
    public static final float DISABLE_MUTATION_CHANCE = 0.1f;
    public static final float ENABLE_MUTATION_CHANCE = 0.2f ;
    public static final float CROSSOVER_CHANCE = 0.1f;

    public static final int STALE_POOL = 10;
}

However, there are way too much species (about 60). I do not know how to reduce this number, given the fact that the COMPATIBILITY_THRESHOLD is already maximized.
So what's my fault?
Note: I am not using: http://nn.cs.utexas.edu/keyword?stanley:ec02
since this algorithm seems not to work in a changing environment (where fitness can vary hardly)

Your species count will increase as the chance of mutation increases. This is because in every generation, so many genes will be mutated that they have little resemblance of each other, and the distance function doesn't factor in historical markings / innovation numbers.
Try lowering the mutation rates.
Below is the distance function from here page 110
$$\delta = \frac{c_1E}{N} + \frac{c_2D}{N} + c_3 \cdot \overline{W}.  $$
If your fitness vary a lot, try ranking the fitnesses in each specie and setting the survival chance based on its rank.
If you mean a large action space by changing environment, you can set the number of output nodes to the total number of actions, and rank each action, best to worst, then pick the best available action for the state.

If the compatibility distances are coming out to be > float.max that tells me something may be wrong with that calculation, I would suggest setting break points and debugging that code, I usually have my threshold set 5-10 and can noticeably tell when a change in species size when i move the threshold to something like 10


text:
High variance in performance of q-learning agents trained with same parameters
I am training an agent to play a simple game using double deep q learning. However, the variance in agent performance is very high, even for agents trained with same model parameters. For example, I can train agent A and agent B using the exact parameters and agent A beats B 800 to 200.
I think I understand why this is happening, when training starts the model is initialized with different weights, and this leads the model to find different local max/min.
The above makes it difficult to find optimal parameters.
What are the strategies to reduce this variance? What parameters should I look at tweaking?
More details about the environment:
This is a two player game (Zombie Dice); however, in my implementation so far the agents are learning to maximize expected score on their turn, so the actions and score of the opponent is ignored.
The variance is higher when I am using purely greedy strategy with no exploitation at all. Though it exists in both cases. I would say roughly 2/3 wins for stronger side with greedy and 3/5 with exploration out of 1000 matches.
The environment is stochastic; I have not done many assessment runs maybe 20 or 30, it is mostly eyeballing, but the differences are fairly large; therefore, I am confident that this is not due to chance.
I tested the models against themselves, and I get scores very close to 50/50. However, two different models trained with same parameters give results very different from 50/50. I tested this with models trained with different types of parameters and it is generally the same problem.

Is this a 2-player game that your agents are learning, or are you comparing scores on separate runs of the environment? Either way, could give a little more detail about the environment and how you are assessing agents - for instance as it is Q-learning, have you swapped to fully greedy use of Q values to set the policy, and if the environment is stochastic, how may assessment runs have you done? If this is a 2-player game, have you checked variance of A vs A and B vs B?

@NeilSlater Thank you for the suggestion, I updated my question.

There is no need to flag edits (interested users on the site can see your revision history). The goal is to have refined question text that makes the question easy to read and understand.


text:
Understanding alpha-beta pruning for simplified NIM
This is a simple version of NIM: Two players alternately remove one, two or three coins from a stack initially containing 5 coins. The player who picks up the last coin loses.
What does alpha-beta pruning look like on the game tree for this game?

Welcome to AI.se! This is a place where we ask and answer questions about AI. Your post looks like a homework problem. Is there a specific part of it you're confused about that we can help you with? Right now, your post is too broad.

@PhilipRaeisghasem I prefer using the standard of the utility of a given question, regardless of it being homework or not.  Let me know if you think the recent edit has rendered it more suitable.

Oh, yes, the recent edit has none of the problems the original question had. The original format of "Find X. Find Y. Show Z" is what I was mostly objecting to as being too broad. It was also, technically, not a question.