text:
What is an appropriate fitness function for a simulated self-driving car?
I have been working for ages on a neuro-evolution AI program, where cars learn how to race around a track.  Presently, I have a rudimentary fitness function that awards points for every degree traveled in the CW direction about the center of the window (removes points in CCW dir) and removes a certain amount of points for every collision that occurs. Cars also lose points for every moment they stay still and/or are colliding with something. My end goal is to create cars that can complete a track full of obstacles, faster than a human controlled car.
Is there a better fitness function that would result in more efficient cars that

make better use of their sensors,
are efficient in getting around the track (don't weave like a drunk driver), and
are faster in general (cars are too cautious, but this is a race!)

?
Half of the population seems to just spin around.
I have fully implemented my neuro-evolution program. Here's my implementation. However, you'll find that it isn't perfect.
How should I alter my fitness function to generate better driving cars?
Currently, the cars will only turn left or right if the outer sensors are activated.  However, if the sensors directly in front are activated, the neural network is configured in such a way that "ignores" this signal. So, the cars crash head-on into obstacles but avoid obstacles in their periphery.  I think this is due to the fact that the fitness function (which gives points for the displacement in 3 second time period) is too generous and removes the incentive for cars to avoid obstacles.  I've tried altering the punishment for collisions and the reward for driving in the right direction but it still isn't performing the way I would like it to.
Here are some screenshots

It would be nice to know how you exactly determine that the car travelled (rotated) 1 degree. Does the car get positive reward every time when it goes around its own little circle?

It seems like you might want to add additional layering for score instead of just the immediate I moved one degree I get a point. Perhaps a moving window that calculates total degrees moved over the last TBD-seconds. I would probably give that more weight than near-term score. I would think that should help prevent weaving and circling. Which might beg the question of why keep a near-term score at all but I would keep the near-term score because I suspect it will help get around obstacles.

You also might need to take into account 'obstacles' movement. That includes speed and direction. Perhaps your cars are circling because they are trying to avoid other cars. However, in a real race, cars race side-by-side because they have to assume the other drivers are going to follow certain 'driving rules'. Thus, they don't need to be concerned about hitting cars driving next to them. Their biggest concern is not hitting the car in front of them.

You can set a series of checkpoints - see this excellent article by Byte Master on codeproject.com: https://www.codeproject.com/Articles/1220276/ReInventing-Neural-Networks

You're correct that it is related to the fitness function, but only indirectly.  Recall that a function is the mathematical embodiment of a conceptual relationship.  The model is a motion model.  Proximity is not simply applying the distance formula $D = \sqrt{(x_2 - x_2)^2 + (y_2 - y_2)^2}$.
Proximity as we "feel it" when driving a car is based on the risks we see in the trajectories of ourselves and the other moving objects along with the stationary objects.  Mathematically, this is an integral over time of a distribution of probable locations.
Look for work in the aeronautics industry about collision avoidance.  The mathematics and algorithms are well developed for antiaircraft weaponry (where the fitness function is the inverse of the one you want) and air traffic control, where the fitness is like yours except with an altitude dimension.


text:
Is there a way to quantify the effect of this floating-point numbers encoding on the robustness ofclassification?
This question is related to the usage of NN in critical systems (those where a failure can cause life threatening situations - autopilots for example) and the need for formal guarantees on their behavior. Here is, for example, a paper that verifies NN used in the controls of an unmanned air-vehicle.
There are numerous tools and techniques for the formal verification (not just testing and simulation, but actual mathematical proof) of, say, floating-point calculation (properties like: will not overflow, will not accumulate an error greater than x, etc).
Now take a DNN where the weights are real numbers. That is the DNN as a concept. But there is the implementation of the NN, where the weights must be encoded as floating-points (some even as low precision as 3-bits integers, apparently). And then, (faster) computation is done on these encodings. Someone might argue that you lose precision, but others might answer that it may be a good thing  since one must prevent a NN for over-fitting anyway...
Questions:

Is there a way to quantify the effect of this encoding on the robustness/precision/stability-of-classification, in a comparable way to what we have in (non learning) software verification?

Are there NN architectures (like Sum-Product Networks(?)) that are more amendable to offer such guarantees?

There is a lot of evidence that floating point precision is not a factor in neural networks, for example here https://arxiv.org/pdf/1412.7024.pdf

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
How does Uber's rating system algorithm work?
I've been trying lately to search the internet to find a result for this but no useful results, unfortunately. We're making an uber-like application, and, during our discussion, we needed a way to validate a rider's rating after he completes his ride with a taxi, but the main question is, does uber use an algorithm to do so? And how?

I’m voting to close this question because it is asking about a specific company's solution, which is probably a trade secret and not allowed to be answered. Also the question lacks some clarity.

I highly recommend that you start reading on the Netflix challenge. It has tonnes of useful and interesting examples dealing with this sort of thing.
You will need an Algorithm that builds a score on both 'quality' and 'quantity'. That is, it needs to add a 'weight' to the final rating based on the number of reviews that an individual has. This is so that, for example, an individual with 50 8 score reviews would be rated higher than an individual with only one 9 score review.
I recommend that you implement Bayesian estimates to calculate weighted voting.
IMDb (Internet Movie Database) utilizes this algorithmn to determine its IMDB top 250 movies. (Robert C 2010)

The formula for calculating the Top Rated 250 Titles gives a true Bayesian estimate:
weighted rating (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C
where:
R = average for the movie (mean) = (Rating)
v = number of votes for the movie = (votes)
m = minimum votes required to be listed in the Top 250 (currently 3000)
C = the mean vote across the whole report (currently 6.9)

Please note that in addition to the rating Algorithm, Uber has a dispatch algorithm which takes into consideration factors i.e. Drivers who are online and drivers who are nearest to the passenger.

Honestly I cant see it being to  much of a sophisticated algorithm, its probably something simple like the above with a touch of normalisation, ie square rooting by the s.d. or something silly and unnecessary like that.  Best algorithms I find are based on the simplest of statistical principles.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What would be the most simple approach to solve crossword puzzles?
I have to model an AI that should be able to understand clues and find the answer from a specified word database. I came across several papers that solve the problem with training neural networks or processing the clues by training several machine learning architectures on clues that exist on several databases. However, they all seem overkill for the course currently I'm taking.
So, what would be the most simple approach to solve that problem?

Artificial neural network to solve Crossword puzzles, this model might help you to get a start on evolving your own.
https://github.com/fh295/DefGen2
A brute force of word combinations would solve the matrix, but might not get the context right and might be the simplest model if you are not in for a neural network.


text:
Inconsistency in TD-Leaf algorithm in KnightCap chess engine
Notice that, in the following formula, at the very right, the term multiplied with $\lambda$ is $d_i$
$$
w := w + \alpha \sum_{i=1}^{N-1} \nabla r(x_i^l, w) \Big \lfloor \sum_{j=i}^{N-1} \lambda^{j-i} d_i \Big \rfloor
$$
Note that $i$ is the index of the first sum.
However, in the following formula, the term is $d_m$, which was $d_i$ in the first equation.
$$
w_j := w_j + \alpha \sum_{i=1}^{N-1} \frac{\partial r(x_i^l, w)}{\partial w_j}  \Big \lfloor \sum_{m=i}^{N-1} \lambda^{m-i} d_m \Big \rfloor
$$
Note that $m$ is the index of the second sum.
In my opinion, first equation seems reasonable. The $\lambda$ in second one actually seems to work like discount factor.
In another chess engine, Giraffe, the author cited same paper, noted the first equation, but then proceeded on to implement second one. See also KnightCap Paper.

The second equation is correct. In TD($\lambda$), the $\lambda$ parameter can be tuned to smoothly vary between single-step updates (essentially what Sarsa does) in the case of $\lambda = 0$, and Monte-Carlo returns (using the full episode's returns) in the case of $\lambda = 1$.
In the first equation, $\sum_{j = i}^{N - 1} \lambda^{j - i} d_i$ could be interpreted as a... summing up exactly the same temporal-difference term $d_i$ a number of times (specifically, $N - 1 - j$ times), but multiplied by a different scalar every time. I'm not sure how that could be useful in any way.
In the second equation, $\sum_{m = i}^{N - 1} \lambda^{m - i} d_m$ can be interpreted as a weighted combination;

$1 \times d_i$: this is the difference between what we predict our returns will be at time $i + 1$, and what we previously predicted our returns would be at time $i$.
$+ \lambda^1 \times d_{i + 1}$: a very similar difference between two of our own predictions, now the predictions at time $i + 2$ and $i + 1$. This time the temporal-difference term is weighted by the parameter $\lambda$ (completely ignored if $\lambda = 0$, full weight if $\lambda = 1$, somewhere in between if $0 < \lambda < 1$).
$+ \lambda^2 \times d_{i + 2}$: again a similar temporal-difference term, now again one step further into the future. Downweighted a bit more than the previous term in cases where $0 < \lambda < 1$.
etc.

For people who are familiar with temporal-difference algorithms like TD($\lambda$), sarsa($\lambda$), eligibility traces, etc. from Reinforcement Learning literature, this makes a lot more sense. The notation is still a bit different from the standard literature on algorithms like TD($\lambda$), but in fact becomes equivalent once you note that in this paper they discuss domains where there are only rewards associated with terminal states, and no intermediate rewards.
Intuitively, what they're doing with the $\lambda$ parameter is assigning more weight (or "credit" or "importance") to short-term predictions / short-term "expectations" (in the English sense of the word, rather than the mathematical sense of the word) or observations of rewards, over long-term predictions/observations. In the extreme case of $\lambda = 0$, you completely ignore long-term predictions/observations and only propagate observed rewards very slowly, one-by-one in single steps. In the other extreme case of $\lambda = 1$, you propagate rewards observed at the end of episodes with equal weight all the way to the beginning of the episodes, through all states that you went to, giving them all equal weight for that observed reward. With $0 < \lambda < 1$, you choose a balance between those two extremes.

Also note that Equation (5) in the KnightCap paper (where they similarly discuss the extreme case of $\lambda = 1$, like I did above) is incorrect if we take the first equation from your question, but is correct if we take the second equation.

Of course, it's the first equation that is correct. d is the temporal difference, and it's the difference between the current and next state. The current state is i, and this difference needs to stay constant inside the summation loop.
The second equation is simply a typo mistake.

If you look at giraffe paper, there is a table at page 23. That looks like 2nd equation. He is multiplying lambda with future errors.

@ShihabShahriar I need little more time to get to the Giraffe paper, but I still believe my answer is correct.

No it's not correct, summing up exactly the same term multiple times, with different weights (different exponents of $\lambda$), does not really seem to have any meaning to me. The second equation, a weighted combination of different temporal-difference terms, has a clear meaning.

If you think it is in fact meaningful to sum up precisely the same term multiple times (with different weights depending on $\lambda$), it would be interesting to explain that. As I see it, the first equation will simply multiply the same temporal-difference term $d_i$ more often for states $i$ that are far away temporally from the terminal state $N$; this is exactly the opposite of what you'll want to do in practice, because  those long-distance returns have high variance. Also, Equation (5) in the KnightCap paper would be completely incorrect if the first equation were correct.

@DennisSoemers This is an old question, I forgot everything. I need some time to digest.

@DennisSoemers I didn’t downvote u

@DennisSoemers as soon as you posted your answer this post bumped to the homepage. Anyone can come in and vote.


text:
Why is the larger value, as opposed to the smaller one, chosen, in the hill climbing algorithm?
In the hill climbing algorithm, the greater value, compared to the current value, is selected, but I cannot understand why it takes the larger value instead of the smaller one. Why is that?
I greatly appreciate the inclusion of figures in your answers.

Answer can be found here

When we climb a hill:

We move higher in altitude. The person who is climbing, will always look for rocks/mud on the hill that are higher, so that he can climb higher.
That is what the algorithm does too. We are assuming that there is a hill of numbers. The larger numbers are placed higher than the smaller numbers. So if we want to climb up the hill, we search for the larger numbers.
Comparing this with the real world hill climbing, simply assume that every rock on the hill has a number written on it. The number will be the altitude of the rock, from sea level height. So if you want to climb up, you just have to search for the rock with a higher number than the rock you are standing on now.

There is no reason why you can't have a hill descending algorithm, instead of finding maxima you will find minima.  If that is what your aim is, it's still called a hill climbing algorithm, I guess...


text:
Can recommendation systems be created for other data other than images?
Can recommendation systems be created (using machine learning) for other data other than images?
For audio or video content, is it necessary to use a dataset of actual audio and video files, respectively, or is text information about the file enough for the recommendation system?

Recommendation systems can be applied for anything, as long as you have sufficient training data.
The most important inputs to the recommendation system are not "audio files or video files".
Wikipedia gives you the list:

Similarity matrix
Item attributes
User activities and behaviours
User profile

https://en.wikipedia.org/wiki/Recommender_system


text:
Is it possible to train a neural network to estimate a vehicle's length?
I have a large dataset (over 100k samples) of vehicles with the ground truth of their lengths.
Is it possible to train a deep network to measure/estimate vehicle length?
I haven't seen any papers related to estimating object size using a deep neural network.

Is your data set images or text? hisairnessag3 is on the right track with porting the pixels/metric approach from OpenCV to an object detection DNN

Yes! This most certainly can be done. Since you have a labeled dataset, that makes it all the more simple!
I would take a look at this project and that should get you where you need to go.
The implementation details should be pretty straightforward. Let me know if I can help further.

It is not necessary to explain why someone is down voting. Many programmers do not have enough time to look at the comments, they look at the votes to see if the answer worth it or not. If you read the question again, you will find that the question is specifically about using DNN and not anything else. the project you shared is about using Opencv ... if you read the comments under the post you shared, you will find that people are asking about DNN, but the post itself does not use DNN. By the way I am not the person who down voted  your answer

It isn't necessary but is helpful. If you search at any depth you will find no projects or papers on this particular use case for DNNs. With that being said, I referenced the closest possible project in the hopes of getting OP started.
Additionally, I answered the question. The question was can this be done.

Thanks for your answer! I think it would be improved by a little explanation of how/why it's possible. At least an outline of the architecture would be very helpful.

I think this paper can help you out: 3D Bounding Box Estimation Using Deep Learning and Geometry
He used 1 VGG-19 (pretrained on ImageNet) to learn the size of cars

Like you, I assumed this is about inferring a 3D bounding box where the data is represented in a 2D image. The project mentioned by hisairnessag3 appears to only address the 2D bounding box with no learned inferential behavior about the 3D nature that the image might contain.

This paper estimates the size and the orientation of the car using a 2D cropped image (only contain the desired object).

Yes it's possible, but first you'll have to recognize some object in the image, either 1) the vehicle itself, and then report that vehicle's known size, or 2) a known object that's the same distance from the camera as the car (a curb, a stop sign, the driver's head, a shetland pony... whatever), and then use that object to calibrate the size of the car that's very close to it.
Any car in an image will be an unknown distance from the camera, making the car object appear larger or smaller from photo to photo.  If you don't recognize the car or at least a referent object that has a known size, the physical size of the car will be uncalibrated -- you'll have no basis for your size estimate.
If the car is unknown, then even if you do have visual clues (there is a referent object present or the distance from camera to car is known), the unknown extent of wide angle-ness of the camera's lens may distort an unknown car's shape (height vs width), further complicating your ability to estimate its apparent dimensions.

Can the license plate of the vehicle be that known object used to calibrate the size of the car? License plates are typically the same size.

You'd have to be careful about the assumptions you make that license plates are the same size. For instance, an Australian plate and US plate are different sizes. If this isn't a problem for your application (it'll only ever run in one country), you could assume a reference size for the plate. You would have to be careful still though, because the plate may not be oriented normally to you. If the car is at 45 degrees, the width of the plate will take up fewer pixels than at 0, even at the same distance.


text:
Is Deep Learning the repeated application of Linear Regression?
Is Deep Learning the repeated application of Linear Regression?

There's no point to fit a linear regression model (such as OLS) with neural network because it's really designed for non-linear models. But if you want to do that, you'll just need to set linear activation units.

A neural network can be reduced to a linear regression model only if we use linear activation functions (i.e. $\sigma(x) = x$), and only if we do not use any  neural network specific techniques such as convolution, residuals, etc., as shown below:
$\text{neural network}(x) = \sigma_n(W_{n} \sigma_{n-1}(W_{n-1}\dots\sigma_1(W_1 x + b_1) + \dots + b_{n-1}) + b_n) \\
= W_n (W_{n-1} \dots (W_1 x + b_1) + \dots + b_{n-1}) + b_n \\
= \left( W_n W_{n-1} \dots W_1 \right) x + \left( W_n W_{n-1} \dots W_2 \right) b_1 + \left( W_n W_{n-1} \dots W_3 \right) b_2 + \dots + W_n b_{n-1} + b_n \\
= W_z x + b_z$
where $W_z = \displaystyle \prod_i W_i$ is a weight matrix and $b_z$ is some vector constant.
This follows the linear regression model form $y = Wx + b$, where $W$ is the weight matrix and $b$ is the vector constant. As a result, you can analytically solve for $W_z$ and $b_z$ using linear regression techniques, and no longer need gradient descent.
Note that for this to work well as a linear regression, you need to check for the OLS data assumptions, such as making sure the regressors have no collinearity, the residuals have no heteroskedasticity, there's no auto-regression, and that the errors are roughly normally distributed (more info). Deep neural networks with non-linear activation functions do not require these assumptions since they are universal approximators, although checking for some conditions may help make the task easier to predict (more info*).
*Note - this link talks specifically about time series data with neural network, but the same concept applies to any task in general.
The neural network specific techniques, such as ReLu, convolutions, residuals, etc., is what allows the network to learn non-linear relationships, and therefore make neural networks something more than just repeated applications of linear regression.

Maybe the penultimate step of your demonstration is not fully clear, so it may be useful to provide a concrete example. Let's say that you have $2$ inputs $x_1$ and $x_2$ in the vector $x = [x_1, x_2]$, then you have 2 hidden layers of $2-3$ units each, and then, say, an output layer with 1 unit. You could then try to show if the NN really collapses to linear regression. Maybe you can find a simpler example that shows that, if you want.

@nbro I tried writing a draft with the example like $x = [x_1, x_2]$, but it became too verbose while not bringing much clarity. I just edited my post to try to make things more clear. If its still unclear, can you explain further exactly where and why?

Generally speaking, you can say this:

there is a relationship between neural network learning (I'm assuming a "vanilla" ANN here, no CNN's or RNN's or anything) and linear/logistic regression.
But they're not the same thing.  Just related.  You could maybe consider them "cousins" to use a real-life analogy.

The big obvious difference is this:  standard linear regression is, well, linear, that is, it's based on a straight line. So it can only separate points on a plane which can be separated by a straight line drawn on that plane.  An ANN however, is non-linear and can fit all sorts of crazy looking curves.  The reason why this is true has to do with a combination of the "activation" functions that are used, as well as the layering effect of your hidden layers.
To be fair, if you extend linear regression to be polynomial regression, you can fit more complicated curves, but that has its own downsides.  And while they are also related, linear regression and polynomial regression aren't - strictly speaking - the same thing (although they may both be special cases of the same general technique).
All of that may be over-simplifying a bit.  If you really want a good explanation of both linear/logistic regression and ANN's and some explanation of how they relate and differ, I recommend Andrew Ng's ML courses on Coursera.  Both the original one and the new DeepLearning.ai ones.

You should probably note that NNs could just have linear activation functions, and, in that case, NNs can "collapse" to a linear regression model, though typically this is almost never done when using NNs.


text:
What are the counterparts of non-linearities and dropout in fully convolutional networks?
I am trying to replicate the fully convolutional networks (FCN) concept described here for semantic segmentation. It seems people have successfully trained such models by removing fully connected layers from the popular pre-trained models such as VGG and adding upsampling and unpooling layers.
I understand that transpose convolution and unpooling in upsampling layers provide counterparts of convolution and max (or average) pooling in earlier downsampling layers respectively, but what are the counterparts for non-linearities such as ReLU? What about dropout? There seems to be no discussion of this in the video.

All-convolutional neural network is a more general concept which can be (and is often) used without deconvolutional and unpolling layers, e.g. for an ordinary classification task. The idea is to replace the pooling and fully-connected layer with particular convolutional layers that do the same. Note that activation functions and dropout are not affected by this transformation. You can read more about it here.
The same will be true for the deconvolutional layers. I didn't find it stated explicitly in the original paper, but here's another work that applies fully-convolutional network for image segmentation. On the figure 2, you can see ReLu activations: they are placed after (simple) convolutional layers, in both upsampling and downsampling parts. They don't use dropout, but batchnorm instead, but if there were a dropout, just as before, it should be placed after the convolutional layer. The keep probability, though, should probably be higher - this relates to hyperparameters tuning and may be different depending on the problem and available data.


text:
How can I leverage artificial intelligence and virtual reality to create intelligent automatic story generation?
How can I leverage artificial intelligence and virtual reality to create intelligent automatic story generation?
My idea is to come up with a system that is empowering the user’s imagination to create and experience stories that engage and immerse them. So they can have the control over the story creation/narrative?
A system that could enable the following use cases:

We sit down in our entertainment room. Select a genre from a menu. Pick the character types for the cast. Decide on a generic plot or give the system the option of picking a random choice. Sit back and enjoy.

Let's say, I want to watch a sci-fi movie set in Paris with Marlon Brando and Jim Carrey. Then the system instantly generates such a movie.

It has come to my attention that SoarTech developed a pattern-of-life capability for the U.S. Army Simulation: Agent-based Patterns of Life for Virtual Environments.
That the University of Twente in the Netherlands has been working on a project called the Virtual Storyteller.
Here is a research paper entitled The automatic generation of narratives.
And more recently that IBM has developed a speech recognition sandbox.
I know that the answer probably requires tech, AI, speech, VR and AR, and Watson integration specialists, but any help is welcome.
Watson probably won't be a necessary component of the solution, as long it is an efficient voice recognition software with speech to animation features.

Comments are not for extended discussion; this conversation has been moved to chat.

The VR and computer speech aspects are entirely corollary.  Adding them in would be relatively trivial in comparison to creating an algorithm that can dynamically generate stories of interest to humans.  Essentially, aesthetic components not related to story structure (images, sounds, speech) are "window dressings".
A story generation algorithm would have to focus on the structural elements of storytelling, possibly in conjunction with human-user reactions, and generate narratives narratives dynamically.
Such a system would be validated by human reaction to the stories, and the validation would not require graphics, sounds or speech.  Think of a movie or video game with a story as the expressions of stories that originally had textual form.
Game theory could certainly be applied to this process--creation of narrative in particular utilizes game theoretic concepts, consciously or unconsciously, in terms of strategies to engage reader/viewers and provide "emotional payoffs".
It should certainly be possible to develop an algorithm with these capabilities, even where the algorithm doesn't not really "understand" the content it is generation.
A similar endeavor might be found in Cameron Browne's work on algorithms that can create simple, combinatorial games of interest to humans: Evolutionary Game Design.
It's important to note that the game created by Ludi are based on pre-existing mechanics, and algorithms don't yet have the capability to make profound leaps in the way that human game designers can. (In other words, it will likely be a very long time before an algorithm can create a game like Magic: The Gathering where no such gaming system previously existed.)
However, this doesn't present much of an issue in the context of procedural narrative generation in that the most successful literary work in terms of audience/sales tends to be based on pre-existing formulas.  So an AI that can merely reconfigure elements in a formula should be able to generate narratives of interest to the average human.


text:
Could a multi-camera SLAM system that is accurate at low driving speeds be equally accurate at high driving speeds?
Could a multi-camera SLAM system for self-driving cars that is accurate to under 10 cm (3.9 in) at parking lot speeds (i.e. very low driving speeds) retain this level of accuracy at high driving speeds (e.g. highway speeds or city driving speeds) if the cameras were a) 60 fps and b) had a global shutter?

It would probably be accurate at higher speeds, if the processing system that processes the information from the camera system can process the information in real time. It looks like it can, but you will also have to account for motion blur. If there is too much motion blur, you will probably not be able to do anything other than speeding up the camera and processing system. However, if the motion blur is not that bad, and the system has been trained on the motion blur, it should be able to classify everything correctly.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Is there a general adversarial network that can take multiple low quality images to create a higher quality image?
Is there a general adversarial network that can take multiple low-quality images of a subject to create a higher quality image of the subject?
SRGAN just takes a single low res image and makes it high res but I need something that can take multiple low quality (not necessarily low res) images to create a higher quality image.
By low quality, I mean you could have one image with a reflection or partially blurry or has a small obstruction that another image might compensate for.  So, it could combine the lower quality images to create a higher quality image.
Is there any GAN that could do this?


text:
What is an activity vector in capsule neural networks?
I was reading the paper Dynamic Routing Between Capsules and didn't understand the term "activity vector" in the abstract.

A capsule is a group of neurons whose activity vector represents the instantiation parameters of a specific type of entity such as an object or object part. We use the length of the activity vector to represent the probability that the entity exists and its orientation to represent the instantiation parameters. Active capsules at one level make predictions, via transformation matrices, for the instantiation parameters of higher-level capsules. When multiple predictions agree, a higher level capsule becomes active. We show that a discrimininatively trained, multi-layer capsule system achieves state-of-the-art performance on MNIST and is considerably better than a convolutional net at recognizing highly overlapping digits. To achieve these results we use an iterative routing-by-agreement mechanism: A lower-level capsule prefers to send its output to higher level capsules whose activity vectors have a big scalar product with the prediction coming from the lower-level capsule.

I thought a vector is like an array of data that you are running through the network.
I started working through Andrew Ng's deep learning course, but it's all new and terms go over my head.

In a traditional neural network, the network's vertices are neurons and the output of a single neuron is a single value (a "scalar"). This number is called its activation. A layer of neurons in the network outputs a vector of activations. We should not confuse this with the activity vectors in a Capsule Network.
Capsule Networks are different since the network vertices are Capsules rather than neurons. They are a higher-dimensional: the output of a Capsule is not a scalar but a vector representing a group of parameters related to the input. Hence the name activation vector.
Motivation
In a neural network, there is no inherent structure between the scalar outputs of the neurons, this is something the following layers have to learn. In Capsule Networks the output of a capsule represents all the parameters related to that together in a vector including a prediction for the activation of deeper layer Capsules. This adds a useful local structure.
For example, consider face recognition. If you have a capsule that knows how to recognize eyes it could output an activity vector representing e.g. "since I have recognized an eye position $(x,y)$ with probability $p=0.97$ I predict the parameters for the whole face will be $(f_1,\dots, f_n)$".
As explained in the Dynamic Routing Between Capsules paper you refer to this information is then used in the way that the capsules in earlier layers (the parts: eye, mouth, nose) predict the activations of deeper layers (face). For example, a face recognizer will only be strongly activated when there is an agreement between the eye, nose and mouth recognizers (the parts) and the face recognizer (the whole) about where the face is located (the $(f_1,\dots, f_n)$ parameters).
Historical Inspiration
Older computer vision algorithms like SIFT work in a similar way where recognition is based on agreement between the configuration of multi-dimensional features (key points) and the reference configuration.

I took it to mean something like "the vector of activations of the neurons in the capsule".  The activation for a given neuron is the weighted sum of its inputs, passed through the activation function (sigmoid, relu, etc).


text:
Are there examples of AI systems that use grammatical rules in order to recognize commands in natural language?
Are there examples of AI systems that use grammatical rules in order to recognize commands in natural language?
For example, is there a program, when given the instruction (either by speech or by text) in the form of "Open file-1", will attempt to understand the command as consisting the parts "open" and "file-1", then respond according to how each part is classified?
The focus here is the fact that these kind of instructions are processed with built-in rules of grammar, rather than the rules being learned through observation of data.

old text adventures? How "natural" does the natural language processing need to be to count as an example for you, and how flexible?

More general than text-adventure; in terms of flexibility, if there is limitation to the variety of sentence forms that the AI can process due to the fact that it has built-in rules, I'd like to know more.

Yes.  You're basically looking for "rule based" NLP.  For reference, see:

https://en.wikipedia.org/wiki/Rule-based_system

https://www.quora.com/Why-are-rule-based-methods-becoming-unpopular-in-NLP-Are-rule-based-methods-still-in-use-If-yes-where-should-I-look-for-them

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3756254/

https://nlp.stanford.edu/software/regexner.html

http://u.cs.biu.ac.il/~yogo/courses/sem2014/intro.pdf


text:
How to design a good evaluation function for a go-like game?
I am trying to find a good evaluation function for a game with:

A 7x7 tile board

2 players, given an equal number(>=3 currently undetermined) of stones placed randomly on the tiles

A turn is consisted of a player moving a stone owned by that player, vertically or horizontally but not diagonally to a very next tile of itself

A player loses when out of moves: a player is out of moves when every stone that player owns, has its very next tiles, except not for diagonals necessarily, occupied either by the board edge or other stones

Right now, my evaluation function's return value increases:

if the total moves available to the player is increasing, and/or

average distance to the middle tile of the board is decreasing

Question: Is there a better strategy? How can I improve my evaluation function?

Welcome to AI!  This is definitely the type of problem studied in Combinatorial Game Theory.  There's even a formal name for your victory condition: "normal play".  It sounds like pathfinding is a factor, so you might want to take a look at that.  I'm not fully understanding your current evaluation function--please describe it comprehensively in full detail.

If you have the best combination of distance between the stones, you should choose the best move to win. In this case, you have to be close to where your opponent plays. It is best to do this by surrounding your opponent's stones. You should always put the first stone in middle or corner of the table.


text:
Is there a way of converting a neural network to another one that represents the same function?
I have read the paper Neural Turing Machines and the paper On the Computational Power of Neural Nets about the computational power of neural networks. However, it isn't still clear to me one thing.
Is there a way of converting a neural network to another one that represents the same function? For example, given a trained recurrent neural network that represents a certain function, is it possible to convert this RNN to e.g. an MLP that represents the same function?
If an answer to the above question exists, can we show the equivalence between neural networks, e.g., all problems solved by a multilayer perceptron can be solved by a recurrent neural network, but the opposite is not true, i.e., MLP is subset of RNN (I do not know if this is true, it is just an example). So, if we obtain this relationship between all neural networks, we can get a neural network $X$ that is more powerful than others, so, we can throw away all other neural networks because $X$ can solve any problem that other NN can. Is this reasoning correct?

To answer this, it's helpful to consider the notion of a neural network architecture – in this context, we can think of the architecture as being the network depth (i.e. number of layers), width (i.e. number of nodes in a layer), and some other structural aspects, such as recurrent layers, convolution layers, pool layers, etc.
Theory
In terms of the theory, there are a few ideas that provide insight into your question. First is the commonly cited universal approximation theorem, which states that a single hidden layer, feed forward neural network with a sigmoid activation and finite width (i.e. the number of nodes in single the hidden layer) can approximate any n-dimensional real-valued function. Superficially, this tells us that we can do a whole lot with one of the most simple architectures and could (in theory) approximate more complex architectures such as LSTMs, CNNs, etc.
Yet, while this seems to indicate that reductions are always possible, research on the asymmetry in representational power between depth and width hints that things may not be so simple (for example, see this paper and this paper). Specifically, network depth provides much greater representational power relative to width. Intuitively, this makes sense – depth allows us to build hierarchies and stack multiple non-linearities. Indeed, this is where we begin to see the outlines of what the answer looks like in practice – depth provides a clear, easy way to bake structure into our network.
So, the upshot from the theoretical perspective is that the universal approximation theorem answers in the affirmative; yes, you could reduce a complex network architecture to a single hidden layer network, but the basis for doing this is a bit naïve – it doesn't consider the efficiency of the network relative to the number of parameters.The two papers linked above imply this.
Practice
In practice, there are issues with performing such a reduction. To date, the most successful models have deep architectures. More importantly though, many of these models are specialized – we have CNNs for computer vision, RNNs for sequence-based data, etc. These architectures were conceived as a way to integrate knowledge about the structure of the problem into the structure of the network to improve learning of the desired representation, and this too requires depth.
In fact, last year a paper took this idea to the extreme. Weight agnostic neural networks are specifically constructed such that their architecture is in essence the representation with a single parameter (weight) that can be varied to generate multiple distinct models with different performance characteristics. The WANNs that were developed in this research weren't especially deep, but they also weren't single layered.
The upshot here is that, for complex problems, depth is necessary for efficient learning and gradient computation and that it can be used to encode problem structure into a network. While it may be possible to reduce a learned representation to a single hidden layer network, it's not inconceivable that you may end up with more parameters condensed into a single hidden layer with very high dimensional gradients. In terms of implementation, this isn't desirable.

As far as I know using a complex model to train a simpler one has been done. Probably by Schmidhuber. I'm not sure they still do it.

See also One Model To Learn Them All (2017) by Lukasz Kaiser et al.


text:
What are examples of resources that describe the basics of Spiking Neural Networks in detail?
I'm very interested in writing a Spiking Neural Network engine (SNN) from scratch, but I can't find the basic information I need to get started.
For example, I've seen pictures of the individual signals that combine to form a neuron pulse in several research papers, with no information on the equations in use.  It's not the focus of the papers, and the authors assume the readers have that knowledge already.  Some papers reference software that provides this foundation (NEST, pyNN, etc.), but the documentation for the software is similarly light on details.
There is a ton of information out there on the more common network types, but SNNs have not yet made it into the mainstream.
So, where do I get this basic information? Has someone pulled together any recipes/examples/tutorials for an SNN, as has been done with all the other network types?

There are so many different versions of spiking neural networks out there. I think it is mainly due to the fact that there has been no dominant successful SNN model with proper learning algorithm like CNN with BP. However, there have been several recent papers(e.g. SuperSpike, SLAYER) on SNN that may lead to the standard framework for SNN. It happened within about 2 years, so it is one of the reasons why there have been no friendly introductions on SNN so far. I found some blogposts on SNN in general sense, but it didn't cover recent important trends in SNN.
Currently, the best way to learn about SNN is by reading papers. One paper that I would recommend is "Surrogate Gradient Learning in Spiking Neural Networks" which comprehensively reviews recent works on supervised learning in SNN with back-propagation.
If you want to implement the SNN from scratch, I would recommend you to checkout BindsNET(github, paper) which is an SNN framework based on PyTorch. To me, it was most intuitive to use and understand compared to other existing SNN libraries. It covers various neuron models and learning rules. But I'm not sure whether it also covers learning rules described in "Surrogate Gradient Learning in Spiking Neural Networks"

There is a book Spiking Neuron Models: Single Neurons, Populations, Plasticity (2002) by Wulfram Gerstner and Werner M. Kistler.
I also found these slides that could be useful.
(Additional resources would be appreciated.)


text:
Where to publish a first article in Deep Reinforcement Learning?
What would be examples of journals that are good for a first publication in the field of Deep Reinforcement Learning?
I am in the process of writing about the research results of DQN-related algorithms.
I have 3 requirements - it should be indexed in one of these databases, otherwise, I cannot receive grant money for research:

https://www.scopus.com/
http://webofknowledge.com

And it should not be very expensive to publish. It should be under 1000EUR to publish, for example, the Open Access license for Elsevier "Artificial Intelligence" journal costs around 2400EUR to publish.
And it should not have a very long review/publishing period. For example, Elsevier's "Information Fusion" journal currently gathers articles for July 2018, which is 8 month period till publishing. Is it normal?
Can you please recommend some journals that qualify & you have had good experience publishing research?

One important consideration here: in the last decade or two the machine learning and artificial intelligence fields, which contains the majority of reinforcement learning work, researchers have considered conferences to be the more impactful publishing venues than journals. The particular venue a researcher chooses depends on the data and/or application domain of his or her use of reinforcement learning, and the conferences are often changing, but to get you started the top tier conferences are (in rough order of exclusivity and importance):

ICML
NeurIPS
ICLR
AAAI
IJCAI
RLDM
UAI (more theory driven)
AISTATS (more theory driven)
COLT (more theory driven)
CoRL (robotics focus)
RSS (robotics focus)
ICRA (robotics focus)
IROS (robotics focus)
CVPR (visual data focus)
ECCV/ICCV (visual data focus)
EMNLP (language focus)
ACL (language focused)

I recommend you focus on quality over quantity. Publishing a paper will boost your reputation and make you more recognised within your academic field (AI); however, this is only if the paper provides useful insights into an important issue.
Your paper is more likely to be accepted if it is well written and easy to understand, stimulates new important questions, uses rigorous methods to explain why the data supports the conclusion and connections to prior work is made and serve to make your paper's arguments clear. (Elizabeth Z Elsevier blog)
Before submitting your paper, ask a mentor or a colleague to proofread it, so that you can make the relevant revisions and changes. Journal editors will look down on your work if it is poorly written or contains substandard grammar.
A way to get published is by writing reviews, especially for researchers in earlier stages of their careers. Most journal editors like to publish replies to previous publications since it stimulates debate.
Remember it is acceptable to challenge reviewers' suggestions with good justification. Many researchers fail to persevere when they are instructed to revise and resubmit their work. Don't give up, however, you can politely decline or even argue why a reviewer is wrong. Editors will accept a rational explanation if it is clear that you have considered all their feedback.
Getting published is never easy, especially in high ranking journals. If you focus on getting published quickly it could derail you from concentrating on the quality of your research. Yes, getting published can be expensive, however, it's much better for your career if you write a high-quality paper than a low-quality paper in a lowly ranked or ungraded journal since it will not be REFable.
Below is a list of Artificial Intelligence Journals that you can submit your papers to and possibly get published.

Artificial Intelligence (Journal) https://www.journals.elsevier.com/artificial-intelligence

Artificial Intelligence Review  https://link.springer.com/journal/10462

Applied Artificial Intelligence http://www.tandfonline.com/action/journalInformation?journalCode=uaai20&

Autonomous Agents and Multi-Agent Systems  https://www.springer.com/computer/ai/journal/10458

International Journal on Artificial Intelligence Tools http://www.worldscientific.com/worldscinet/ijait

International Journal of Pattern Recognition and Artificial Intelligence http://www.worldscientific.com/worldscinet/ijprai

Journal of Experimental and Theoretical Artificial Intelligence  http://www.tandfonline.com/action/journalInformation?journalCode=teta20

Journal of Machine Learning Research  http://www.jmlr.org/

IEEE Intelligent Systems  http://publications.computer.org/intelligent-systems/

Journal of Automated Reasoning  https://www.springer.com/computer/theoretical+computer+science/journal/10817

Minds and Machines  https://www.springer.com/computer/ai/journal/11023

Autonomous Agents and Multi-Agent Systems  https://www.springer.com/computer/ai/journal/10458

Applied Intelligence  https://link.springer.com/journal/10489

Journal of Intelligent Manufacturing  https://link.springer.com/journal/10845

Journal of Intelligent Information Systems   https://link.springer.com/journal/10844

AI & Society  https://link.springer.com/journal/146


text:
What's the difference between model-free and model-based reinforcement learning?
What's the difference between model-free and model-based reinforcement learning?
It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In that case, when would model-free learners be appropriate?

How do you mean that you could reframe a model-free learner as a model-based?

What's the difference between model-free and model-based reinforcement learning?

In Reinforcement Learning, the terms "model-based" and "model-free" do not refer to the use of a neural network or other statistical learning model to predict values, or even to predict next state (although the latter may be used as part of a model-based algorithm and be called a "model" regardless of whether the algorithm is model-based or model-free).
Instead, the term refers strictly as to whether, whilst during learning or acting, the agent uses predictions of the environment response. The agent can use a single prediction from the model of next reward and next state (a sample), or it can ask the model for the expected next reward, or the full distribution of next states and next rewards. These predictions can be provided entirely outside of the learning agent - e.g. by computer code that understands the rules of a dice or board game. Or they can be learned by the agent, in which case they will be approximate.
Just because there is a model of the environment implemented, does not mean that a RL agent is "model-based". To qualify as "model-based", the learning algorithms have to explicitly reference the model:

Algorithms that purely sample from experience such as Monte Carlo Control, SARSA, Q-learning, Actor-Critic are "model free" RL algorithms. They rely on real samples from the environment and never use generated predictions of next state and next reward to alter behaviour (although they might sample from experience memory, which is close to being a model).

The archetypical model-based algorithms are Dynamic Programming (Policy Iteration and Value Iteration) - these all use the model's predictions or distributions of next state and reward in order to calculate optimal actions. Specifically in Dynamic Programming, the model must provide state transition probabilities, and expected reward from any state, action pair. Note this is rarely a learned model.

Basic TD learning, using state values only, must also be model-based in order to work as a control system and pick actions. In order to pick the best action, it needs to query a model that predicts what will happen on each action, and implement a policy like $\pi(s) = \text{argmax}_a \sum_{s',r} p(s',r|s,a)(r + v(s'))$ where $p(s',r|s,a)$ is the probability of receiving reward $r$ and next state $s'$ when taking action $a$ in state $s$. That function $p(s',r|s,a)$ is essentially the model.

The RL literature differentiates between "model" as a model of the environment for "model-based" and "model-free" learning, and use of statistical learners, such as neural networks.
In RL, neural networks are often employed to learn and generalise value functions, such as the Q value which predicts total return (sum of discounted rewards) given a state and action pair. Such a trained neural network is often called a "model" in e.g. supervised learning. However, in RL literature, you will see the term "function approximator" used for such a network to avoid ambiguity.

It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based.

I think here you are using the general understanding of the word "model" to include any structure that makes useful predictions. That would apply to e.g. table of Q values in SARSA.
However, as explained above, that's not how the term is used in RL. So although your understanding that RL builds useful internal representations is correct, you are not technically correct that this can be used to re-frame between "model-free" as "model-based", because those terms have a very specific meaning in RL.

In that case, when would model-free learners be appropriate?

Generally with current state of art in RL, if you don't have an accurate model provided as part of the problem definition, then model-free approaches are often superior.
There is lots of interest in agents that build predictive models of the environment, and doing so as a "side effect" (whilst still being a model-free algorithm) can still be useful - it may regularise a neural network or help discover key predictive features that can also be used in policy or value networks. However, model-based agents that learn their own models for planning have a problem that inaccuracy in these models can cause instability (the inaccuracies multiply the further into the future the agent looks). Some promising inroads are being made using imagination-based agents and/or mechanisms for deciding when and how much to trust the learned model during planning.
Right now (in 2018), if you have a real-world problem in an environment without an explicit known model at the start, then the safest bet is to use a model-free approach such as DQN or A3C. That may change as the field is moving fast and new more complex architectures could well be the norm in a few years.

A small correction, normally the terms "model based" or "model free" are not used for planning algorithms such as MCTS. It is only used to classify learning algorithms.

@MiguelSaraiva: I'm not 100% certain about that, but have removed the reference to MCTS. Out of interest, where would you place DynaQ regarding this limitation of the usage of the terms? I think it becomes tricky, when the algorithms all share such a common view of the MDP model and improving policies, to tell where the boundaris are between planning and learning.

I could be wrong, I'm a novice in the area. I just remember a teacher from the field making that comment after I had done the same remark.

@NeilSlater, what would you recommend for an environment which is fully observable, but also gives back a model of it -- the exact reward and transition functions are known. I think there could be a benefit in using something beyond  DQN or A3C to take advantage of that knowledge. I cannot use dynamic programming since state-space is too large. thanks

@d56: I think that could be a new question on the site. In brief, you can still use most RL methods, and add planning to refine estimates of return values.

Thank you for informative answer! I want to clarify one thing, for model-free algorithms you've said they "never use generated predictions of next state and next reward to alter behaviour" however in Q-learning agent may take greedy action(isn't this generated?) or random action even though it updates Q-value assuming agent took greedy action, isn't greedy action generated since it is not actually taken by the agent?

@haneulkim It is the prediction of next reward and state that are part of a model of the environment. The action is treated separately, using a policy of some kind, and yes all learning agents must have a method to do that somehow. The difference is important: An agent can control its action choice, but cannot control any other aspect of the environment rules. When looking at the difference between action taken (exploring behaviour policy) and the value update (greedy target policy), it doesn't matter whether you consider this "generated", it has nothing to do with the model of the environment

@NeilSlater You say that "if you have a real-world problem in an environment without an explicit known model at the start, then the safest bet is to use a model-free approach such as ". You mean that you should use real world simulations?  usually, when using a simulation even if the "learning agent" does not use the model, the model is necessary -and present-in order to excute the script and study the algorithm performance. So, apparently it is correct to say that in real world problem not only the "learning algorithm" is the problem.

Continuation: You also need to construct a correct "real world simulation", except if you can experiment in the real world.  In this sense, RL is not model free. Am I wright?

@HermesMorales There is a complex relationship between models, simulation and planning, in terms of when you might consider that you are using one or the other. One important detail is whether you have a sampling model or a distribution model. Model-free methods are often paired with simulations which are effectively sampling models. If the end goal is to then use the trained agent in a real environment, you might consider the training session as planning

Other sources suggest TD-learning is model-free though.

@Sam: Yes TD learning is itself model-free. It doesn't require any model. However, it is often used in situations where there is a model in the combined system. E.g. DynaQ.

@NeilSlater so is the statement 'Basic TD learning, using state values only, must also be model-based in order to work as a control system and pick actions.' still correct (I was confused by it)?

@Sam - the learning system in that case must be model-based, yes. Without a model, TD learning using state values cannot make decisions. You cannot run value-based TD learning in a control scenario otehrwise, which is why you would typically use SARSA or Q learning (which are TD learning on action values) if you want a model-free TD learner. TD on state values still works model-free in predicion scenarios though.

Model-based reinforcement learning has an agent try to understand the world and create a model to represent it. Here the model is trying to capture 2 functions, the transition function from states $T$ and the reward function $R$. From this model, the agent has a reference and can plan accordingly.
However, it is not necessary to learn a model, and the agent can instead learn a policy directly using algorithms like Q-learning or policy gradient.
A simple check to see if an RL algorithm is model-based or model-free is:
If, after learning, the agent can make predictions about what the next state and reward will be before it takes each action, it's a model-based RL algorithm.
If it can't, then it’s a model-free algorithm.

Comments are not for extended discussion; this conversation has been moved to chat.

In reinforcement learning (RL), there is an agent which interacts with an environment (in time steps). At each time step, the agent decides and executes an action, $a$, on an environment, and the environment responds to the agent by moving from the current state (of the environment), $s$, to the next state (of the environment), $s'$, and by emitting a scalar signal, called the reward, $r$.  In principle, this interaction can continue forever or until e.g. the agent dies.
The main goal of the agent is to collect the largest amount of reward "in the long run". To do that, the agent needs to find an optimal policy (roughly, the optimal strategy to behave in the environment). In general, a policy is a function which, given a current state of the environment, outputs an action (or a probability distribution over actions, if the policy is stochastic) to execute in the environment. A policy can thus be thought of as the "strategy" used by the agent to behave in this environment. An optimal policy (for a given environment) is a policy which, if followed, will make the agent collect the largest amount of reward in the long run (which is the goal of the agent). In RL, we are thus interested in finding optimal policies.
The environment can be deterministic (that is, roughly, the same action in the same state leads to the same next state, for all time steps) or stochastic (or non-deterministic), that is, if the agent takes an action in a certain state, the resulting next state of the environment might not necessarily always be the same: there is a probability that it will be a certain state or another. Of course, these uncertainties will make the task of finding the optimal policy harder.
In RL, the problem is often mathematically formulated as a Markov decision process (MDP). A MDP is a way of representing the "dynamics" of the environment, that is, the way the environment will react to the possible actions the agent might take, at a given state. More precisely, an MDP is equipped with a transition function (or "transition model"), which is a function that, given the current state of the environment and an action (that the agent might take), outputs a probability of moving to any of the next states. A reward function is also associated with an MDP. Intuitively, the reward function outputs a reward, given the current state of the environment (and, possibly, an action taken by the agent and the next state of the environment). Collectively, the transition and reward functions are often called the model of environment. To conclude, the MDP is the problem and the solution to the problem is a policy. Furthermore, the "dynamics" of the environment are governed by the transition and reward functions (that is, the "model").
However, we often do not have the MDP, that is, we do not have the transition and reward functions (of the MDP associated the environment). Hence, we cannot estimate a policy from the MDP, because it is unknown. Note that, in general, if we had the transition and reward functions of the MDP associated with the environment, we could exploit them and retrieve an optimal policy (using dynamic programming algorithms).
In the absence of these functions (that is, when the MDP is unknown), to estimate the optimal policy, the agent needs to interact with environment and observe the responses of the environment. This is often referred to as the "reinforcement learning problem", because the agent will need to estimate a policy by reinforcing its beliefs about the dynamics of the environment. Over time, the agent starts to understand how the environment responds to its actions, and it can thus start to estimate the optimal policy. Thus, in the RL problem, the agent estimates the optimal policy to behave in an unknown (or partially known) environment by interacting with it (using a "trial-and-error" approach).
In this context, a model-based algorithm is an algorithm that uses the transition function (and the reward function) in order to estimate the optimal policy. The agent might have access only to an approximation of the transition function and reward functions, which can be learned by the agent while it interacts with the environment or it can be given to the agent (e.g. by another agent). In general, in a model-based algorithm, the agent can potentially predict the dynamics of the environment (during or after the learning phase), because it has an estimate of the transition function (and reward function). However, note that the transition and reward functions that the agent uses in order to improve its estimate of the optimal policy might just be approximations of the "true" functions. Hence, the optimal policy might never be found (because of these approximations).
A model-free algorithm is an algorithm that estimates the optimal policy without using or estimating the dynamics (transition and reward functions) of the environment. In practice, a model-free algorithm either estimates a "value function" or the "policy" directly from experience (that is, the interaction between the agent and environment), without using neither the transition function nor the reward function. A value function can be thought of as a function which evaluates a state (or an action taken in a state), for all states. From this value function, a policy can then be derived.
In practice, one way to distinguish between model-based or model-free algorithms is to look at the algorithms and see if they use the transition or reward function.
For instance, let's look at the main update rule in the Q-learning algorithm:
$$Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha (R_{t+1} + \gamma \max_{a}Q(S_{t+1}, a) - Q(S_t, A_t))$$
As we can see, this update rule does not use any probabilities defined by the MDP. Note: $R_{t+1}$ is just the reward that is obtained at the next time step (after taking the action), but it is not necessarily known beforehand. So, Q-learning is a model-free algorithm.
Now, let's look at the main update rule of the policy improvement algorithm:
$$Q(s,a) \leftarrow  \sum_{s' \in \mathcal{S}, r\in\mathcal{R}}p(s',r|s,a)(r+\gamma V(s'))$$
We can immediately observe it uses $p(s',r|s,a)$, a probability defined by the MDP model. So, policy iteration (a dynamic programming algorithm), which uses the policy improvement algorithm, is a model-based algorithm.

Already upvoted. Very well explained sir. Just a confusion, that why can't we just learn a reward and transition function associated with environment and then find optimal policy by dynamic learning. And if we are given this environment model by another agent then why can't we just apply dynamic learning to find optimal policy?

@user_3pij In principle, one can do what you are suggesting. However, as I say in my answer, if you first need to estimate the transition and reward functions, then you necessarily introduce more approximations in your calculations, which may not be desirable.

When you are comparing the equations of update rules, I don't understand why the model-free still uses $R_{t+1}$. You explained that this is not necessarily known beforehand, but then how is it eventually known? What are the starting conditions of this reward if we don't have a model?

@mesllo What I mean by "not known beforehand" is that $R_{t+1}$ is only known after you have taken the action $A_t$. $R_{t+1}$ is determined by the reward function, if $p(s', r \mid s, a)$ is not known. Note that the environment still behaves according to $p$ and the reward function, but $p$ may not be known.

Although there are several good answers, I want to add this paragraph from Reinforcement Learning: An Introduction, page 303, for a more psychological view on the difference.

The distinction between model-free and model-based reinforcement learning algorithms
  corresponds to the distinction psychologists make between habitual and goal-directed
  control of learned behavioral patterns. Habits are behavior patterns triggered by appropriate
  stimuli and then performed more-or-less automatically. Goal-directed behavior,
  according to how psychologists use the phrase, is purposeful in the sense that it is controlled
  by knowledge of the value of goals and the relationship between actions and their
  consequences. Habits are sometimes said to be controlled by antecedent stimuli, whereas
  goal-directed behavior is said to be controlled by its consequences (Dickinson, 1980,
  1985). Goal-directed control has the advantage that it can rapidly change an animal’s
  behavior when the environment changes its way of reacting to the animal’s actions. While
  habitual behavior responds quickly to input from an accustomed environment, it is unable
  to quickly adjust to changes in the environment.

It keeps going from there, and has a nice example afterwards.
I think the main point that was not always explained in the other answers, is that in a model-free approach you still need some kind of environment to tell you what is the reward associated with your action. The big difference is that you do NOT need to store any information about the model. You give the environment your chosen action, you update your estimated policy, and you forget about it. On the other hand, in model-based approaches, you either need to know the state transitions history as in Dynamic Programming, or you need to be able to calculate all possible next states and associated rewards, from the present state.

Model-Free RL
In Model-Free RL, the agent does not have access to a model of the environment. By environment I mean a function which predicts state transition and rewards.
As of the time of writing, model-free methods are more popular and have been researched extensively.
Model-Based RL
In Model-Based RL, the agent has access to a model of the environment.
Main advantage is that this allows the agent to plan ahead by thinking ahead. Agents distill the results from planning ahead into a learned policy. A famous example of Model-Based RL is AlphaZero.
The main downside is that many times a ground-truth representation of the environment is not usually available.

Below is a non-exhaustive taxonomy of RL algorithms, which may help you to visualize better the RL landscape.

Image is from: https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html
Please credit sources.

According to OpenAI – Kinds of RL Algorithms, algorithms which use a model of the environment, i.e. a function which predicts state transitions and rewards, are called model-based methods, and those that don’t are called model-free. This model can either have been given the agent or learned by the agent.
Using a model allows the agent to plan by thinking ahead, seeing what would happen for a range of possible choices, and explicitly deciding between its options. This may be useful when faced with problems that require more long-term thinking. One way to perform planning is by using some kind of tree search, for example Monte Carlo tree search (MCTS), or—which I suspect could also be used—variants of the rapidly exploring random tree (RRT). See e.g. Agents that imagine and plan.
The agent can then distill the results from planning ahead into a learned policy – this is known as expert iteration.
A model can also be used to create a simulated, or "imagined," environment in which the state is updated by using the model, and make the agent learn inside of that environment, such as in World Models.
In many real-world scenarios, the ground-truth model of the environment is not available to the agent. If an agent wants to use a model in this case, it has to learn the model, which can be challenging for several reasons.
There are however cases in which the agent uses a model that is already known and consequently doesn't have to learn the model, such as in AlphaZero, where the model comes in form of the rules of the game.


text:
How can artificial intelligence techniques be used in diagnosing mental health issues?
I've seen events, like CogX, and articles that describe how machine learning techniques or algorithms can be used to diagnose mental health issues.
Here's my question.
How can artificial intelligence and machine learning algorithms or techniques be used in diagnosing mental health issues, besides, for example, Facebook using machine learning algorithms to detect people who may commit suicide?

Check this paper http://ieeexplore.ieee.org/document/7293425/?reload=true

Try to go through this paper for your benefit https://thesai.org/Downloads/Volume7No1/Paper_76-Prediction_of_Mental_Health_Problems_Among_Children.pdf

just a comment, but i think this is a very bad idea. mental health issues are very complex and our understanding of it is very limited

@k.c.sayz'k.csayz' that's a fair point, but there are commonalities within disorders that would likely be identifiable for an advanced AI.  (See my answer on the successful results of the application of Machine Learning in predicting suicide.)

There's a strong sentiment towards the idea that medical diagnosis is largely Abductive Reasoning. See this presentation for additional details.  One approach to automated abductive reasoning is parsimonious covering theory.
If you want a relatively in-depth look at all of this, check out the book Abductive Inference Models for Diagnostic Problem-Solving .
Expert systems have also been shown to work very well for medical diagnosis. As far back as the 1970's, systems like MYCIN could beat human experts in terms of diagnosis and treatment plans.
From what I can tell, there doesn't seem to be any reason in principle to think that AI/ML can't be used across the board for medical diagnosis, mental health or otherwise.

Thanks for the historical perspective!  (See my answer for recent ML application)

It's already being done, and apparently with very good results.
See: Predicting Risk of Suicide Attempts Over Time Through Machine Learning, Walsh, Ribiero, Franklin
Here is the abstract from the paper:

Traditional approaches to the prediction of suicide attempts have limited the accuracy and scale of risk detection for these dangerous behaviors. We sought to overcome these limitations by applying machine learning to electronic health records within a large medical database. Participants were 5,167 adult patients with a claim code for self-injury (i.e., ICD-9, E95x); expert review of records determined that 3,250 patients made a suicide attempt (i.e., cases), and 1,917 patients engaged in self-injury that was nonsuicidal, accidental, or nonverifiable (i.e., controls). We developed machine learning algorithms that accurately predicted future suicide attempts (AUC = 0.84, precision = 0.79, recall = 0.95, Brier score = 0.14). Moreover, accuracy improved from 720 days to 7 days before the suicide attempt, and predictor importance shifted across time. These findings represent a step toward accurate and scalable risk detection and provide insight into how suicide attempt risk shifts over time.

This has gotten national press attention quite recently.  Here are a couple of prior articles on the result and endeavor:
Artificial Intelligence is Learning to Predict and Prevent Suicide (Wired)
Artificial intelligence can now predict suicide with remarkable accuracy (Quartz)
Is strongly suspect ML can be used in an array of application related to human mental health.  For instance, it is entirely possible a app would be able to discern the mood swings in people with bipolar disorder based on activity or lack thereof.

You may infer some mental problems using text processing. Any text written by an individual contains a lot of clues about their mental state, probably including any health problems.
However, I believe pursuing this would prove to be difficult due to lack of ground truth. Not many people makes it public that they have mental problems and without a reliable database it is quite difficult to make research on the field.


text:
Is there a rigorous proof that AGI is possible, at least, in theory?
It is often implicitly assumed in computer science that the human mind, or at least some mechanical calculations that humans perform (see the Church-Turing thesis), can be replicated with a Turing machine, therefore Artificial General Intelligence (AGI), defined as a human-like AI, may be possible.
I do not know of any other argument that AGI is possible, and the foregoing argument is extremely weak.
Is there a rigorous proof that AGI is possible, at least, in theory? How do we know that everything the human mind can do can be encoded as a program?

It seems you're asking if it is possible to develop AI that operates at a similar level to humans (terms like "Artificial general intelligence" or "Strong AI" are often used to describe this). It's a fabulous question, but it is poorly defined. It begs another question: how do you define human-level intelligence? Is it the ability to convince other human agents that you're human (this is somewhat circular logic)? Is it the ability to write music or create a painting? Depending on your definition, the answer varies wildly, and it must be clarified before proceeding.

A strong reason why people think the mind can be implemented on a Turing Machine stems from the Computational Theory of Mind (CTOM), which is the leading theory of mind for now.
There are lots of reasons for supporting the CTOM, one of which being that the language of belief/desire psychology (propositional attitudes over mental representations) seems to fit nicely to a computational framework.
But most simply is that the computation analogy is very helpful in fields such as psychology and neuroscience. When we know of an input/output pair, but don't know how it is implemented, we could say "its performing the relevant computation".
And since Turing showed that any computation can be performed on an appropriate Turing Machine, the natural extension is that the mind can be implemented on a computer.
However, the CTOM is more of a useful idea than a complete theory. We still don't know how to analyze thought in a logical syntax, which can be implemented in a computer. And we also don't know how/why "computation" (whatever that means in this sentence) is performed in the brain.

Comments are not for extended discussion; this conversation has been moved to chat.

4 years later: Turing did /not/ show that "any computation can be performed on a Turing machine", but that "Turing machines can compute a very large class of functions, beyond which we do not know whether exists functions that are not computable by a Turing machine." this is known as the Church-Turing thesis (that TMs accurately describe what we know as "computation"). check out the Busy Beaver function for a negative example (which fails (i assume) due to being defined as an implicit function)

Consciousness is not well-understood
As an AI practitioner and philosopher, I don't think that humans will be able to create a truly conscious silicon-based AGI.

Humans are incapable of creating some "thing" from fiat (a decree). It's never happened in human history. The innovation cycle must begin with some "thing" (some "stuff" of some kind), and consciousness is not a thing.

The essence of consciousness is imperceptible (it is unseen), like gravity, and attraction. Humans are incapable of creating things that they are unable to observe. Even if they are able to observe it, the human perceptive ability is unable to actually perceive the true essences of things seen, much less those unseen.

Humans do not adequately understand the "essence" and "nature" of consciousness - which is a fundamental prerequisite to creating "anything" at all.

The "easy" problems, those physical by nature, although not yet solved by empirical domains of psychology, cognitive science, and neuroscience, are expected to be solved in time. Regardless, they are "not" yet solved today.

The "hard" problems, those determining why or how consciousness occurs given the right arrangement of brain matter, might not ever be solved, since it must explain why certain physical mechanism gives rise to consciousness instead of "something else" or "nothing at all". This is significant and is the most damning of all arguments against the idea of humans creating true existential consciousness in silicon creatures as a whole.

Dualism vs physicalism
The greatest philosophical debate on consciousness has focused on the distinction between dualism and physicalism.

Dualism is the theory that consciousness somehow falls outside the domain of the physical (these are the hard problems)

Physicalism holds that consciousness is entirely physical. (significant arguments below view it as false).

Problems with dualist views

Why would one be motivated to hold a dualist view?

How can something that is not part of the physical world interact with the physical world? That seems impossible!

The physical world is a closed system, how can you have a consciousness that is not part of a closed system?

Consciousness is a lot like mass or charges, it's a philosophically "fundamental" thing, you either "have it or you don't", you can simulate them, but you cannot existentially "be" them unless you have those specific "properties", and behavior "simulating" human consciousness is not a fundamental thing.
So, despite the sensationalist tendencies of rogue journalists "parroting" wildly spectacular concepts from the fringe camps of the transhumanists (aka science fiction) - a quick perusal of the more rigorous communities of the grounded and thoughtful philosophers camp strongly and convincingly argues otherwise.
More musings on physicalism
Actually, consciousness has never been properly explained by the biomechanical, which is more or less the key issue of all philosophical studies of the mind - which is essentially the study of consciousness.
Physicalists have trouble explaining several aspects of consciousness in a way that is consistent with our "observations" of how physical properties interact. Let me list a few more problems, with a reference to the titans of philosophy.
Arguments against physicalism

It is impossible to imagine how mere neuronal tissue could produce conscious experience (Huxly)

Failures of supervenience, such as zombies and inverted spectra, are conceivable (David Chalmers, John Locke, etc.).

Mary learns something (Frank Jackson).

Brains have mass, volume, and other physical properties, but experiences do not.

Paranormal phenomena (near-death experiences NDErs, ESP, etc) are real, and involve consciousness implemented in a nonphysical substrate.

If shrunken so I can stroll around your brain and look about, I will observe neuronal processes, not experiences (G. W. Leibniz).

The soul is the seat of consciousness, and the soul is not physical. (Theological constraints recognized BTW...).

Conscious experiences have intrinsic qualities, but science can only tell us about relational qualities (Russell, Rosenberg).

Consciousness cannot be observed; there will never be a consciousness detector that can tell you if a given creature is conscious.

Conscious experiences are not simply the movement of molecules, consciousness is more than mass in motion (Mill, Ward).

I edited this interesting answer to structure/format it properly and to include links to some relevant references that you mention. However, I have not included links to all references, so you may want to finish the work ;)

I'm going to go out on a limb and suggest that this is a matter of evolution, that humans are in no way exceptional in the grand scheme, and that AGI will manifest so long as technology advances, because human consciousness is simply a matter of complexity of the system.
The idea comes out of emergent complexity in Conway's Game of Life. In Conway's words:

"There are Life patterns which behave like self-replicating animals… It’s probable, given a large enough Life space, initially in a random state, that after a long time, intelligent self-replicating animals will emerge and populate some parts of the space.” Source: Winning Ways for Your Mathematical Plays

I came across a paper Computation in Cellular Automata:
A Selected Review, which I am still working my way through, and which you may find interesting.

For those who use philosophical arguments to make the case algorithmic consciousness is not possible, I'd posit the question "how do we know we're conscious?", not because I'm interested in the answer, but merely to throw a wrench into that line of inquiry.
Because ultimately it doesn't matter.
Consciousness in the sense of human awareness is not a requirement of life, and the most basic definition of consciousness is awareness of any kind, no matter how trivial.
I find the idea that there is something "magical" about human consciousness, that ideas are not things because they do not have material form, to be problematic.
Intangibility I don't have a problem with, as intangibles clearly interact with the physical world.
(As an analogy, I studied for many years with a famous Tai Chi teacher who never talked about "chi".  I suspect this disinclination derived from the way in which the concept of "chi" leads to magical thinking, which is illusory as opposed to practical. The practice and application of Tai Chi techniques is purely a matter of physics and physiology, even when such applications seem to defy natural laws. Possibly there is something going on that we don't understand, but if that were the case, such phenomena are natural in origin.)
We know there is randomness in nature at the quantum level, and if this proves to be a component of human consciousness, we can use quantum computing to provide a medium for artificial consciousness.

Comments are not for extended discussion; this conversation has been moved to chat.

Rather than prove that Artificial General Intelligence is possible, I would consider an argument for why it is impossible.
We start by defining what we mean by AGI. You state that the human mind can be replicated by a Turing Machine, and therefore AGI should be possible. This seems to imply that humans have `General' (capital G) intelligence. By this I mean that you are implying that with enough time, humans can learn any task or problem. However, if you are asserting that humans minds are machines replicable by Turing machines, you must also concede that they have some finite representational power. Finite representational power implies that there will always be problems or tasks where our intelligence will fail (a consequence of the No Free Lunch Theorem).
Fortunately (maybe unfortunately), finite representational power is what allows us to learn at all: VC Dimension (a measure of the complexity or representational power of a class of functions that a learning algorithm can learn [also here and here]) implies that a learning algorithm that can learn any problem is actually useless, as the ability to explain any set of data yields the requirement that the algorithm see an infinite amount of examples in order to generalize. While this result comes from the relatively constrained class of binary classification problems in the statistical learning setting, the intuition seems to apply more broadly.
To summarize, I would refer to this quote from Shalev-Shwartz and Ben-David (2014):

If someone can explain every phenomenon, his explanations are worthless.

It truly is the case that our decision to systematically ignore some possible outcomes is the only thing that allows use to learn useful representations of real-world problems.

just seeing this now, I really like this answer.

Although it is not a rigorous proof, Marvin Minsky's book, The Society of Mind gives us a blueprint for creating a "mind" (general intelligence).  In his book, he posits that by combining mindless components ("agents") together in various competing and cooperative structures, we can create actual minds.
IMHO, the recent popularity of Boosting, Bagging, Stacking, and other ensemble techniques will eventually evolve (through research) into Marvin Minsky's "agent" metaphor. Subsequently, as we learn to make these agents compete and cooperate (looks like this has recently begun with Generative Adversarial Networks), we will be able to write "programs" that mimic (or surpass) the human mind.


text:
Can a second network take as input the weights of a first network and help training the first network?
I understand that as a network learns about an output with regards to an input, weights are updated according to how wrong the guess was for that node. So, over time, the weights move in the "direction" towards the correct value.
Is it possible to use a separate neural network that takes as input the weights of the first network while it trains to try approximating that "direction" and, in effect, pushing the weights in that direction faster?

That is similar to a chicken-and-egg problem; If the second network could figure out the direction and learn faster, why couldn't the first network do that too and learn even faster.
However, I would recommend looking up Actor Critic Methods. Here there are two networks but one (the critic) evaluates the other (the actor) to head it in the right direction.


text:
Are machine learning and self-learning really possible?
Some AI's, such as some chess players, are extremely well coded and have defeated humans in several matches. But I think that they won simply because computers can make calculations way faster than humans can not because they learned from their opponents.
If you put an AI against itself, who will win? Will the game continue indefinitely or will the game eventually finish because the AI plays randomly?
So, are machine learning and self-learning really possible?

You're asking many questions here, which makes this post too broad.

Yes of course they are possible, I have built some! You are correct that there is an aspect of randomness to the process of machine learning but it is more accurate to describe this as trial and error. Each successive try in a machine learning system is evaluated against a goal and if it is an improvement or is closer to the goal, then this try is stored and some aspect that made it successful is incorporated into similar trys for this type of input. Therefore, machines learn by trying all possible combinations of a problem, albeit with some clever human described short cuts or "heuristics" to make the task easier.

But will machines learn by trying all possible combinations ?

@BrnPer There are different types of AI with different learning mechanisms. If a problem can be fully analyzed and all possible combinations considered (take TicTacToe for example), the AI can play in a perfect way. Most problems in modern AI research are not of that nature and therefore the AIs must learn to judge situations without knowing all possibilities.

Machine learning and self-learning are of course possible, and there're many successive cases!
You need to know this: machines won't think like humans. Machines form a statistical model and calibrate the model. A good model is a model that does what it's supposed to do accurately.

But will statistical models predict every decision? Will humans give all the models machine need or will machine make their own models? Because if it's self-learning, it can learn by itself...its some questions I've. Sorry if I wasn't  clear enough

I like your focus on optimization (re: "One of my productive days was throwing away 1,000 lines of code";)
I think the problem with this question is it your supposition is incorrect. Check out Giraffe Chess for more info on self-learning. [Note that Giraffe Chess is a result, not a hypothesis, and Lai was subsequently tapped by DeepMind.]
I'd also recommend getting familiar with concepts like non-deterministic polynomial time and obsessing over intractability.
Regarding random choices, "monte carlo" has had major successes of late, but I suspect the success is related to processor speed, and may not be sufficient in greater complexity spaces, or problems where rationality is heavily bounded.


text:
How could an alien probe learn the basics of a language with only broadcasting signals?
While conducting research, I recently stumbled upon the deep learning and natural language processing concepts. In this question they say that the "grammar induction" is a "supervised learning" mode. So I was wondering:
Let's say that there's a way-more-than-human intelligent alien probe orbiting our planet. It can receive, decode and analyze all the broadcasting signals leaving the Earth.  For all we know at this moment, how could it learn the basics of a language with nothing else but our broadcasting signals, so without the help from a "supervisor"? How would an artificial intelligence human engineer (theoretically?) face the problem?
I’m interested on the more "technical" side of the issue.

Given that even human scientists could not decipher Egyptian until Rosetta Stone was discovered, it does not sound plausible that such an agent could find the meaning. What it could do, however, is to extract grammar to some extent and learn how to predict next word in a given text. So it really depends on what you mean by "the basics of a language".

Not sure what is the level of abstraction that you use in your novel, but my thoughts about aliens interpreting our languages by capturing the telecommunication signals is far more complicated than that. They should figure out that the analog waves actually contain digital information, understand the bytecode, filter out the overhead, keep the text information among images, sound etc., and then maybe we can talk about approaches to learn the language.

@Tasos: let's say that the decoding is not a trouble at all. Leitasat get the point. By "basic" I mean the basics to be understood, like a bad english speaking tourist asking for direction.

@Lupetto it's an interesting question.though it sounds a little bit scientifically fictitious.

I think I understand where your confusion comes from, and the reason for your question.
When we talk about a supervised problem, it means that there is some feedback of how well the machine did at a certain task. Of course, we always need this kind of feedback in some way or another (otherwise, the machine would never learn, because it would have no incentive to change anything that it does), but the feedback can be classified into "supervised" if we have a clear way of telling when something was right/wrong, or "unsupervised" if we just roll out with the results and try to make it better each time.
For example, an AI performing translation like in grammar induction in the other question will take as input both the phrase and the syntactic tree. After performing the prediction of what the syntactic tree for the phrase should be, it can be compared against the real syntactic tree given as input and based on the accuracy of the prediction, the weights (think of it as "knobs" that adjust the result) can be tweaked a little bit to give better predictions next time.
This type of learning is considered supervised, not because of the existence of a supervisor, but rather because there is labeled data that we can use to test predictions and better them next time.
Unsupervised problems don't have this kind of labeled data, and just work with what they have. They cannot tell what's right and what's wrong in terms of predictions, they only have raw data and try to make sense out of it by extracting correlations or common properties.

For the case of a language, I will go on a limb here but I'll say that this is mostly likely a task for supervised algorithms. Unsupervised techniques could properly analyze language and determine that they actually have a particular structure, but this is also true for lots of other sources of data that are not language. An alien race, knowing nothing about humanity, could end up deducting that cars honking are some sort of language too, albeit much simpler than human-produced noises.
Also, the natural flexibility that languages have make it extra complex, because they don't follow a correct structure completely, and to make it worse, they are constantly changing.
As leitasat mentioned in their comment, if even human scientists (which we certainly deem intelligent) could not decipher ancient Egyptian on their own without any context, it's very unlikely that a machine would do it.

Finally, notice that translator machines or inference machines don't actually derive meaning in a way that's meaningful. Without going much in much detail, you should think of them as "correlation detection machines" -- so a machine might notice a high enough correlation between how "Hola" is used in Spanish as to how "Hi" is used in English, and by giving you that information, we call it a translator. However, the internal learnt structure is actually just a probability distribution for an output given a set of inputs.
Not to say that any of this is not useful -- it definitely is -- but with something as separate as an alien race of super-human intelligence and without any way to derive meaning from just a bunch of sounds, it's very unlikely that a machine would prove useful in such correlations.
And still, I cannot find any hard stop to your idea -- if these aliens have a language and they express the same concepts that human express, then there is correlation. There just needs to be a way to find it.
Sorry for the wall of text and the huge rambling, I hope I provided some context on the points you were looking clarification for.