text:
What heuristic to use when doing A* search with multiple targets?
Usually, using the Manhattan distance as a heuristic function is enough when we do an A* search with one target. However, it seems like for multiple goals, this is not the most useful way. Which heuristic do we have to use when we have multiple targets?

Is your goal to visit all of the multiple targets, or is the goal simply to reach at least one of the multiple targets?

For readers, please, next time you see a question that needs to be clarified, you should ask for clarification before providing an answer. Meanwhile, you should flag (or vote) the question to be closed. In this case, Dennis was doing the right thing. It's not completely whether the problem is to find the shortest path to multiple targets or to one of the targets. Another answer seems to assume something different. So, this shows that it's very important that we ask for clarification.

By the way, here is a related question.

If by "visit multiple targets", you mean "visit several points in the fastest order", you are no longer in a simple path-finding-style search problem, but instead in an optimization problem. This is roughly the difference between chapters 3 & 6 of Russell & Norvig's section on search.
To do this, you can't just change your heuristic, instead you need to reframe your problem:

Instead of states in your search being locations, they should be tours. Each state is a list of all the places you need to visit, in a specific order.
Instead of actions being movements from one location to another, they need to be transformations from one tour to another. For example, if you swap the order that you visit two adjacent locations, then you'll get a different tour. This gives you a way to "move" between tours.
A solution means visiting all the locations as fast as possible. If you know how to get between two locations, just store the distances, and then sum all the distances together to get the cost of a tour. If you don't know, you can just run A* from each place to each other place once, and then cache the distances afterwards.
A heuristic will depend on your domain. A reasonable start might be to assume that you can visit each location from the nearest other location you've already visited. Generally, heurstics based on the idea of minimum spanning trees are effective for this domain.

The real answer though, is to try a technique that is meant for this kind of problem, like a local search algorithm. Notice that if we know the cost of moving between any two points, we can just adopt a greedy approach: make the move that improves things the most each time. This is often faster than waiting for A* in practice if you just want a good solution, but it doesn't need to be the very best one.

Genetic algorithms are giving promising results for problems with multiple objectives[goals].
http://www.iitk.ac.in/kangal/Deb_NSGA-II.pdf
the above paper will give best algorithm for multi objectives

Welcome to AI.SE Sreedhar! While NSGA-II is a useful algorithm for multi-objective optimization, this question describes multi-goal path planning, which is a different topic. Multi-goal path planning has only one objective (minimize distance), so NSGA-II is not a suitable technique.

I see that the objective is to achieve multiple goals using A* algorithm.
So if your problem is more like a Traveling Salesman Problem, which is what it kind of sounds like, you can refer to this post:
https://stackoverflow.com/questions/4453477/using-a-to-solve-travelling-salesman
The problem can be converted to a graph search problem, and could utilize Minimum Spanning Tree. A-star can be used to compute edge weight.


text:
How do I calculate the policy in the Proximal Policy Optimization algorithm?
I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are calculated.
In the paper Proximal Policy Optimization Algorithms (at page 5), the pseudocode of the PPO algorithm is shown:

It says to run $\pi_{\theta_{\text{old}}}$, compute advantage estimates and optimize the objective. But how can we calculate $\pi_\theta$ for the objective ratio, since we have not updated the $\pi_{\theta_{\text{old}}}$ yet?

You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\theta)={\pi_\theta(a|s)\over\pi_{\theta old}(a|s)}$) by the advantage (so $loss=-r_t(\theta)A_t$).
However, with PPO you run multiple epochs of training on the same data. So after your first update you do the whole thing again (without exploring the environment any more) and this time $\pi_{\theta old}$ is different to $\pi_\theta$.
Here's a great explanation of the algorithm: https://stackoverflow.com/questions/46422845/what-is-the-way-to-understand-proximal-policy-optimization-algorithm-in-rl

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How are "lags" and "exogenous factors" accounted for in reinforcement learning?
In reinforcement learning, the system sets some controllable variables, and then determines the quality of the result of the dependent variable(s); using that "quality" to update the algorithm.
In simple games, this works fine because for each setting there is a single result.
However, for the real world (e.g. an airflow system), the result takes some time to develop and there is no single precise "pair" result to the conditions set.  The flow change takes time and even oscillates a bit as flow stabilizes to a steady-state.
In practical systems, how is this "lag" accounted for?  How are the un-settled (false) results ignored?  How is this noise distinguished from exogenous factors (un-controlled system inputs e.g. an open window exposed to wind)?

One of Reinforcement Learning's core features is the ability to deal with delayed rewards/punishments - i.e. rewards that may occur as a consequence of a decision that occurred multiple time steps ago. That is because the value that it optimises is defined as a long-term sum of immediate rewards. This value is often called the return or utility and although it will take short term rewards into account, they do not have to be the dominant factors, and often are not.
All RL solvers are designed to solve this problem, it is core to the MDP formalism used in RL theory. You can decide to focus on immediate rewards or longer-term ones by adjusting using a discount factor (often represented as $\gamma$ in RL equations). Low discount factors will cause an agent to prefer short-term rewards, high discount factors will cause the agent to prefer for longer-term sums of reward. In continuous problems you must use $\gamma \lt 1$ or instead use average reward as the objective to be maximised.
One classic example of this kind of delay problem is the Mountain Car environment. Here, reward is only granted for reaching a certain point, and it is not possible to reach it by just taking the obvious action of moving directly towards the objective. Most RL algorithms can solve this problem, and get close to optimal solutions, purely from experiencing the final reward and associating it progressively with earlier and earlier decisions.

In practical systems how is this "lag" accounted for?

As above, this is inherent to RL, as opposed to simplified versions such as Contextual Bandits which don't have time steps and evolving state.
In practice, it may take some experimentation to find the best RL algorithm and solve a problem efficiently. Environments with very large delays and lots of noise are harder to solve.
It is also really important to set the reward mechanism for your true goals, and not some intermediate ones. If your goal is a stable state, you need to reward that, and not necessarily maximum throughput. If both throughput and stability are important to the end result, you may need to reward both - e.g. reward throughput, but penalise short-term fluctuations.

How are the un-settled (false) results ignored?

In general handling short-term vs long term effects of actions is known as the assignment problem.
This is most often solved statistically, over long term experience, as the agent associates not just immediate reward with an action, but the long term return after making each decision. Different RL solvers have different mechanisms for effectively moving this association back through time.
The key thing here is that an RL solver learns to maximise expected return or utility from any given state. Working with the expected value allows for smoothing out of variations, provided the agent gains enough experience to approach a statistical mean. Extremely rare and large variations can cause problems with this, and might need to be explicitly coded for as opposed to learned.

How is this noise distinguished from exogenous factors (un-controlled system inputs e.g. an open window exposed to wind)?

This is trickier, and depends more critically on the problem.
A RL-based system that is exposed to these variations whilst learning, and some way to represent their effects in the state variables should learn to pick the best action when these external factors impact it.
RL systems based on maintaining a stable state can sometimes do well when they are moved out of that stable state. A good example here might be pole balancing, where the agent learns enough variations of position and speed that if something knocks the pole out of balance, it will still right itself even though the agent was not trained to react to such events directly.
However, in some cases, if knowledge is not available to the agent, or it has not been trained in that particular scenario, it will fail to generalise and may behave in counter-productive ways.
Either way, if you are concerned about specific scenarios, you need to include them in training, or at least test for them after training.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is the correct way to use the implication in first-order logic?
I know the implication symbol, $\rightarrow$, is used for conditions like

If $A$ is true, then $B$ will be true.

which can be written as
$$
A \rightarrow B
$$
However, sometimes the implication symbol is also used in other contexts. For example, if we want to say that

All $A$'s are $B$.

We could write
$$\forall X (A(X) \rightarrow B(X))$$
I don't understand why the implication is used here? And if the implication is necessary to use here, then why isn't the implication used in the example written below?

Some $A$'s are $B$'s.

$$\exists X (A(X) \land B(X))$$

When we state in English that "All As are Bs", this means that we gain information as soon as we observe an A, we can immediately deduce that it must also be a B. These are the kinds of situations where we use an implication. So, this would be written in formal logic as:
$$\forall X \left( A(X) \rightarrow B(X) \right)$$
When we state in English that "Some As are Bs", we do not gain any new information just from observing that something is an A, we cannot deduce anything about that A. It might happen to be one of the As that simultaneously is a B, but it also might happen not to be one of those examples. So, it would be wrong to use an implication here. The only information that the English sentence gives us is that there is at least one thing somewhere that happens to be an A as well as a B, which is written formally as:
$$\exists X (A(X) \land B(X))$$

Suppose that we would have written the following in logic:
$$\exists X (A(X) \rightarrow B(X))$$
This would be translated to English as follows:

There exists some $X$ such that, if it is an $A$, it is also a $B$.

That bolded part is very important there. Note that this logical statement is also true as soon as I find one example $X$ that is not an $A$. For example, the following statement is true in the real world:

There exists some human $X$ such that, if $X$ can fly, $X$ can also shoot fireballs from his or her hands.

(this is true in the real world, because I can come up with many examples of humans who cannot fly)


text:
How do I optimize a specific function using a genetic algorithm?
I recently learned about genetic algorithms and I solved the 8 queens problem using a genetic algorithm, but I don't know how to optimize any functions using a genetic algorithm.
$$
\begin{array}{r}
\text { maximize } f(x)=\frac{-x^{2}}{10}+3 x \\
0 \leq x \leq 32
\end{array}
$$
I want a guide on how to find chromosomes and fitness functions for such a function? And I don't want code.

Here you can find another similar question.

Yours is a really nice and easy question: you've seen how to use GAs in a complex problem but you're missing how to apply them to the most basic of all. I'll show you:
In a real-world implementation we have to structure our problem to be solved with GAs; we need to modify it, finding an equivalent representation that accepts individuals, each one of these is built from a genome and must be evaluated using a fitness function.
If you see this graphically you'll discover that the fitness function is just describing an area (or line or volume, depending on the grade) on our space and we are randomly dropping individuals on it, awarding the ones that fell higher than the others. We then try to modify the genomes of these guys to move them towards peaks of this fitness function.

In practice the fitness function is our world, our ground truth and we are exploring it.
Now to the function optimization part: well, we do not need any abstraction here, any strange individual representation or transformation of the problem; we just want to find a maximum, and this is exactly what GAs are for!
So, let's have a look at the elements you need to solve your problem with GAs:

Fitness function => you have it already! It's the function you want to find the max of.
Individual => as the maximum is enough, so the individual will be just a point in space.
Genome => like every respective point in space, the genome will just be a collection of real numbers (one for each dimension).

Now, like in any GA, you will instantiate randomly an initial population (random points in the domain). The evaluation of course is the easy part, just put them into the function and you'll se the value of the individuals.
What about crossover and mutation though? You'll have to use techniques that work with real-number genes, like BLXalpha, BLXalphabeta.
These are just defining ranges between values and picking random values inside these ranges, I wrote a pretty detailed answer about this, you can check it out at: https://ai.stackexchange.com/a/6323/15530

In my opinion, this answer provides little insight into how to really solve this problem. For instance, you say that the fitness function is the function you're trying to find the maximum of. However, this is not really true, although the fitness function is a function of the original function. See my answer.

People usually say that genetic algorithms are used to solve optimization problems, but when it comes to optimizing a specific function given in an analytic form (i.e. when it comes to finding a maximum or minimum of such a function), it may not be clear how to proceed. I have created a complete but simple implementation and explanation of how to solve this problem here, but let me also describe here the main idea behind the approach. Before that, let's briefly review genetic algorithms (GAs).
Genetic algorithms
Genetic algorithms are composed of

a population (i.e. a set) of individuals (also known as chromosomes or genotypes), which represent the solutions to some problem

a fitness function that evaluates each individual (i.e. how "good" it is, maybe compared to other individuals, where "good" depends on the problem)

genetic operations to stochastically change the individuals in the population: typically, these operations are the mutation and cross-over

a method to select individuals for the cross-over (where you combine 2 or more individuals to produce other individuals); the selection is also a genetic operation

How to solve your problem?
To solve any problem with genetic algorithms, you first need to address all the four points above, i.e. define what your individuals (i.e. solutions) are, how to compute the fitness of a solution (i.e. how good it is), and define the specific evolutionary operations (specifically, mutation, cross-over and selection).
In your specific problem, the solutions are $\hat{x} \in \mathbb{R}$, such that
$$f(x)=\frac{-x^{2}}{10}+3 x$$
is a (local or global) maximum, i.e. $f(\hat{x}) \geq f(x)$, for all $x \in  \mathbb{R}$ in a neighbourhood of $\hat{x}$.
(Note that this is just the definition of the problem of function maximization: if you are not familiar with it, you should probably get familiar with it before trying to understand this answer or even trying to solve this problem with GAs).
Therefore, in this case, the individuals are real numbers (which are the inputs to $f$).
The fitness function can be a function that computes $f(x_i)$, for all $x_i$ in your population, then compares $f(x_i)$ to $f(x_j)$ for all $i \neq j$. The higher the $f(x_i)$, the closer it is to a maximum.
The genetic operations can be implemented in different ways. You should think about it. If you are familiar with GAs and you know now that solutions are real numbers, at least one way of implementing these genetic operations should come to your mind at this point. Keep in mind that your solutions should be in the range $[0, 32]$, i.e. this is a constrained optimization problem. If you do not have any idea on how to implement them, take a look at my implementation/explanation.

I have a .csv file with 5 inputs (columns) with different parameters. And one output (column). How can I change the input values to optimize my output? Although I have understood the general idea of Genetic Algorithms, I cannot understand how to use them in my case... (I don't feel there is need to open a new thread, because my problem  is quite related to the current question/thread)

Genetic algorithms are quite simple, although they may not always be effective. So, first, you need to identify what a solution is. What is a solution in your case, i.e. what do you want to find? A number? A vector? By convention, we call a solution an individual. The idea of genetic algorithms is to maintain a set (or population) of individuals, randomly change and combine them, in such a way that better solutions are more likely to be kept in this set (population) than worse solutions. Now, you see I used the word "better". That's another thing you need to define and it's very important:

basically, it's the objective function, but in genetic algorithms it's called the fitness function (just another name for the same concept). So, in your case, you need to understand how you can or need to evaluate an individual. If you provide more info about how the 5 numbers are related to the output value and what exactly you want to find, then I could try to be more specific.


text:
Where can I find the original paper that introduced RNNs?
I was able to find the original paper on LSTM, but I was not able to find the paper that introduced "vanilla" RNNs. Where can I find it?

The two tech reports below both call RNNs explicitly "recurrent net(work)s".

Rumelhart, David E; Hinton, Geoffrey E, and Williams, Ronald J (Sept. 1985). Learning internal representations by error propagation. Tech. rep. ICS 8504. San Diego, California: Institute for Cognitive Science, University of California.

Jordan, Michael I. (May 1986). Serial order: a parallel distributed processing approach. Tech. rep. ICS 8604. San Diego, California: Institute for Cognitive Science, University of California.

Jordan was a student of Rumelhart, so I would lean on identifying 1 as the paper introducing RNNs, with the caveat that the first sentence in the section "Recurrent Nets" of 1 reads:

We have thus far restricted ourselves to feedforward nets. This may
seem like a substantial restriction, but as Minsky and Papert point
out, there is, for every recurrent network, a feedforward network with
identical behavior (over a finite period of time).

This is interesting for two reasons:

After this sentence, he then goes on to show how RNNs can be unrolled and the error propagated back. Not a full-fledged BPTT yet, though.
The sentence shows that the idea of recurrence (and unrolling) has been around since at least 1969.

Unfortunately, I don't have access to Minsky and Papert (1969), so I cannot follow this line any further.

You can find Minsky and Papert's book here: https://russell-davidson.arts.mcgill.ca/e706/Perceptrons.pdf. So, you may want to check that part. I have only found that this book mentions "Multilayer machines with loops" (section 13.2 Other Multilayer Machines, pp. 231-232), so they were at least aware of the possibility of augmenting neural networks with recurrent connections, but I didn't find the term "recurrent" (although I didn't really search for it extensively).

As that link no longer seems to work, here a link to the Internet Archive version: https://archive.org/embed/perceptronsintro00mins

Hopfield networks, a special case of RNNs, were first proposed in 1982: https://www.pnas.org/content/79/8/2554
Otherwise (shameless plug, I am the author) a non-technical timeline for NLP can be found here: https://blog.exxcellent.de/ki-machine-learning

You say "a non-technical timeline for NLP can be found here:". Are you talking about a timeline for when RNNs were applied to NLP tasks?

No, the timeline shows the dates of the papers describing them. TR-808 was the introduction of the famous drum computer, referred to in the German text. 1982-86 were the papers on Hopfield networks and RNNs. 1995-97 the papers on LSTMs. And 1999 is the date the first GPU was launched. If you have corrections or comments, I would love to hear them.

Warren McCulloch and Walter Pitts talk about recurrent neural nets in their paper McCulloch, W.S., Pitts, W. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5, 115–133 (1943). https://doi.org/10.1007/BF02478259.
They finish their introduction with the paragraph:

The nervous system contains many circular paths, whose activity so regenerates the excitation of any participant neuron that reference to time past becomes indefinite, although it still implies that afferent activity has realized one of a certain class of configurations over time. Precise specification of these implications by means of recursive functions, and determination of those that can be embodied in the activity of nervous nets, completes the theory.

Their paper contains a section titled:

The Theory: Nets Without Circles.

in which they introduce feed-forward (nets without cycles) and recurrent (nets with cycles) networks, and the next section, titled

The Theory: Nets with Circles.

in which they prove a few theorems about recurrent neural networks.
Marvin Minsky quotes them, and discusses recurrent neural networks extensively throughout his book, Computation: Finite and Infinite Machines (1967). Prentice Hall, ISBN: 0131655639,9780131655638
I am not sure, are there earlier references.

According to this meta paper, "vanilla" RNN of today are based on Elman's work on networks with dynamic memory: Finding structure in time


text:
Should you reload the optimizer for transfer learning?
For example, you train on dataset 1 with an adaptive optimizer like Adam. Should you reload the learning schedule, etc., from the end of training on dataset 1 when attempting transfer to dataset 2? Why or why not?

When doing transfer learning it makes sense to have different update policies for "inherited" parameters and the "new" parameters. "Inherited" parameters are pre-trained on dataset1 and they typically form the front end of the deep model. The "new" parameters are trained from scratch and they typically produce the desired predictions on dataset2. It would be sensible to restart the learning schedule for the "new" parameters. However, most often we would avoid doing that for "inherited" parameters in order to avoid catastrophic forgetting.

Adam reduces the learning rate over time. When you change to the new training data, you want to reset the learning rate. But Adam might not be the best choice for the second round of training - it can make big changes to the inherited weights, which prevents the transfer of previous learning. It can be good to switch to simple SGD for the second round.


text:
Why is the state value function sufficient to determine the policy if a model is available?
In section "5.2 Monte Carlo Estimation of Action Values" of the second edition of the reinforcement learning book by Sutton and Barto, this is stated:

If a model is not available, then it is particularly useful to estimate action values (the values of state– action pairs) rather than state values. With a model, state values alone are sufficient to determine a policy; one simply looks ahead one step and chooses whichever action leads to the best combination of reward and next state, as we did in the chapter on DP.

However, I don't see how this is true in practice. I can see how it'd work trivially for discrete state and action spaces with deterministic environment dynamics, because we could compute $\pi(s) = \underset{a}{\text{argmax}}\ V(\text{step}(s, a))$ by just looking at all possible actions and choosing the best one. As soon as I think about continuous state and action spaces with stochastic environment dynamics, computing the $\text{argmax}$ seems to be become very complicated and impractical. For the particular case of continuous states and discrete actions, I think estimating an action value might be more practical to do even if a forward model of the environment dynamics is available, because the $\text{argmax}$ becomes easier (I'm especially thinking of the approach taken in deep Q learning).
Am I correct in thinking this way or is it true that if a model is available it's not useful to estimate action values if state values are already available?

With a model, state values alone are sufficient to determine a policy; one simply looks ahead one step and chooses whichever action leads to the best combination of reward and next state, as we did in the chapter on DP.

As soon as I think about continuous state and action spaces with stochastic environment dynamics, computing the $\text{argmax}$ seems to be become very complicated and impractical.

For stochastic dynamics the calculations would be more complex, but will often be quite tractable. It depends on size of the distribution, and ease of calculating probabilities to make the correct weighted sums. Instead of $|\mathcal{A}(s)|$ calls to $Q(s,*)$ you will need to make roughly $|\mathcal{S'}(s)| \times |\mathcal{A}(s)|$ calls to $V(s')$ where $\mathcal{S'}(s) \in \mathcal{S}$ are all possible states that might result from the starting state. In the worst case this is $|\mathcal{S}| \times |\mathcal{A}|$
Despite the additional work here, you have been more efficient earlier. By only evaluating $V(s)$, you have removed the action dimension, which otherwise splits up your estimates. Your value function estimates will, all else being equal, converge faster because of this. So this is sometimes a compromise you might be willing to make.
For continuous states, this might still be practical, if both action choices and possible transitions are still discrete.
Once either action space is continuous, or transitions are continuous over some probability density function, then finding the maximising action via the model becomes impractical.


text:
How do temporal-difference and Monte Carlo methods work, if they do not have access to model?
In value iteration, we have a model of the environment's dynamics, i.e $p(s', r \mid s, a)$, which we use to update an estimate of the value function.
In the case of temporal-difference and Monte Carlo methods, we do not use $p(s', r \mid s, a)$, but then how do these methods work?

The main idea is that you can estimate $V^\pi(s)$, the value of a state $s$ under a given policy $\pi$, even if you don't have a model of the environment, by visiting that state $s$ and following the policy $\pi$ after that state. If you repeat this process many times, you'll get many samples of trajectories starting at $s$ with some total return associated with them. If you average them, you'll have a sample-based estimate of $V^\pi(s)$. In a similar way, you can do a sample-based approach to estimate $Q^\pi(s, a)$, where you start at $s$, take some arbitrary action $a$, and then continue following $\pi$. After you collect many samples, you can also average the returns to get a sample-based estimate of $Q^\pi(s, a)$. Once you have a good enough estimate of $Q^\pi(s, a)$, you can see that it's easy to improve the current policy $\pi$ to get a new policy $\pi'$ that always chooses the best action on any given state: $\pi'(s) = \underset{a}{argmax}\ Q^\pi(s, a)$.
The procedure I described in the last paragraph where you sample an entire trajectory and wait until the end of the episode to estimate a return is the Monte Carlo approach. In contrast, TD exploits the recursive nature of the Bellman equation to learn as you go, even before the episode ends. For example, you can estimate the value function like this: $V^\pi(s_0) = r_0 + \gamma V^\pi(s_1)$, where $r_0$ comes from experience and instead of collecting more $r_t$ until the end of the episode, you rely on your estimate of the value of the next state (which you also get from experience). At the beginning, your estimates might be random, but after many iterations the process converges to good estimates. The same idea of exploiting this recursive structure can be used to estimate $Q^\pi(s, a)$. I guess this approach is known as temporal difference because you sample $r_0$ which is in some sense the temporal difference between $s_0$ and $s_1$.

This looks like a solid answer. I think you may want to lead with the intuitive relationship between expectations of distributions and samples from those distributions in a first paragraph, due to how basic the OP's question is. There is also the concept of generalised policy iteration - http://www.incompleteideas.net/book/ebook/node46.html - which helps explain how repeated sample-based estimates and changes to policy can converge.


text:
Should I model my problem as a semi-MDP?
I have a system (like a bank) that people (customers) are entered into the systems by a Poisson process, so the time between the arrival of people (two consecutive customers) will be a random variable. The state of the problem is related to just the system (bank), and the action, made inside the system, can be e.g. offering the customer promotion or not (just based on the state of the system, not the status of customers).
To model the problem through RL, 1) it is possible to discretize time horizon into very short time interval (for example 5 minutes as a stage) such that in each time interval, just a single customer enter to our system. On the other hand, 2) it is possible that stages are defined as the time when a customer enters our system.
My questions are:

Is the second approach an semi-MDP (SMDP)? If I want to solve it with RL, should I use hierarchical RL?
In the first approach, if a customer enters in a time interval, it is easy to update the Q values. However, what should we do, if we are in state $S$ and take action $A$, but no customer enters our system, so we do not receive any reward for the pair of $(S, A)$? There would be no difference if we would take action $A_{1}$, $A_{2}$, and so on. This can happen for several consecutive time intervals. I think it is more challenging when we consider eligibility traces.

To model the problem through RL,

it is possible to discretize time horizon into very short time interval (for example 5 minutes as a stage) such that in each time interval, just a single customer enter to our system.
On the other hand, it is possible that stages are defined as the time when a customer enters our system.

1) Is the second approach an SMDP and if I want to solve it with RL, I should use Hierarchical RL?

The second approach as you described it does not sound like an SMDP to me... it sounds to me like a regular MDP where the discrete time steps in the MDP do not have any meaningful connection anymore to real-world time in the real-world problem being modelled. This is the approach I would recommend taking if long periods of "inactivity" in between customers are irrelevant and/or if you do not expect your actions to have any influence on the duration of those periods of inactivity.
If you do expect your actions to have meaningful influence on the duration of inactivity, I would not recommend this approach. For example, your actions may make a customer angry and reduce their likelihood of quickly returning. Or your actions may indirectly affect the likelihood of other customers visiting due to interaction between your population of customers, they may or may not recommend your bank to their friends based on your actions.

2) In the first approach, if a customer enters in a time interval, it is easy to update $Q$-values. However, what should we do, if we are in state $S$ and take action $A$ but, no customer enter our system so we do not receive any reward for the pair of $(S, A)$ there would be no difference if we would take action $A_1$, $A_2$ and so on. This can happen for several consecutive time interval. I think it is more challenging when we consider eligibility trace.

I would actually argue that your first approach can be treated as an SMDP, and that it may be a good idea to do so (personally I prefer the Options framework, it's a bit more general than SMDPs, a bit more flexible, and a bit closer to "standard" RL terminology; there are plenty of free versions of that paper available on goole too).
Whenever you encounter a customer, you could select an option (or "macro-action") in the SMDP, which executes the one "real" action for your current customer once, and is subsequently forced to automatically select no-op actions (which do nothing) until the next customer arrives; that is the point in time where your previous option ends, and a new option can be selected.
An RL algorithm that operates on the SMDP / hierarchical level, i.e. one that only learns to optimize the selection of options / macro-actions and completely skips over all the forced no-op actions, would suffice. This is basically the approach described in Concurrent Reinforcement Learning from Customer Interactions.


text:
How do you implement NEAT by taking into account the loops?
I'm working on my own implementation of NEAT algorithm based on the original 2002 paper called "Efficient Reinforcement Learning through Evolving Neural Network Topologies" (by Kenneth O. Stanley and Risto Miikkulainen). The way the algorithm is designed it may generate loops in connection of hidden layer. Which obviously will cause difficulties in calculating the output.
I have searched and came across two types of approaches. One set like this example claim that the value should be calculated like a time series usually seen in RNNs and the circular nodes should use "old" values as their "current" output. But, this seems wrong since the training data is not always ordered and the previous value has nothing to do with current one.
A second group like this example claim that the structure should be pruned with some method to avoid loops and cycles. This approach apart from being really expensive to do, is also against the core idea of the algorithm. Deleting connections like this may cause later structural changes.
I my self have so far tried setting the unknown forward values as 0 and this hides the connection (as whatever weight it has will have no effect on the result) but have failed also for two reasons. One is my networks get big quickly destroying the "smallest network required" idea and also not good results.
What is the correct approach?

NEAT does not employ a feed forward concept and also it does not take any special action to avoid loops.
Network is evaluated in non-recursive model. The only non-deterministic loop, the evaluation has is the loop for activating all the outputs. Pseudo code is something like this,
Until all the outputs are active
    for all non-sensor nodes
        activate node
        sum the input
    for all non-sensor and active nodes
        calculate the output

NOTE1: You can use a defensive mechanism (like a counter) to avoid a infinite loop
NOTE2: When summing the input, outputs of nodes which were at least evaluated/calculated once are considered, otherwise their outputs are assumed to be zero.
This is the note from the author of NEAT about identifying loops,

Note that checking networks for loops in general in not necessary and therefore I stopped writing this function
bool Network::integrity()

I don't think that we are on the same page. If node A has a connection to node B and node B has a connection to node A you can't get the input of node A because it depends on it self. You added a note to stop infinite loops but how would you calculate the initial iteration?

@Emad, I have added another note regarding how NEAT handles the dependent value issue. That is: When summing the input, outputs of nodes which were at least evaluated/calculated once are considered, otherwise their outputs are assumed to be zero.

For example, suppose both A and B has links from sensors (actual input layer), in the first iteration of outer loop, A's input summing disregards B's output since B is not evaluated yet. And B does the same since A's output is not evaluated yet. Then both A and B calculate their outputs in the second inner loop. Then in the second iteration of outer loop, both A and B consider each other's outputs from first iteration. So you see this continues without an issue. Let me know if you need more clarification

Ok great. Now I understand. Can you please cite any reference for this. This is same as the first approach that I mentioned in the OP. But I fail to verify this in literature.

Literature fails to express all these implementation details I guess. However next valid reference we can take is the original source code, which is http://nn.cs.utexas.edu/?neat-c

You can use a feed forward style network, so that every node outputs to a higher node except output nodes. This will eliminate connection loops.

"higher" in what sense? Innovation number? or something else?

Yes, identification number or innovation number.

Ok but innovation number is not actually a measure of being later in the feed network. i.e. a new connection can form before one of the existing ones yet take higher innovation number.

Whoops it is actually this: a new connection is formed only when the first node is is closer to the input then the second node, or the fewest number of connections to get to the input is greater in the second node. Another question simailar to this can be found here https://ai.stackexchange.com/questions/6231/how-to-evaluate-a-neat-neural-network

I see. Yet in the literature it's a bit different. I appreciate your solution.

This is what I do:
During evaluation, I add visited counters to nodes and keep the paths.
If we go [1i-5h-7h-4o-5h], 5h is visited twice, so
I detect the loop [5h-7h-4o-5h] and mark the [4o-5h] connection as a loop.
Then I let it continue evaluating.
[1i-5h-7h-4o-5h-7h-4o] and 4o can't go to 5h again because it was marked as a loop.
This way you don't avoid or ignore loops, you add them to your evaluation as a kind of memory.
This is consistent with the stanley 02 paper
Introduction Paragraph 1
In addition, memory is easily repre-
sented through recurrent connections in neural networks, making NE a natural choice
for learning non-Markovian tasks (Gomez and Miikkulainen, 1999, 2002).

He references recurrent connections being used as "memory".
Page 122 Figure 8 subtext:
A NEAT solution to the DPNV problem. This clever solution works by taking
the derivative of the difference in pole angles. Using the recurrent connection to itself,
the single hidden node determines whether the poles are falling away or towards each
other.

So this recurrent connection should not be avoided, in fact it is required for the given solution.
This is why I evaluate them once.


text:
Is the optimal policy always stochastic if the environment is also stochastic?
Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic?
Intuitively, if the environment is deterministic (that is, if the agent is in a state $s$ and takes action $a$, then the next state $s'$ is always the same, no matter which time step), then the optimal policy should also be deterministic (that is, it should be a map from states to actions, and not to a probability distribution over actions).

Here's a related question: https://mathoverflow.net/q/44677.

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic?

No.
An optimal policy is generally deterministic unless:

Important state information is missing (a POMDP). For example, in a map where the agent is not allowed to know its exact location or remember previous states, and the state it is given is not enough to disambiguate between locations. If the goal is to get to a specific end location, the optimal policy may include some random moves in order to avoid becoming stuck. Note that the environment in this case could be deterministic (from the perspective of someone who can see the whole state), but still lead to requiring a stochastic policy to solve it.
There is some kind of minimax game theory scenario, where a deterministic policy can be punished by the environment or another agent. Think scissors/paper/stone or prisoner's dilemma.

Intuitively, if the environment is deterministic (that is, if the agent is in a state  and takes action , then the next state ′ is always the same, not matter which time step), then the optimal policy should also be deterministic (that is, it should be a map from states to actions, and not to a probability distribution over actions).

That seems reasonable, but you can take that intuition further with any method based on a value function:
If you have found an optimal value function, then acting greedily with respect to it is the optimal policy.
The above statement is just a natural language re-statement of the Bellman optimality equation:
$$v^*(s) = \text{max}_a \sum_{r,s'}p(r,s'|s,a)(r+\gamma v^*(s'))$$
i.e. the optimal values are obtained when always choosing the action that maximises reward plus discounted value of next step. The $\text{max}_a$ operation is deterministic (if necessary you can break ties for max value deterministically with e.g. an ordered list of actions).
Therefore, any environment that can be modelled by a MDP and solved by a value-based method (e.g. value iteration, Q-learning) has an optimal  policy which is deterministic.
It is possible in such an environment that the optimal solution may not be stochastic at all (i.e. if you add any randomness to the deterministic optimal policy, the policy will become strictly worse). However, when there are ties for maximum value for one or more actions in one or more states then there are multiple equivalent optimal and deterministic policies. You may construct a stochastic policy that mixes these in any combination, and it will also be optimal.

Comments are not for extended discussion; this conversation has been moved to chat.

I would say no.
For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and $n$. This is a simple stochastic environment: this is a one state environment, but it is still an environment.
But obviously the optimal policy is to choose the arm with the highest $p_i$. So this is not a stochastic policy.
Obviously, if you are in an environment where you play against other agent (a game theory setting), your optimal policy will certainly be stochastic (think of a poker game, for example).

Why would it be obvious to always choose the arm with the highest $p_i$? $p_i$ is a probability, so it's not certain you will always obtain the highest amount of reward (at least, in finite time) if you always choose the arm $i$.

@nbro: It is certain in expectation, which is what the optimal policy maximises. Policies don't try to second-guess random number generators, that is assumed impossible (if it were possible due to some internal state of the system, you must either add that internal state to the model, or treat as a POMDP)

@NeilSlater Ok. But would the conclusion change if time is finite? If you have a limited amount of time to play, then the expectation, I guess, must also consider the available time to play.

@nbro: That may change your decisions, but is not really about the optimal policy. The optimal policy for the bandit arms is still deterministic, about using the best arm, but you don't know it. This is about exploration vs exploitation. You could phrase that as having "an optimal policy for exploring a bandit problem" perhaps. Not the terminology used in e.g. Sutton & Barto, but perhaps some parctioners do say that, I don't know . . .

Why do you say that the environment contains only one state in the "multi-armed bandit problem"?

This is not a correct summary of the multi-armed bandit problem, which is interesting precisely because the reward probability of any given arm isn't initially known, and requires exploration to determine. I agree with the general idea of the answer however.

The environment contains only one state in which you face the same decision over and over : which arm I have to choose ?

I'm thinking of a probability landscape, in which you find yourself as an actor, with various unknown peaks and troughs.  A good deterministic approach is always likely to lead you to the nearest local optimum, but not necessarily to the global optimum.  To find the global optimum, something like an MCMC algorithm would allow to stochastically accept a temporarily worse outcome in order to escape from a local optimum and find the global optimum. My intuition is that in a stochastic environment this would also be true.


text:
How do I write a genetic algorithm to solve the knapsack problem?
I am trying to write a genetic algorithm that generates 100 items, assigning random weights and utilities to them. And then try to pick items how out these 100 items while maximising the utility and not picking items over 500ks. The program should return an array of boolean values, where true represents items to be picked and false represents item not to be picked.
Can someone help with this or point me to a link of something that has been written like this before?

This is a common example of an application of genetic algorithms: showing how to solve the knapsack problem. Haven't you found anything on the web?


text:
Where are reinforcement algorithms used in financial services?
One of the most common misconceptions about reinforcement learning (RL) applications is that, once you deploy them, they continue to learn. And, usually, I'm left having to explain this. As part of my explanations, I like to show where it is being used and where not.
I've done a little bit of research on the topic, but the descriptions seem fairly academic, and I'm left with the opinion that reinforcement learning is not really suitable for financial services in regulated markets.
Am I wrong? If so, I would like to know where RL is being used? Also, in those cases, are these RL algorithms adapting to new data over time? How do you ensure they are not picking up on data points or otherwise making decisions that are considered to be unacceptable?

Have a look at Wealth Wizards; I believe they are using ML for financial advice. https://www.wealthwizards.com/ -- Disclaimer: I know a few people who work/used to work there, but have nothing to do with the company myself.

My opinion (heavily based on personal experiences and anecdotes) is that RL is not used that much in real production environment. I've heard that some bigger tech companies used it successfully but they won't post their methods and results for obvious reasons. And even then, I'm quite skeptical whether they use 100% RL in their systems.

I know the new bot in JP Morgan uses reinforcement learning:
Machine Learning in FX
There a lot of studies of Reinforcement learning can apply in crypto currency trading. Here is one of many examples:
A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem

authors: Hášíl Páůďýál


text:
What is the most biologically plausible representation for the actor and critic?
Which representation is most biologically plausible for actor nodes? For example, actions represented across several output nodes which may be either

mutually exclusive with each other (e.g., go north, go south, etc),
achieved by winner-takes-all.
NOT mutually exclusive with each other (e.g. left leg forward, right leg forward); these actions may occur concurrently. To go north, the correct combination of nodes must be active.

Similarly which representation is most plausible for critic output nodes?

A single output node that outputs a real number representing the
reward.
A set of output nodes each representing a separate value, achieved by winner-takes-all.

Or do other representations better align with real brains ?

To make it clear, you want to represent both the actor and the critic, in the actor-critic architecture, as a neural network. However, it is not clear to me what your actual problem is. I understand you want to be consistent with the biology (if possible), but what exactly must be consistent with the biology, in this context? What nodes of the last layer of the neural network (both for the actor and the critic)?

Hi @nbro,
To be biologically plausible the model should really apply to all problems, rather than a specific problem.
Some interesting example..
This model attempts to tie in with brain regions, however it uses a winner-takes-all rule for the action.
This model uses plausible local learning rules for neurons, but uses a winner-takes-all rule for the action and I think requires a separate critic neuron for each state.

For the actor, I'd say the 'not mutually exclusive' option is more biologically plausible in the context of muscle systems, where the actions can be seen as simultaneous muscle activations. Maybe at a higher level, an agent thinks of the action as 'go north' or 'go south', but the final outputs which have to control muscles at a lower level have to represent simultaneous muscle activations.
For the critic, I'd say the 'single output node' is more biologically plausible. Agents perceive the world in the form of high dimensional inputs, such as images. The approach where a value function is learned in a tabular fashion where you know the value for every single state doesn't really scale very well and is limited to small discrete state spaces. For biological agents, it makes sense to have a function that senses the current state of the environment and outputs a single number that represents the value, which gives the agent an idea of how things are going so far given the actions it took in the past.

Hi @yewang,
Yes I agree, in fact the mutually exclusive actor (1) will also not scale well in my opinion.
-Do you know of any models with plausible (eg local) learning rules that support the plausible representations (2&3) ?
thanks

For example, if you use A2C to control a robot, a common approach is to train a neural network that outputs a probability distribution over a number of actuators, often as a multivariate gaussian distribution, where the actor outputs the mean of the multivariate distribution (compatible with (2)). For the value function, it's common to model it as a function that takes an n-dimensional vector (for example joint angles) and outputs a single number representing the value of the state (compatible with (3)). Since the state space is continuous, you can't store a value for every state in a table.

Hi @yewang, thanks. The examples I looked at for A2C I think are using back-propagation to train the value function. -  Do you know of a paper that achieves this using only local rules ?

what do you mean by local rules? I think backpropagation can be thought of as local rules if you consider that the update rule for a neuron depends only on immediately close neurons. As you propagate the gradients layer after layer you get a global behavior in an indirect way, but the basic rules are local.

Hi @yewang, I appreciate your feedback :) By local rules I meant it only uses local information available to each synapse. Principally hebbian type rules, including 3-factor rules (pre, post & neuromodulator). So in the case of A2C I believe the actor could be implemented with a local 3-factor rule (eg a Policy Gradient method such as ARP). But the critic value function appears to use backprop. I have not found a hebbian type rule to support that single node representation.FYI However I happen to have devised a simple method to do this, I can give you the details if you're interested.


text:
What is the definition of a heuristic function in the BayesChess paper?
I am reading BayesChess: A computer chess program based on Bayesian networks (Fernandez, Salmeron; 2008)
It is a chess-playing engine using Bayesian networks. The following is mentioned about the heuristic function in section 3.

Here the heuristic is defined in terms of 838 parameters.
There are 5 parameters indicating the value of each piece (pawn,
queen, rook, knight, and bishop -the king is not evaluated, as it must
always be on the board), 1 parameter for controlling whether the king
is under check, 64 parameters for evaluating the location of each
piece on each square on the board (i.e., a total of 786 parameters,
corresponding to 64 squares × 6 pieces each colour × 2 colours) and
finally 64 more parameters that are used to evaluate the position of
the king on the board during the endgame.

The above sentence contains the parameters used by the heuristic function. But I didn't find the actual definition. What is the actual definition of the heuristic function?

By far the most common form of heuristic evaluation functions for Chess-playing (or, really, any game-playing) agents are simple linear functions. At least when we're talking about handcrafted features that's the case, of course all the hype with Deep Neural Networks in more recent years is different. So, when it's not specified in a paper like this exactly what their heuristic evaluation function looks like, you can relatively safely assume it's just a linear function.
With linear function, I mean that you have vectors of features $\boldsymbol{\phi}(s)$ for your states $s$, and a vector of weights $\boldsymbol{\theta}$, and the evaluation $f(s)$ of a state $s$ is simply given by the dot product (summing up all the multiplications of feature values with their corresponding weights):
$$f(s) = \boldsymbol{\phi}(s)^{\top} \boldsymbol{\theta} = \sum_i \phi_i(s) \times \theta_i(s),$$
where the subscript $i$ indicates taking the $i^{th}$ element of a vector.

Heuristics can be understood aas rules.  Typically heuristics are thought of as problem-specific strategies.  Expert systems were an early form of AI that utilized rules-based decisions.  In a game-playing context, heuristics can be pure strategies.
A heuristic function would be one that includes a some predefined decision rules.  Russell and Norvig have a nice chapter on informed (heuristic) search strategies.

Heuristic functions are the most common form in which additional knowledge of the
  problem is imparted to the search algorithm.
   Artificial Intelligence: A Modern Approach; 3.5 pdf

h(n) as opposed to f(n) or g(n)

Heuristic just means that it is hand constructed by a human. Suppose you have the value of each piece given an initial value. That is a heuristic because it was just defined by a human. But if the bayesian process is going to modify that initial value given by the human that would be a non-heuristic thing.
So there is no function definition to search for except to find what the initial parameters given by the human are.

but in the last paragraph of page 5, it was mentioned as a heuristic function


text:
If we collected a very large labelled dataset from multiple sensors, then train a neural network with that data, could that lead to an AGI?
I've come up with an idea on how we could use a combination of Deep Learning and body sensors to create a walking talking living humanoid. Here goes:
First, we will recruit 1 billion people and have them wear a special full face mask and suit. This suit will contain touch sensors along the skin, cameras, smell sensors, taste sensors on the mask, basically every data and information that a human receives will be collected electronically, whether it is what they see, what they smell what they feel and so on.
These suits will also have potentiometers and other sensors to measure the movement made by the person. Every hand movement, leg movement, muscle movement will be recorded and saved in a database as well.
After 50 years or so, all collected input and output data from every single person who participated in this experiment will be saved in a computer. We then create a neural network and then train it on the input and output data from the database.
Next, we create a robot that has motorized muscles and hand/leg joints that are the same specs as to our previous suits and also the touch, smell, sight and other sensors integrated inside of this robot.
Once everything is ready, we will load the trained neural network onto the robot and switch it on. During inference, the neural network will take data from sensors all over the robot's body and translate it into movement in legs, hands, and muscles.
Could these techniques in conjunction with the data collection I describe, produce an AGI?
Essentially, how feasible is current technology to produce a robot that will behave, speak, live, move like a normal human being?

I would say this is more philosophy (there's a tag for that) than answerable using objective facts, and there are a few potential angles to it. (1) Is it possible to consider such a project theoretically using known technology? (2) Are there limitations of "brute force" training using deep learning? (3) Would a trained robot be considered "human" or just a sophisticated puppet - would it have self awareness, or just regurgitate sentences claiming that it had? I think the wording of your last sentence implies (3) is not of interest, but please clarify the kind of answer you are looking for

How would you label all that data?

No need to label, just append all data into one giant vector and throw it at the neural network

By mapping sensorial information directly to output actions you are "skipping" cognitive mechanisms that depend on experience, cultural context and other factors. Given a specific stimulus, people may react in very different ways. If you just provide input-output pairs coming from a billion people I think that the main risk would be to have just a random behaviour. This is just my first thought


text:
Batch PTA stopping condition
I am reviewing my Neural Network lectures and I have a doubt: My book's (Haykin) batch PTA describes a cost function which is defined over the set of the misclassified inputs.
I have always been taught to use MSE < X as a stopping condition for the training process. Is the batch case different? Should I use as stopping condition size(misclassified) < Y (and as a consequence when the weight change is very little)?
Moreover, the book uses the same symbol for both the training set and the misclassified input set. Does this mean that my training set changes each epoch?

What does "MSE < X" actually mean? I can infer that MSE is the mean-squared error and X is maybe the input, but what kind of operation is that? Less than? It is not clear what your stopping condition is. What is the name of the book? To prevent overfitting, you can often detect it by seeing an improvement in the performance (e.g. accuracy) on the training dataset, but a decrease in performance on the evaluation dataset. This is often a sign of overfitting. See: https://stats.stackexchange.com/q/131233/82135.

Sorry for the lack of clarity. for "MSE < X" I meant "MSE lower than a certain value X". I always been taught to use this stopping condition to prevent overfitting (as you told me) and stop the training when a certain precision on the train set (let's say 95%) is met.


text:
Can neuroevolution be used for solving tasks other than games?
I'm seeing a lot of examples of neuroevolution techniques involving games or robot problems. Can neuroevolution be used for solving tasks other than games? For example, how could you transform a CSV file of psychological data to determine the best life actions you can get from a self-report questionnaire?

Medical Diagnostics is a promising area for AI in general: https://en.wikipedia.org/wiki/Artificial_intelligence_in_healthcare#Current_Research


text:
What should be saved in SARSA prioritized sweeping?
In the book "Reinforcement Learning: An Introduction", by Sutton and Barto, they provided the "Q-learning prioritized sweeping" algorithm, in which the model saves the next state and the immediate reward, for each state and action, that is, $Model(S_{t},A_{t}) \leftarrow S_{t+1}, R_{t+1}$.
If we want to use "SARSA prioritized sweeping", should we save "next state, immediate reward, and next action", that is, $Model(S_{t},A_{t}) \leftarrow  S_{t+1}, R_{t+1}, A_{t+1}$?

SARSA is an on-policy method. Your historical choices of action will have been made using older Q values, and thus from a different policy. In addition, the action that was taken at the time may not have been typical for the agent (it may have been exploring). So you don't usually want to re-use historical action choices to calculate TD targets in single-step SARSA, because these may introduce bias.
Provided you are performing single-step SARSA, then you can re-generate the action choice sampling from the current best policy. This is similar to generating the max Q value in the TD target value $R_{t+1} + \text{max}_{a'} Q(S_{t+1},a')$.
You could do this using a regular SARSA sampling the action choice:
$$R_{t+1} + Q(S_{t+1}, a' \sim \pi(\cdot|S_{t+1}) )$$
Or you could use Expected SARSA and take a weighted mean of all possible actions:
$$R_{t+1} + \sum_{a' \in \mathcal{A}(S_{t+1})} \pi(a'|S_{t+1})Q(S_{t+1}, a')$$
Technically these would both be on-policy with respect to evaluating the TD Target, but off-policy with respect to the distribution of $S_t, A_t$ that you are running the update for. That's already the case due to prioritised sweeping focussing more updates on certain transitions, but could be a big difference when using a neural network to approximate Q. Bear in mind that making TD learning methods off-policy can have a negative impact on stability.
If you want to process multi-step updates, then you do have to reference $A_{t+1}$ and adjust for when the historical data makes a different action choice than your current estimate of the best policy. This would commonly use importance sampling. This is true for Q-learning as well however, so there would still be no difference in what you store between Q-learning and SARSA.

Dear @Neil Thank you so much. To make sure that I understood your points, you mean that in a tabular single step SARSA, the model stores $Model(S_{t},A_{t}) \leftarrow S_{t+1}, R_{t+1}$, as the same as "Q-Learning", and then for each step in repeat, we re-generate a new action for $A_{t+1}$, and we do not use importance sampling at all.
$s,a \leftarrow first(PQueue)$
$r,s^{\prime} \leftarrow Model(s,a)$
$Take action a^{\prime} , e.g. \epsilon -greedy from Q(s^{\prime},.) $
$Q(s,a) \leftarrow Q(s,a) + \alpha . (r+\gamma Q(s^{\prime},a^{\prime}) - Q(s,a))$

@KatatoniaSh: Yes, that's it.

@Dwait: Ideally you would re-sample $A_{t+1}$ as part of any re-calculation of the priority, although that could be inefficient. Possibly though you could still get away with it, since the priorities may still loosely apply due to a particular state being interesting in the given problem.

@NeilSlater Thanks! And sorry I meant to edit my comment but deleted it instead. Posting it again so your response isn't out of context for other readers:
The order of making these updates is still governed by their priorities in the PQueue which are decided by Rt+1+γQ(St+1,At+1)−Q(St,At), where At+1 is the historical action chosen while moving along the trajectory, right? Here the target is different from the one that will be used for making the actual update. So the central idea of Prioritized Sweeping - making the bigger updates first - is somewhat spoiled with SARSA isn't it?


text:
How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?
In this video, the lecturer states that $R(s)$, $R(s, a)$ and $R(s, a, s')$ are equivalent representations of the reward function. Intuitively, this is the case, according to the same lecturer, because $s$ can be made to represent the state and the action. Furthermore, apparently, the Markov decision process would change depending on whether we use one representation or the other.
I am looking for a formal proof that shows that these representations are equivalent. Moreover, how exactly would the Markov decision process change if we use one representation over the other? Finally, when should we use one representation over the other and why are there three representations? I suppose it is because one representation may be more convenient than another in certain cases: which cases? How do you decide which representation to use?

In general the different reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ are not equivalent mathematically, so you will not find any formal proof.
It is possible for the functions to resolve to the same value in a specific MDP, if, for instance, you use $R(s, a, s')$ and the value returned only depends on $s$, then $R(s, a, s') = R(s)$. This is not true in general, but as the reward functions are often under your control, it can be the case quite often.
For instance, in scenarios where the agent's goal is to reach some pre-defined state, as in the grid world example from the video, then there is no difference between $R(s, a, s')$ or $R(s)$. Given that is the case, for those example problems you may as well use $R(s)$, as it simplifies the expressions that you need to calculate for algorithms like Q-learning.
I think the lecturer did not mean "equivalent" in the mathematical sense, but in the sense that future lectures will use one of the functions, and a lot of what you will learn is going to be much the same as if you had used a different reward function.

Finally, when should we use one representation over the other and why are there three representations?

Typically, I don't use any of those representations by default. I tend to use Sutton & Barto's $p(s', r|s, a)$ notation for combined state transitions and rewards. That expression returns probability of transitioning to state $s'$ and receiving reward $r$ when starting in state $s$ and taking action $a$. For discrete actions, you can re-write the expectation of the different functions $R$ in terms of this function as follows:
$$\mathbb{E}[R(s)] = \sum_{a \in \mathcal{A}(s)}\sum_{s' \in \mathcal{S}}\sum_{r \in {R}}rp(s', r|s, a)\qquad*$$
$$\mathbb{E}[R(s,a)] = \sum_{s' \in \mathcal{S}}\sum_{r \in {R}}rp(s', r|s, a)$$
$$\mathbb{E}[R(s,a, s')] = \sum_{r \in {R}}r\frac{p(s', r|s, a)}{p(s'|s, a)}$$
I think this is one way to see how the functions in the video are closely related.
Which one would you use? Depends on what you are doing. If you want to simplify an equation or code, then use the simplest version of the reward function that fits with the reward scheme you set up for the goals of the problem. For instance, if there is one goal state to exit a maze, and an episode ends as soon as this happens, then you don't care how you got to that state or what the previous state was, and can use $R(s)$
In practice, what happens if you use a different reward function is that you need to pay attention to where it appears in things like the Bellman equation for theoretical treatments. When you get to implement model-free methods like Q-learning, $R(s)$ or its variants don't really appear except in the theory.

* This is not technically correct in all cases. The assumption I have made is that $R(s)$ is a reward granted at the point of leaving state $s$, and is independent of how the state is left and where the agent ends up next.
If this was a fixed reward for entering state $s$, regardless of how, then it could be written around $R(s')$ as follows:
$$\mathbb{E}[R(s')] = \sum_{s \in \mathcal{S}}\sum_{a \in \mathcal{A}(s)}\sum_{r \in {R}}rp(s', r|s, a)$$
i.e. by summing all the rewards that end up at $s'$

Sutton & Barto define $r(s, a, s')$ differently, which does not seem to be equivalent to your definition. See equation 3.6 of the 2nd edition. They also use a different notation to denote these reward functions, but it's also true that, in my original question, I also used a notation different than the one in this version of the book (so maybe it's not a good idea to edit your answer to be consistent with the book).

@nbro: As you'd expect Sutton&Barto are right here. I missed that fixing $s'$ means needing to normalise the values. I have fixed it, though I could also expand what $p(s'|s,a)$ is, I think it is fairly clear given other definitions.

Let $R(s)$ denote a probability distribution over rewards that our agent may get in some MDP as a reward for entering a state $s$. The easiest case is to demonstrate that we can also choose to write this as $R(s, a)$ or $R(s, a, s')$: simply take $\forall a: R(s, a) = R(s)$, or $\forall a \forall s': R(s, a, s') = R(s)$, as also described in Neil's answer.

Let $R(s, a)$ denote a probability distribution over rewards that our agent may get as a reward for executing action $a$ in state $s$. The easy case of demonstrating equivalence to $R(s, a, s')$ is already handled above, but can we also construct an MDP in which we only use the $R(s)$ notation?
The easiest way I can think of to do so (may not be the cleanest way) would be to construct a new MDP with a bunch of "dummy" states $z(s, a)$, such that executing action $a$ in state $s$ in the original MDP deterministically leads to a dummy state $z(s, a)$ in the new MDP. Note that I write $z(s, a)$ to make the connection back to the original MDP explicit, but this is a completely independent MDP and you should view it as just a state "$z$".
Then, the reward distribution $R(s, a)$ that was associated with the state-action pair $(s, a)$ in the original MDP can be written as $R(z(s, a))$, which is now only a function of the state in the new MDP. In this dummy state $z(s, a)$ in the new MDP, every possible action $\alpha$ should have exactly the same transition probabilities towards new states $s'$ as the original transition probabilities for executing $a$ in $s$ back in the original MDP. This guarantees that the same policy has the same probabilities of reaching certain states in both MDPs; only in our new MDP the agent is forced to transition through these dummy states in between.
If you also have a discount factor $\gamma$ in the original MDP, I guess you should use a discount factor $\sqrt{\gamma}$ in the new MDP, because every step in the original MDP requires two steps (one step into a dummy state, and one step out of it again) in the new MDP.

The final case for $R(s, a, s')$ could be done in a very similar way, but it would get even more complicated to write out formally. The intuition would be the same though. Above, we pretty much "baked" state-action pairs $(s, a)$ from the original MDP into additional dummy states, such that in the new MDP we have states that "carry the same amount of information" as a full state-action pair in the original MDP. For the $R(s, a, s')$ case, you'd need to devise an even uglier solution with even more information "baked into" dummy states, such that you can treat full $(s, a, s')$ triples as single $z(s, a, s')$ states in the new MDP.

Finally, when should we use one representation over the other and why are there three representations? I suppose it is because one representation may be more convenient than another in certain cases: which cases? How do you decide which representation to use?

I would recommend always using the simplest representation that happens to be sufficient to describe how the rewards work in your environment in a natural way.
For example, if you have a two-player zero-sum game where terminal game states give a reward of $-1$, $0$, or $1$ for losses, draws, or wins, it is sufficient to use the $R(s)$ notation; the reward depends on the terminal game state reached, not on how it was reached. Another example would be a maze with a specific goal position, as described in Neil's answer. You could use the more complex $R(s, a)$ or $R(s, a, s')$ notations... but there wouldn't be much of a point in doing so really.
If you have an environment in which the reached state and the played action both have influence on the reward distribution, then it's much more sensible to just use the $R(s, a)$ notation rather than trying to define a massively overcomplicated MDP with dummy states as I've tried to do above. An example would be... let's say we're playing a quizz, where $s$ denotes the current question, and different actions $a$ are different answers that the agent can give. Then it's natural to model the problem as an MDP where $R(s, a)$ is only positive if $a$ is the correct answer to the question $s$.

I would like to note that $R(s)$, $R(s, a)$ and $R(s, a, s')$ can be intuitively interpreted differently. We can think of $R(s)$ as the reward you obtain for entering, exiting or staying in the state $s$ (no matter what action we take or which next state we may end up in). $R(s, a)$, intuitively, can be thought of as the reward you obtain for taking action $a$ in state $s$. Finally, $R(s, a, s')$ can be thought of as the reward you obtain for taking action $a$ in state $s$ and ending up in state $s'$.

I recall that the environment may be stochastic, so, in general, if we take action $a$ in state $s$, we may not end up always in the same next state, that is, if we take action $a$ in the state $s$ twice, the resulting next state may be different.


text:
How to perform structure learning for Bayesian network given already partially constructed Bayesian network?
Let's assume that we have a dataset of variables (random events)I apriori would like to set dependency conditions between some of them and perform structure learning to figure out the rest of the Bayesian network.
How can this be done practically (e.g. some libraries, like bnlearn) or, at least, in theory?
I was trying to google it, but haven't found anything related.

The hill climbing or the constraint-based structure learning algorithms accept whitelist or blacklist arguments permitting or prohibitting some arcs.

This is an old question, but are you referring to the implementation of these algorithms from this library https://www.bnlearn.com/? Please, edit this answer to include these details.


text:
How to handle infeasibility caused due to crossover and mutation in genetic algorithm for optimization?
I have chromosomes with floating-point representation with values between $0$ and $1$. For example
Let $p_1 = [0.1, 0.2, 0.3]$ and $p_2 = [0.5, 0.6, 0.7]$ be two parents. Both comply with the set of constraints. In my case, the major constraint is $$ p_1[1]*p_1[2] - k*p_1[0] \geq 0 $$ for any chromosome $p_1$. For the example above we can take $k=0.3$, which renders $c_2$ infeasible.
However, the children produced by 1 point crossover, we get $c_1 = [0.1, 0.6, 0.7]$ and $c_2 = [0.5, 0.2, 0.3]$ out of which 1 or both may not comply with the given constraints.
A similar scenario can also occur with a small perturbation of values due to mutation strategy. Correct me if I am wrong in the belief that such kind of scenarios might arise irrespective of the strategy employed for crossover and mutation.
What are the options to handle such kinds of cases?

You have two broad categories of options, prevention and repair.
Prevention means defining a crossover and mutation operator that try to be more intelligent about respecting the constraints. Suppose you have an encoding where each individual is a list of integers, and the constraint is that there can't be duplicates. You might define a crossover operator that did something like the following. For each position in the offspring, choose randomly from one parent such that, if possible, you choose a value that hasn't already appeared in the offspring. If both parents have values at that position that are already used in the offspring, choose a random value.
That operator would avoid violating the constraint in the first place, while attempting as much as possible to have the offspring inherit information from the parents.
Another option is to just let the operator violate constraints and then deal with the ramifications afterward. You don't go into details about what your constraints actually are, just that c1=[0.1, 0.6, 0.7] violates them. Let's say the constraint is that the third position should not be more than 4x greater than the first one. OK, so then let's take this offspring and adjust either the first or third item. Maybe we make the new individual into c1=[0.2, 0.6, 0.7].
Often, you want some element of randomness in either option so that you don't strongly bias the production of new search points. In my example, don't always make the first element larger. Sometimes, make the third element smaller or produce some random combination of both to repair the constraint violation.
Finally, both options typically strongly benefit from domain knowledge. Design an operator that understands your problem and tries to intelligently solve the problem.


text:
How do I convert table-based to neural network-based Q-learning?
I've used a table to represent the Q function, while an agent is being trained to catch the cheese without touching the walls.
The first and last row (and column) of the matrix are associated with the walls. I placed in last cell a cheese that agent must catch while being training.
So far, I've done it with dynamic states and, when necessary, I resized matrix with new states. I've used four actions (up, left, right and down).
I would like now to use an ANN to represent my Q function. How do I do that? What should be the input and output of such neural network?

A neural network (NN) is a "function approximator", that is, it is a model that can be used to approximate functions. In fact, a neural network with at least one hidden layer is a "universal" function approximator (that is, it can approximate any function).
In mathematics, a function $f$ is usually represented as a mapping of the form $f: D \rightarrow C$, where $D$ and $C$ are respectively the domain (inputs) and codomain (outputs) of $f$, and $\rightarrow$ means that $f$ "maps" $D$ to $C$. A NN (with at least one hidden layer) can thus approximate any function $f$ of this form.
In your context, the $Q$ table is a function: it is a mapping between states and actions (inputs) and Q values (outputs), which are the "expected future cumulative reward" (that you will obtain if you take a certain action from a certain state and then continue to follow the same policy). The Q function can thus more formally be denoted by $Q: (S, A) \rightarrow \mathcal{R}$, where $S$ is the "space of states" and $A$ is the "space of actions" in your problem. Initially, your Q table does not contain the correct (optimal) values. However, after training (or learning), hopefully, your Q table will be an approximation to the optimal Q function for your specific problem.
How do you then represent this table using a NN? What should be the inputs and outputs of the NN?
Let's suppose that your $Q$ table is implemented as matrix $M$. Then $M[s, a]$ is the $Q$ value for the state $s$ and action $a$. So, in this case, the combination of $s$ and $a$ is the input, whereas $M[s, a]$ is the output of your $Q$ function.
To represent this table as a NN, you can thus have as input of the NN the state and action, and as output the $Q$ value. You will then train the NN (using e.g. back-propagation) to learn the $Q(s, a)$, given a state $s$ and an action $a$ as input, for all $s \in \mathcal{S}$ and $a \in \mathcal{A}$. So, during your $Q$-learning algorithm, instead of using $M[s, a]$ to represent $Q(s, a)$, you will simply use the current output of your neural network.
Note that, in practice, I think, you will likely encounter problems while training the NN, if you update the weights of your NN at every time step (because e.g. NNs do not cope well with correlated data, and, in general, the "experience" data you will obtain from time step to time step will be highly correlated). Anyway, this is the basic idea of how to use a NN to represent a $Q$ function. There are other ways, but this is the simplest one, at least, conceptually.

I've used 2D-array that represent Q-Matrix and all elements of array are iterating to form a string that's called state of Q-Matrix.So how can I set state as input of neuron while neuron input must be in float type and weight values are also used between -1 and 1 that is to update later on.

Why are you creating a string out of the 2d array?

So,How can I create states?It's represented matrix state where the agent exist recently and I'm adding it into list with resizing of matrix in case of dynamic states or newer states.

State of matrix can not be presented in floating points.It's possible only for distance based applications or other related to positions of agents in 2d-3d surfaces.

how to get state of matrix in float Number

@ArslanAli I am not sure I understand what you just said above. Why can't you create a matrix of floating point numbers?

I'm saying how I will send state as input while state and action must be float number with respect to weights.

@ArslanAli You could e.g. encode your actions (up, down, etc.) as numbers: e.g. up could be 0, down could be 1, and so on. The states could e.g. be encoded as the indices you use to access the state. You could then combine actions and states (which, at this point, should be numbers) in a vector. I am not sure if this is the best way to do it, but this could be a way to do it.

But input for NN must be between 0 and 1 while states can be greater than 1 in record while four action are choiced 0-3.So how these inputs can be given to NN while weights are also between 0 and 1 with respect to each input.

Rows are states while columns are actions in Q-Table.So,in that case where I have to save the states and corresponding learning Q-Values in NN.

@ArslanAli Each input to a NN must be a real number number (a scalar). But you can have more than one input to the NN. In that case, you can pack these numbers into a vector.

Output will not be checked while updating weights

@ArslanAli What do you mean? What does this have to do with what I said?

I have to pack each action and state in vector then send as input?

As new states will come then I have to increase input in NN.That's meant by you

@ArslanAli Yes, I suggested that you could pack the state and the action (both encoded as real numbers) in a vector, which would be the input to the NN. As I explain in my answer above, this NN would represent your $Q(s, a)$ function. So, at every point of your Q-learning algorithm, the $Q$ function only requires two inputs ($s$ and $a$), so I am not sure why you're saying that the inputs to the NN would increase. The dimensions of the inputs remain fixed. What changes is the actual input (i.e. the encoded $s$ and $a$) that you pass.

It will require new table while pre results will be saved somewhere.It is because when I will find Max from Q(s',a') that will be needed for equation.

@ArslanAli No, you will not require a new table. You may want to use two NNs (where one for the target). Please, read the paper "Human-level control through deep reinforcement learning" by DeepMind. There you will find an explanation of a network which is used to represent the Q function.


text:
Which neuron represents which part of the input?
In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this digit.

What determines which neuron will represent which part of the digit?
Is it possible that if we pass in the same input multiple times, each neuron can represent different parts of the digit?
How is this related to the back-propagation algorithm and chain rule? Is it the case that, before training the neural network, each neuron doesn't really represent anything of the input, and, as training proceeds, neurons start to represent some part of the input?

The determination of what each neuron represents is dictated by the initial weights of the network. As you may know, the common practice is to initialize the weights randomly. This means that given the same input data, the network may switch what each neuron means. It may also function differently, taking longer or shorter times to train. However, if you set the random seed to some constant, you can make the network deterministic.

It is the combination of the output of all neurons that determines the output of the neural network. In the case of convolutional neural networks (CNNs), the term "feature" is used because it is associated with the feature maps (or activation maps) and filters (or kernels) of the CNN. However, this terminology might not be accurate (because these might not be features in the intuitive sense) and it is used more to interpret the inner workings of the CNN.
After training, the weights of the neural network are fixed (unless you perform online and continual training), so the output of each neuron will be the same given the same input, thus the output of the neural network will also be the same (unless there is some random operation being performed). During training, the weights and thus the output of each neuron (and of the neural network) often change.
The contribution of each neuron to the output of the neural network is determined by the weights of the connections between the neurons, which change during training and can be initialised in different ways, which might affect differently the final weights (after training).
There several ways of visualising the contribution of each neuron to the output of the neural network. See also this article Visualize Features of a Convolutional Neural Network and the paper Visualizing and Understanding Convolutional Networks (2013).


text:
My Gaussian Naive Bayes classifier is too slow
I am trying to build a film review classifier where I determine if a given review is positive or negative (w/ Python). I'm trying to avoid any other ML libraries so that I can better understand the processes. Here is my approach and the problems that I am facing:

I mine thousands of film reviews as training sets and classify them as positive or negative.
I parse through my training set and for each class, I build an array of unique words.
For each document, I build a vector of TF-IDF values where the vector size is my number of unique words.
I use a Gaussian classifier to determine: $$P(C_i|w)=P(C_i)P(w|C)=P(C_i)*\dfrac{1}{\sqrt{2\pi}\sigma_i}e^{-(1/2)(w-\mu_i)^T\sigma_i^{-1}(w-\mu_i)}$$ where $w$ is the my document in a vector, $C_i$ is a particular class, $\mu_i$ is the mean vector and $\sigma_i$ is my covariance matrix.

This approach seems to make sense. My problem is that my algorithm is much too slow. As an example, I have sampled over 1,500 documents and I have determined over 40,000 unique words. This means that each of my document vectors has 40,000 entries and if I were to build a covariance matrix, it would have dimensions 40,000 by 40,000. Even I were able to generate the entirety of $\sigma_i$, but then I would have to compute the matrix product in the exponent, which will take an extraordinarily long time just to classify one document.
I have experimented with a multinomial approach, which is working well. I am very curious about how to make this work more efficiently. I realise the matrix multiplication runtime can't be improved, and I was hoping for insight on how others are able to do this.
Some things I have tried:

Filtered any stop words (but this still leaves me with tens of thousands of words)
Estimated $\sigma_i$ by summing over a couple of documents.


text:
How do probabilistic graphical models factor into modern machine learning?
I just finished the three-part series of Probabilistic Graphical Models courses from Stanford over on Coursera. I got into them because I realized there is a certain class of problem for which the standard supervised learning approaches don't apply, for which graph search algorithms don't work, problems that don't look like RL control problems, that don't even exactly look like the kind of clustering I came to call "unsupervised learning".
In my AI courses in the Institute, we talked briefly about Bayes Nets, but it was almost as if professors considered that preamble to hotter topics like Neural Nets. Meanwhile, I heard about "Expectation Maximization" and "Inference" and "Maximum Likelihood Estimation" all the time, like I was supposed to know what they were talking about. It frustrated me not to be able to remember statistics well enough to feel these things, so I decided to fill the hole by delving deeper into PGMs.
Throughout, Koller gives examples of how to apply PGMs to things like image segmentation and speech recognition, examples that seem completely dated now because we have CNNs and LSTMs, even deep nets that encode notions of uncertainty about their beliefs.
I gather PGMs are good when:

You know the structure of the problem and can encode domain knowledge that way.

You need a generative model.

You want to learn more than just one $X \rightarrow Y$ mapping, when you instead need a more general-purpose model that can be queried from several sides to answer different kinds of questions.

You want to feed the model inputs that look more like probability distributions than like samples.

What else are they good for?
Here are a few more related questions.

Have they not been outstripped by more advanced methods for lots of problems now?

In which domains or for which specific kinds of problem are they still the preferred approach?

How are they complementary to modern advanced methods?

This question was also asked here. Incredibly, this question without an answer got more views than the one there with an answer. I will not close this post for now, as the original question was too broad (so it could have been closed), while the current one is more acceptable.


text:
How are vectors and matrices multiplied in supervised machine learning?
I've recently started reading a book about deep learning. The book is titled "Grokking Deep Learning" (by Andrew W Trask). In chapter 3 (pages 44 and 45), it talks about multiplying vectors using dot product and element-wise multiplication. For instance, taking 3 scalar inputs (vector) and 3 vector weights (matrix) and multiplying.
From my understanding, when multiplying vectors the size needs to be identical. The concept I have a hard time understanding is multiplying vectors by a matrix. The book gives an example of an 1x4 vector being multiplied by 4x3 matrix. The output is an 1x3 vector. I'm am confused because I assumed multiplying vector by matrix needs the same number of columns, but I have read that the matrices need rows equal to the vectors columns.
If I do not have an equal number of columns, how does my deep learning algorithm multiply each input in my vector by a corresponding weight?

I the deep learning algorithm; I might have vector of three(columns) inputs. I also would have three weights. Each weight would have a vector of three columns to correspond With each input. These weights would be stored in matrix. The box gives an example of element wise multiplication with a 1x4 times 2x5 and states they don’t have same number of columns so it throws error. On the next page it give example of 1x4 times 4x3 and states the output is a 1x3 vector. This is contradicting.

In general, when people do not explicitly state it, a vector $v \in \mathbb{R}^n$ is usually considered a "column vector", that is, you can think of it as the matrix $v \in \mathbb{R}^{n \times 1}$ (that is, a matrix with $n$ rows and $1$ column).
If it is not explicitly stated and you assume that the given vector is a column vector, but the dimensions do not match, then you should check that the dimensions match if you consider the given vector as a "row vector" (because it might be the case that the author is implicitly considering the vectors as row vectors).
Having said that, you can multiply a vector $v \in \mathbb{R}^n$ by a matrix $A \in \mathbb{R}^{n \times m}$ from the left, that is, you can do $v^T A$. Here, I considered $v$ has a column vector (that is, $v \in \mathbb{R}^{n \times 1}$), even though I have not explicitly stated it: you can and you often need to deduce this from the context! If I transpose $v$, I obtain $v^T \in \mathbb{R}^{1 \times n}$, and, thus, you can see that you can indeed perform the operation $v^T A = u \in \mathbb{R}^{1 \times m}$. Note that, at this point, $u$, the vector resulting from the operation $v^T A$, is actually considered a matrix, but you can still use it as a vector, if you need and that is permitted according to the mathematical rules of the operations you need to perform. Note also that I cannot multiply $v$ from the right of $A$, because there is no way of making the dimensions match. Have a look at this question, if you do not know how to multiply a vector by a matrix from the left.
Similarly, you can multiply $v \in \mathbb{R}^n$ by the matrix $B \in \mathbb{R}^{m \times n}$ only from the right. If $v$ is a column vector (that is, $v \in \mathbb{R}^{n \times 1}$), you need to do $B v \in \mathbb{R}^{m \times 1}$, but, if $v$ is a row vector (that is, $v \in \mathbb{R}^{1 \times n}$), you will first need to transpose it, so that you can perform the operation: $B v^T\in \mathbb{R}^{m \times 1}$.
Furthermore, note that, if you multiply a vector by a matrix from the left, that vector needs to be a "row vector", so, if you initially assume that the vector is a column vector (or that is explicitly stated), you will need to transpose it first, before the multiplication. However, if the vector is already a row vector, you won't have to transpose it. Similarly, if you multiply a vector by a matrix from the right, you will need a column vector.
To conclude, you can multiply a vector either from the left or right of a matrix, but you need to make sure that the dimensions match: if you multiply from the left, you will need to check that the dimensions of the vector match the number of the rows of the matrix; if you multiply the vector from the right of the matrix, you will need to check that the dimensions of the vector match the number of columns of the matrix.

@DuttaA I read the question. I tried to give an answer which is as general and thus, hopefully, useful as possible. In my answer, I explain all the details that the OP is confused about.

@DuttaA I will wait for the feedback of the OP. If he's not happy with my answer, I will modify it. If he is also confused about how to structure his weight matrix, then this is another question, which either can be asked separately, or maybe I could also answer it here.

@DuttaA Also note that the OP stated "The concept I’m have hard time understanding is multiplying vectors by a matrix". In my answer, I give him the general guidelines/rules to follow. I also address his statement " Im am confused because I assumed multiplying vector by matrix needs the same number of columns as well but have read that the matrixes need rows equal to the vectors columns.", so the confusion lies in how to multiply vectors by matrices: this is what I supposed from his/her statements.

@DuttaA To me, it was clear (apart maybe from the last paragraph). He's confused about how to multiply matrices with vectors.

Thank you nBro. I feel slightly more knowledgeable but don’t understand how weighted matrix don’t need equal columns. In a book example; The weighted matrix has 3 rows that represent each type of weight. The columns provide the weighted percentage for the  corresponding input(the 1x3 vector). If my matrix did not have a third column then the input vector would not have a percentage weighed for that third column input from vector. Also the the rows of my matrix represent different types of weights.If I only wanted to use two weights, Now I would have two row matrix. This means I cant multiply

@zEyeland In a NN, weights are usually associated with connections between neurons. In a fully-connected feed-forward neural network, every neuron is connected to every other neuron. So, say, that you 3 neurons in the first layer and 3 neurons in the second layer, then you would need a $3 \times 3$ matrix, say $M$, to hold all $9$  "weights" of all possible connections between these neurons.

What the columns or rows represent in that matrix $M$ depends on the context: it might be that each row contains the weights associated with one neuron from the connections to all other neurons in the previous (or next) layer, or it might be that each column represents and holds such weights. It depends on the context (i.e. it may be explicitly stated or you need to deduce it from the math).

@zEyeland Please, if you have another question other than how matrix-vector multiplication works, then you should ask it separately and clearly.