text:
What is the space complexity of breadth-first search?
When using the breadth-first search algorithm, is the space complexity $O(b^d)$, where $b$ is the branching factor and $d$ the length of the optimal path (assuming that there is indeed one)?

The space complexity of the breadth-first search algorithm is $O(b^d$) in the worst case, and it corresponds to the largest possible number of nodes that may be stored in the frontier at once, where the frontier is the set of nodes (or states) that you are currently considering for expansion.
You can take a look at section 3.5 (page 74) of the book Artificial Intelligence: A Modern Approach (3rd edition, by Norvig and Russell) for more info about the time and space complexity of BFS.


text:
Off-policy full-random training in easy-to-explore environment
Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe).
In those environments, using off-policy algorithm, is it a good practice to train using exclusively random actions, instead or epsilon-greedy, Boltzmann or whatever ?
For my mind, it seems logical, but I have never heard about it before.

What makes you think that, given that DQN uses the experience replay (which is what I assume you mean by "DQN is offline"), then you never need to exploit? Can you also confirm that by "offline" you mean that it uses the experience replay or maybe you meant something else?

By "offline" I mean training a policy with samples not collected by this policy (isn't it the literature meaning ?). When you ask "then you never need to exploit" ? this is my question too. If we are using offline algo, and if we are able to collect data from all possible states with only random actions, is there a need for standard exploration strategies like e-greedy or Boltzmann, instead of a simple random exploration ?

That's "off-policy" and not "offline". Check this.

Thank you for this correction ! I confused, I have updated my question.

Ok, but I still don't fully understand the relationship between "being off-policy" and "always taking random actions" that you have in mind. Maybe you're thinking that, given that they are off-policy, in principle, you can use any behaviour policy, so you can just take random actions with off-policy algorithms. I think that would be possible. In fact, as far as I know, tabular Q-learning converges provided that you explore all states enough, which a random policy should do. However, note that, with function approximation (i.e. DQN), "enough exploration" may not be enough for convergence.

See this post: https://ai.stackexchange.com/q/11679/2444.

I've edited your post to put in the title what I think is your main question/concern. Please, make sure that's the case. Feel free to edit again your post to correct it, if that's not the case.

There is a misunderstanding, my question was exclusively about environments where a random exploration can explore every states, you didn't mention that in your edit (that is the purpose of my post)

As far as I know, an off-policy RL algorithm need to exploit to discover states it couldn't reach with random moves.
The purpose of my question is to know whether or not it should be necessary in environments where all states can be reached with random moves.
If you agree there is a misunderstanding, can you please restore the post ?

Do you know the difference between "exploitation" and "exploration" in RL? From what you said (i.e. you said "need to exploit to discover states"), it seems you don't know that these terms have specific meanings in RL. See this answer.

Thank you but I perfectly know the difference (same as offline/off-policy, it was just a typo yesterday because I wrote the post late). I did not say "need to exploit to discover states", I said "need to exploit to discover states it couldn't reach with random moves".
In most environments, you cannot reach complex states with only random exploration. So, a good exploration/exploitation tradeoff allow to further explore difficult-to-reach states. And so, I am asking why exploitation should be necessary in an env where random exploration can reach all states.

If you have infinite time and lives and all states are reachable from the start state, you could reach all states by random actions, in all environments where these conditions are met. I still don't understand what your question is, to be honest. The main requirement for convergence in tabular Q-learning is that you explore enough (i.e., in principle, you can just take random actions to "explore enough"), and not that you exploit enough (i.e. take the best current action), so that "you update the values of all states" (this is the idea).

Note that by taking random actions, you eventually end up in all different states, so you eventually reach all states (implied by taking random actions) and perform all actions (by definition of your behaviour policy). That's why you can "explore enough" with a random policy. Now, the question is: is it a good idea? Probably not, given that exploitation may help to converge faster (i.e. you should be updating values for actions that are best to reach to some, at least, local optima).

Why are you talking about tabular Q-learning ? The question was about algorithms such as DQN and environments where you cannot store all the states in a lookup table, and thus prefer using an approximator Q-function.
"you can just take random actions to "explore enough" -> that is an answer to my question. So you are saying exploitation is not necessary in those cases.
"If you have infinite time and lives", I really don't see why it should be more time expensive than an optimized policy, in envs such as tic-tac-toe easily explorable with random actions

I was talking about tabular Q-learning because in that case Q-learning is guaranteed to converge. DQN is Q-learning with function approximation and experience replay. As far as I know, DQN is not guaranteed to converge to the optimal value function, that's why I was restricting my reasoning to the tabular case.

"Note that by taking random actions, you eventually end up in all different states" -> The condition of my question is that we are in environments where an agent that takes random moves can reach all states as well as any other possible policies

I posted this on Reddit (didn't know there was a RL channel), so I will delete this post. I expect with your highlighting the post is better formulated.
I would be curious to know whether it is clearer or not.
Thank you for the time you spent to help me
https://www.reddit.com/r/reinforcementlearning/comments/jqvuwm/fullrandom_exploration_in_specific_environments/

To be honest, in that Reddit post, you're still apparently mixing the terms exploration and exploitation, so that post/question is not fully clear, as well as this one. I suggest that you reformulate your questions based on my suggestions.

Well so I think there is still a misunderstanding.
By "I removed the exploration strategy" I mean "use a random policy as exploration strategy" (this is my only mention of exploration so I suppose you're talking about that). I still don't see the mixing you are talking about.
Btw, I get the exact responses I was looking for, and the others seem to understand it pretty well so never mind. Thank you and good continuation

Yes, exactly, I was referring to that part "exploration strategy" (you should have said "I removed the exploitation"). Of course, I know what you mean, but that wording doesn't make the question very clear. That's what I mean.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the space complexity of bidirectional search?
Is the space complexity of the bidirectional search, where the breadth-first search is used for both the forward and backward search, $O(b^{d/2})$, where $b$ is the branching factor and $d$ the length of the optimal path (assuming that there is indeed one)?

Norvig & Russell's book (section 3.5) states that the space complexity of the bidirectional search (which corresponds to the largest possible number of nodes that you save in the frontier)
$$O(2b^{d/2}) = O(b^{d/2}).$$
The intuition behind this result is that (as opposed to e.g. uniform-cost search or breadth-first search, which have space (and time) complexity of $O(b^{d})$) is that the forward and backward searches only have to go half way, so you will not eventually need to expand all $b^{d}$ leaves, but only half of them.
However, this space complexity is correct if you use a breadth-first search for the forward and backward searches (which is your scenario!), given that breadth-first search, assuming a finite branching factor, expands one level at a time, so it's guaranteed that both the forward and backward searches meet in the middle. This can be seen in figure 3.17 of the same book, where you can see that both searches have the same "radius". Moreover, the only nodes that you need to store in the frontier are the ones on the circumference (not all nodes that you see in the image)

However, if you used another search algorithm to perform the forward and backward searches, the space complexity may be different. This is true if e.g. the searches do not meet and then they end up exploring all the state space.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
If uniform cost search is used for bidirectional search, is it guaranteed the solution is optimal?
If uniform cost search is used for both the forward and backward search in bidirectional search, is it guaranteed the solution is optimal?

UCS is optimal (but not necessarily complete)
Let's first recall that the uniform-cost search (UCS) is optimal (i.e. if it finds a solution, which is not guaranteed unless the costs on the edges are big enough, that solution is optimal) and it expands nodes with the smallest value of the evaluation function $f(n) = g(n)$, where $g(n)$ is the length/cost of the path from the goal/start node to $n$.
Is bidirectional search with UCS optimal?
The problem of bidirectional search with UCS for the forward and backward searches is that UCS does not proceed layer-by-layer (as breadth-first search does, which ensures that when the forward and backward searches meet, the optimal path has been found, assuming they both expand one level at each iteration), so the forward search may explore one part of the search space while the backward search may explore a different part, and it could happen (although I don't have the proof: I need to think about it a little bit more!), that these searches do not meet. So, I will consider both cases:

when the forward and backward searches do not "meet" (the worst case, in terms of time and space complexity)

when they meet (the non-degenerate case)

Degenerate case
Let's consider the case when the forward search does not meet the backward search (the worst/degenarate case).
If we assume that the costs on the edges are big enough and the start node $s$ is reachable from $g$ (or vice-versa), then bidirectional search eventually degenerates to two independent uniform-cost searches, which are optimal, which makes BS optimal too.
Non-generate case
Let's consider the case when the forward search meets the backward search.
To ensure optimality, we cannot just stop searching when we take off both the frontiers the same $n$. To see why, consider this example. We take off the first frontier node $n_1$ with cost $N$, then we take off the same frontier node $n_2$ with cost $N+10$. Meanwhile, we take off the other frontier node $n_2$ with cost $K$ and the node $n_1$ with cost $K + 1$. So, we have two paths: one with cost $N+(K + 1)$ and one with cost $(N+10)+K$, which is bigger than $N+(K + 1)$, but we took off both frontiers $n_2$ first.
See the other answer for more details and resources that could be helpful to understand the appropriate stopping condition for the BS.

It depends on the stopping condition.  If the stopping condition is "stop as soon as any vertex is encountered by both the forward and backward scan", then bidirectional uniform-cost search is not a correct algorithm -- it is not guaranteed to output the optimal path.  But it is possible to adjust the stopping condition to make bidirectional uniform-cost search guaranteed to output an optimal solution.
See the following resources for details, and the correct stopping condition:
Computing Point-to-Point Shortest Paths from External Memory.  Andrew V. Goldberg, Renato F. Werneck.  ALENEX/ANALCO 2005.
Point-to-point shortest path algorithms with preprocessing.  Andrew V. Goldberg.  International Conference on Current Trends in Theory and Practice of Computer Science, 2007.
Efficient Point-to-Point Shortest Path Algorithms.  Andrew V. Goldberg, Chris Harrelson, Haim Kaplan, Renato F. Wemeck.
I found these resources by looking at the Wikipedia article on bidirectional search; it mentions that the termination condition has been articulated by Andrew Goldberg et al and cites the third reference above.  Then a quick search on Google Scholar immediately turned up the other papers as well.
Lesson for the future: It can be useful to spend a little time checking standard resources (such as Wikipedia and textbooks), and checking the literature (e.g., with Google Scholar).  Many natural questions have already been answered in the literature.

It may be helpful to know that UCS is essentially the same thing as Dijkstra's algorithm.


text:
Thompson sampling with Bernoulli prior and non-binary reward update
I am solving a problem for which I have to select the best possible servers (level 1) to hit for a given data. These servers (level 1) in turn hit some other servers (level 2) to complete the request. The level 1 servers have the same set of level 2 servers integrated with them. For a particular request, I am getting success or failure as a response.
For this, I am using Thompson Sampling with Bernoulli prior. On success, I am considering reward as 1 and, for failure, it is 0. But in case of failure, I am receiving errors as well. In some error, it is evident that the error is due to some issue at the server (level 1) end, and hence reward 0 makes sense, but some error results from request data errors or issue at level 2 servers. For these kinds of errors, we can't penalize the level 1 servers with reward 0 nor can we reward them with value 1.
Currently, I am using 0.5 as a reward for such cases.
Exploring over the Internet, I couldn't find any method/algorithm to calculate the reward for such cases in a proper (informed) way.
What could be the possible way to calculate reward in such cases?

Let me try to clarify. Is this a bandit problem, i.e. do you have only 1 state or multiple ones? From your description, it seems that you have to select actions only once at the beginning, i.e. you have to select only 1 server (from a pool of servers), and this selection represents an action. However, it's not fully clear what you mean by "These servers (level 1) in turn hit some other servers (level 2) to complete the request.". How do they "hit" other servers? Do they do it automatically, stochastically or what?

Eaxctly, I have a collection of servers at level 1 from which I have to select one. I will send a request to these servers. These servers at level 1 will do their processing and generate another request which will be sent to server at lev 2. So these servers at lev 2 are the ultimate destination. The problem is identify best servers at lev 1 which will optimize my success rate. And I want to reward these servers(lev 1) for the error at their end (0 for failure and 1 for success) but there are cases where lev 1 servers are not responsible for failure and I want to calculate reward in such case


text:
Find repeating patterns in sequence data
I have database of sequential events for multiple animals.
The events are represented by integers so it looks something like:
Animal A: [1,6,4,2,5,7,8]
Animal B: [1,6,5,4,1,6,7]
Animal C: [5,4,2,1,6,4,3]

I can manually see that for each event 6 event 1 first happens.
And event 4 happens quickly after a 1,6 combination.
But these are easy to spot in such a small dataset, the real lists are 10000+ events per animal.
Is there a way to use an algorithm or machine learning to search for these kinds of patterns?

So, you do not know in advance how many patterns a sequence may have and the type of patterns, right?

@nbro I do not no. I could manually search for patterns but I want to also find some deeper hidden patterns which may not be obvious

Is the apriori algorithm not fit for this job?

@random_npc, yeah you are right. I wish I figured that out 2 years ago though :)


text:
Object detection noise filtering
In my project, I am detecting only one class, which is "airplane", using yolov5. However, at some frames, the neural network labels some of the buildings as airplanes, which obviously are not. This noise happens like 1 frame among 60 frames. How should I treat this issue? Which algorithms can be applied to filter out?

I had a similar issue with faces detection and Yolo. The best solution from an accuracy standpoint was in adding simple classification network for 2 classes 0-not a face and 1 -face. And passing resized block detected by you to double-check by this net. It perfectly filters almost all Yolo's false detections. Please let me know if you consider it can suit your case - I can share more details.

Hello @StepanNovikov actually I think this is what I need. Best way to see that is to test and see I believe. I trained my object detection yolov5 with the repository https://github.com/ultralytics/yolov5. I might be wrong but what you means is adding some extra layers at the end of yolov5 network,right?

Can you please rephrase this sentence "However, at some frames network boxes, some of the building around my perspective which obviously not an airplane." so that it is clear? What does "frames network boxes" mean? Then you say "some of the building around my perspective", what your perspective?

however at some of the frames, network tags some of the buildings as airplane which they are not. @nbro. Stepan understood btw.

@BerkantAy I am not sure how Stephan understood, but I didn't and my supposed job (according to me) is to make this community great and all the posts understandable to all users, including myself. Anyway, so, are you saying that your neural network detects some buildings as airplanes, every 1 over 60 frames/images? So, I suppose, you're trying to perform object detection on a video. Is this right? And what is the noise here? Is the noise the wrong detection? That's not noise, but your neural network performing bad.

It was an additional ANN simple classifier trained to determine if given input picture is a face or not. Just 2-3 conv layers + 2-3 MLP layers with SoftMax for 2 classes (for better accuracy). After you have this ANN trained, you need to verify boxes detected by Yolo if it really aircrafts or false detection by resizing detected box to the input of your checking ANN  and checking by it. In my case with faces I was able to get rid of majority false detected faces by Yolo. I guess it's better to create an answer with samples and links than continue here. Please confirm understanding correctness.

@StepanNovikov yes I got the idea but does this also mean that I should also label the buildings as non-airplane?

Yes, Berkant, exactly, all non-airplane objects, including building would be marked as belonging to non-airplane class and I would recommend including some or better yet all false detections to a training set of this additional checking ANN we discussing as examples of non-airplane object's class. In my experience where were some objects, especially small, which resemble a bit human faces or animals' faces considered as human faces by Yolo in my case, but easily filtered by this simple additional checking ANN.

Thanks @StepanNovikov I will try your solution and let you know.

If you are processing a video (because you used frames). Why not leverage the temporal correlation in the videos for filtering out FP? Use some recurrence man

I have been working on this for just 3 months for real world applications so I am trying to improve my skills on machine learning. Thanks a lot @JVGD


text:
Is it possible to improve the average precision of YOLO trained on Open Images Dataset by fine-tuning it with COCO?
I consider pre-training a YOLOv5 with Google Open Images Object Detection dataset. The dataset includes general domain categories with ~15 M box samples. After the pre-training is done, I will fine-tune the model on MS COCO dataset.
I would like to do it, if I can improve AP by ~7%. Do you think that it is possible, and I have a logical expectation? Unfortunately, I could not find anywhere anyone has tried an Open Images pre-trained object detector with MSCOCO training.

To clarify, by MSCOCO, you mean Microsoft's COCO dataset, right?

yes, that was right


text:
Finding or creating a dataset for Neural Text Simplification
I'm currently starting a research project focused on NLP.
One of the steps involved in this project will be the development of a text simplification system, probably using a neural encoder-decoder architecture.
For most Text Simplification research available, the most commonly used dataset is one derived from pairing Wikipedia entries in both English and Simplified English. My problem arises from the fact that the focus of my research is not on the English Language, but rather in Portuguese, specifically Portugal Portuguese.
There exists no Simple Portuguese Wikipedia page and it seems that there exists no publicly available text simplification dataset in Portugal Portuguese at all. Due to this fact I'm curious if there would be any way of tackling this problem. Maybe having a dataset simply of complex Portuguese and simple portuguese, but with no pairings, although I'm not quite sure how that could be formulated to train a NN with.
So my question is if there are any text simplification datasets in Portugal (or maybe Bazil, as a last resource) Portuguese and, if not, what would be the optimal way to build that dataset.
Thank you.

What is your specific question? Please, edit your post to clarify this (and put the specific question in the title). It seems that your problem is about training a model with a very small dataset (that you're planning to build). Have you heard of 1. one-shot learning or 2. transfer learning? If not, you may want to have a look at these concepts (then, as I said, please, edit the post to ask a specific question that also takes into account the info that I just gave you.)

Reformulated the question

If you're asking for datasets, then this is off-topic. See https://ai.stackexchange.com/help/on-topic. I'm not sure you're asking for datasets though. Can you clarify this? If you're looking for datasets, maybe https://opendata.stackexchange.com/ (or eventually Data Science SE) is a better place to ask this question. Here we focus on theoretical aspects of AI.


text:
Recent deep learning textbook (i.e. covering at least GANs, LSTM and transformers and attention)
I am searching for an academic (i.e. with maths formulae) textbook which covers (at least) the following:

GAN
LSTM and transformers (e.g. seq2seq)
Attention mechanism

The closest match I got is Deep Learning (2016, MIT Press) but it only deals with part of the above subjects.

There are a few more books that were published after 2016 that cover some of the topics you are interested in. I've not read any of them, so I don't really know whether they are good or not, but I try to summarise if they cover some of the topics you may be interested in.

Deep Learning with Python (2017), by Francois Chollet (author of the initial Keras library), which covers GANs in section 8.5 (p. 305), but it does not seem to cover transformers and attention mechanisms, although it covers other intermediate/advanced topics (not sure to which extent), such as text generation with LSTMs, DeepDream, Neural Style Transfer and VAEs

Grokking deep learning (2019), by Andrew Trask, which seems to cover some intermediate/advanced topics (such as LSTMs and related tasks), but no transformers or GANs (unless I missed them); you can find the accompanying code here

Generative Deep Learning: Teaching Machines to Paint, Write, Compose, and Play, by David Foster, which covers many variants of GANs, VAEs and other stuff

The first transformer was published in 2017, so I guess there may not yet be a book that extensively covers it and other related models, such as the GPT models (if you're interested in CV, check this blog post, although it seems to list books that cover mostly traditional CV techniques). The attention mechanisms are older and can probably be found in textbooks that cover machine translation topics (such as seq2seq models with LSTMs), such as this one.

I recommend Introduction to Deep Learning by Eugene Charniak ISBN 978-0-262-03951-2 (MIT 2018). It mentions GAN & LSTM & Attention (all three occurs in the index).
But read also Pitrat's last book: Artificial Beings: The Conscience of a Conscious Machine - it does cover machine learning (but not in the "deep learning" sense) but was published before 2016.
And see also RefPerSys. If you speak French, also see this and more generally the AFIA organization.

Why the downvote? Charniak book mention deep learning and is after 2016!

And it seems that Blincer lives (like I do) in France

The question is about books that cover GANs and transformers. I downvoted because you're promoting other books here, which have little to do with the question. Remove those books and I will remove my downvote. Also, you should explain if the DL book you're suggesting contains chapters (or at least some sections) dedicated to these topics. It should contain, otherwise, you should not be suggesting it. So, you should specify which chapters are dedicated to GANs and transformers.

My understanding is that it about books that cover machine learning, with GAN or transformers.

Deep learning typically refers to techniques that involve neural networks, so not all machine learning techniques/models. Please, do not excessively promote some textbook, project, or person in your answers. Try to stay focused and answer the actual question.

According to the index of that Charniak book, it is relevant (to all of  GAN, LSTM, Attention). And Pitrat's book has more than a chapter dedicated to machine learning (without GAN; LSTM, but with some attention).

Ok, the first book may be relevant, but the other 2 things? As I said, DL is not exactly ML, but a specific subset of it.

Then the question should be improved

I agree with you. Let's wait for feedback. In any case, it seems to me that the OP is looking for books that cover those mentioned topics.

Introduction to Deep Learning is a pretty good match thank you! The only issue is that it covers attention in only a specific case. I don't see how the rest of the answer is relevant.

There aren't really any current books, it's very strange. There isn't even a good computer vision book. Most are kind of ancient and don't include deep learning.

Here is an answer from 2023 for newcomers that may run into this page:
Dive into Deep Learning: Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola
Transformers for Machine Learning - A Deep Dive: Uday Kamath, Kenneth L. Graham, Wael Emara


text:
What is the effect of too harsh regularization?
While training a CNN model, I used an l1_l2 regularization (i.e. I applied both $L_1$ and $L_2$ regularization) on the final layers. While training, I saw the training and validation losses are dropping very nicely, but the accuracies aren't changing at all! Is that due to the high regularization rate?

The effect as you saw was that the algo is giving more priority in keeping it's weights low and sparse as compared to improving accuracy. Remember algo minimizes loss, it doesn't improve accuracy. Accuracy is generally related to loss unless you bring in regularizers.

So, I should use them carefully, right?

Maybe you should also provide the plots of the losses and accuracy, specify the training and validation datasets that you're using (including the number of examples that they contain), and specify the loss function you're using, and also the problem you're solving (binary classification)?


text:
How to frame this problem using RL?
How should this problem be framed in the domain of RL for preventing users from exceeding their bank account balance and being overdrawn?
For example, a user has 1000 in an account, and proceeds to withdraw 300, 400, 500, making the user overdrawn by 200 :((300+400+500) - 1000).
Treating this as a supervised learning problem, I could use logistic regression. The input feature is the transaction amounts. The input features for a training instance, 300,400,500 and the output feature occurs if the account is overdrawn or not overdrawn with corresponding values of 1 and 0 respectively. For simplicity, we will assume the number of transactions is consistent and is always 3.
For RL, a state could be represented as a series of transactions, but how should the reward be assigned?
Update:
Here my RL implementation of the problem:
import torch
from collections import defaultdict
gamma = .1
alpha = 0.1
epsilon = 0.1
n_episode = 2000
overdraft_limit = 1000

length_episode = [0] * n_episode
total_reward_episode = [0] * n_episode

episode_states = [[700,100,200,290,500] , [400,100,200,300,500] , [212, 500,100,100,200,500]]

def gen_epsilon_greedy_policy(n_action, epsilon):
    def policy_function(state, Q):
        probs = torch.ones(n_action) * epsilon / n_action
        best_action = torch.argmax(Q[state]).item()
        probs[best_action] += 1.0 - epsilon
        action = torch.multinomial(probs, 1).item()
        return action
    return policy_function

def is_overdrawn(currentTotal):
    return currentTotal >= overdraft_limit

# Actions are overdrawn or not, 0 - means it is not overdrawn, 1 - means that it will be overdrawn
def get_reward(action, currentTotal):
    if action == 0 and is_overdrawn(currentTotal):
        return -1
    elif action == 0 and not is_overdrawn(currentTotal):
        return 1
    if action == 1 and is_overdrawn(currentTotal):
        return 1
    elif action == 1 and not is_overdrawn(currentTotal):
        return -1
    else :
        raise Exception("Action not found")

def q_learning(gamma, n_episode, alpha,n_action):
    """
    Obtain the optimal policy with off-policy Q-learning method
    @param gamma: discount factor
    @param n_episode: number of episodes
    @return: the optimal Q-function, and the optimal policy
    """
    Q = defaultdict(lambda: torch.zeros(n_action))
    for ee in episode_states :
        for episode in range(n_episode):
            state = ee[0]
            index = 0
            currentTotal = 0
            while index < len(ee)-1 :
                currentTotal = currentTotal + state
                next_state = ee[index+1]
                action = epsilon_greedy_policy(state, Q)
#                 print(action)
                reward = get_reward(action, currentTotal)
                td_delta = reward + gamma * torch.max(Q[next_state]) - Q[state][action]
                Q[state][action] += alpha * td_delta

                state = next_state
                index = index + 1

                length_episode[episode] += 1
                total_reward_episode[episode] += reward

    policy = {}
    for state, actions in Q.items():
        policy[state] = torch.argmax(actions).item()
    return Q, policy

epsilon_greedy_policy = gen_epsilon_greedy_policy(2, epsilon)

optimal_Q, optimal_policy = q_learning(gamma, n_episode, alpha, 2)

print('The optimal policy:\n', optimal_policy)
print('The optimal Q:\n', optimal_Q)

This code prints:
The optimal policy:
 {700: 0, 100: 0, 200: 1, 290: 1, 500: 0, 400: 0, 300: 1, 212: 0}
The optimal Q:
 defaultdict(<function q_learning.<locals>.<lambda> at 0x7f9371b0a3b0>, {700: tensor([ 1.1110, -0.8890]), 100: tensor([ 1.1111, -0.8889]), 200: tensor([-0.8889,  1.1111]), 290: tensor([-0.9998,  1.0000]), 500: tensor([ 1.1111, -0.8889]), 400: tensor([ 1.1110, -0.8890]), 300: tensor([-1.0000,  1.0000]), 212: tensor([ 1.1111, -0.8888])})

The optimal policy is to inform us if 700 is added to the balance, then the customer will not overdraw (0). If 200 is added to the balance, then the customer will overdraw(1). What avenues can I explore to improve upon this method as this is quite basic, but I'm unsure as to what approach I should take in order to improve the solution.
For example, this solution just looks at the most recent additions to the balance to determine if the customer is overdrawn. Is this a case of adding new features to the training data?
I'm just requesting a critique on this solution so I can improve it. How can I improve the representation of the state?

In RL there's an agent that interacts with the environment by selecting actions. In your case, the agent would be pretty much giving yes/no answers, so there would be two actions. How are you input in the sequences? Do you put them in at once or do you want to have a training loop where agent receives them one by one?  If you put the whole input at once, then agent just selects whether it thinks the amount is overdrawn and the episode is done. You can give positive award (1, for example) for getting the answer right and negative award for getting it wrong.

@markmark the sequences are input one by one.

@markmark I'm not trying to train the agent to determine if the account is overdrawn which is what I think your suggesting, I'm attempting to train the agent ti to determine if the account will be become overdrawn based on a series of transactions.

You don't need supervised learning, reinforcement learning or any kind of learning. The problem is elementary school algebra?? If not then formulate your question better because right now it doesn't make any sense what you want to accomplish here with learning approaches.

If you're going with RL, I think you need to define action space (what actions can be taken by agent, in this case action is deciding whether overdrawing will happen), observation space (what information is available to agent at every step? A sequence and current accounts balance), what an episode of training/evaluation is like (do you receive multiple sequences for different accounts in every episode?) and the reward(did agent succeed at guessing? give positive reward).

Do you want to put numbers of the sequence one by one? Then you can have episodes give account balance and current withdraw amount as an observation. The agent then evaluates at every step whether the current balance is overdrawn. And the episode ends when the sequence ends.

@markmark please see question update.

(Disclaimer -- I'm kinda new to this). Q-learning is for discrete action/observation space. The amount of money can be floating point value. Even if you keep it just to integers, the observation space will grow too large. You may need to use deep-Q-learning. Deep Lizard made good videos on that (starting from video 6 in this playlist https://www.youtube.com/watch?v=qhRNvCVVJaA&list=PLZbbT5o_s2xoWNVdDudn51XM8lOuZ_Njv&index=6).

Right now, your agent just looks at the number and decides whether it's negative or not. If you want to make your agent more sophisticated, you probably would want it to learn the relationship between two numbers - current balance and the next amount to withdraw.

With deep-q-learning, you would be able to pass these two numbers as features to the neural network, which after training should be able. This problem can be solved with RL, but RL is more for cases where agent actually interacts with the environment -- here your agent's actions don't really affect the state of the environment.

I hope this helps

The question is very unclear. To clarify, you should give the rules that the agent is working with - the state description, the action space, the reward (or goal that you are hoping to achieve). I cannot see anything that looks like a machine learning problem of any kind so far. Predicting the minimum amount required to cover expected withdrawals could be a machine learning problem - of predicting likely withdrawals - if you had some other data describing the account owner's spending habits.


text:
Is there any rule of thumb to determine the amount of data needed to train a CNN
I am training an AlexNet Convolutional Neural Network to classify images in a dataset. I want to know if there is any general rule for using data augmentation in training a neural network. How can I make sure about the amount of data, and how can I know if I need more data?

It's not very clear the question in the title though. What do you mean "rule for number of applying"? Maybe your question is: "Is there any rule of thumb to determine the amount of data needed to train a CNN? Moreover, it seems that you're asking another question: "Which data augmentation techiniques should you use? Is there any rule?". Please, if you have two question, ask each of them in its separate post.

@nbro No, I just want to know about the amount of data, I edited the title again, sorry for my english.

So, why did you write "I want to know if there is any general rule for using data augmentation in training a neural network"?

Your question seems quite broad and unclear – especially the added question about data augmentation. My understanding is that there is no "rule of thumb"; the amount of data you need depends on the complexity of the images that you are / will be dealing with. Here are some sources on data augmentation: https://journalofbigdata.springeropen.com/articles/10.1186/s40537-019-0197-0 , https://arxiv.org/abs/1712.04621 , https://arxiv.org/abs/1609.08764


text:
How do we interpret the images of weights in logistic regression
The following images are
a) The weights of a logistic regression model trained on MNIST.
b) The sign of the weights of a logistic regression
How do these images  represent the weights?
Would be grateful for any help.
Source of the research paper

It depends on what you are training the model for. If you are trying to classify a face weights will be given to things like eyes nose etc. Just think how would you recognise something, and thats how the LR model will distribute the weights, how the features affect the classification (this is an overgenralization, but you get the idea)

I understand, maybe you can write it as an answer

Sorry it takes a long time to write an answer, and I am very short of time. But this actually a very common concept. You can find this in any introductory NN book. This is basically what we call Hebbian Learning. The very first paradigm of computational learning.

Thank you very much!


text:
Time series analysis using computer vision principles
I'm just starting to explore topics within computer vision and curious if there are any concepts in that area that could be applied to segmenting multivariate time series with the goal of grouping individual data points similar to how a human might do the same.  I know that there are a number of time series segmentation methods, but in-depth explanations of multivariate methods are more scarce and it seems like somewhat of an underdeveloped topic overall. Since segmentation is such a fundamental part of CV and is inherently multidimensional, I'm wondering if concepts there can be modified to apply to time series.
Specifically, I'd like to be able segment a time series and reformulate a prediction problem as something closer to a language processing problem.  The process would look something like this:

Segment a multivariate time series into near-homogenous segments of variable length.  Some degree of preprocessing might be required but I can worry about that separately.
Encode the properties of each segment based on summary statistics (e.g., mean, variance, derivative values, etc.) such that the segments fall into discrete buckets.
Each bucket will represent a "word" and the goal of the model will be to predict the next word given a series of words, i.e., the next segment given a series of segments.

In a few days of reading about CV, it seems like there's a ton to learn.  If there are traditional time series segmentation techniques that are more suitable, that would be of interest, but I'd still be curious about a CV approach since that approach likely better aligns with how a person might look at a graph to identify segments.

Your idea of representing the time series as words reminds me of Symbolic Aggregate approXimation (SAX) (https://jmotif.github.io/sax-vsm_site/morea/algorithm/SAX.html).

You could have a look at This  and this


text:
How to draw a 3-dimensonal shape's neural network
I am reading an exam question about NN (that I cannot publish, for copyright reasons). The question says: 'Construct a rectangle in 2D space. Define the lines, and then define the weights and threshold that will only fire for points inside the rectangle.'
I understand that this is an example of a rectangle drawn as a NN (i.e. this NN will fire, if the point is in the rectangle, where the rectangle is defined by the lines X = 4; X = 1, Y = 2, Y = 5).

In this diagram, since it's a rectangle, the equations of the line in this example are x = 4, x =1, y=2, y=5, so I left the other weights out (as they equal to 0).
I'm now wondering how this could be translated to a 3D structure. For example, if a 3D shape was defined by the points:
(0,0,0), (0,1,0), (0,0,1), (0,1,1), (1,0,0), (1,1,0), (1,0,1), (1,1,1)
I wanted to draw a hyperplane that separates the corner point (1,1,1) from the other points in this cube. Can this 3D shape be drawn similarly to below (maybe it would be easier to understand, if there were other numbers except 1 and 0 in the co-ordinates)?
Would I draw this with 3 nodes in the input layer, still one node in the output layer, I just don't understand what the hidden layer should look like? Would it have 24 nodes? One for each surface of the cube, with relevant X and Y values?

In your neural network diagram, there are no weights on the connections between the $x$ input and the 2 bottom hidden neurons (in the second layer). I assume that those weights are zero. Right? Similarly, I assume that the connections that come from the input $y$ to the top 2 hidden neurons are zero.

I didn't really think about what this neural network does, but this is what comes to my mind in order for that to make sense to me, given that, if you're checking whether $x$ is within the range $[1, 4]$ (which is the $x$ range of the rectangle, you don't need to care about the values of the $y$ range. In any case, even if this reasoning was correct, I don't fully understand what the output of this NN is supposed to be to consider that a point is inside the rectangle. Maybe you need to clarify that. Where did you take this NN from?

Can you provide a link to the resource that claimed that this NN computes whether a point is inside a NN, so that we have more context?

@nbro, regarding your first question, sorry yes the weights not shown are 0, since it's a rectangle, the equations of the line in this example are x = 4, x =1, y=2, y=5, so I left the other weights out.

For your second and third point, I appreciate what you're saying, I just can't for copyright. But for your second point, I'm reading a not-publicly-available exam question that says (and this is the whole question): 'construct a rectangle in 2D space. Define the lines, and then define the weights and threshold that will only fire for points inside the rectangle'. So I did that (answer above), but I want to understand how this could be pushed further, to a 3D shape?

Please, edit your post to include these details.


text:
Multivariate time-series classification with many variables
I am attempting to use time-series classification algorithms for fraud detection applications. I have came across several works in the literature that propose novel techniques for multivariate time-series classification, however, most of these approaches treat each feature as an individual signal.
Now, my processing of my data transforms a transactions dataset into a tensor; 1 dimension where each observation is an account, 1 dimension where each element is a transaction and 1 dimension for the transaction attributes. The transactions dataset has a large number of features, many of which are one-hot encoded categorical variables. Therefore, I am not really sure that multivariate time-series classification algorithms such as CNN or LSTM will work in this case, since it will treat every one-hot encoded feature as a signal on its own.
What would be an alternative approach in this case? Would applying PCA on the data to capture the most significant features help instead of the ordinary features?

Why are you not using ordinal encoding?

Why haven't you tried CNN and LSTM to find out?  It doesn't take that much effort.

My only issue with ordinal encoding is figuring out how many encodings per categorical feature I need to generate. I have some categorical features with 20 unique values, and a few other features have over 300. The reason I haven't attempted the methods yet is because I need to preprocess and generate more features, which will be a time consuming step. I want to know if deep learning in this case is feasible, given that there is a huge class imbalance in my dataset as well.


text:
What are good techniques for continuous learning in production?
I was wondering which AI techniques and architectures are used in environments that need predictions to continually improve by the feedback of the user.
So let's take some kind of recommendation system, but not for a number of $n$ products, but for some problem of higher space. It's initially trained, but should keep improving by the feedback and corrections applied by the user. The system should continue to improve its outcomes on-the-fly in production, with each interaction.
Obviously, (deep) RL seems to fit this problem, but can you really deploy this learning process to production? Is it really capable of improving results on-the-fly?
Are there any other techniques or architectures that can be used for that?
I'm looking for different approaches in general, in order to be able to compare them and find the right one for problems of that kind. Of course, there always is the option to retrain the whole network, but I was wondering whether there are some online, on-the-fly techniques that can be used to adjust the network?

Each type of system will have its own way of coping with this issue. Even within recommender systems there will be different approaches, and prgamatic solutions such as partial online learning - e.g. online updates to each user's profile can be done in real time, but training the aggregate model for all users is likely done offline on some routine (e.g. overnight).

I agree here with nbro, the question needs more focus in order to be answerable. Please use [edit] to make your question clearer. If you have a specific type of recommender system in mind it could help.

Thanks for commenting.
The point is, that I'd like to compare different approaches on that, so I'm quite free with the technique. It's more about how in general, something like this can be done, like offline retraining, or on-the-fly-adjustments


text:
Correct way to work with both categorical and continuous features together
I have a time series with both continuous and categorical features, and I want to do a prediction task.
I will elaborate:
The data is composed of 100Hz sampling of some voltages, kind of like an ecg signal, and of some categorical features such as "green", "na" and so on.
In total, the number of features can reach 300, of which most are continuous.
The prediction should take in a chunk of frames and predict a categorical variable for this chunk of frames.

I want to create a deep learning model that can handle both categorical and continuous features.
Best I can think of is two separate losses, like MSE and cross entropy, and a hyperparameter to tune between them, kind of like regularization.
Best I could find on this subject was this, with an answer from 2015.
I wonder if something better was invented, since then, or maybe just someone here knows something better.

Can you please provide more info about your actual data?

@nbro sure, like what? I can describe everything, but not sure what exactly is needed, please say and I will add.

Ok, now, after thinking about it a little more, I realize that you want to predict what's going to come next in the time-series data, and that's why you thought about using two loss functions. That wasn't initially clear to me, and that's why I asked you to provide more details about the data, but I guess that's no longer strictly needed. In any case, it may be useful to know the nature of your data, e.g. if they are sequences of images and numbers or whatever.

@nbro I now see I was not clear. I edited, hope it is better now.


text:
Is there a reference that describes Recurrent Neural Networks for NLP tasks?
I would like some references of works that try to understand the functioning of any kind of RNN in natural language processing tasks. They can be any work that tries to explain the functioning of the model by studying the structure of the model itself. I have the feeling that it is very common for researchers to use models, but there is still little theory about how they work in solving natural language processing tasks.

Did you do any research yourself?

I think that this request "works that try to understand the functioning of any kind of RNN in natural language processing tasks." is a bit broad. There are many NLP tasks. So, I suggest that you reformulate this post to ask a more specific question (maybe focus on a specific NLP task) or, at least, explain why you're interested in "all NPL" tasks.

Moreover, as suggested in the previous comment, there are several references on the web that already explain how RNNs work (such as this), even if you say "but there is still little theory about how they work" (which is incredible, as there are many resources explaining how RNNs work).


text:
Is it possible and if so does it make sense to have dense layers in between LSTM layers?
I am new to LSTMs and I was wondering if it is possible to have LSTM layer then dense then LSTM again and does it make sense?

Hello. Welcome to AI SE! Can you please provide a little bit more context? Why were you wondering about this? Are you asking whether this is theoretically possible or if this could be useful in some cases or has been used in some cases? Please, edit your post to include more details and address these questions. Please, take a look at https://ai.stackexchange.com/help/on-topic.

I just know it's possible and you can easily implement it, but I don't know if it's good or not


text:
Is there any neural network model that can perform multiple NLP steps at once?
I realize most NLP algorithms have multiple steps. (e.g. OCR/speech rec > syntax > semantics > response logic > semantic output > natural language output)
Is there any NN model that can perform multiple steps in NLP at once? For example, a single network which accepts audio input and returns a semantic analysis of the given speech, or a single network which accepts text input and returns natural language output?

Is T5 the sort of thing you're looking for? https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html

Yes! That's very close to what I had in mind. Do you know of anything similar that is capable of OCR/speech recognition as well?


text:
Can ML/DL solve my classification problem?
I'm new to AI but would still like to try and get a project off the ground.
I've read a lot about ML/DL the past few days but I just can't figure out if my problem can be solved with ML/DL. What I'm trying to do looks like a classification job to me but maybe isn't.
I have 100s of images of compacted soil samples, on these images there may be multiple layers visible.
I will include a picture below, this sample had a sticker on it, normally they don't. On the image there are 3 layers, separated above and under the sticker.
With every image there is data (xml file) available on the size of the layer(s) and the type of soil in that layer, which costs a lot of time to produce, so I want to automate this classification in the future.
The data files contains info like:
layer0:
    type 004
    2cm
    12cm
layer1:
    type 003
    12cm
    25cm

If there would be just one layer, the AI could learn what these layers look like and sort them in the right soil class.
But I don't know if my problem can be solved with AI as there could be 1, 2, 3 or 4 different layers (classes) on one image and I haven't seen any examples on classification where there can be multiple classes in one image.
As AI is quite a steep learning curve I would like to know if my problem is suited for ML/DL before I spend more of my nights reading for something that might not work.
I've read numerous websites and a few short books but can't find an answer to my questions.
Can ML/DL solve my multi-class single-image classification problem and which strategy should I read into?

I don't understand what a layer is in your image that you're showing us. I only see a black and white image with some patterns that look like Perlin noise. So, can you clarify what a layer is here? Also, what do you mean by "sticker" here? Do you mean the big rectangular thing?

Hey @nbro thnx for commenting, as specified in my question there are 3 layers and thus 2 separation lines, these lines can be seen with the naked eye just a little above the upper side of the rectangle, and a little above the underside of the rectangle, and yes, the rectangle is the sticker. The 'Perlin noise' is the soil composition and its the data I'm trying to analyse. I will try to get a second image that I can make a little clearer with some CV.

A simple sanity-check on whether an image classifier can perform a task in theory is:

Can a human expert, using the same image plus a list of catgeories that they are familiar with, perform the same task?

It is important you only consider the contents of the image (or in general the data you are prepared to supply to the classifier) and the expert's general knowledge. The expert is not allowed to collect more data for instance, or interact with the sample other than maybe take a few measurements on the pixels.
This sanity check doesn't tell you how hard the problem is. It also does rule out problems that are very hard or impossible for humans but actually quite easy for computers. However, it is a good start because nowadays single-purpose computer vision classifier tasks often rate similarly to or better than humans performing the same task. You are effectively checking "is the data I need for the inference actually in the image?"
Multi-class classifiers are possible in several ways. One way would be to have separate heads to the neural network to classify each layer, and maybe a layer present binary flag to allow for varying numbers of layers. This is similar to an architecture called YOLO which classifies 0 or 1 objects and their locations over multiple grid squares within an image. Your architecture would need to be different to YOLO, but you could use a lot of the ideas from it, such as having an output with multiple multi-class classifiers, one for each soil layer.

I have 100s of images of compacted soil samples

One issue you will face is that you would need a very large number of labelled images in order to train a neural network with cutting-edge performance for this task from scratch. So you will want to look into transfer learning, which involves taking an existing image classifier trained on e.g. ImageNet and adapting it to your problem before training with your smaller dataset.
The small amount of sample image data you have will be a major limiting factor in your case. Sadly no-one can tell you before you attempt the project whether you have enough for a deep learning approach. That is probably where your largest risk of failure is.

Thank you for this very informative answer, @Neil Slater! You've given me some to think about! cheers


text:
Why do I get the best policy before Q values converge using DQN?
I have implemented DQN algorithm and wonder why during testing, the best performance is achieved by a policy from about 300 episode, when mean Q values converge at about 800 episode?

Mean Q-values are calculated on a fixed set of states by taking mean of max Q-values for each state.
By convergence I mean that the plot of mean Q-values converge to some level (those values does not increase to infinity).

It can be seen in here (page 7) that mean Q-values converge and average rewards plot is quite noisy. I get similar results and in tests, the best policy is where the peaks are during training (average reward plot). I don't understand why don't I get better average scores over time (and better policies) when Q-values converge.

Even if the mean of the maximum Q-value increases from episode 300 onwards, it doesn't mean that the relative order of the Q-values of the actions that you can take in the states change, which means that the policy may not change, even though the value function changes, assuming you're acting greedily with respect to the value function.
More concretely, suppose that you can take one of two actions $\{ a_1, a_2\} = \mathcal{A}$ in each state $s \in \mathcal{S}$. Let's say that you pick $s_1, s_2$ to calculate the average of the maximum Q-value. Without loss of generality, suppose that the action associated with the highest Q-value in these states is $a_2$. If your policy is the greedy policy with respect to the state-action value function, then your policy will choose $a_2$ in $s_1$ and $s_2$. If all Q-values $\hat{q}(s_1, a_1)$, $\hat{q}(s_1, a_2)$, $\hat{q}(s_2, a_1)$ and $\hat{q}(s_2, a_2)$ increase (or even decrease) but their relative order remains the same (i.e. $\hat{q}(s_1, a_2) > \hat{q}(s_1, a_1)$ and $\hat{q}(s_2, a_2) > \hat{q}(s_2, a_1)$, from episode 300 onwards), the greedy policy also remains the same.
So, I think that what you observed is theoretically possible, although I cannot guarantee that you don't have another issue.


text:
Is there a reason why no one combines word embeddings with the median?
Could you combine word embeddings with the median per dimension to get a document embedding? In my case I have a huge amount of words to build one document, which in turn should describe a topic. I feel like using the median is the right thing to do, as I get the most common parameter value per dimension. However, I cannot find anyone trying it before. This is why I'm wondering, is there something speaking against it?

What do you mean by "combine word embeddings with the median"? Median of a sequence? Which sequence? How how do you want to combine "word embeddings" with the "median". I don't understand this question maybe because I'm misinterpreting the word "median".

I think @Stephan means combining a number of embeddings into one by computing the median per dimension. I remember reading a paper that showed mean generally works best, but I can't find it now. In any case, median may work well because you can think of each embedding dimension as a distribution. If the distribution is skewed, the median will have a higher posterior density than the mean, but intuitively these dimensions would be approximately normal so the mean and median would be similar. Try it out!


text:
Data Augmentation of store images using handwritten labels
I am new to AI and NN. I've started learning using Geron's book on Tensorflow.
My first project ("Smart Shelf") is to determine which items in a store have been purchased and need refilled. The store camera periodically takes pictures of the tops of items on store shelves. To start, we have only 5 distinct products.
We have created ~250 handwritten images of product-labels that cover these 5 distinct products. So far, the training results are way below our expectation.
I am thinking to augment the training data and see whether it would make any difference.
I have thought about the following strategies:

Train the model again using grayscale images. https://stackoverflow.com/questions/45320545/impact-of-converting-image-to-grayscale/45321001
Invert images, translate them horizontally or vertically https://www.tensorflow.org/tutorials/images/data_augmentation, https://nanonets.com/blog/data-augmentation-how-to-use-deep-learning-when-you-have-limited-data-part-2/

Which of the above will yield better results and why? I am curious. Thanks for any help. I feel that I know various data augmentation techniques, but not sure how and why to apply them.

It seems this is a popular question, as learned from Choosing Data Augmentation smartly for different application etc.

Hi! Sorry but I'm not understand about this part "~250 handwritten images that cover these 5 distinct products", can you edit your question and give more information or examples?

Sure Malioboro. Those handwritten images are of data labels.


text:
Understanding policies in helicopter control in the paper by Andrew Ng et al
I was going through this paper on helicopter flight control using reinforcement learning by Andrew Ng et al.
It defines two policy classes to learn two policies, one for hovering the helicopter and another for maneuvering (aka trajectory following). The goal for hovering policy is defined as follows:

We want a controller that, given the current helicopter state and a desired hovering position and orientation $\{x^*, y^*, z^*, \omega^*\}$ computes controls $a\in [-1,1]^4$ to make it hover stably.

The goal for maneuvering policy is given in term of hovering policy as follows:

Given a controller for keeping a system’s state at a point $(x^*,y^*,z^*,\omega^*)$, one standard way to make the system move through a particular trajectory is to slowly vary $(x^*,y^*,z^*,\omega^*)$ along a sequence of set points on that trajectory points.

The neural network for these policy classes is shown as follows:

In this, $(\dot{x},\dot{y},\dot{z})$ are velocity estimates, $(\phi,\theta)$ is helicopter roll and pitch estimates and $\dot{w}$ is angular velocity component estimate (more on this at the bottom of page 1 of the linked paper).
Each edge with an arrow in the picture denotes a tunable parameter. The solid lines show the hovering policy class. The dashed lines show the extra weights added for trajectory
following (maneuvering). With this observation, I had following doubts:
Q1. Does addition of dashed lines to hovering policy to get maneuvering policy makes manuvering policy superset of hovering policy?
Q2. Rephrasing Q1: can we use maneuvering policy for hovering task (say by setting weights corresponding to dashed lines to zero)?
Q3. If maneuvering policy is indeed a superset of hovering policy, why the authors dont use just maneuvering policy for both tasks or maneuvering policy for hovering task also? Is it because it involves computation involving helicopter's additional sub dynamics represented by dashed line and this additional computation is not required for hovering task?
Or am I completely getting wrong with all these questions?

Too many questions in the same post. Please, edit this post to leave only 1 question. Ask the other questions in their separate post, even though they are all related.

All questions are very tightly related. So I felt asking them together will help answerer to stay in the larger context and answer more holistically. Asking them as separate post will require me to copy paste everything (the related context from the paper) before Q1 in three different posts. Is such duplication preferred on stackexchange?

It's fine to provide enough context to be able to answer your question in each post, even if that means somehow that there will be a bit of duplication. In my opinion, what's not fine is that you ask us to put the effort to answer multiple questions. Unless someone has read or reads the full paper, which requires some time and effort, it's difficult to answer any of the questions at all. So, that's why I'm suggesting you to simplify this post as much as possible and ask only 1 main specific question. Moreover, I would suggest that you put your main specific question in the title.

As a rule of thumb, if you can't do that, then it means that your post is too broad. Right now, "Understanding policies in helicopter control in the paper by Andrew Ng et al" is like the title of a blog post where you explain everything of one paper.


text:
Incorrect node expansion in game board with A* search
I have the following game board below, and we're using A* search to find the optimal path from the agent to the key. There are 8 directions. Up, down, left, right have a cost of 1, and diagonal directions have cost 3. We will be using a priority queue with function $f(v) = g(v) + h(v)$ where $g(v)$ is the backwards cost from the goal through the given edges and up to the vertex v while $h(v)$ is the optimal least cost distance from v to the goal node.

So I calculated the f(s) for the different states, assuming no prior edges specified:

And then I started the search and these are the steps I took: expand C: (CD,3), (CE,3), (CF,3), (CA,5), (CB,5)
expand CD: (CDF,3),(CE,3), (CF,3), (CA,5), (CB,5), (CDB,5)
expand CDF: (CDFH,3), (CE,3), (CF,3), (CA,5), (CB,5), (CDB,5), (CDFG,6)
expand CDFH: (CE,3), (CF,3), (CA,5), (CB,5), (CDB,5), (CDFG,6)
So I only expanded, C,D,F,H. I got the correct answer for the optimal path, but not the correct answer for nodes expanded, which is supposed to be C, D, E, F, G, H. What am I doing wrong?

In the first iteration of $A*$ (and basically many other state-space search algorithms), you will only add to the frontier the nodes reachable from your current node, but $H$ and $G$ are not reachable with a single action from $C$ (if I interpret correctly your state-space), so I'm not sure how you calculated $f(n)$ for those (in the table). So, you should probably try to think about this issue before proceeding. I would also suggest that you draw your state space as a graph, so that to avoid this type of confusion again.

those f(s) values are just there for the question, I dont use those exact values for the queue. are my expansion steps right?

If I understand correctly, diagonal moves cost 3. So the first expand: CF=3+1, CB=3+3

@Manaal As I said above, it looks like you didn't really understand how A* (or any other state-space search algorithm) expands the nodes or adds them to the frontier, so I would suggest that you do what I said. Draw your state space as a graph, where only reachable nodes are connected by an edge (so no edge between C and H, e.g.). Moreover, someone also pointed out above that you didn't calculate correctly f(n) for several nodes. I would suggest that you delete this post, do what I suggested, try to solve the problem again, then ask another question, if you still have a question.


text:
How to deal with dynamically changing input tensor in neural networks without padding?
I have a dataset about the monitored health/growth of a community of people. The dataset has tensor shaped (batch_size, features, person, window), where:

person==10 means there are 10 people in the community
features==9 means that there are 9 features being monitored, for example, blood pressure, sugar level, ..etc
window==15 means the recorded value of each feature every day for 15 days (time dimension)

Moreover, people can join/leave the community, so the person dimension would increase/decrease over time. For simplicity, the window dimension is fixed at 15, then a new person that joined has to be in the community for a minimum of 15 days to be included in the dataset as 1 data point/sample. Also, say the number of features is fixed at 9. Hence, for this problem, only the number of people at an instance may change over each input interaction.
For example, assume batch_size==1 then the input dimension into the neural network would be something like:
Iter 1: (1, 9, 7, 15)
Iter 2: (1, 9, 7, 15)
Iter 3: (1, 9, 7, 15)
Iter 4: (1, 9, 8, 15) # 1 person joins the community
Iter 5: (1, 9, 8, 15)
Iter 6: (1, 9, 7, 15) # 1 person left the community
Iter 7: (1, 9, 6, 15) # 1 person left the community
Iter 8: (1, 9, 6, 15)
Iter 9: (1, 9, 10, 15) # 4 person joins the community

Is there a way to deal with this dynamically changing input tensor in neural networks without padding? As we won't know in advance how many people will join/leave the community (related to continual learning?) hence won't know the maximum pad.
Also, how to deal when batch_size is not 1?

So, each community contains certain number of people, and each of them has specific characteristics? You want to infer something about the community, based on it's residents?

What is the target variable you are trying to predict?


text:
What is a "learned policy" in Q-learning?
I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and the task is to apply the Q-learning algorithm to fill out a Q-table (which I've done) but confused on what it may mean by the learned policy.
So, what is a "learned policy" in Q-learning?

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed something about the policy.
The most common policy scenarios with Q learning are that it will converge on (learn) the values associated with a given target policy, or that it has been used iteratively to learn the values of the greedy policy with respect to its own previous values. The latter choice - using Q learning to find an optimal policy, using generalised policy iteration - is by far the most common use of it.
A policy is not a list of values, it is a map from state to actions. The question wants you to show the policy that you have learned the Q values for.
The policy in your case is therefore likely to be to pick the action that has the highest action value in each state. You may be able to decribe your answer in text ("always turn left unless next to the exit") or as a graphic (draw arrows on a grid world to show the preferred direction). Or you could write out a table of states showing the chosen action in each one.
The maths notation for how you derive the policy from a Q table can be written:
$$\pi(s) = \text{argmax}_a Q(s,a)$$
Or a bit more formally:
$$\pi: \mathcal{S} \rightarrow \mathcal{A} = \text{argmax}_{a \in \mathcal{A}(s)} Q(s,a)\qquad \forall s \in \mathcal{S}$$


text:
Accuracy goes straight for about 200 epochs then start increasing
Can anyone explain the following observation?
Why did the accuracies keep to be a straight line with a very smooth decrease of loss?
Is this because of the learning rate or other reasons?
Some info:
The input is in the dimension of (319,50,40) as (batch, step, features)
The dataset consists of 319 samples. It was split using train_test_split() to yield 0.2 test size.
I used a self-attention LSTM model with 4 Dense layers and 1 self-attention LSTM layer. The codes are long, I will post them if it is needed.
The hyperparameters:
batch_size= 100
epochs =1400
learning_rate = 0.00014
optimizer = 'RMS'
num_classes = y_train.shape[1]
dropout=0.37

In addition,
If I don't set random seed and keep shuffle=True, Sometimes, I get a horizontal line till the endo of training without any increasing.

Well these type of questions can be best answered on datascience stack exchange, you may not find much people active on this forum.

that's strange, loss is reducing quick but accuracy stays horizontal

can you give more context? it's possible if the model is trained with different behavior after some epochs ( changing learning rate, etc.)

@malioboro No, every parameter stays the same and increasing the learning rate shorts the horizontal straight line.

how about your data validation size? if your data validation is small, it's also possible you have the same accuracy score for some epochs, please edit your post and tell us more about your data, problem, model, etc.


text:
How to design a fitness function for a problem where there are 2 objectives?
I am told to express a fitness function for a question I have been presented. I am unsure how I would express the function. In words, what I have written down makes sense but turning this into a mathematical formula is proving a bit difficult. My understanding is:
The fitness function for this scenario will want to ensure that the best offer for building the computers is chosen whilst the price of the final optimal offer is low.
The fitness function in this case would want to consider a few factors. The first factor is that the quantity of the computer parts was enough that each offers that were returned had a sufficient quantity of parts. Ideally, it would be best if we did not have any duplicates of parts in the offers. The cost is low too, but all parts have been found amongst the different offers that we have.
The fitness function will need to ensure all of this is factored in.
The scenario and question are below:

For the production of a number of laptops, a computer company needs a quantity of each component such as screens  (S),  hard drives  (HD),  optical drives  (OD),  RAM,  video cards  (VC),  CPU,  Ports,  etc.  The company received a number of priced offers. Note that offers do not contain all components. As examples:

Offer 1: 1000 RAMs, 800 HDs, 2000 ODs – £75K
Offer 2: 1850 S, 1570 OD - £40K
Offer 3: 3000 HD, 2000 RAM – £70K
Offer 4: 1500 RAM, 2000 VC, 1700 S – £55Ketc.

The company would be interested to accept cheaper offers in the first place. Answer the following: Give the expression of the fitness function.

Any help would be greatly appreciated .

If we assume that each laptop requires 1 component of each type, so 1 screen, 1 hard drive, 1 RAM, etc. (i.e. the company has no preference for the type of component), then the company, to maximize the number of laptops it can build (which is supposedly the ultimate goal of the company), it should

maximize the number of instances of the least available component in the offer, and

minimize the cost of the offer

So, one possible fitness function would then need to take these 2 objectives into account, so this would be a multi-objective optimization problem.
Recall that a fitness function evaluates individuals in the population. If we assume that the individuals are the offers and the goal would be to find the best offer in the space of offers, then we can devise some fitness function $f$ of the form
$$
f(o) = (1 - \alpha)\frac{1}{1 + p(o)} + \alpha \min(o), \label{1}\tag{1}
$$
where

$o$ is an offer, so $f(o)$ is the fitness of the offer $o$, i.e. how much it is desirable (i.e. we want the fitness to be high, usually).
$p(o)$ is the price/cost of the offer $o$
if we assume that $o$ is an array of the form $o = [1000, 800, 2000, \dots]$, where $o_i$ is the number of items of type $i$ (e.g. $i$ is RAM, then $o_i$ would be the number of RAMs in the offer $o$), then $\min(o)$ would be the number associated with the least available item in the offer (e.g. if the indidivudal was $[12, 4, 5]$, then $\min(o) = 4$)
$\alpha \in [0, 1]$ is a hyper-parameter that trades off the two objectives (i.e. prices and the smallest number)

The fitness function $f$ in equation \ref{1} will be maximal when $p(o) = 0$ and $\min(o) = N$, where $N$ is some maximum threshold of possible number of items of the same type that an offer can have.
I have just come up with this fitness function. I don't know whether it will work in practice or not, but this is the idea of what you have to do.  You can design other similar fitness functions (for example, how would you design a fitness function that takes into account that the number of components of each type should be more or less the same?). You may want to try to implement this e.g. with DEAP and see how it behaves. You probably also want to make sure that $o$ are arrays of integers (and not floating-point numbers). You probably also want to read this answer.


text:
Is it a good idea to use different width and height of the kernel in a CNN?
I always see that the width and height of the kernel are the same. But is it a good idea to use different numbers?
Recently I tried to use GoogLeNet (which expects images to be 224x224) on my images (500x150) and I got an error:

Negative dimension size caused by subtracting 7 from 5 for 'average_pooling2d_5/AvgPool'...

I know that this error is because the height of my image is too small. If I use the height of about 200, then everything is ok. So, maybe, in this situation, I could just use a smaller height and bigger width in the kernel. For example (5, 3).
Is it a good idea in this case? Or in general? How can it affect the accuracy of the network and the ability to extract different features?

It depends on your application. In case of text recognition, non-uniform kernels are used since the information about text is less on the horizontal axis and more on the vertical axis.
If in your case it is applicable then, it will be good idea. But, if it is not you are better off using a smaller uniform kernel (2x2, maybe). You can also zero-pad your image to make it uniform before putting it through convolutions. Also, check if you are doing 'valid' or 'same' padding in your convolutions since 'valid' convolutions chip away at your image dimensions.

My task is to create an autonomous car (a bot that play car game). And I use Googlenet which is designed very well so probably changing parameters like kernel size is not a good idea. So it looks like the best solution is to zero pad the image

Yes, that would be the best option for you then. Also, accept the answer. Much thanks.

interesting, depends on the task to be done such as non-square objects in image


text:
Is there any research work on known malware detection systems based on AI?
I'm working on writing an article about the possibilities of modern AI-based algorithms to produce invisible self-learning malware, that can distribute itself throughout the internet and create flexible botnets.
So far, I can not find any additional information about that, except this one.
Is there any research work on known malware detection systems based on AI?

Hello. It seemed to me that you were only interested in "neural networks" (and not every possible "AI technique"), so I edited your post to reflect that. If that assumption/supposition was incorrect, feel free to edit your post and clarify your question again. Moreover, can you clarify what you mean by "self-learning malware"? Do you mean NNs that can detect malwares, right?

@nbro: My interpretation of OP's question before you edited it, is that they are looking for research about creating malware, and the detection part is about whether anyone has seen real world evidence that AI-created malware is in the wild. Whilst the linked article is about using AI systems for smart deployment of software payloads that would be very difficult to analyse without valid trigger input.

@NeilSlater The original question was "May be anyone can help me to find any information about known detections of malware based on AI?", so it seems that the current question is at least consistent with the original question. It's also true that this post wasn't fully clear, so I don't exclude that you may be right and the OP simply didn't formulate the question very clearly. Only the OP can confirm this.

Thanks for editing your post and clarifying that you're interested in any AI technique, but can you clarify this part "to produce invisible self-learning malware"? What do you mean by that? It's not clear what you mean by that because of this part "that can distribute itself throughout the internet and create flexible botnets.", and I think that's why Neil thought the question was another one.


text:
How to implement the deconv which is used in “Visualizing and Understanding Convolutional Networks”
I'm trying to understand the deconv referenced in the paper Visualizing and Understanding Convolutional Networks
The paper states (section 2, p. 3):

the deconvnet uses transposed versions of the same filters, but applied to the rectified maps

Is it possible to implement this step in a short code example? Given an unpooled, rectified map; how would the transposed filter be applied against it?
I did try looking at the referenced paper Adaptive Deconvolutional Networks for Mid and High Level Feature Learning. However, I'm not wrapping my head around its explanations too well; and it references a third paper with regard to its work on "layers of convolutional sparse coding" (deconvolution [M. Zeiler, D. Krishnan, G. Taylor, and R. Fergus. Deconvolutional networks. In CVPR, 2010]), but this 2010 paper appears to require access to download.

Hi. Please, put your specific question in the title. "Understand “Visualizing and Understanding Convolutional Networks”" is not a question and it would be too broad anyway. It seems that you're asking here "How does deconvolution work?". Have you tried to read something about the topic before asking this question? If no, it's probably a good idea to read something about the topic before proceeding. If yes, please, edit your post to explain what you have not understood.

Thanks for the suggestions for clarifying the question and edits!


text:
Handling imbalanced data with multiple targets
I have the model which has 3 outputs (it is a regression task, I have the angle of the steering wheel, brake and acceleration). I can divide my values to some smaller bins and in this way I can change this into classification problem. I can balance data to have the same number of data points in each bin.
But now I wonder how to balance this data correctly.
I found some good resources and libraries
imbalanced-learn | Python official documentation
multi-imbalance | Python official documentation
Multi-imbalance | Poznan University of Technology
But to my understanding, these algorithms can deal with imbalanced data (in normal and multi class classification) only if you have one output.
But I have 3 outputs. And these outputs can be correlated somehow.
How to balance them correctly?
I thought about 2 ideas:

Creating tuples consist of 3 elements and balancing in such a way that you have the same number of different tuples
But you can have this situation:
(A, X, 1), (A, Y, 2), (A, Y, 3), (B, Z, 3)
These tuples are different, but you can see that we have a lot of tuples with the value A at first position. So the data is still quite imbalanced.

Balancing data iteratively considering only one column at a time. You balance first column, then you balance second column etc.

Are these ideas good or not? Maybe there are some other options for balancing data if you have multiple targets?

$1)$ I contest the notion that imbalance is a problem for you. $//$ $2)$ Binning continuous variables is often a poor practice.

You can try weighting your training data instances. So, if for example class A has proportion $p_A$, you weight every instance of class A with $1/p_A$. There also exists more sophisticated approaches to train on unbalanced data, such as generating synthetic samples to create a balanced dataset and so on. You can start learning more here.

These are reasonable approaches.  In this specific example, instead of weighting based the proportion of the class, you would weight based on the proportion of each tuple.


text:
Why is my siamese network learning very well in e.g. 1 out of every 5 runs?
Why is my siamese network learning very well in e.g. 1 out of every 5 runs? The rest of the time it's not learning and maintains an accuracy of 0.5.
Any explanations? Is the contrastive loss taken in the embedded space to loose of a constraint?
The task is greyscale signature matching.
Additionally, trying the model on facial matching gives a constant 0.5 accuracy, no learning at all - the images are RGB, and maybe it's a higher-order task in general.
Anyways, would appreciate any and all enlightenment in this matter.
P.S. I'm thinking to try a variational autoencoder for the face dataset, where I then use the trained encoder as the siamese network "head".
I would appreciate any guidance or thoughts on this approach as well.

Are your loss values changing? Check the input, output and loss calculation. Or post some more data here.

My loss is fluctuating but not steadily growing or decreasing - which makes sense because my accuracy is steadily hovering around 0.5 (random)

Did you check your pipelines for input and output. Also, if the loss calculation is right? Also, have you set all parameters to trainable?

Its something with the problem being too hard for the loss or network model

I think its an implementation mistake or dataset is too easy. I need to see code first. Post it here.

dataset too hard more likely

show me the code then

What's a "greyscale signature matching" task? Can you please provide a link to a reference that describes this task?


text:
Resources for Computer Vision Algorithms and Applications
Are there any videos or other books/notes/slides that anyone has come across that follow Computer Vision Algorithms and Applications by Richard Szeliski? We are using this book in class but the professor did a bad job explaining and I have some trouble getting through the book. Thanks a lot!

What do you mean by "that follow Computer Vision Algorithms and Applications by Richard Szeliski"

Notes or slides that follow along with the book.


text:
Proving existence or non existence of reward function to make given policy "uniquely" optimal when reward function is dependent only on S or both S,A
I was going through paper titled "Algorithms for Inverse Reinforcement Learning" by Andrew Ng and Russell.
It states following basics:

MDP $M$ is a tuple $(S,A,\{P_{sa}\},\gamma,R)$, where

$S$ is a finite seto of $N$ states
$A=\{a_1,...,a_k\}$ is a set of $k$ actions
$\{P_{sa}(.)\}$ are the transition probabilities upon taking action $a$ in state $s$.
$R:S\rightarrow \mathbb{R}$ is a reinforcement function (I guess its what it is also called as reward function) For simplicity in exposition, we have written rewards as $R(s)$ rather than $R(s,a)$; the extension is trivial.

A policy is defined as any map $\pi : S \rightarrow A$

Bellman Equation for Value function $V^\pi(s)=R(s)+\gamma \sum_{s'}P_{s\pi(s)}(s')V^\pi(s')\quad\quad...(1)$

Bellman Equation for Q function $Q^\pi(s,a)=R(s)+\gamma \sum_{s'}P_{sa}(s')V^\pi(s')\quad\quad...(2)$

Bellman Optimality:  The policy $\pi$ is optimal iff, for all $s\in S$, $\pi(s)\in \text{argmax}_{a\in A}Q^\pi(s,a)\quad\quad...(3)$

All these can be represented as vectors indexed by state, for which we
adopt boldface notation $\pmb{P,R,V}$.

Inverse Reinforcement Learning is: given MDP $M=(S,A,P_{sa},\gamma,\pi)$, finding $R$ such that $\pi$ is an optimal policy for $M$

By renaming actions if necessary, we will assume
without loss of generality that $\pi(s) = a_1$.

Paper then states following theorem, its proof and a related remark:

Theorem: Let a finite state space $S$, a set of actions $A=\{a_1,..., a_k\}$, transition probability matrices ${\pmb{P_a}}$, and a discount factor $\gamma \in (0, 1)$ be given. Then the policy $\pi$ given by $\pi(s) \equiv a_1$ is optimal iff, for all $a = a_2, ... , a_k$, the reward $\pmb{R}$ satisfies  $$(\pmb{P}_{a_1}-\pmb{P}_a)(\pmb{I}-\gamma\pmb{P}_{a_1})^{-1}\pmb{R}\succcurlyeq 0 \quad\quad ...(4)$$
Proof:
Equation (1) can be rewritten as
$\pmb{V}^\pi=\pmb{R}+\gamma\pmb{P}_{a_1}\pmb{V}^\pi$
$\therefore\pmb{V}^\pi=(\pmb{I}-\gamma\pmb{P}_{a_1})^{-1}\pmb{R}\quad\quad ...(5)$
Putting equation $(2)$ into $(3)$, we see that $\pi$ is optimal iff
$\pi(s)\in \text{arg}\max_{a\in A}\sum_{s'}P_{sa}(s')V^\pi(s')  \quad...\forall s\in S$
$\iff \sum_{s'}P_{sa_1}(s')V^\pi(s')\geq\sum_{s'}P_{sa}(s')V^\pi(s')\quad\quad\quad\forall s\in S,a\in A$
$\iff \pmb{P}_{a_1}\pmb{V}^\pi\succcurlyeq\pmb{P}_{a}\pmb{V}^\pi\quad\quad\quad\forall a\in A\text{\\} a_1 \quad\quad ...(6)$
$\iff\pmb{P}_{a_1} (\pmb{I}-\gamma\pmb{P}_{a_1})^{-1}\pmb{R}\succcurlyeq\pmb{P}_{a} (\pmb{I}-\gamma\pmb{P}_{a_1})^{-1}\pmb{R} \quad\quad \text{...from (5)}$
Hence proved.
Remark: Using a very similar argument, it is easy to show (essentially by replacing all inequalities in the proof above with strict inequalities) that the condition $(\pmb{P}_{a_1}-\pmb{P}_a)(\pmb{I}-\gamma\pmb{P}_{a_1})^{-1}\pmb{R}\succ 0 $ is necessary and sufficient for $\pi\equiv a_1$ to be the unique optimal policy.

I dont know if above text from paper is relevant for what I want to prove, still I stated above text as a background.
I want to prove following:

If we take $R : S → \mathbb{R}$—there need not exist $R$ such that $π^*$ is the unique optimal policy for $(S, A, T, R, γ)$
If we take $R : S × A → \mathbb{R}$. Show that there must exist $R$ such that $π^*$ is the unique optimal policy for $(S, A, T, R, γ)$.

I guess point 1 follows directly from above theorem as it says "$\pi(s)$ is optimal iff ..." and not "unique optimal iff". Also, I feel it also follows from operator $\succcurlyeq$ in equation $(6)$. In addition, I feel its quite intuitive: if we have same reward for any given state for every action, then different policies choosing different actions will yield same reward from that state hence resulting in same value function.
I dont feel point 2 is correct. I guess, this directly follows from the remark above which requires additional condition to hold for $\pi$ to be "uinque optimal" and this condition wont hold if we simply define $R : S × A → \mathbb{R}$ instead of $R : S → \mathbb{R}$. Additionally, I feel, this condition will hold iff we had $=$ in equation $(3)$ instead of $\in$ (as this will replace all $\succcurlyeq$ with $\succ$ in the proof). Also this also follow directly from point 1 itself. That is we can still have same reward for all actions from given state despite defining reward as $R : S × A → \mathbb{R}$ instead of $R : S → \mathbb{R}$, which is the case with point 1.
Am I correct with the analysis in last two paragraphs?
Update
After some more thinking, I felt I was doing it all wrong. Also I feel the text from the paper which I specified is of not much help in proving these two points. So let me restate new intuition for proofs for the two points:

For $R: S\rightarrow \mathbb{R}$, if some state $S_1$ and next state $S_2$ has two actions between them, $S_1-a_1\rightarrow S_2$ and $S_1-a_2\rightarrow S_2$, and if optimal policy $π_1^*$ chooses $a_1$, then $π_2^*$ choosing $a_2$ will also be optimal, thus making NONE "uniquely" optimal since both $a_1$ and $a_2$ will yield same reward as reward is associated with $S_1$ instead of with $(S_1,a_x)$.

For $R: (S,A)\rightarrow\mathbb{R}$, we can assign large reward say $+∞$ to all actions specified in given $π^*$ and $-∞$ to all other actions. This reward assignment will make $π^*$ a unique optimal policy.

Are above logics correct and enough to prove given points?

Cross-posted: https://cs.stackexchange.com/q/136665/755, https://datascience.stackexchange.com/q/90656/8560, https://ai.stackexchange.com/q/26821/1794.  Please do not post the same question on multiple sites.

Could you please simplify this post as much as possible, but still leave the relevant details to answer your specific question? This simplifies our life. For example, do we really need to quote all those details from the paper? Moreover, could also focus on 1 question per post? If you have more than one question, ask each of them in their separate post. Finally, put your specific question in the title, so that we immediately know what your specific question is? "providing facts in IRL" is very vague and not a question.

The two questions are very related. If I have to ask them on separate post, then I have to copy paste the whole background, which will be a lot of redundancy. Also any future visitor will gain a lot by reading the two problems together, instead of reading one and not realising that there is separate post for other very closely related problem. Even if he/she realizes, he will have to figure out what is difference between two by comparing them. Btw, I have added updates which may make theorem, proof and remark irrelevant, though not sure and hence I havent removed them.


text:
Multi-target regression using scikit-learn without ytrain
I would like to use the multi-target regression with scikit-learn. However, the examples I've seen use Xtrain and ytrain?
What is ytrain in regression?
I know y it is used for classes in classification. My data is composed of two columns of data, and I want to predict both values independently (but using a single MTR). So, I have clear X is a training set of n number of samples from those two values, however, I don't know how to create y.
My data is composed of two attributes not correlated, I guess this is Xtrain. What should I use as ytrain?
I'm based on this source https://machinelearningmastery.com/multi-output-regression-models-with-python/.
Any clue?

It's not fully clear what you're asking here. Is your question "What are the labels supposed to be in a multi-target regression problem?", or are you asking what are the labels supposed to be in any regression problem and not just "multi-target regression problem"? Could you also explain what you mean by "multi-target" here? Moreover, the article that you mention does not contain the words Xtrain or ytrain, after a quick search.

Right, doesn't say Xtrain or ytrain, but X and y. In my case, X, is composed for two non-correlated variables (latitude and longitude) as input, I want the same as output (latitude and longitude but predicted. So, I have N number of samples and I want to predict M number of samples (in both cases composed for the same number of variables). There are not more input neither labels. I don't know if it is clear.


text:
Why doesn't U-Net work with images different from the dataset?
I have implemented a U-Net, similar to this implementation, but for a different dataset, this one, to segment roads.
It works fine using the test folder images, but, for example, when I pick a print from bing maps and try to infer with the trained model, this is returned:

Why this is happening?
I already tried to change the thresholding values, normalization, etc.
Tensorboard

One thing to check - image could be with value range in [0, 255] and [0, 1]. Make sure this range is identical for train and test images)

When you say in the title "with images different from the dataset", you mean "test dataset", right? It may be a good idea to show us an example of an image from the test dataset too (even though you already provide a link to the dataset).


text:
How is the discounted maximum entropy objective obtained for soft-q-learning and SAC
In the soft q-learning paper, they provide an expression for the maximum entropy objective that takes discounting into account.
My main question is: can someone explain how they incorporated discounting into the objective?
I've also got a few other questions related to the form of the discounted objective as well.
The first one being is: they first define the objective in way of obtaining $\pi_{\text{MaxEnt}}^*$.
In this first expression,
$$
\pi_{\mathrm{MaxEnt}}^{*}=\arg \max _{\pi} \sum_{t} \mathbb{E}_{\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right) \sim \rho_{\pi}}\left[\sum_{l=t}^{\infty} \gamma^{l-t} \mathbb{E}_{\left(\mathbf{s}_{l}, \mathbf{a}_{l}\right)}\left[r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\alpha \mathcal{H}\left(\pi\left(\cdot \mid \mathbf{s}_{t}\right)\right) \mid \mathbf{s}_{t}, \mathbf{a}_{t}\right]\right],
$$
I don't really understand the purpose of the inner expectation. If it's an expectation over $(s_l,a_l)$, the terms within the expectation are constants, so they can be taken out of the expectation and even the inner sum too. So, I think the subscript might be wrong, but was hoping someone could confirm this.
My second issue is: they rewrite the maximum entropy objective using $Q_{soft}$ in (16)
$$J(\pi) \triangleq \sum_{t} \mathbb{E}_{\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right) \sim \rho_{\pi}}\left[Q_{\mathrm{soft}}^{\pi}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\alpha \mathcal{H}\left(\pi\left(\cdot \mid \mathbf{s}_{t}\right)\right)\right]$$
I'm not sure how they do this. If someone could provide a proof of this connection, that would be much appreciated.

Edit your question to only contain one question. Make separate questions for your others.

Please, do as suggested in the previous comment. Split this post into 3: one for each of your 3 questions.


text:
Is vectorizing backpropagation feasible?
Does it make sense to have the backpropagation of a neural network layer happen all at once if the learning rate is lowered? This would mean the new weights of that layer would be independent of each other, but, it would be extremely fast. Is this method feasible in any way for a neural network, or would it create a cost threshold which the network can't reach because of it's independent inaccuracy?

What do you mean by "all at once"? If I'm not mistaken, the way that backpropagation normally works is that all of the gradients are calculated (which is a very parallel process, but not 100% parallel) and then all of the gradients are applied at the same time. How does your idea differ from that?

Oh. The way I did it involved optimizing each weight one by one so that the other weights are related to them. I didn't know parallelization was the normal way to do it... No wonder it was so slow. Thank you so much! You just saved me from wasting 3 hours trying to find small optimizations.

I am still confused about how this works however. If the weights get optimized all at once, couldn't the previous influence of the weights on the layers' outputs make the new weights' outputs significantly different?

Yes, absolutely. If the learning rate is too large, then that can be a big problem. I think that there are many possible ways to solve that problem. Using a smaller learning rate is one way. I don't know enough about machine learning to be able to name any other ways.