text:
How are nested bounding boxes handled in object detection (and in particular in the case of the SSD)?
The basic approach to non-maximum-suppression makes sense, but I am kind of confused about how you handle nested bounding boxes.
Suppose you have two predicted boxes, with one completely enclosing another. What happens under this circumstance (in particular, in the case of the Single Shot MultiBox Detector)? Which bounding box do you select?


text:
What are the pros and cons of 3D CNN and 2D CNN combined with optical flow for action recognition?
For action recognition or similar tasks, one can either use 3D CNN or combine 2D CNN with optical flow. See this paper for details.
Can someone tell the pros/cons of each, in terms of accuracy, cost such as computation and memory requirement, etc.? In other words, is the computation overhead of 3D CNN justified by its accuracy improvement? Under what scenarios would one prefer one over another?
3D CNNs are also used for volumetric data, such as MRI images. Can 2D CNN + optical flow be used here?
I understand 2D CNNs and 3D CNNs, but I do not know about optical flow (my background is not computer-vision).

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Research into social behavior in Prisoner's Dilemma
I've been working on research into reproducing social behavior using multi-agent reinforcement learning. My focus has been on a GridWorld-style game, but I was thinking that maybe a simpler Prisoner's Dilemma game could be a better approach. I tried to find existing research papers in this direction, but couldn't find any, so I'd like to describe what I'm looking for in case anyone here knows of such research.
I'm looking for research into scenarios where multiple RL agents are playing Iterated Prisoner's Dilemma with each other, and social behaviors emerge. Let me specify what I mean by "social behaviors." Most research I've seen into RL/IPD (example) focuses on how to achieve the ideal strategy, and how to get there the fastest, and what common archetypes of strategies emerge. That is all nice and well, but not what I'm interested in.
An agent executing a Tit-for-Tat strategy is giving positive reinforcement to the other player for "good" behavior, and negative reinforcement for "bad" behavior. That is why it wins. My key point here is that this carrot-and-stick method is done individually rather than in groups. I want to see it evolve within a group.
I want to see an entire group of agents evolve to punish and reward other players according to how they behaved with the group. I believe that fascinating group dynamics could be observed in that scenario.
I programmed such a scenario a decade ago, but by writing an algorithm manually, not using deep RL. I want to do it using deep RL, but first I want to know whether there are existing attempts.
Does anyone know whether such research exists?

Is that https://arxiv.org/pdf/1803.00162.pdf maybe something you are looking for?

I've seen this research before, though I've only read the abstract, as I did again now. It seems to be talking about a continuum of choices between Defect and Cooperate, but that's not really what I'm looking for here.

I've asked this on rl-list and got lots of interesting references to research papers.
So far the most promising one I've seen is this:
Nicolas Anastassacos, Stephen Hailes and Mirco Musolesi. Partner Selection for the Emergence of Cooperation in Multi-Agent Systems using Reinforcement Learning. In AAAI 2020. New York City, NY, USA. February 2020.

Maybe you can briefly explain why this paper is useful/promising.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
For the generalised delta rule in back-propogation, do you subtract the target from the obtained output, or vice versa?
When I look up the generalised delta rule equation for back-propogation, I am seeing two conflicting equations.
For example, here (slide 20), given $o$ (the output, defined in slide 18), $z$ (the activated output) and a target $t$, defined in slide 17, then:
$\frac{\delta E}{\delta Z} = o(1-o)(o-t)$
When I look for the same equation else, e.g. here, slide 14, it says, given $o$ the output and $y$ the label, then (using slightly different notation $\beta_k$):
$\beta_k = o_k(1-o_k)(y_k-o_k)$
I can see here that these two equations are almost the same, but not quite. One subtracts the output from the target, and one subtracts the target from the output.
The reason why I'm asking this is I'm trying to do question 29 and 30 of this paper, and they are using the second equation ($\beta_k$) but my college notes (that I can't copy and paste due to copyright) define the equation according to the first equation $\frac{\delta E}{\delta Z}$. I'm wondering which way is correct, do you subtract the target from the obtained output, or vice versa?

This may depend on the definition of the loss function. You should check that the loss function in the second case is equivalent to the loss function in the first. If you find the answer to your own question and nobody provides a formal answer meanwhile, feel free to write a formal answer to your own question below.


text:
How can I perform the forward pass in a neural network evolved with NEAT, given that some connections may not exist or there may be loopy connections?
I have a problem that arose as part of a NEAT (Neuro Evolution Through Augmenting Topologies) implementation that I am writing. I am wanting it to produce topologies or graphs that describe neural networks, similar to the one below.

Here, nodes 0 and 1 are inputs, and 4 is the output node, the rest of the nodes are hidden nodes. Each of these nodes can have some activation function defined for them (not necessary that all the hidden nodes have the same activation function)
Now, I want to perform the forward pass of this neural network with some data, and, based on how well it performed in that task, I assign it with a fitness value, which is used as part of the NEAT evolutionary algorithm to move towards better architectures and weights.
So, as part of the evolution process, I can have connections that can cause internal loops in the hidden layers and there is the possibility that a skip connection is made. Because of this, I feel the regular matrix-based forward pass (of fully connected MLPs) will not work in order to perform the forward pass of these evolved neural networks, and hence I want to know if an algorithm exists that can solve this problem.
In short, I want this neural network to just take the inputs and provide me outputs - no training involved at all, so I'm not interested in the back-propagation part now.
The only way to solve this that I see is to use something on the lines of a job queue (the queue will consist of the nodes that needs processing in order). I feel this is extremely inefficient and I cannot allocate this simulation method a proper stop condition. Or, even when to take output from the neural network graph and consider it.
Can anybody at least point me in the right direction?

The thing, I am extending the NEAT implementation. I have already implemented NEAT in two languages and those work. This is a interesting problem that I came upon when I considered evolving the neurons along with the network - I strongly think that it must have better convergence and should be much faster. The simple NEAT implementation is very straight forward - not much to think in there. Can I take my stack overflow reputation and bounty this question using that.


text:
How to derive compact convex set K and its diameter D to program Accelegrad algorithm in practice?
Given the original paper (https://arxiv.org/pdf/1809.02864.pdf), I would like to implement the Accelegrad algorithm for which I report the pseudocode of the paper:

In the pseudocode, the authors refer to a compact convex set $K$ of diameter $D$. The question is whether I can know such elements. I think that they are theoretical conditions to satisfy some theorems. The problem is that the diameter $D$ is used in the learning rate and also the convex set $K$ is used to perform the projection of the gradient descent. How can I proceed?


text:
How can I model a problem as an MDP if the agent does not follow the successive order of states?
In my problem, the agent does not follow the successive order of states, but selects with $\epsilon$-greedy the best pair (state, action) from a priority queue. More specifically, when my agent goes to a state $s$ and opens its available actions $\{ a_i \}$, then it estimates each $(s,a)$ pair (regression with DQN) and stores it into the queue. In order for my agent to change to state $s'$, it picks the best pair from the queue instead of following one of the available actions $\{ a_i \}$ of $s$. I note that a state has a partially-different action set from the others.
However, in this way, how can I model my MDP if my agent does not follow the successive order of states?
More specifically, I have a focused crawler that has an input of a few seeds URLs. I want to output as many as possible relevant URLs with the seeds. I model the RL framework as follows.

State: the webpage,
Actions: the outlink URLs of the state webpage,
Reward: from external source I know if the webpage content is relevant.

The problem is that, while crawling, if the agent keeps going forward by following the successive state transition, it can fall into crawling traps or local optima. That is the reason why a priority queue is used importantly in crawling. The crawling agent does not follow anymore the successive order of state transitions. Each state-action pair is added to the priority queue with its estimated action value. For each time, it selects the most promising state-action pair among all pairs in the queue. I note that each URL action can be estimated taking into account the state-webpage where it was extracted.

Could you clarify - is this non-sequential ordering of states only part of the training process that you have implemented, or is it integral to the environment? Is there any meaningful state transition rule?

The environment says that if you are in a state s then estimate all its available actions {a} and then choose the best action from all past states, including current, and move to s'.

That doesn't appear to be a description of an environment, but some combination of agent and environment. Ignoring the evaluation step (which is not part of the environment definition), are you saying that the state of the environment is the current state plus all historical states, and the agent can choose to act within the context of any state seen so far (i.e. it can effectively extend the trajectory from any point in history)?

Or another way to put this is the agent's action effectively both choosing which state to use (out of states it has seen) plus a choice that it can make within that state?

Yes, I guess. But the current point-state can only estimate its actions and not all actions seen in trajectory

How can I insert in my MDP formula the priority queue for the previous (state,action) pairs that tells which is the next state ?

That's not a restriction that the environment can impose. Clearly if it has no data, any value estimate may be incorrect, but evaluation (or not) of a state or state/action pair is not an action, it is integral to the agent and cannot be restricted. Unless you have some interaction allowed with the environment which fetches some additional values . . . if so, may be better not to call that "estimates" or "evaluation"?

I think you are going to need to explain more about your environment in your question and how/why you have modelled it this way. It is not clear why you have done this, and you are likely to be stretching the meaning of RL terminology to cover what you are doing - it might be fine, but it is hard to understand what you are attempting to do

Is it clarified more now ?

Yes it is clearer. You could clarify one more thing - is the backtracking/priorising of earlier links something you want the agent to do only during training (to prevent blocks during training), or is it also the behaviour that you want in the optimal agent?

I want the non-succesive move in the optimal agent too

The idea, except for finding relevant pages in a direct way, is to identify hubs ; new webpages (states) that are possibly irrelevant but extract relevant urls (actions).

Maybe I can assume that each state(webpage) has an action(URL) for all pages(states).So there is a transition probability for all states.Assuming that we are in s, the DQNAgent would output a Q-value for each action a and with this value the pair (s,a) would be inserted in queue.New_state comes from the action that the best pair (state,action) in queue has.So currentstate retains the action values of the past states for all actions seen and only estimates its own actions.In ER store the record (state,action,reward,new_state) with state being the webpage truely extracted this action.Is it fair?

I've changed the title of this post. Make sure that the title is your actual question.

Your problem fundamentally is that you are confusing what the state and actions are in this setting. Webpages are not your states; your state is the entire priority queue of (website-outlink) pairs + the (new_website-outlink) pairs. Your action is which pair you select.
Now this is a variable sized state-space and variable sized action-space problem setting at the same time. To deal with this lets start by noting that state==observation need not be (in general). So what is your observation? Your observation is a variable-sized batch of either:

(website-outlink) pairs or
next_website (where each next_website is determined by its corresponding pair)

Both of these observations could work just fine, choosing between one or the other is just a matter of whether you want your agent to learn "which links to open before opening them" or "which links are meaningful (after opening them)".
What your priority queue is essentially doing is just adding a neat trick that:

Saves the computational complexity of keeping the state ordered (remember that your state is not a website, but the list/batch of website-outlink)
Avoids needlessly recomputing the Q-values for each of your actions (remember that an action is not selecting an outlink from new_website, but selecting an outlink from all available choices in the queue)

Note however that to actually have the second saving it is crucial to store the Q-values for each pair!!!
Last important thing to note is that in a scenario where you use a Replay Buffer (which I guess is likely given that you chose a DQN), you can't use the priority queue whilst learning from the RB. To see why (and to see in detail how your learning process actually looks like), start by remembering that your Q-value updates are given by the formula here; your state s_t is a (quasi-ordered1) batch of pairs. Q(s_t, a_t) is just the output of running your DQN regression on just the best website/pair in this batch (you have to add an index to denote the best choice when adding transitions to the RB, in order to be consistent about which action was taken from this state). To compute the estimate of the optimal future value however you will have to recompute the Q-value of every single website/pair in the next state. You CANNOT use the priority queue when training from the RB.
1 You have the priority queue ordered for all websites you had in it whilst looking at the last website, but all the new_website-outlink pairs you are now adding are not ordered yet. You still have to run the agent on them and then you can order them with the rest of the priority queue to generate the next state (which still will not be ordered because you will have new_new_website-outink pairs).

Thanks for the answer. With my modeling of MDP, what is wrong? State: webpage that has possible actions for all other states. Action: single url. A state has the actions for all URLs directly extracted by the current state webpage plus the actions seen in past states. In queue, store pairs (s,a) with a directly extracted from s. And in ER store records <(s,a), r, s'> with a directly extracted from s and not necessary from the last state. Update DQN with the max operator using only actions directly extracted from s' and not all actions seen. After this, update all pairs in queue. etc.

That is, in order to change state, update synchronously all pairs in frontier and select with e-greedy.

Well the issue is that you can't just select whatever you want as a state. MDP requires that the transitions be state-action-nextstate. If you define a state in a way such that this isn't true (because in your description nextstate could be relative to a previously seen website) then you have defined your state wrong. This is why nothing will work or make sense. You have to expand your state. This is what my proposed solution does by inlcuding the entire queue as the state while still implementing some computational shortcuts at run time


text:
In few-shot classification, should I use my custom dataset as the validation dataset and mini-ImageNet as the training dataset?
I am new to few-shot learning, and I wanted to get a hands-on understanding of it, using Reptile algorithm, applied to my custom dataset.
My custom dataset has 30 categories, with 5 images per category, so this would be a 30 way 5 shot.
Given a new image, I wish to be able to classify it into one of 30 categories. I changed train_shots = 5, classes = 30 in the linked example, and got the training output as
batch 0: train=0.050000 test=0.050000
batch 1: train=0.050000 test=0.050000

Should the custom dataset be used as a validation set, with mini-ImageNet as a training dataset, so that the knowledge is transferred? Or can I use only a custom dataset with only $30*5=150$ images for training?


text:
How can an "architectural motif" be extracted from a trained MLP?
I am trying to reproduce the paper Synthetic Petri Dish: A novel surrogate model for Rapid Architecture Search. In the paper, the authors try to reduce the architecture of an MLP model trained on MNIST (2 layers - 100 neurons) by initializing a motif network from it, that is, 2 layers, 1 neuron each, and extracting the sigmoid function. I have been searching a lot, but I have not found the answer of how can someone extract an 'architectural motif' from a trained neural network.

Hi. Welcome to AI SE. Have you already looked into https://github.com/uber-research/Synthetic-Petri-Dish? If you find the answer in this Github repo or somewhere else, feel free to write a formal answer below to your own question ;)

@nbro Hello, thanks for welcoming me. Indeed I did but unfortunately they did not include the motif extraction part in their code. What they do only is display the motif training points - which they define in their paper as tuple of slope values and validation accuracy.

Maybe the best thing to do is open an issue in the issue tracker of that repo, if nobody provides an answer meanwhile.


text:
Given a 2-layer GCN, can we choose the dimensions of the 2nd weight matrix, such that this architecture has the same capacity as a 1-layer GCN?
This might be more of a question about nested function classes:
For $k$ class node classification in a graph with $n$ nodes, and $d$ feature vector.
I want to compare
Architecture I: the GCN model of Kipf/ Welling with two graph convolutional layers:
$$
\mathbf{Y}=\operatorname{softmax}\left(\mathbf{A} \xi\left(\mathbf{A X W}_{1}\right) \mathbf{W}_{2}\right)
$$
where

$\mathbf{X}$ is $n \times d,
- $ $\mathbf{Y}$ is $n \times k$
$\mathbf{A}$ is a fixed $n \times n$ graph diffusion matrix,
$\mathbf{W}_{1}, \mathbf{W}_{2}$ are learnable weight matrices of size $d \times d^{\prime}$ and $d^{\prime} \times 2,$ respectively, shared across all nodes, and
$\xi$ is a nonlinearity.

Architecture II: a single-layer graph neural network of the form:
$$
\mathbf{Y}=\operatorname{softmax}\left(\mathbf{A}^{2} \mathbf{X W}\right)
$$
where $\mathbf{W}$ is a learnable weight matrix of size $d \times 2$.

Now I'm wondering

$\operatorname{Can} \xi, d^{\prime}$ be chosen in a way that both architectures have the same expressive power? (i.e. can represent the same class of functions)?

$\operatorname{Can} \xi, d^{\prime}$ be chosen in a way that Architecture II is more expressive?

What would be the advantage in training complexity of Architecture II when applied to large-scale graphs.

Could you please focus on 1 specific question (eventually, you can add 1-2 more sub-questions, provided they are sub-questions and not distinct questions)? Then could you please put that specific question in the title? To me, it seems that your questions are quite distinct (and a bit broad) and require a lot of effort to answer properly. So, I actually suggest that you focus on 1 of those questions and remove the others. If you have more questions, ask them in other separate posts.


text:
What are the advantages of RL with actor-critic methods over actor-only methods?
In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?
This is not a comparison with the Q-learning series, but probably a method of learning the game with only the actor.
I think it's effective to use only actors, especially for sparse rewards. Is that correct?
Please, let me know if you have any specific use cases that use only actors.

You may also be interested this question.

In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods?

One practical benefit is that critics can use TD learning to bootstrap, allowing them to learn online on each step taken, plus learn in continuing problems. Pure actor algorithms like REINFORCE, cross-entropy method, and non-RL policy-only learners, such as genetic algorithms, require episodic problems. The smallest unit those can learn from is an entire episode. That is because without a critic providing value estimates, the only way to estimate return is to sample an actual return from the end of an episode.
A TD-based critic may also have lower variance, which can aid in fast learning and stability, although this is not always a benefit. TD-based critics also have bias, which can cause instability. See Why do temporal difference (TD) methods have lower variance than Monte Carlo methods? on Cross Validated for more details on this.
In practice, RL algorithm choice is a hyperparameter. As well as impacting difficulty of implementation, CPU and other resource costs, it can impact how well learning occurs depending on the problem being attempted. Usually, the only way you can tell a method is better for your problem is to try all the valid ones and measure their performance.

I think it's effective to use only actors, especially for sparse rewards. Is that correct?

The sparsity of reward is not a major factor here. The hard credit assignment problem that this brings into play means that the agent has to either assign a value, or pick an action, in states which have no direct feedback. All else being equal, both the value function and optimal action choice are hard to resolve when they depend on a large number of future policy decisions and state transitions which may vary.
Which approach is best will depend on whether it is easier for a statistics-based learner to approximate the mapping from state to expected future reward, or to the action. The two functions can have different degrees of complexity.
For an example where a policy function (for an actor) is simpler than a value function (for a critic or value-based method), you could consider a simple chase environment where a wolf tries to catch a rabbit on a simple continuous plane surface. The agent is the wolf, and gets a reward of +1 for catching the rabbit (I won't bother with other details, there are plenty of variations you could make).
In the example environment, a simple strategy for the wolf is to turn to face the rabbit and move forwards. This is easy to map from current locations and facings of the wolf and the rabbit - for ultimate ease you can express the state as the rabbit's position and velocity in polar coordinates from the wolf's perspective. Compare that to the value function - it has to predict the time it will take to reach the rabbit given a current action choice. This is a far harder function to express based on the state, and as a result may also be harder to learn.

Maybe you should at least briefly (or link to some reference that) describe(s) why 1. actor-only methods (such as REINFORCE) require episodic problems and 2. TD-based methods may have lower bias.

@nbro I expanded the first paragraph with an explanation and linked a related question on cross-validated to address your second point

Ok, thanks! That makes this answer clearer. Another thing that I think is missing from this answer is that you seem to focus on the comparison of actor-only methods and critic-only methods, though it seems to be implied that all the info mentioned is applicable to actor-critic methods as well as critic-only methods (such as Q-learning). In any case, it may be a good idea to differentiate actor-critic methods and critic-only methods.


text:
Why does batch norm standardize with sample mean/variance, when it also learns parameters to scale the mean/variance?
Batch norm is a normalizing layer that is shown to help deep networks learn faster and with higher generalization accuracy. It normalizes the activations of the previous layer to a mean $\beta$ and variance $\gamma^2$ to prevent things like activations from exploding or shifting during the learning process.
More specifically:
$$\hat{x} = \displaystyle \frac{x - \mu_t}{\sqrt{\sigma_t^2 + \epsilon}}\label{1}\tag{1}$$
$$ BatchNorm_{\mu_t, \sigma_t}(x) = \gamma \hat{x} + \beta \label{2}\tag{2}$$
where

$x$ is the layer input of the layer
$\mu_t, \sigma_t$ is the sample mean and standard deviation at time step $t$
$\epsilon$ is a small constant, and
$\gamma$ and $\beta$ are learnable parameters so that the output is not necessarily standardized to mean $0$ and variance $1$, but possibly to another mean and variance that may be better for the neural network.

My question is, why does BatchNorm first standardize the input $x$ to $\hat{x}$ before applying the learnable parameters $\gamma$ and $\beta$? Isn't this redundant? The parameters $\gamma$ and $\beta$ could learn to standardize the input themselves right?
In fact, as training progresses, $\mu_t$ and $\sigma_t$ becomes updated to new values $\mu_{t+1}$ and $\sigma_{t+1}$, so the learned parameters at that time step, $\gamma_t$ and $\beta_t$, no longer apply for time step $t+1$ since that involves a different standardization process with a different mean and variance. So by adding this standardization step, it may even hurt the convergence of the layer during learning, since it is adding the gradient of $BatchNorm_{\mu_{t+1}, \sigma_{t+1}}(x)$ to $BatchNorm_{\mu_t, \sigma_t}(x)$, which are two different functions right?
Why not just simply make it like this?
$$BatchNorm(x) = \gamma x + \beta \label{3}\tag{3}$$
This would simplify the calculation of the gradients, which would make learning faster to compute.
BatchNorm is one of the most successful developments of deep learning, so I know my intuition on these things is wrong -- I'm just curious as to what I am missing.

That second formula reminds me of the re-parametrization trick (used e.g. in VAEs). Not sure if it's related or not, though, because I'm not currently familiar with the details of batch normalization. So, you could investigate this option.

Theoretically, Yes and batchnorm can be omitted. Practically, No. Most of the intermediate features aren't normalized. For example, after a 2x2 Max pool of a normalized feature, the SD and mean would be vastly shifted to 0.8 and 1.2 (I don't remember the exact number) of the originals. It will take an immense amount of iterations for subsequent learnable parameters to absorb these scales and offsets. Before the absorption happens, those low SD feature map will be suppressed and lowering the overall network capacity. Other parameters are trained according the condition where some features are suppressed. The overall dynamics of unnormalized features are unpredictable. Batch Norm first normalize x, then the learnable gamma and beta are updated under the same scale of other normalized features.


text:
What happens if there is no activation function in some layers of a neural network?
What if I don't apply an activation function on some layers in a neural network. How will it affect the model?
Take for instance the following code snippet:
def model(x):
    a = Conv2D(64, (3, 3))(x)
    x = Conv2D(64, (3, 3), activation = 'relu')(x)
    b = Conv2D(128, (3, 3))(x)
    x = Conv2D(128, (3, 3), activation = 'relu')(b)
    return x, a, b

If you do not specify an activation for a layer you are effectively creating a linear transformation through that layer.  From the documentation:

activation: Activation function to use. If you don't specify anything, no activation is applied (see keras.activations).


text:
Why won't my model train with CTC loss?
I am trying to train an LSTM using CTC loss, but the loss does not decrease when I train it. I have created a minimal example of my issue by creating training data where the network simply has to copy the current input element at each time step. Moreover, I have made the length of the label the same as the length of the input sequence and no adjacent elements in the label sequence the same so that both CTC loss and categorical cross-entropy loss can be used. I found that when using categorical cross-entropy loss the model very quickly converges, whereas when using CTC loss it gets nowhere.
I have uploaded by minimal example to colab. Does anyone know why CTC loss is not working in this case?

Could you please provide the mathematical formula of the CTC loss that you implemented, to have more context?

I did not implement the loss function myself, but rather am using a built-in function from tensorflow, tf.nn.ctc_loss. The mathematical formulation of this is available in the paper Connectionist Temporal Classification - Labeling Unsegmented Sequence Data with Recurrent Neural Networks.

Thanks! I would like to note that programming issues are off-topic here. So, next time, if you think your problem is a bug in your code and not a conceptual issue, then you should ask your question on Stack Overflow. Here, we focus on the theoretical, philosophical, and social aspects of Artificial Intelligence. Take a look at https://ai.stackexchange.com/help/on-topic.

The problem was the dimensions of the logit_length argument to tf.nn.ctc_loss was incorrect.
It was this:
tf.repeat(tf.shape(y_pred)[-1], tf.shape(y_pred)[0])

But it should have been
tf.repeat(tf.shape(y_pred)[-2], tf.shape(y_pred)[0])


text:
Can AI be understood as a generalized statistics tool?
I am a (soon-to-become, to be honest) theoretical physicist. I want to learn a bit about AI. So as you know in physics we develop theories based on as few and as simple basic equations as possible which shall explain as much of the experimental results and observations as possible. I feel that this is kind of not how AI solves problems.
My understanding is that AI can be understood as a very generalized and abstract statistics software package handling input data in a general way to find the "best fit" to some form of problem. Is that correct? I know it isn't. But is it vaguely correct?
I give you an example. In weather prediction there is a technique called MOS (model output statistics). It collects output from numerical weather prediction models (simulation software) as well as observational data and finds statistical relations between them to correct the model output for errors. For example, it might be that the intensity of precipitation in London is on average underestimated by the model by 10 %, so MOS will correct for that. Over time, it improves itself, because it collects more and more data. Is this already a form of AI?

My understanding is that AI can be understood as a very generalized and abstract statistics software package handling input data in a general way to find the "best fit" to some form of problem. Is that correct? I know it isn't. But is it vaguely correct?

No. It's not correct, in my opinion, not even vaguely and in many ways.

AI is not (necessarily) abstract (although there are examples of theoretical frameworks of AI, such as AIXI, which may be of interest to you, given that you are about to become a theoretical physicist)
AI does not necessarily generalize statistics or statistical concepts
AI is not just machine learning or, more precisely, supervised learning. You're very likely referring to supervised learning when you say "best fit", but there are other forms of learning (such as reinforcement learning) and there are other aspects of AI apart from learning, such as perception, control, etc. Some people sometimes claim that machine learning is glorified statistics, but, although the two are similar and related, I don't think that's exactly correct.

To know what AI is, you should read the book Artificial Intelligence: A Modern Approach. If you want to know what machine learning is, then there are many books that you can read, such as Machine Learning (1997) by Tom M. Mitchell. If you are interested in the relationship between machine learning and statistics, you may be interested in this post. You should also read this answer, which describes what AI is or may refer to. This answer gives a nice brief description of the difference between AI and ML.


text:
How do I learn the value function for a POMDP with a single-step horizon (bandit)?
Consider a POMDP with a finite number of environment states, $|\mathcal{S}| = N$, but the number of belief states is uncountably infinite. The belief state space is the convex hull of an $N$ simplex. Each turn this space is sampled with a flat probability distribution. As you are sampling from an uncountably infinite set of belief states, the probability of a belief state recurring in a finite number of samples is zero.
Now, let's suppose that there are a finite number of episodes, and each episode ends after 1-time step. At the only time step of the episode, the agent receives a belief state $b(s)$ over some fixed set of contexts $s$, selects a single action, and receives a single reward, before the episode ends.
I understand that the belief state value function, $V(b)$, is piecewise linear and convex, with a single hyperplane for each action (see e.g. [1]).
My question is, given that I only observe the belief states and the sampled rewards, how do I identify the value function, given that a belief state $b(s)$ has an infinitesimally small probability of occuring again?
The expected reward for a given belief state $b$ is just a linear function $\alpha \cdot b$, where $\alpha$ is the vector of the rewards for each state of the environment. But I cannot simply learn a linear model here because $\alpha \cdot b$ gives me the expected reward for a given belief state, but I may never start with this belief state again and so cannot simply calculate the sample mean expected reward.


text:
Is pre-processing used in deep learning?
I'm new to deep learning. I wanted to know: do we use pre-processing in deep learning? Or it is only used in machine learning. I searched for it and its methods on the internet, but I didn't find a suitable answer.

Yes, sure, data pre-processing is also done in deep learning. For example, we often normalize (or scale) the inputs to neural networks. If the inputs are images, we often resize them so that they all have the same dimensions. Of course, the pre-processing step that you apply depends on your data, neural network, and task.
Here or here are two examples of implementations that perform a pre-processing step (normalization in the second case). You can find more explanations and examples here and probably here too.

Adding to nbro's solution, the less you have to normalize/balance/preprocess/augment, the better, e.g. because then you know for sure that the accuracy is the achievement of the model rather than data combing. For example, if you can achieve the same accuracy using two approaches (e.g. with the image dataset):

for each image, subtract global mean, divide by global standard deviation
1)+random flips, random crops, color jitter, etc,

then 1), if you can achieve a comparable accuracy, is a better solution, as the model is more general. The same refers to the balancing of the data - if you can train a good model without it, it's an additional strength.

I wouldn't say that the model is "more general". I would say that the model has learned from a different range of inputs and possibly a more complicated function. So, I'm not really sure about the correctness of this answer, so I would like to read some reliable research paper that goes in the direction of what you're saying.

This can be verified on running the model on the out-of-sample data

Out-of-sample data, of course, can cause problems in both cases (normalization or not). So, would you care to show some code/research that illustrates the issue and really supports your claim?

Preprocessing in deep learning is about how to make data compatible to the model architecture's input tensor shape and output tensor shape.
So the main critical aspect of preprocessing in deep learning is how to encode the data into the tensor with a specified shape of model input so that it can be fed into the deep learning model.
As you already know, deep learning is intended to avoid any human intervention or feature engineering (opposite what traditional machine learning does). Therefore, normalization and feature scaling in deep learning are optional things, it's because they are considered as feature engineering. Unless this is the requirement to make compatibility shape of model input. Mainly if your data contains physical units that represent the real world. You can just let the model train that real-world thing.
When you encode the data to the tensor, you might lose some information such as metadata or index mapping, this is common mistake. You should keep the metadata or hard coded it into code.
The metadata supposed to decode the tensor back to the data that represent original data in post-processing step or inferring process.
So, this is the overall deep learning pipeline:
Training

Inference


text:
Can someone explain to me this implementation of Tile Coding using Hash Tables?
The code below is adapted from this implementation.
from math import floor

basehash = hash

class IHT:
    "Structure to handle collisions"
    def __init__(self, sizeval):
        self.size = sizeval
        self.overfullCount = 0
        self.dictionary = {}

    def count (self):
        return len(self.dictionary)

    def getindex (self, obj, readonly=False):
        d = self.dictionary
        if obj in d: return d[obj]
        elif readonly: return None
        size = self.size
        count = self.count()
        if count >= size:
            if self.overfullCount==0: print('IHT full, starting to allow collisions')
            self.overfullCount += 1
            return basehash(obj) % self.size
        else:
            d[obj] = count
            return count

def hashcoords(coordinates, m, readonly=False):
    if type(m)==IHT: return m.getindex(tuple(coordinates), readonly)
    if type(m)==int: return basehash(tuple(coordinates)) % m
    if m==None: return coordinates

def tiles(ihtORsize, numtilings, floats, ints=[], readonly=False):
    """returns num-tilings tile indices corresponding to the floats and ints"""
    qfloats = [floor(f*numtilings) for f in floats]
    Tiles = []
    for tiling in range(numtilings):
        tilingX2 = tiling*2
        coords = [tiling]
        b = tiling
        for q in qfloats:
            coords.append( (q + b) // numtilings )
            b += tilingX2
        coords.extend(ints)
        Tiles.append(hashcoords(coords, ihtORsize, readonly))
    return Tiles

if __name__ == '__main__':
    tc=IHT(4096)
    tiles = tiles(tc, 8, [0, 0.5], ints=[], readonly=False)
    print(tiles)

I'm trying to figure out how the function tiles() works. It implements tile coding, which is explained in "Reinforcement Learning: An Introduction" (2020) Sutton and Barto on page 217.
So far I've figured out:

qfloats rescales the floating numbers to be the largest integer less than or equal to the original floating number, these are then re-scaled by the number of tilings.

Then, for each tiling, a list is created, the first element of the list is the tiling number, followed by the coordinates of the floats for that tiling, i.e. each of the re-scaled numbers is offset by the tiling number, b and integer divided by numtilings.

Finally, hashcoords first checks the dictionary d to see if this list has appeared before, if it has it returns the number relating to that list. It not it either creates a new entry with that list at the key and the count as the value or if the count is more than or equal to the size it adds one to overfullCount and returns basehash(obj) % self.size.

I'm struggling to understand two parts:

What is the tilingX2 part doing?

Why is tilingX2 added to b after the first coordinate has been calculated? It seems to me that each coordinate should be treated separately

And why by a factor of 2?

What is the expression basehash(obj) % self.size doing? I'm quite new to the concept of hashing. I know that, generally, they create a unique number for a given input (up to a limit), but I'm really struggling to understand what is going on in the line above.

Hello. Welcome to AI SE :) Your post was not properly formatted, so I took the time to format it properly. Please, take a look at https://ai.stackexchange.com/help/formatting and at the current version of your post so that you know how a "good-looking" posts looks :) Take also a look at this https://ai.meta.stackexchange.com/q/1654/2444 if you have some time.

To partially answer your 4th question. The operator a % b is the modulo operator, i.e. it returns the remainder of the integer division between a and b. So, say that you have a = 4 and b = 3, then a % b = 4 % 3 = 1, because 3 * 1 + 1 = 4.

So, basically, basehash(obj) % self.size is a number that is ensured to never be greater or equal to self.size. So, if you have an list of size self.size, you can index it with basehash(obj) % self.size and be sure you will not get an IndexError.


text:
How to implement REINFORCE with eligibility traces?
The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set $\lambda^{\theta}=1$ and replace $\delta$ with the immediate reward $R_t$, do I get a backwards implementation of REINFORCE?

Well, I think, if you set $\lambda^{\theta}$ to 1, there will be no sense in eligibility traces. In this case you are going to update $\theta$ directly based on policy gradient multiplied by TD-error $\delta$

Well, not quite. $z^{\theta}$ will still hold $\nabla \pi(A|S, \theta)$ from previous steps, and will decay them based on $\gamma$. That means past updates will be credited in a decaying manner going forward. Since ET implement a backward view of the $\lambda$-return, and setting $\gamma=1$ there should result in MC updates. Is there anything wrong in this reasoning?

Now that I think about it, setting both $\lambda^{\theta}=1$ and $\lambda^w=1$ should make this method become REINFORCE with baseline, shouldn't it?


text:
What are acting as weights in a convolution neural network?
Looking at some old notes I took on CNN's and I wrote down that the weights in a CNN are acting like filters in a CNN but to be honest I don't really know what the weights are acting as in a CNN and was wondering if someone could explain that clearly to me.

You asked me to re-open this post because you wanted to get a specific answer to your question, although this question is already although somehow implicitly answered in other posts. In any case, if my answer below answers your question, you should accept it.

hi @PlayerUnknown_12, check out this minicourse about CNNs by Andrew Ng, if you're interested. I found it very helpful. Starting from second video he discusses how filters of CNNs can learn to detect edges of the objects in the images

There are many resources that answer your question, but, given that you're apparently new to machine learning (ML), deep learning (DL), and neural networks (NN), let me provide a simple answer that should clarify your doubts.
The term weight in the context of ML, DL, and NN is a synonym for parameter (sometimes, in some contexts, such as linear regression, it is also known as coefficient), which can be constant (i.e. do not change during, for example, learning/training) or learnable (i.e. change during the training/learning process). In a feedforward neural network (FFNN), with and without recurrent connections, the weights are the numbers on the connections between neurons in different or same layers. They are called weights to emphasize that their role is to weigh the effect of one neuron on the other.
In a CNN, the weights are the kernels/filters of the CNN, i.e. the matrices that you use to perform the convolution (or cross-correlation) operation in a convolutional layer. So, given that CNNs perform an operation that seems to be different than the linear combination followed by the non-linear activation function in FFNNs, you could think that the weights in a CNN are not very similar to the weights in FFNNs. However, this is not fully true, as CNNs can be viewed as FFNNs (with some specific structure): maybe this answer will provide you more info about this topic. So, your notes are right!
In my opinion, the best way to understand the role of the kernels (i.e. weights) of a CNN and all the details behind the convolution (or cross-correlation) operation is to first study some traditional computer vision and image processing techniques, so maybe you should pick a book that covers them.
In this answer, I briefly try to describe what a CNN is and what it can be used for. There are other answers on this site, such as this, this, and this, which you may want to read, if not now later, to understand even more all the details of CNNs. A  decent reference book that covers CNNs (and other DL topics) is Deep Learning by Ian Goodfellow et al.


text:
In UCB, is the actual upper bound an upper bound of an one-sided or two-sided confidence interval?
I'm a bit confused about the visualization of the upper bound (following the notation of (c.f. Sutton & Barto (2018))
$$Q_t(a)+C\sqrt{\frac{\mathrm{ln}(t)}{N_t(a)}}$$
In many blog posts about the UCB(1)-algorithm, such as visualized in the following image (c.f. Link ):

Isn't the upper (confidence) bound simply the upper bound of a one-sided confidence interval instead of a two-sided confidence interval as shown in the image above?
A lower bound of the interval is completely useless in this case, or am I wrong?

As far as I am aware it is for a two sided confidence interval. I only have a brief knowledge of MAB's so I don't want to write an official answer but I am fairly certain a paper I read on them a few years ago also proved a lower bound (I guess for completeness) because you are right in saying that a lower bound would be useless for a MAB.

In the MAB domain, it is possible to specify a lower as well as an upper bound for the regret of an algorithm. Probably you mix it up with that?

The upper bound used here is derived from Hoeffding's inequality, which provides a symmetric, two-sided confidence interval. A good pair of blog posts on how this bound used in UCB for bandits is derived can be found here:

First steps: Explore-then-Commit
The Upper Confidence Bound Algorithm

Indeed, in practice when using this UCB for bandits, we do not actually care about the lower bound. We only need the upper found for the exploration mechanism. But the lower bound does still exist, even if we don't use it.


text:
Current research on Gödel machines
Is there any current research on Gödel machines? It seems that the last article by Jürgen Schmidhuber on this topic was published in 2012: http://people.idsia.ch/~juergen/goedelmachine.html

This paper Can Machines Design? An Artificial General Intelligence Approach (2018, presented at AGI-18 and published in the related proceedings here), which proposes the design Gödel machine, may be useful to you.
After a quick search, I have not found other relevant papers, so I suppose that the research on GMs is currently not very active.


text:
Is better to reward short- or long-term progress in Q-learning?
I have been training some kind of agent to reach a target using a Q-learning based approach, and I have tried two different types of rewards:

Long-term reward: $\mathrm{reward} = - \mathrm{distance}(\mathrm{agent,target})(t+1)$

Short-term reward: $\mathrm{reward} = \mathrm{distance}(\mathrm{agent,target})(t) - \mathrm{distance}(\mathrm{agent,target})(t+1)$

In the first case, I am rewarding the current progress. In the second case, I am rewarding direct progression, but this may lead to less progression in the future. My question is, what kind of reward does Q-learning need?
I understand that the $\gamma$ factor should incorporate long term rewards, so it makes more sense to reward direct progression. However, using long-term rewards gave better results for my scenario...

perhaps you could take a weighted average of both? or use one to regularise the other? this is just off the top of my head, it would need some thought to make sure there's no way the agent could exploit something like this to maximise the returns without achieving your objective.


text:
How are the parameters $\alpha_i$ of hard attention trained?
I have a question about Show, Attend and Tell: Neural Image CaptionGeneration with Visual Attention paper by Xu. The basic mechanism of stochastic hard attention is that each pixel of the input image has a corresponding parameter $\alpha_i$, which describes the probability that this pixel will be chosen for further processing.
But I don't see an explanation of how to train or define this parameter in the paper. Can someone explain how to train this $\alpha_i$ for each pixel?


text:
How and why do state-of-the-art models in medical segmentation differ from general segmentation models?
I am just getting into medical image segmentation and have been able to understand the state-of-the-art architectures, like Double UNet, UNet++, and Multiresunet.
What I haven't understood yet: Why are these approaches better for medical segmentation than, for example, HRNet-OCR, which currently tops the rankings of the Cityscapes dataset, and vice versa?

Why do you think that UNet++ is better than HRNet-OCR? Can you please provide a reference that supports this conclusion? It may also be a good idea to include links to the papers that introduced all these models and to some site or whatever that supports this claim "HRNet-OCR, which currently tops the rankings of the Cityscapes dataset".


text:
Is using a LSTM, CNN or any other neural network model on top of a Transformer(using hidden states) overkill?
I have recently come across transformers, I am new to Deep Learning. I have seen a paper using   CNN and BiLSTM on top of a transformer, the paper uses a transformer(XLM-R) for sentiment analysis in code-mixed domain. But many of the blogs only use a normal feed formal network on top of the transformer.
I am trying to use transformers for sentiment analysis, short text classification.
Is it overkill to use models like CNN and BiLSTM on top of the transformer considering the size of the data it is trained on and its complexity?

Hello. Welcome to AI SE. It would be better if you describe the problem you're trying to solve with the transformer and LSTM and what that paper is using them for.

The paper is using transformers for sentiment analysis on code mixed domain

It may be useful to also know what those blog posts that you mention use the LSTM + transformer for. Do they use it for the same task? Maybe provide a link to one such blog post.

https://medium.com/atheros/text-classification-with-transformers-in-tensorflow-2-bert-2f4f16eff5ad

The hugging face implementation of transformers for sequence classification uses a feed-forward layer on top of the transformer

How would using a LSTM after a transformer improve the performance? I mean, the transformers does all the magic a LSTM does (it even sees the whole context at the same time). A pretrained transformer + a linear layer is the normal way to do it, to the best of my knowledge..

Using transformer as a feature extractor for the LSTM

But all the magic a LSTM does is captured by attention. So when you finetune on your target task the properties of memory (that lstms provide) are already captured by the transformer.


text:
Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?
I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs.
To ask my question straightforwardly: is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$? If I need to be more restrictive, assume a game setting, for example, chess.
My first hypothesis would be to define the terminal state as the state $s_T$ such that $p(s',r|s_T,a) = p(s',r|s_T)$, a state from which the transition is independent of the agent's actions. But that does not seem quite right. First, there is no particular reason why this state should be unique. Second, from this definition, it could also just be an intermittent state of "lag".

Wait, are you asking how would the transition model be defined for a terminal state of an MDP? From the equations, that's what you seem to be trying to do, but that's somehow a different question than defining the "terminal state from the transition model" (which is your question in the body). So, I suggest that you edit this post to clarify what your question really is.

I don't know if there is a general definition of the terminal state based on the MDP transition probabilities.
But remember that we define our MDP problem in a $\mathbb{S}$ set of all possible states and a $\mathbb{A}(s)$ representing the set of all possible actions for each state. Based on that, probably there aren't any possible actions for the terminal state, so the transition probability $p(s', r |s_T, a)$ can be undefined. Based on that, in episodic tasks we need to define a the set of all possible terminal states $\mathbb{S}^+$ and distinguish this with $\mathbb{S}$.

As far as I remember, terminal state is a state from which agent cannot escape, i.e if the agent reached this state, he will never escape. In mathematical notation can be written as:
$$
p(s^{'}, r|s_T,a) = \delta_{s^{'}s_T} \delta_{rr_{S_T}}
$$
Where $\delta_{ab}$ is a Kronecker symbol, and by $r_{S_T}$ I mean the reward collected by the agent sitting in the terminal state from now till the end of the episode.
This state doesn't have to be unique. Imagine a Markov chain as a set of points, representing states, and arrows between the states with associated probabilities. Nothing prevents you from defining MDP where several nodes have only one outgoing arrow pointing to itself with the probability 1.


text:
Is there any situation in which breadth-first search is preferable over A*?
Is there any situation in which breadth-first search is preferable over A*?

Could you please add more context? Why are you interested in this question?

The only general situation that comes to my mind where BFS could be preferred over A* is when your graph is unweighted and the heuristic function is $h(n) = 0, \forall n \in V$. However, in that case, A* (which is equivalent to UCS) behaves like BFS (except for the goal test: see section 3.4.2 of this book), i.e. it will first expand nodes at level $l$, then at level $l+1$, etc., this is because nodes at level $l+1$ are farther away from the initial node than nodes at level $l$, i.e. $f(n') = g(n') > f(n) = g(n)$, for all $n' \in V_{l+1}$ and $n \in V_{l}$, where $g(n)$ is the cost of the shortest path from the initial node to $n$ and $V_{l}$ is the subset of nodes of the search space that belong to level/layer $l$ of the search tree.
So, if you think that your goal is close to the initial node in terms of levels/layers and each node has a small branching factor (i.e. not many children), then BFS may be a good idea (but you could also use A*).
BFS can also be used as a sub-routine in other algorithms, such as the Ford–Fulkerson algorithm. So, in these cases, BFS may also be preferable.

There is an inherent assumption in heuristic search that the heuristic function points you in the right direction.
A* largely depends on how good the heuristic function is. Two nice properties for the heuristic function are for it to be admissible and consistent. If the latter stands, I can't think of any case where BFS would outperform A*. However, this property doesn't stand in every case.
If you select a misleading heuristic function (i.e. a function that points you in the wrong direction) then BFS should outperform A*.

I drew a dumb example to illustrate my point. You want to reach from the Initial State (IS) to the Goal State (GS). The shade of green indicates the value of the heuristic (the greener the better).

In the third case, where you have a misleading heuristic, A* would tend to explore areas near the top right first, then it would go down towards the GS. This case is actually worse than having no heuristic and using a BFS.

I just want to point out that having "no heuristic" would be like having a heuristic $h(n) = 0$, for all $n$, which is an admissible heuristic, so, in that case, A* should also be optimal and, as I say in my answer, it should behave like BFS, although I didn't try it out (with an example to confirm it) and some details may be different.


text:
Is there an ideal range of learning rate which always gives a good result almost in all problems?
I once read somewhere that there is a range of learning rate within which learning is optimal in almost all the cases, but I can't find any literature about it. All I could get is the following graph from the paper: The need for small learning rates on large problems

In the context of neural networks trained with gradient descent, is there a range of the learning rate, which should be used to reduce the training time and get a good performance in almost all problems?

The 2015 article Cyclical Learning Rates for Training Neural Networks by Leslie N. Smith gives some good suggestions for finding an ideal range for the learning rate.
The paper's primary focus is the benefit of using a learning rate schedule that varies learning rate cyclically between some lower and upper bound, instead of trying to choose a single fixed learning rate value. For this to work, you still need to select good lower and upper bounds, and Smith suggests training the model for a few epochs while increasing the learning rate between a large range of values. At first, the learning rate will be too small to make any progress at all. As the learning rate increases, eventually, the loss will begin to decrease, but, at some point, the learning rate will get too large, and the loss will stop decreasing and even begin increasing.  Your ideal range consists of the learning rate values where the loss was decreasing steeply.  After finding your range, you can reset the weights and biases on your model and restart training using whatever learning rate schedule you plan to use for training.
Here is a concrete example from one of my experiments:

In this case, I start my learning rate search at 1e-09 and plan to end with a learning rate of 0.99 (although I am actually able to stop sooner than that).  Your experiment may require different search bounds, but you could always start with that and adjust things as needed.  At first, the loss plot is flat, and then it begins to decrease, but is too gradual.  At the first red line, loss starts to decrease sharply, and once it reaches the second red line, the plot has begun to level off, so I can end my search.  For this particular experiment, my ideal learning rate range had a minimum of 4.01e-4 and a maximum of 2.58e-2.
For more information, I suggest reading this Keras Learning Rate Finder post, which contains more information on how the process works and a tutorial for how to program it using Keras and Tensorflow.

Edit: I recently learned that in newer versions of Tensorflow / Keras, the on_train_batch_end callback now returns an aggregated average loss instead of the raw loss for the given batch, which gives poor results for the learning rate finder.  See this Github Issue for more information and the workaround I am currently using.

The visualisation can be found in The need for small learning rates on large problems. This paper by D. Randall Wilson and Tony R. Martinez from 2001 investigates the role of learning rates in gradient descent algorithms.
In general, different algorithms assign different meaning to the same word 'learning rate'. For example, the learning rate in a gradient descent algorithm is not comparable to the learning rate in a tabular reinforcement learning algorithm such as Q-learning.
This means that at a particular 'best' does not exist considering the different concepts denoted with the term `learning rate' in different algorithms.
Additionally, the learning rate is typically considered a part of the learning algorithm. The no free lunch theorem of machine learning tells us that no particular learning algorithm performs best across tasks. Because the learning rate is part of the solution, no particular learning rate is 'best' across tasks either.
In practice, you should set the learning rate sufficiently low to not 'overshoot' the optimal solution which will be evidenced by oscillations in the error (no convergence). But you also should also set it high enough to obtain reasonable performance given the amount of available training time.
What learning rate gives you the right trade-off typically requires a combination of domain knowledge and experimentation on the training set.

"In general, different algorithms assign different meaning to the same word 'learning rate'." This is wrong for your example. The meaning of "learning rate" in Q-learning is very similar to the meaning of "learning rate" in GD (if not the same thing): just look at the pseudocode!! In fact, in deep Q-learning, they are exactly the same thing. Maybe you need to clarify that the ideal range for the LR in one case may be different than in another case.

Moreover, note that now the question focuses on neural networks trained with gradient descent.  Moreover, this answer doesn't not answer the question at all. The question is "is there an ideal range of learning rate which always gives a good result almost in all problems?". The question is not "How do I set my ideal learning rate?"


text:
When using PCA for dimensionality reduction of the feature vectors to speed up learning, how do I know that I'm not letting the model overfit?
I'm following Andrew Ng's course for Machine Learning and I just don't quite understand the following.
Using PCA to speed up learning

Using PCA to reduce the number of features, thus lowering the chances for overfitting

Looking at these two separately, they make perfect sense. But practically speaking, how am I going to know that, when my intention is to speed up learning, I'm not letting the model over-fit?
Do I've to find a middle-ground between these two scenarios when applying PCA? If so how exactly can I do that?

After reading your post more than once, I think I got your question, so I edited your post to clarify what your question seems to be and I put it in the title. Please, make sure that the question in the title is your question.

Yes @nbro...that's exactly what I wanted to ask..Thanks for the edit :)

I'm not sure if I understood your question correctly, but here's my take anyway.
So, PCA is a technique that you can apply to data to reduce the number of features.
In return, (i) this can speed-up training, as there are less features to do computation with, (ii) and can prevent overfitting, as you lose some information on your data.
To detect overfitting, you usually monitor the validation and training losses during the training. If your training loss decreases, but your validation loss stays constant or increases, it's likely that your model is overfitting on the training data. In practice, this means your model generalizes worse and you can observe this by measuring the test accuracy.
All in all, you can apply PCA, train a new model, and measure your model's test accuracy to see if PCA has successfully prevented overfitting. In case it didn't, you can re-train with other regularization techniques such as weight decay and so on.
After your edit that put the slides
Basically, what slides claim is, PCA could be a bad way to prevent overfitting when compared to using standard regularization methods. To actually see whether this is the case, the standard way would be measuring your model's performance on a validation dataset. So, if PCA throws away lots of information, and hence causes your model to underfit, your validation accuracy should rather be poor wrt to using standard regularization techniques.


text:
Can I do topic classification of Arabic text (software requirements) without a training dataset?
I am trying to make a text classification for Arabic data. The problem is that there is no labeled Arabic dataset for this data. My question is then: is possible to do a classification without a training dataset? If yes, what methods can I use?

Hi. It's not clear to me why you have "software requirements"  in the title. Can you clarify that?

the data that I want to classify is a software requirements

Yes, it is possible to do it. This can be posed as an unsupervised text classification. You can look at TF-IDF, Neural networks (BERT) etc. for creating embedding of your text and then, use clustering techniques like K-Means, KNN etc. for classification.

as i know  unsupervised text classification needs an unlabeled dataset which I don't have

if you even don't have data, how do you expect to create a model. unsupervised is albeit a safety net when supervised is not possible. no data = no model. anyways, the BERT link is for Arabic language, maybe use that to do it directly but that would still need data.

https://lionbridge.ai/datasets/best-arabic-datasets-for-machine-learning/ and you do have datasets. or you need to create a small dataset yourself. but dataset is a precondition for a model.


text:
How can I estimate the minimum number of training samples needed to get interesting results with WGAN?
Let's say we have a WGAN where the generator and critic have 8 layers and 5 million parameters each. I know that the greater the number of training samples the better, but is there a way to know the minimum number of training examples needed? Does it depend on the size of the network or the distribution of the training set? How can I estimate it?

Hello. To clarify, are you only interested in WGAN? In any case, it seemed to me that was the case, so I tried to clarify that in your post. So, are you also just interested in the task of learning a probability distribution to generate samples? In other words, which machine learning task are you interested in? It may also be a good idea to be more specific when you say "interesting results", although, in this case, this detail should be obvious (although dependent on the task and performance measure).

Sorry, I just started with machine learning less that a month ago. With interesting I mean like what WDCGANs usually do, generate elements that look similar to the samples without being the same. I made the quiestion thinking about GANs but if someone gives guidelines for any ANN, that would be good.


text:
In classification, how does the number of classes affect the model size and amount of data needed to train?
When solving a classification problem with neural nets, be it text or images, how does the number of classes affect the model size and amount of data needed to train?
Are there any soft or hard limitations where the number of outputs starts to stall learning?
Do you know about any analysis of how the number of classes scales the model?
Does the optimal size increase proportionally with the number of outputs? Does it increase at all? If it does increase, is the relationship linear or exponential?

Model/network design has multiple guidelines, a basic one is: The solving capacity of the network should be larger than the possibility space of the problem to be solved.
Solving capacity (learning capacity) of a network (dense usually) can be calculated as the product of number of neurons in all layers, for example:
Input shape: 10 values
Network shape: [layer1 30 units, layer2 20 units, output 1 unit] should have learning capacity of $30 \times 20 \times 1 = 600$, it learn roughly max 600 different inputs (each input holds 10 values).
Another consideration, the separation lines, even when the inputs of the problem to be learnt are unlimited, but the 2 classes (just example) are always separated on 2 sides of a line without mixing up, just a single neuron can solve the problem.
One neuron can make 1 separation line, 1 layer makes a poly-line with segments are by the neurons in that layer, another layer makes another poly-line.
Thus, more classes, more separations to be done, and more classes would mean the input variety is large, so surely training data are a lot and model size needs to be large.

Very good point with the seperation lines. I am thinking about testing it out by making a dataset generator for n-classes (n-arm spiral?) and see how the size of minimal viable model changes when n increases.
Imo the number of different inputs a network "can hold" depends on the complexity of the problem. If dataset is seperatable by a line, then it can hold infinite amount of points, because it generalized well. If dataset is completely random, then the network can't generalize and points can only be memorized. Maybe in the second case your product-based method holds, but idk.

yeah, those 'completely random' datasets are crazy

The most obvious way more classes increase the network size is the output layer, but I don't believe there is a rule of thumb for the size of the entire network.
As I understand it, there is no clear answer to how big a network needs to be to achieve a certain performance with regard to the number of layers compared to the number of classes. This is a very active research field, just as an example compare the size of EfficientNet to other State of the art models when it was introduced and you can see the size difference.
Regarding the data needed, in The Deep Learning Book (which is a few years old now) they state as a general that with the models available at that time, you needed ~5000 examples per label for acceptable performance, while to exceed human performance (their words) you would need about 10 million labeled examples.

I didn't expect the difference between acceptable and super-human to be three orders of magnitude. Do you remember in which chapter of the book this statement was made?

As I mentioned, it is a rather old book (with regard to DL, since everything develops so fast) so this might have changed since it's a rather active research field. However, I think it's in the introductory chapter, under some title with Data in the name.


text:
Is data leakage relevant when scaling across samples?
I have a question about data leakage when pre-processing data for a neural network and whether data leakage actually applies in my instance.
I have variance stabilising transformed genomic data. Because it is genomic data we know apriori that lower numbers translate to lower levels of a gene being made and vice versa. Before input into the neural network, the data are squashed to between 0 and 1 using sklearn:
preprocessing.minmax_scale(data, feature_range=(0,1), axis=1)

The min_max scaling needs to be done across sample (axis=1) as opposed to features because of this apriori assumption of gene levels - low genes need to remain low and vice-versa...
Because of this, my question is: do training samples still need to be scaled separately from test samples as it doesn't seem there is a risk of data leakage here? Is this the correct assumption to make?

Considering that you are making a minmax scaling, the only time in which there would be no risk of data leakage is if the minimum value on your training set equals the minimum value of the test set, and your maximum value on the training set equals the maximum value on the test set.
In that circumstance the result of your scaling would be exactly the same as fitting the scaler to the training set, and applying it to the test set.
There is basically no advantage in scaling them separately, and you take the risk of data leakage if their value ranges differ. I don't know much about genomic data, but even if your assumption was correct I would recommend not to do that.

Thank you for your detailed response... So recommend to not scale separately? As in scale all data together and then split?

Fit your scaler to the train data and then use the same scaler to the test data. If you scale all data together then you are also taking the risk of leakage.

Thank you for clarification... I think I am still none the wiser given that I am scaling per sample (axis=1) - not per feature because of the nature of the data.. I think the justification is lacking for me, I don't really see why... Also the range per sample is slightly different, it's not the same across....


text:
Why is Openai's PPO2 implementation differentiable?
I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: loss = pg_loss - entropy * ent_coef + vf_loss * vf_coef.
First question: The computation of pg_loss requires to use operations like tf.reduce_mean and tf.maximum. Are these two functions differentiable? Apparently, they are, otherwise, it would not work. Can someone explain why so I can understand the implementation?
Second question: During training, an action is sampled by using the Gumbel Distribution: Noise from such a distribution is added to the logits and then tf.argmax is applied. This index is then used to calculate the negative log-likelihood. However, the tf.argmax should also not be differentiable, so how can this work?

Please, edit your post to include a link to the implementation you're referring to.


text:
Can the hidden state of an RNN be a matrix?
If I'm dealing with a sequence of images as the input (frame by frame), and I want to output a matrix at each timestamp, can the hidden state be a matrix?

Please add more context, so that it's easier for people to help you.

Hi Paul. Welcome to Artificial Intelligence Stack Exchange! As someone suggested above, could you please explain why are you asking whether the hidden state of an RNN can be a matrix? Why do you need it to be a matrix? This doesn't  seem to necessarily follow from what you wrote in your post so far.

Yes, I would say more, that hidden state can be a tensor of arbitrary dimensionality. For vanilla RNN the update rules of the hidden state and the output are:
$$
h_t = \sigma_h(W_h x_t + U_h h_{t-1} + b_h)
$$
$$
y_t = \sigma_y(W_y h_t + b_y)
$$
Here $W_h$ is input to hidden state matrix, and $U_h$ is hidden state to hidden state, $W_y$ is hidden state to output. One can simply upgrade these matrix products to more general case of tensor products, such that they could handle multidimensional input data and hidden state.
The contraction of the $m$-dimensional tensor $X$ with $n$-dimensional Y (we assume $m \geq n$) by maximal amount of indices will be $m-n$ dimensional tensor $Z$:
$$
X_{i_1 \ldots i_m} Y_{i_1 \ldots i_n} = Z_{i_{n+1} \ldots i_m}
$$
Namely, imagine, that the $x_t = (x_t)_{i_1 \ldots i_X}$ is an $d_x$-dimensional tensor. Then take $W_h$ to be  $d_{wh}$ dimensional tensor, hidden state $h$ to be $d_h = d_{wh} - d_{x}$ -dimensional, $U_h$ is $d_{uh}$ - dimensional, $W_y$ is $d_{wy}$ -dimensional. The output $y_t$ of the RNN will be $d_{wy} - d_{h}$ dimensional tensor.
Here note, that in order for the sum to be make sense in the first equation, dimensions need to match:
$$
d_{uh} - d_{h} = d_{h}
$$

The question is, why you specifically want a matrix? I assume you mean per-frame features. In such case you can use a ConvNet as a feature extractor, i.e. it outputs a vector of features of fixed size. This vector is an input in your LSTM. If you have $C$ frames, the output of LSTM is
$$
Output, (h, c) = LSTM (frame \ features)
$$
where frame features are size $(1, C, v)$, where $v$ is the feature vector length from ConvNet. $Output$ is size $(1, C, w)$ and $h$ is $(1, 1, w)$ where $w$ is the size of the hidden layer in LSTM, and $h$ is the last hidden layer. Now, you can resize $Output$ to $(C, w)$ and make it an input in some linear layer that outputs a batch size $C$. This is your predictions per frame.


text:
Is it okay to calculate the validation loss over batches instead of the whole validation set for speed purposes?
I have about 2000 items in my validation set, would it be reasonable to calculate the loss/error after each epoch on just a subset instead of the whole set, if calculating the whole dataset is very slow?
Would taking random mini-batches to calculate loss be a good idea as your network wouldn't have a constant set? Should I just shrink the size of my validation set?

I would also suggest that you explain a little bit the model you're using, how much time it takes to compute the validation loss and what is the task you're trying to solve.

I assume you intended to write compute the evaluation metric over the validation set in batches; you do not compute loss over the validation set!
That is quite a standard practice in many academic implementations (because, when the validation set is large enough, the memory will be a constraint), however, be sure to take the average of the values over all the batches. Using a K-fold setup will increase the confidence in the reported values.


text:
How should I model the state and action spaces for a problem where the goal is to draw a line between two points?
I have a problem where the goal is for the agent to draw a single line between two points on a  $500 \times 500$ white image.
I have built my DQN. For now, the output layer's size of the network is $[1, 500 * 500]$. This way, when a Q value is given within a single step, it can be mapped to a single coordinate within the space of the image.
So, with that, I'm able to get a starting point for the line. However, what that doesn't give me is the location of the second point for completion of the line.
One thing I have tried is drawing a line for every two steps. However, this means that the state of the environment does not change for each step.
Should the goal be to change the environment/state for each step or does this not matter? Maybe I have not found the ideal way of modelling the state and action spaces for this problem.

hi @junfanbl, what actions can the agent take?

I have edited this post to try to save it by asking the question "How to model the action and state space in this problem?". Make sure that you agree with these changes. If not, edit the post again.


text:
How exactly is Monte Carlo counterfactual regret minimization with external sampling implemented?
I have read many papers, such as this or this, explaining how external sampling works, but I still don't understand how the algorithm works.
I understand you divide $Q$, which is the set of all terminal histories into subsets $Q_1,..., Q_n$.
What is the probability of reaching some subset $Q_i$? Is it just the product of chance probability, the opponent's probability, and my probability?
As I understand it, the sampling only occurs in the opponent's information sets. How does that work? If there are two players, player 1 strategy is based on what strategy I use.
What happens after you have determined a subset $Q_i$ you want to sample? How many times do you iterate over the subset $Q_i$?
I have searched around and I cannot find any Python code that uses external sampling, but plenty of papers that give formulas, but do not explain the algorithm in detail. So, a Python example of MC-CFR external sampling would probably make it a lot easier for me to understand the algorithm.

Hello. Welcome to Artificial Intelligence Stack Exchange. Please, ask only 1 question per post. If you have multiple questions, ask each of them in its separate post. Of course, in your case, the questions are all related to the same algorithm, but it's still fine to ask them in separate posts, provided that you provide the necessary context in each post to understand the question.

External sampling and outcome sampling are two ways of defining the sets $Q_1, \dots, Q_n$. I think your mistake is that you think of the $Q_i$ as fixed and taken as input in these shampling schemes. It is not the case.
In external sampling, there is as many sets $Q_{\tau}$ as there are pure strategies for the opponent and the chance player (a pure strategy is a deterministic policy). Think of it as "the set of terminal nodes that I can reach if my opponent and chance play in this fixed way".
Sampling a set $Q_{\tau}$ thus means sampling a pure strategy for the opponent and chance node. An alternative is to sample on the fly the opponent's policy $\sigma^{-i}$ (and chance) when needed. It gives identical probabilities for q(z), while does not correspond to a full definition of a deterministic policy $\tau$ (which would need a definition on all the information nodes).
There are several implementations on Github, one can be found here: https://github.com/bakanaouji/cpp-cfr/blob/master/RegretMinimization/Trainer/Trainer.cpp

Do you know about any implementation code that is in python? That is the only programming language that i currently know

Do I have to update strategies for all info sets doesn't that take up a lot of memory?


text:
How would one disambiguate between two meanings of the same word in a sentence?

The boy lifted the bat and hit the ball.

In the above sentence, the noun "bat" means the wooden stick. It does not mean bat, the flying mammal, which is also a noun. Using NLP libraries to find the noun version of the definition would still be ambiguous.
How would one go about writing an algorithm that gets the exact definition, given a word, and the sentence it is used in?
I was thinking you could use word2vec, then use autoextend https://arxiv.org/pdf/1507.01127.pdf to differentiate between 2 different lexemes e.g. bat (animal) and bat (wooden stick).
Then the closest cosine distance between the dictionary definition and any of the words of the sentence might indicate the correct definition.
Does this sound correct?

I'd suggest BERT for this. It is essentially a word-embedding model that uses at local context to determine the appropriate embedding for each word. This means it would assign "bat" a different embedding in a sentence containing "hit the ball" vs. a sentence containing "flies and eats bugs". On top of that, Google has released a number of pre-trained versions of BERT, which can be used directly without additional training (depending on your task of course). BERT as a service is great if you just want embeddings. The Python transformers library makes it exceedingly simple to incorporate BERT into your task-specific model.

Could you please elaborate on how to utilise BERT? Which functions to use to achieve the task outlined?
From what I've read, BERT isn't good for cosine similarity. So how can this be used?

Sorry I think I misunderstood your use-case the first time around. It seems you want to match tokens in context to dictionary definitions? This is a non-trivial problem. You're right that BERT may not be the best choice because the embeddings for the same word change depending on context. Autoextend may work. You may also use BERT to average the embeddings for the words of the same definition on a corpus where the ground truth meanings are known and annotated, then for new sentences compute the cosine sim between the new occurrence of the word and the averaged embedding.


text:
What is the definition of the hinge loss function?
I came across the hinge loss function for training a neural network model, but I did not know the analytical form for the same.
I can write the mean squared error loss function (which is more often used for regression) as
$$\sum\limits_{i=1}^{N}(y_i - \hat{y_i})^2$$
where $y_i$ is the desired output in the dataset, $\hat{y_i}$ is the actual output by the model, and $N$ is the total number of instances in our dataset.
Similarly, what is the (basic) expression for hinge loss function?

The hinge loss/error function is the typical loss function used for binary classification (but it can also be extended to multi-class classification) in the context of support vector machines, although it can also be used in the context of neural networks, as described here.
The hinge loss function is defined as follows
$$
\ell(y) = \max(0, 1-t \cdot y) \tag{1}\label{1},
$$
where

$t = \{-1, 1\}$ is the label (so, if your labels are in the set $\{0, 1 \}$, you will have to first map them to $\{-1, 1\}$)
$y$ is the output of the classifier (e.g. in the context of the linear SVM, $y=\mathbf{w} \cdot \mathbf{x}+b$, where $\mathbf{w}$ and $b$ are the parameter of the hyper-plane)

This means that the loss in equation \ref{1} is always non-negative. If you're familiar with the ReLU, this loss should look familiar to you. In fact, their plots are very similar.
For more details, you probably should start with the related Wikipedia article, then maybe one of the many machine learning books that covers support vector machines, for example, Pattern Recognition and Machine Learning (2006) by  Christopher Bishop, chapter 7 (page 325).


text:
Is the framework provided by this paper for checking the constraints of AI systems really new?
The authors of this paper present a framework for checking the constraints of AI systems using formal argumentative logic between 2 agents: an interrogating agent and a suspect agent. The interrogating agent is attempting to find contradictions in the responses of the suspect agent by querying about information and the suspect agent must provide all the relevant information.
Is this framework really new? I am pretty certain that I saw a very similar framework in the context of program verification some years ago.


text:
Which metric should I use to assess the quality of the clusters?
I have a model that outputs a latent N-dimensional embedding for all data points, trained in a way that clusters data-points from the same class together, while being separated from other clusters belonging to other different classes.
The N-dimensional embedding is projected down to 2D using UMAP. At each epoch, I wish to test the clustering capability of the model on these 2D projections for use as validation accuracy. I have the labels for each class.
How should I proceed?

You can compute Silhouette Coefficient for your aim. Its values mean:

1: Means clusters are well apart from each other and clearly distinguished.
0: Means clusters are indifferent, or we can say that the distance between clusters is not significant.
-1: Means clusters are assigned in the wrong way.

Other measures, such as purity and mutual information, are also possible by computing

an external criterion that evaluates how well the clustering matches the gold standard classes

One more popular metric for this is the Davies Bouldin Score.
You can also take a look at the clustering metrics in scikit documentation.


text:
What kind of deep learning model does latest version of AlphaFold use for protein folding problem?
I understand there are multiple versions used in AlphaFold. What kind of deep learning model does the more advanced version use? CNN, RNN, or something else?
(Additionally, is there an open-source reference model for the protein folding problem?)

The accurate answer is 'we don't know for sure'. But that said, the Deepmind article about AlphaFold 2, which won the CASP 14 competition, says this:

A folded protein can be thought of as a “spatial graph”, where residues are the nodes and edges connect the residues in close proximity. This graph is important for understanding the physical interactions within proteins, as well as their evolutionary history. For the latest version of AlphaFold, used at CASP14, we created an attention-based neural network system, trained end-to-end, that attempts to interpret the structure of this graph, while reasoning over the implicit graph that it’s building. It uses evolutionarily related sequences, multiple sequence alignment (MSA), and a representation of amino acid residue pairs to refine this graph.

A reasonable assumption that can be formed from this paragraph is that it is using some kind of a Transformer-based architecture. Yannic Kilcher's review video on AlphaFold discusses this assumption, along with a detailed explanation of the previous version of AlphaFold.
The source code for this previous version of AlphaFold, the one that was used in CASP 13, seems to have been open-sourced by Deepmind in their Github repository:
https://github.com/deepmind/deepmind-research/tree/master/alphafold_casp13


text:
Can the quality of randomness in neural network initialization affect model fitting?
This is a topic I have been arguing about for some time now with my colleagues, maybe you could also voice your opinion about it.
Artificial neural networks use random weight initialization within a certain value range. These random parameters are derived from a pseudorandom number generator (Gaussian etc.) and they have been sufficient so far.
With a proper sample simple, pseudorandom numbers can be statistically tested that they are in fact not true random numbers. With a huge neural network like GPT-3 with roughly 175 billion trainable parameters, I guess that if you would use the same statistical testing on the initial weights of GPT-3 you would also get a clear result that these parameters are pseudorandom.
With a model of this size, could in theory at least the repeatable structures of initial weights caused by their pseudorandomness affect the model fitting procedure in a way that the completed model would be affected (generalization or performance-wise)? In other words, could the quality of randomness affect the fitting of huge neural networks?

I think theoretically yes, using a TRNG instead of a pseudo one obviously yields better values in anything that relies on randoms (if speed is not a concern).
However, I think there's a lot going on when training a NN for this to make a big impact. One aspect to consider is that the range of the initial random weights might be 'smoothed out' with initialization techniques such as Xavier (sigmoid, tahn) and He (ReLU).
Another important thing to consider is that randomness is not present only in weight initialization (source):

Randomness being present throughout the entire training process, and not only on weight initialization, should reduce the concern of not having true random numbers from the get go. Also, while using a hardware-based random number generator seems feasible to initialize weights, I think it would be too much of a costly I/O operation to happen in the middle of training.