text:
How to generalise over multiple simultaneous dependent actions in Reinforcement Learning
I am trying to build an RL agent to price paid-for-seating on commercial flights. I should reiterate here - I am not talking about the price of the ticket - rather, I am talking about the pricing you see if you click on the seat map to choose where on the plane you sit (exits rows, window seats, etc). The general set up is:

After choosing their flights (for a booking of n people), a customer will view a web page with the available seat types and their prices visible.
They select between zero and n seats from a seat map with a variety of different prices for different seats, to be added to their booking.
The revenue from step 2 is observed as the reward.

Each 'episode' is the selling cycle of one flight. Whether the customer buys a chosen seat or not, the inventory goes down as they still have a ticket for the flight so will get a seat at departure. I would like to change prices on the fly, rather than fix a set of optimal prices throughout the selling cycle.
I have not decided on a general architecture yet. I want to take various booking, flight, and inventory information into account, so I know I will be using function approximation (most likely a neural net) to generalise over the state space.
However, I am less clear on how to set up my action space. I imagine an action would amount to a vector with a price for each different seat type (window seat, exit row, etc). If I have, for example, 8 different seat types, and 10 different price points for each, this gives me a total of 10^8 different actions, many of which will be very similar. In a sense, each action is comprised of a combination of sub-actions - the action of pricing each seat type.
Additionally, each sub-action (pricing one seat type) is somewhat dependent on the others, in the sense that the price of one seat type will likely affect the demand (and hence reward contribution) for another. For example, if you set window seats to a very cheap price, people will be less likely to spend a normal amount for the other seat types. Hence, I doubt the problem can be decomposed into a set of sub-problems.
I'm interested if there has been any research into dealing with a problem like this. Clearly any agent I build needs some way to generalise across actions to some degree, since collecting real data on millions of actions is not possible, even just for one state.
As I see it, this comes down to three questions:

Is it possible to get an agent that can deal with a set of actions (prices) as a single decision?
Is it possible to get this agent to understand actions in relative terms? Say for example, one set of potential prices is [10, 12, 20], for middle seats, aisle seats, and window seats. Can I get my agent to realise that there is a natural ordering there, and that the first two pricing actions are more similar to each other than to the third possible action?
Further to this, is it possible to generalise from this set of actions - could an agent be set up to understand that the set of prices [10, 13, 20] is very similar to the first set?

I haven't been able to find any literature on this, especially relating to the second question - any help would be much appreciated!

I may not have been clear about the seat price combinations. I need to set 8 different prices, each which have 10 different price points they could take, making 10^8 combinations. To take a smaller example, if we have seat types A, B and C, that can take value 10 or 15, we get combinations: 10, 10, 10 | 10, 10, 15 | 10, 15, 10 | 10, 15, 15 | 15, 10, 10 | 15, 15, 10 | 15, 10, 15 | 15, 15, 15. This is eight combinations, 2^3 - the number of price points to the power of the number of seat types. I hope this clears up the confusion. Thanks.

If you want to treat the problem as a full Reinforcement Learning problem, I'd recommend to try avoiding the combinatorial explosion of the action space by treating every sub-action as a separate decision point, a separate full action. If you have, for example, already selected 4 sub-actions for a particular customer, you can try to include those in some way in the state representation / input when moving on to the 5th sub-action. By including already-selected sub-actions in the state space, your algorithm can learn to take into account that optimal prices for some seat types will depend on what prices were already selected for others.
I do suspect such a full RL formulation will still be a difficult problem to learn though, will require huge amounts of experience. It may be worth considering to simplify it anyway, treat it as a Contextual + Combinatorial Multi-Armed Bandit Problem. That way you won't be able to learn long-term effects across multiple different customers as you described in the comments, but you will likely at least be able to learn something that works decently well with less experience. Recently, an interesting new book appeared on MAB problems, which is available for free here. You will find many chapters on Contextual MABs there, and also one chapter (chapter 30) on Combinatorial MABs.
Note that with either of these "combinatorial" approaches, you can also try to play around with the order in which you select sub-actions. For example, the sub-action / price point selected for seat type A might have a significant influence on what the optimal remaining policy would be for other seat types, whereas the sub-action for seat type B might have no influence on other seat types. It would then be useful to always prioritize selecting sub-actions for seat type A. You can try to identify these kinds of effects by keeping track of (co)variances in observed returns.

Is it possible to get an agent that can deal with a set of actions (prices) as a single decision?

The solutions I proposed above do not do this explicitly, they circumvent the issue by taking multiple sequential decisions. There does appear to be some research in RL with vector-valued actions. For example, the paper Clipped Action Policy Gradient briefly mentions vector-valued actions in Subsection 3.2. I am not personally familiar enough with RL + vector-valued actions to make a direct recommendation as to what approaches do or don't work well, but maybe this can at least help you find more relevant literature if this is a direction you'd like to pursue.

Is it possible to get this agent to understand actions in relative terms? Say for example, one set of potential prices is [10, 12, 20], for middle seats, aisle seats, and window seats. Can I get my agent to realise that there is a natural ordering there, and that the first two pricing actions are more similar to each other than to the third possible action?
Further to this, is it possible to generalise from this set of actions - could an agent be set up to understand that the set of prices [10, 13, 20] is very similar to the first set?

This kind of generalization should again come naturally from using function approximation with either of the solutions proposed above.

Hi Dennis thanks a lot for your answer. I don't think I was clear about the prices, hopefully my comment in reply to your earlier one on the question clears up what I mean about the combinations. I've also edited the question a bit since I don't think I properly explained that I am looking to change prices dynamically over the selling cycle of the flight (and hence why I think a full MDP makes sense). I did consider a bandit type formulation, and indeed I think that would work best if there were not inventory constraints.

However, the action of selling seat reservations does affect the future state, since it reduces seat inventory that can be sold/reserved in future booking events. For example, I might be able to sell exit row reservations for a higher price later in the booking process since business customers start booking. Because of this I might decide to price them high early in the window, to discourage demand at a price lower than I believe I can get later. This may not give me the best expected reward from this one action, but may end up being optimal over the life of the flight.

@domdomdom Ah yes you're right, it seems like I was sleeping. Just edited the answer.

Thanks for the edit Dennis, it's been very useful. Was the answer supposed to contain a link to that book about MABs? If so it doesn't seem to work, would be great if you could link it in the comments, thanks!

@domdomdom Yes, the link was broken because it didn't start with http://, fixed now. I'll try to find time later today to edit in a bit more of a complete answer, addressing again the specific questions you asked about generalization.

The idea of using a bandit formulation is especially interesting, thanks for your help.


text:
Why not go another layer deeper with Auto-AutoML?
So I'm finding AutoML to be pretty interesting but I'm still learning how it all works. I've played with the incredibly broken AutoKeras and got some decent results.
The question is, if you are using a NN to optimize the architecture of another network, why not take it another layer deeper and use another network to find the optimum architecture for your Parent network with a grand-parent network?
The problem doesn't necessarily need to expand exponentially as the grand-parent network could do few-shot training sessions on the parent network which itself is doing few or one-shot training.

Logically it is possible, but you will just end up complicating the entire task.
The aim of AutoML is to provide a drop in solution to the customers. To do this, a trained network decides and generates the model architecture. This is done so that anyone with basic experience is able to integrate the solution into their systems.
Currently, the complicated architectures and networks require experienced data scientist to build, train and deploy. To overcome this bottleneck and make ML accessible to all, AutoML is being developed.
So adding another grand-parent network to optimize the autoML network will just complicate the task in terms to computation time and hyper parameter optimization.
In case we decide to add another network, now the researchers must look at this network and tune it in regards to the inner network and the model both. This means more work and no direct way to understand how the hyper parameters are affecting the final results.

My slight retort that AutoML is or will be more than that in a very short amount of time if not already. I'd argue its not simply a tool for those "lowly code monkeys" who don't know ML. I say that because Googles AutoML has already beat ML experts in a Kaggle competition. And according to some studies published this last year or so AutoML is able to beat industry experts at a number of ML designs.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How does reinforcement learning handle measured disturbances?
I recently encountered an interesting problem and was wondering how RL would solve it.  The objective of the problem is to maximize the coffee quality, given by box X.  The coffee quality objective function is defined by the company.
To maximize the quality of the coffee, we can perform 2 actions:

change stirring speed of the coffee machine
change the temperature of the coffee machine

Now to the tricky part.  The coffee bean characteristic coming into the coffee machine is random.  We can measure its characteristics before sending them to the coffee machine, but we cannot change them.
I formulated this problem into a control problem, such that $X$ is a function of my previous states, $X(t - k)$, the control input , $U$, and the measured disturbance, $D$. Given a constant $D$, the problem is trival to solve because the disturbance is a constant part of the environment.  However, during times when $D$ changes rapidly, the policy is no longer optimal.
How do I inject the information of the measured disturbance into my RL agent?

Use of reinforcement learning makes most sense over LSTM if control parameters can change during brewing to optimize results, which is not standard brewing policy in low end brewers but more precise control of brewing may produce superior results, especially with regard to temperature control.
Policy does not necessarily need to be predisposed to brewing, if the search is guided by a deep artificial network (DNN), with parameters $\theta$ randomly initialized and defined as follows.
$$ (p, v) = f_{\theta}(s) $$
The brewing process parameters comprise state $s$. The vector of control change probabilities $p$ have components $p_a = \Pr(a|s)$ for each action $a$. A scalar $v$ estimates the value of the expected outcome $o$ from control state $s$, where $v \approx \mathbb{E} [o|s].
Rather than characterize the variations in brewing stock as disturbance $D$, it is more consistent with most policy learning approaches to consider the measured properties of brewing stock as features $\vec{b}$ and feed a normalized signal $\mathcal{N}(b)$ into additional inputs to the deep network. This way, sensitivity to variations in $\vec{b}$ is learned as a critical factor in brewing policy.
In such a case, a vanilla MCTS (Monte-Carlo tree search algorithm) can be employed. Using a deep network removes the need to extract domain specific knowledge or discover heuristics to effectively guide control change policy.
Control values must be made discrete, based on granularity that fully (but not excessively) covers brewing possibilities. The control state probabilities that guide the search can be learned using coffee drinker feedback collected in any number of ways, depending the relative costs of each form of outcome data acquisition.

A focus group
A group of independent testers
Repurchase statistics at a selected set of coffee shops
Feedback controls at a set of offices that provide employees with coffee

For the comprehensiveness in control response to variations in brewing stock properties, the distribution of component values in $\vec{b}$ and their statistical correlation over time may need to be controlled during the training process such that trends and distributions resemble likely real world variations in brewing stock.
Each search during the training process is guided by MCTS control changes, $a_t ∼ \pi_t$. The final state at the end of training is $s_T$, rated using a fixed scale of taster feedback. Selection of each state $s$ can be based on a control operation search advantage based on three characteristics.

Low past experience (a low count of previous visits in the search)
High probability $p$
High average of values $\{v, \; ...\}$ over the leaf states of brewing trials that selected $a$ from $s$ according to the converging neural network parameters $f_{\theta}$

The output of the search is $\vec{\pi}$ representing a probability distribution over control changes. The deep network parameters $\theta$ converge via gradient descent using a loss function $\ell$.
$$ \ell = {(o − v)}^2 − {\pi}_T \, \ln{p} + c \, {||\theta||}^2 $$
Loss is thus an aggregation of two optimality criteria.

Mean square estimated disparity between the predicted outcome $v_t$t and the taste outcome $o$
Cross-entropy of policy vector $p_t$ and search probabilities ${\pi}_t$
Hyper-parameter $c$ weighted L2 regularization

The updated parameters are used in subsequent brew and taste trials. All control operations must be included in each trial. Success using this approach has been obtained using either greedy or proportionality with respect to the root visit counts.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why aren't neural networks contractions?
I'm not sure I understand why neural networks aren't considered contractions, as Geoffrey J. Gordon says in his paper: Stable Function Approximation in Dynamic Programming:

"Our theorems in the following sections will be based on two views of function approximators.
First, we will cast function approximators as expansion or contraction mappings; this distinction captures the essential difference between approximators that can exaggerate changes in their training values, like linear regression and neural nets, and those like k-nearest-neighbor that respond conservatively to changes in their inputs.
Second, we will show that approximate temporal difference learning with some function approximators is equivalent to exact temporal difference learning for a slightly different problem.
...
Definition: A function $f$ from a vector space $S$ to itself is a contraction mapping if, for all points $a$ and $b$ in $S$, $|| f (a) - f(b) || \le \gamma || a-b ||.$ Here $\gamma$, the contraction factor or modulus, is any real number in $[0,1]$. If we merely have $|| f (a) - f(b) || \le || a-b ||$, we call $f$ a nonexpansion.
$\qquad$ For example, the function $f(x) = 5 + \frac{x}{2}$ is a contraction with contraction factor $\frac{1}{2}$. The identity function is a nonexpansion. All contractions are nonexpansions.

I understand that at least in practice they've been shown to greatly exaggerate differences between similar target functions, but I suppose there's a theoretical explanation for this?

Can you please define what you mean by "contraction" in this case?

why neural networks aren't considered contractions, as Geoffrey J. Gordon says in his paper.

I am not sure how strongly you mean aren't considered, if you mean it in a strong sense or weak sense.
Contraction theory is probably not very well known in the AI community, but I did find some material to suggest contraction theory is being applied to NNs.
Perspectives on Contraction Theory and Neural Networks, Bullo 2022
Learning-based Adaptive Control using Contraction Theory, Tsukamoto, 2021

The use of DNNs permits real-time
implementation of the control law and broad applicability to a
variety of nonlinear systems with parametric and nonparametric
uncertainties. We show using contraction theory that the aNCM
ensures exponential boundedness of the distance between the
target and controlled trajectories in the presence of parametric
uncertainties of the model, learning errors caused by aNCM
approximation, and external disturbances.

In the weak sense specific NNs can be considered contractions, but in general contractions must be uniformly continuous and it's not clear that in the strong universally true sense for all cases of NNs. There may be some Networks, that for some set of inputs may reveal some discontinuity.
Also function f must be a mapping of S to itself.
In image classification we are not mapping S -> S but instead are mapping R^MxNxC -> R^n  where n is the number of classes (or 1 when using integer encoding)
Its also not clear that all networks would satisfy the condition:

|| f(a) - f(b) ||
---------------   <= k < 1
|| a - b ||

Suppose we had a vector a of size n of all 1's [1,1,1,...] and a vector b of all zeros of size n [0,0,0,...] then their norm || a - b || = sqrt(n)
a neural network would need to map

a -> a'= [u,u,u,...]
b -> b'= [v,v,v,...] such that:
|| a'-b' ||/sqrt(n) = sqrt(n*(u-v)) = sqrt(n)*sqrt(u-v) < 1 for all n

So to summarize I would say:

It depends on how strongly you mean aren't considered
The AI community at large is not familiar with what a contraction mapping is so the connection isn't well developed yet.
It may be true on case by case basis, but not universally true.

Please, don't provide screenshots. Copy and paste the actual relevant text (and provide a reference to the original).


text:
When exactly does the split into different heads in Multi-Head-Attention occur?
I am confused by the Multi-Head part of the Multi-Head-Attention used in Transformers. My question concerns the implementations in Pytorch of nn.MultiheadAttention and its forward method multi_head_attention_forward and whether these are actually identical to the paper. Unfortunately, I have been unable to follow along the original code of the paper. So I could not check whether the implementations in Pytorch are acutally identical to the paper.
Please forgive the excessive use of illustrations. However, I hope it will improve understanding my problem.
What is the correct order for calculating the Queries Q, Keys K and Values V and splitting the operation into the individual Attention-Heads? Unfortunately most explanations I found online while helpful for understanding the general principle and intuition of Multi-Head-Attention did not go into the details of the implementation.
In the original paper Attention is all you need Multi-Head-Attention is explained as followed:

First, according to my current understanding, if we have a sequence of vectors with 512-dimensions (like in the original Transformer) and we have $h=8$ Attention-Heads (again like the original), every Attention-Head attends to $512/8=64$ entries of the input vector used to calculate the Attention in the corresponding head. So the first Attention-Head attends to the first 64 entries, the second to the second 64 entries and so on. However, if the split is conducted before calculating Q,K,V this would refer to the first 64 entries of X (this does not seem match the explanation in the paper I believe) while in the other case it would refer to the first 64 entries of Q,K,V.
In the text they say "project the queries, keys and values h times with different, learned linear projections to $d_k,d_k$ and $d_v$ dimensions and since they set $d_k=d_v=d_{model}/h=512/8=64$. Therefore, if we actually have single matrices for every Attention-Head h we would have
$W^Q_i,W^K_i,W^V_i \in \mathbb{R}^{512x64} \forall i \in h$.
This matches the illustration found here https://jalammar.github.io/illustrated-transformer/

It is explained that the input X is transformed into the Queries, Keys and Values for the different attention heads by using different projection matrices which are learned during training.
This seems to indicate that the split into the individual Attention-Heads is conducted after the calculation of $Q,K,V$ (or rather during the calculation). Since we have $h=8$ this leads in sum to $3*8*512*64=3*512*512$ learnable parameters in total (if we ignore the bias). Thus as far as the overall number of parameters is concerned we would have the same number if we would instead use three big matrices which concatenate the matrices of the individual Attention-Heads.
$W^Q=[W^Q_1,W^Q_2,...,W^Q_h] \in \mathbb{R}^{512x512}$
$W^K=[W^K_1,W^K_2,...,W^K_h] \in \mathbb{R}^{512x512}$
$W^Q=[W^V_1,W^V_2,...,W^V_h] \in \mathbb{R}^{512x512}$
In the explanation from the same author of GPT-2 (this model has an embedding dimension of 768 and 12 Attention-Heads, instead of 512 and 8 like the original Transformer) found here https://jalammar.github.io/illustrated-gpt2/#part-2-illustrated-self-attention

the Queries Q, Keys K and Values V are first calculated by multiplying the input with one big matrix which is the concatenation of $W^Q,W^K,W^V$. If the input for calculating Q, K and V is identical (which is the case for self-attention), it is clear to me that you can use $W_{concat}=[W^Q,W^K,W^V]$ and obtain $[Q,K,V]$, since you essentially still multiply the input with each weight matrix separately.
Then you can split the result to again obtain $Q,K,V$ as individual matrices (The image displays $q_9,k_9,v_9$ as an example, not the complete matrices). Then $q_9,k_9,v_9$ are again split into 12 vectors which results in a matrix of dimension $(12x64)$.
So overall here we did not use individual matrices per Attention-Head but only one larger matrix.
Is this method mathematically identical to the one using individual smaller matrices per Attention-Head?
It appears that this is the way the calculation is implented in Pytorch, if $d_{model}=kdim=vdim$ Though here unlike in the paper which used $d_k$ and $d_v$ as names, $kdim$ and $vdim$ refer to the dimension of all Attention-Heads summed up, e.g. $kdim=d_k*num_{heads}$(=512 for the original Transformer).
So  In the documentation of nn.modules.MultiheadAttention the model either creates three separate projection matrices to generate the Queries, Keys and Values or one big matrix (if the dimensions are identical). The following is part of the _init_ function.
if self._qkv_same_embed_dim is False:
            self.q_proj_weight = Parameter(torch.empty((embed_dim, embed_dim), **factory_kwargs))
            self.k_proj_weight = Parameter(torch.empty((embed_dim, self.kdim), **factory_kwargs))
            self.v_proj_weight = Parameter(torch.empty((embed_dim, self.vdim), **factory_kwargs))
            self.register_parameter('in_proj_weight', None)
        else:
            self.in_proj_weight = Parameter(torch.empty((3 * embed_dim, embed_dim), **factory_kwargs))
            self.register_parameter('q_proj_weight', None)
            self.register_parameter('k_proj_weight', None)
            self.register_parameter('v_proj_weight', None)

If we stay in the standard case of _qkv_same_embed_dim=True the input is passed through a nn.linear as part of _in_projection_packed which is using self.in_proj_weight as the weight
if not use_separate_proj_weight:
        assert in_proj_weight is not None, "use_separate_proj_weight is False but in_proj_weight is None"
        q, k, v = _in_projection_packed(query, key, value, in_proj_weight, in_proj_bias)
    else:
        assert q_proj_weight is not None, "use_separate_proj_weight is True but q_proj_weight is None"
        assert k_proj_weight is not None, "use_separate_proj_weight is True but k_proj_weight is None"
        assert v_proj_weight is not None, "use_separate_proj_weight is True but v_proj_weight is None"
        if in_proj_bias is None:
            b_q = b_k = b_v = None
        else:
            b_q, b_k, b_v = in_proj_bias.chunk(3)
        q, k, v = _in_projection(query, key, value, q_proj_weight, k_proj_weight, v_proj_weight, b_q, b_k, b_v)

def _in_projection_packed(
    q: Tensor,
    k: Tensor,
    v: Tensor,
    w: Tensor,
    b: Optional[Tensor] = None,
) -> List[Tensor]:
    r"""
    Performs the in-projection step of the attention operation, using packed weights.
    Output is a triple containing projection tensors for query, key and value.
    Args:
        q, k, v: query, key and value tensors to be projected. For self-attention,
            these are typically the same tensor; for encoder-decoder attention,
            k and v are typically the same tensor. (We take advantage of these
            identities for performance if they are present.) Regardless, q, k and v
            must share a common embedding dimension; otherwise their shapes may vary.
        w: projection weights for q, k and v, packed into a single tensor. Weights
            are packed along dimension 0, in q, k, v order.
        b: optional projection biases for q, k and v, packed into a single tensor
            in q, k, v order.
    Shape:
        Inputs:
        - q: :math:`(..., E)` where E is the embedding dimension
        - k: :math:`(..., E)` where E is the embedding dimension
        - v: :math:`(..., E)` where E is the embedding dimension
        - w: :math:`(E * 3, E)` where E is the embedding dimension
        - b: :math:`E * 3` where E is the embedding dimension
        Output:
        - in output list :math:`[q', k', v']`, each output tensor will have the
            same shape as the corresponding input tensor.
    """
    E = q.size(-1)
    if k is v:
        if q is k:
            # self-attention
            return linear(q, w, b).chunk(3, dim=-1)
        else:
            # encoder-decoder attention
            w_q, w_kv = w.split([E, E * 2])
            if b is None:
                b_q = b_kv = None
            else:
                b_q, b_kv = b.split([E, E * 2])
            return (linear(q, w_q, b_q),) + linear(k, w_kv, b_kv).chunk(2, dim=-1)
    else:
        w_q, w_k, w_v = w.chunk(3)
        if b is None:
            b_q = b_k = b_v = None
        else:
            b_q, b_k, b_v = b.chunk(3)
        return linear(q, w_q, b_q), linear(k, w_k, b_k), linear(v, w_v, b_v)

def _in_projection(
    q: Tensor,
    k: Tensor,
    v: Tensor,
    w_q: Tensor,
    w_k: Tensor,
    w_v: Tensor,
    b_q: Optional[Tensor] = None,
    b_k: Optional[Tensor] = None,
    b_v: Optional[Tensor] = None,
) -> Tuple[Tensor, Tensor, Tensor]:
    r"""
    Performs the in-projection step of the attention operation. This is simply
    a triple of linear projections, with shape constraints on the weights which
    ensure embedding dimension uniformity in the projected outputs.
    Output is a triple containing projection tensors for query, key and value.
    Args:
        q, k, v: query, key and value tensors to be projected.
        w_q, w_k, w_v: weights for q, k and v, respectively.
        b_q, b_k, b_v: optional biases for q, k and v, respectively.
    Shape:
        Inputs:
        - q: :math:`(Qdims..., Eq)` where Eq is the query embedding dimension and Qdims are any
            number of leading dimensions.
        - k: :math:`(Kdims..., Ek)` where Ek is the key embedding dimension and Kdims are any
            number of leading dimensions.
        - v: :math:`(Vdims..., Ev)` where Ev is the value embedding dimension and Vdims are any
            number of leading dimensions.
        - w_q: :math:`(Eq, Eq)`
        - w_k: :math:`(Eq, Ek)`
        - w_v: :math:`(Eq, Ev)`
        - b_q: :math:`(Eq)`
        - b_k: :math:`(Eq)`
        - b_v: :math:`(Eq)`
        Output: in output triple :math:`(q', k', v')`,
         - q': :math:`[Qdims..., Eq]`
         - k': :math:`[Kdims..., Eq]`
         - v': :math:`[Vdims..., Eq]`
    """
    Eq, Ek, Ev = q.size(-1), k.size(-1), v.size(-1)
    assert w_q.shape == (Eq, Eq), f"expecting query weights shape of {(Eq, Eq)}, but got {w_q.shape}"
    assert w_k.shape == (Eq, Ek), f"expecting key weights shape of {(Eq, Ek)}, but got {w_k.shape}"
    assert w_v.shape == (Eq, Ev), f"expecting value weights shape of {(Eq, Ev)}, but got {w_v.shape}"
    assert b_q is None or b_q.shape == (Eq,), f"expecting query bias shape of {(Eq,)}, but got {b_q.shape}"
    assert b_k is None or b_k.shape == (Eq,), f"expecting key bias shape of {(Eq,)}, but got {b_k.shape}"
    assert b_v is None or b_v.shape == (Eq,), f"expecting value bias shape of {(Eq,)}, but got {b_v.shape}"
    return linear(q, w_q, b_q), linear(k, w_k, b_k), linear(v, w_v, b_v)

def linear(
    input: Tensor, weight: Tensor, bias: Optional[Tensor] = None,
    scale: Optional[float] = None, zero_point: Optional[int] = None
) -> Tensor:
    r"""
    Applies a linear transformation to the incoming quantized data:
    :math:`y = xA^T + b`.
    See :class:`~torch.nn.quantized.Linear`
    .. note::
      Current implementation packs weights on every call, which has penalty on performance.
      If you want to avoid the overhead, use :class:`~torch.nn.quantized.Linear`.
    Args:
      input (Tensor): Quantized input of type `torch.quint8`
      weight (Tensor): Quantized weight of type `torch.qint8`
      bias (Tensor): None or fp32 bias of type `torch.float`
      scale (double): output scale. If None, derived from the input scale
      zero_point (long): output zero point. If None, derived from the input zero_point
    Shape:
        - Input: :math:`(N, *, in\_features)` where `*` means any number of
          additional dimensions
        - Weight: :math:`(out\_features, in\_features)`
        - Bias: :math:`(out\_features)`
        - Output: :math:`(N, *, out\_features)`
    """
    if scale is None:
        scale = input.q_scale()
    if zero_point is None:
        zero_point = input.q_zero_point()
    _packed_params = torch.ops.quantized.linear_prepack(weight, bias)
    return torch.ops.quantized.linear(input, _packed_params, scale, zero_point)

Then later the Queries, Keys and Values are split up into the individual Attention-Heads:
    #
    # reshape q, k, v for multihead attention and make em batch first
    #
    q = q.contiguous().view(tgt_len, bsz * num_heads, head_dim).transpose(0, 1)
    if static_k is None:
        k = k.contiguous().view(k.shape[0], bsz * num_heads, head_dim).transpose(0, 1)
    else:
        # TODO finish disentangling control flow so we don't do in-projections when statics are passed
        assert static_k.size(0) == bsz * num_heads, \
            f"expecting static_k.size(0) of {bsz * num_heads}, but got {static_k.size(0)}"
        assert static_k.size(2) == head_dim, \
            f"expecting static_k.size(2) of {head_dim}, but got {static_k.size(2)}"
        k = static_k
    if static_v is None:
        v = v.contiguous().view(v.shape[0], bsz * num_heads, head_dim).transpose(0, 1)
    else:
        # TODO finish disentangling control flow so we don't do in-projections when statics are passed
        assert static_v.size(0) == bsz * num_heads, \
            f"expecting static_v.size(0) of {bsz * num_heads}, but got {static_v.size(0)}"
        assert static_v.size(2) == head_dim, \
            f"expecting static_v.size(2) of {head_dim}, but got {static_v.size(2)}"
        v = static_v

Therefore, it appears to me that both ways of calculations should be equal and the one using the bigger matrices is just more efficient to compute. Is this correct?
In this case, I ask myself why nn.MultiheadAttention requires that embed_dim is divisible by num_heads, since the split into the individual Attention-Heads is actually conducted after generating $Q,K,V$. Should then not $d_q,d_k,d_v$ be made to be divisible by num_heads? Since these dimensions do not have to be equal to the dimension of the inputs?
Thanks for any advice in advance!

The queries, keys and values are calculated then chunked so that each chunk depends on (is a linear combination of) all values of the input embedding.
As for understanding an implementation, I didn't bother with pytorch but instead understood this code http://einops.rocks/pytorch-examples.html although there are differences as I understand that pytorch expects input in the form (L, N, C) where L=words, N=batch, C=embedding for performance reasons.
Transformer's attention needs more attention
class MultiHeadAttentionNew(nn.Module):
    def __init__(self, n_head, d_model, d_k, d_v, dropout=0.1):
        super().__init__()
        self.n_head = n_head

        self.w_qs = nn.Linear(d_model, n_head * d_k)
        self.w_ks = nn.Linear(d_model, n_head * d_k)
        self.w_vs = nn.Linear(d_model, n_head * d_v)

        nn.init.normal_(self.w_qs.weight, mean=0, std=np.sqrt(2.0 / (d_model + d_k)))
        nn.init.normal_(self.w_ks.weight, mean=0, std=np.sqrt(2.0 / (d_model + d_k)))
        nn.init.normal_(self.w_vs.weight, mean=0, std=np.sqrt(2.0 / (d_model + d_v)))

        self.fc = nn.Linear(n_head * d_v, d_model)
        nn.init.xavier_normal_(self.fc.weight)
        self.dropout = nn.Dropout(p=dropout)
        self.layer_norm = nn.LayerNorm(d_model)

    def forward(self, q, k, v, mask=None):
        residual = q
        q = rearrange(self.w_qs(q), 'b l (head k) -> head b l k', head=self.n_head)
        k = rearrange(self.w_ks(k), 'b t (head k) -> head b t k', head=self.n_head)
        v = rearrange(self.w_vs(v), 'b t (head v) -> head b t v', head=self.n_head)
        attn = torch.einsum('hblk,hbtk->hblt', [q, k]) / np.sqrt(q.shape[-1])
        if mask is not None:
            attn = attn.masked_fill(mask[None], -np.inf)
        attn = torch.softmax(attn, dim=3)
        output = torch.einsum('hblt,hbtv->hblv', [attn, v])
        output = rearrange(output, 'head b l v -> b l (head v)')
        output = self.dropout(self.fc(output))
        output = self.layer_norm(output + residual)

Understand MultiHeadAttentionNew first then understand MultiHeadAttentionOld, and then maybe look at the pytorch code.

Thank you for the link! It is really helpful. While I do not understand the details of some of the einops functions, it is clear enough. Since also bigger matrices are used in MultiHeadAttentionNew it really seems that you can just use bigger matrices and it gets you the same result.
I'll try to write it down mathematically.

@Aushilfsgott The design of multi-head attention is for performance rather than having multiple single head attentions with the same input. Also I should have mentioned in my answer its worth really understanding einops and einsum, rather using the older pytorch functions

Thanks again Tom. I got the basic principle of einsum so it is fine! The link is definitely very good for understanding the function.


text:
Survey on non-machine learning object detection algorithms
I am working on a project in which I will be performing object detection on deformed objects. Unfortunately, there isn't enough data sets to train them on some neural network. I am looking for reference on computer vision algorithms that does not require learning or training on datasets. Rather, topics arising from "structure from motion" that are popular within the computer vision community.
Some note: The object detection will take place in an open environment rather close environment.

"... I am looking for reference on computer vision algorithms that does not require learning or training on datasets. Rather, topics arising from "structure from motion" ...".

Here is a list of papers, blogs, and code that you might find helpful:

The website PapersWithCode's search engine returns several hits for "NRSfM", a few of the projects specifically mention limited datasets:

"Deep Interpretable Non-Rigid Structure from Motion", (Feb 28 2019), by Chen Kong and Simon Lucey. GitHub.

"Once the network's weights are estimated (for a non-rigid object) we show how our approach can effectively recover 3D shape from a single image -- outperforming comparable methods that rely on direct 3D supervision.".

"Local Non-Rigid Structure-From-Motion From Diffeomorphic Mappings",
(CVPR 2020), by Shaifali Parashar, Mathieu Salzmann, and Pascal Fua. GitHub.

"We show that it is nevertheless sufficient to establish local correspondences between the surface in two different images and therefore to perform point-wise reconstruction using only first-order derivatives. ".

"MHR-Net: Multiple-Hypothesis Reconstruction of Non-Rigid Shapes from 2D Views", (July 19, 2022), by Haitian Zeng, Xin Yu, Jiaxu Miao, and Yi Yang. (No code implementations yet).

"We propose MHR-Net, a novel method for recovering Non-Rigid Shapes from Motion (NRSfM). MHR-Net aims to find a set of reasonable reconstructions for a 2D view, and it also selects the most likely reconstruction from the set. To deal with the challenging unsupervised generation of non-rigid shapes, we develop a new Deterministic Basis and Stochastic Deformation scheme in MHR-Net.".

arXiv's search engine returns 31 hits for "NRSfM", a few of the papers are:

The arXiv paper "Organic Priors in Non-Rigid Structure from Motion", (July 16, 2022), by Suryansh Kumar and Luc Van Gool, advocates the use of organic priors in classical non-rigid structure from motion (NRSfM). They outline how to effectively recover motion and non-rigid 3D shape using the proposed organic prior based approach and demonstrate results that outperform prior-free NRSfM performance by a significant margin.
A quote from Suryansh Kumar's blog "Non-Rigid Structure from Motion":

"To date, no algorithm can solve NRSfM for all kinds of conceivable motion. Consequently, additional constraints, priors, and assumptions are often employed to solve NRSfM.".

His GitHub contains some NRSfM code.

"A Closed-Form Uncertainty Propagation in Non-Rigid Structure from Motion", (May 10, 2020), by Jingwei Song, Mitesh Patel, Ashkan Jasour, and Maani Ghaffari. YouTube Video, GitHub.

"Deep NRSfM++: Towards Unsupervised 2D-3D Lifting in the Wild", (Jan 27, 2020), by Chaoyang Wang, Chen-Hsuan Lin, Simon Lucey. GitHub without code, GitHub with code.

"Deep NRSfM++, achieves state-of-the-art performance across numerous large-scale benchmarks, outperforming both classical and learning-based 2D-3D lifting methods.".

Some other papers are:

In the arXiv paper: "DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image" (Aug 11, 2017), by Andrey Kurenkov, Jingwei Ji, Animesh Garg, Viraj Mehta, JunYoung Gwak, Christopher Choy, Silvio Savarese, they promise to make code available later.

The website x-mol's search engine returns 65 hits for "NRSfM". That site doesn't clearly present which papers have code available and which do not, so there's a lot to dig through.
Adding the word "code" to the search balloons the number of search results. On the arXiv website adding the extra word greatly reduces the number of returned results; that gives more focus but may also miss some helpful papers.
For example the paper "A Benchmark and Evaluation of Non-Rigid Structure from Motion", (Dec 29, 2020), by Sebastian Hoppe Nesgaard Jensen, Mads Emil Brix Doest, Henrik Aanæs & Alessio Del Bue, only offers psuedo-code.


text:
MobileNetV2 - Some particularities
So I was studying MobileNetV2 architecture and came across this table from the original paper that represents its architecture:

Table Description: "Table 2: MobileNetV2 : Each line describes a sequence of 1 or more identical (modulo stride) layers, repeated n times. All layers in the same sequence have the same number c of output channels. The first layer of each sequence has a stride s and all others use stride 1. All spatial convolutions use 3 × 3 kernels. The expansion factor t is always applied to the input size as described in Table 1."

However I don't really understand why the last pointwise convolution (conv2d 1x1) is applied after the average pooling.
Shouldn't the output from the avgpool go to a dense layer so that it can then perform its predictions?
What is the point of performing convolution, more precisely pointwise conv. after pooling? Also, why is k not a fixed number of kernels/filters, like the other layers?
Thanks in advance!

This is largely a matter of semantics I think. After the avg pool layer, the feature map becomes $1\times 1\times 1280$ as indicated on the table, so when you apply a pointwise convolution (a $1 \times 1$ conv), it essentially serves as a fully-connected layer operating on the channel dimension since there are no spaital dimensions to work with anymore.
If you want, you can think of the last layer as a dense layer with weights shaped $1280 \times k$, where the $1280$ input channels are interpreted as input features to the dense layer, and where $k$ is the number of classes.

I was having a hard time understanding the purpose of that final '1x1 conv layer' but your answer really made it clear for me! Thanks for that! I guess I didn't understand the semantics behind the usage of '1x1 conv layer' instead of a simple 'dense layer'. Is this a common practice when writting papers? Thank you again!

@BlueRoss I don't know if this convention is common, but I've seen multiple conventions (e.g. 1x1 conv, dense, fc, etc.) being used to describe the classification layer of a convnet. You may want to look at other major convnet papers (e.g. ResNet, other mobilenet versions) to see what they use


text:
How to define a fitness function to make sure the best fitness value is 'close to 9' in genetic algorithm
I am learning about genetic algorithms (GA), but I encountered a question about the definition of the fitness function used in GA.
I understand that the fitness function should return a scalar value (e.g. to make sure to return the maximum value for a given individual).
For example, if one wanted to optimize a small molecule $logP$ (to ensure that the returned $logP$ has the maximum values), one can define the fitness value thusly: $J(m) = logP$, and then use GA to optimize the $logP$.
But my question is that I want to make sure that the $logP$ has a value close to 9 (rather than the larger the better), the more closer to 9 the better.
In this case how can I define the fitness function used in GA?
Thanks.

I tried to use this one: 1 / abs(logP - 9) as the fitness function. Also considering denominator is zero the corner case. If abs(logP - 9) = 0, then one can set it as very small number like 0.0000000001.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?
In Sutton and Barto (PDF, page 265), 2nd edition, Figure 13.1 applies REINFORCE to the "short corridor with switched actions" environment from Example 13.1. The figure looks like this:

My question is, why is the initial performance so poor? I believe it should be close to optimal even without any training.
My reasoning is this: If we initialize the algorithm with $\boldsymbol{\theta} = \boldsymbol{0}$, then the initial policy has
$$\pi({\tt right}|s,\boldsymbol{\theta}) = \frac{e^{\theta_1}}{e^{\theta_1} + e^{\theta_2}} = 0.5.$$
And from the figure in Example 13.1 (p. 323) --

-- we know that a policy that goes $\tt{right}$ with probability 0.5 in every state has an expected value that is very close to that of the optimal policy, around $-12$ or so. So shouldn't the policies produced by REINFORCE start close to optimal with essentially no training, and then just improve little to none after that?
My own experiment supports my hunch:

But it's possible I'm doing something wrong, both theoretically and computationally.

I'm actually working on this example too, implemented the REINFORCE algorithm, and got the same result as you. My only guess is that the authors chose a different initial $\theta$ value to show the convergence properties for different choices of $\alpha$. (For example maybe something like $\theta_0 = [-3; 0]$ so the initial probability of right action is ~0.05 and the initial expected cost is quite large.)

Yes, I think you are right. I tried setting the initial theta to $(-3, 0)$ and got a curve much more like Sutton and Barto's. The curves are nearly identical for the three values of $\alpha$, so that's another question mark, but I'll leave that one for now.

This is exactly right. In a toy example such as the above it is clear what a good starting point could be, but in eg a deep neural network this is not the case.

I guess what was confusing is that the pseudocode for the algorithm suggests initializing $\theta$ to 0, which just happens to be a good starting point for this toy problem, but the plot shows something different and there's no comment about why.


text:
When do you know that your neural network is learning something when metrics are garbage?
When training a neural network for binary classification on a highly imbalanced set its training loss decreases, however validation loss increase even though accuracy is very high (due to highly imbalanced dataset) and its other metrics (auc, recall, precision) are bad.
I want to figure out whether there is something wrong with the model architecture, therefore building a new model would be the right choice or is the model learning something; however you just need to tweak some hyperparameters.
By plotting activation outputs and weights at each layer we can see if there is a vanishing/exploding gradient or saturation at a certain layer (activation output being on extremes when using saturating activation functions like tanh, sigmoid).
My question is, is that only thing we can interpret from weights and activation output values?
What can weights distribution of layers at each layer tell us?
I'm very confused since from the DataCamp - TensorBoard Tutorial it says: “Visualizing network weights is important, because if the weights are wildly jumping here and there during learning it indicates something is wrong with the weight initialization or the learning rate.”,
however from:  Stack Overflow - Understanding TensorBoard (weight) histograms I’ve understood the answer as: "changes in weight imply learning" because the answerer looks at the first 3 layers (where their weights do not change at each iteration) and says that it is not learning, and looks at last layer (weight changes) and says that it is learning.

Running the risk of oversimplifying your problem, I still want to point out the obvious: A decreasing training loss, but increasing validation loss is bad.

You ask different questions in the title and in the body of the question, so I'll start from the title.
When do you know your neural network is learning?
The surprisingly simply answer to this question is to just TEST the model as frequently as you can during training. And testing does not mean to just compute metrics, those serve the purpose of quantifying the performance of your model, rather what you want to do is to set up a quick pipeline to apply your checkpoints to some real unseen data and check yourself if the predictions make sense. Doesn't matter if your dataset is unbalanced or not, you are in charge of selecting as many testing instances as you want to test the model, and the more edge/difficult cases you select the better.
Also, save intermediate results during training. You have access to input images and labels, so for x amount of wrong classification you can easily save those input images, or save to a csv file the index of those training instances. Once you have a small sample of false positive and negatives, you can inspect specifically those samples for patterns.
What can we interpret from weights and activation output values?
It is true that plotting the histogram of weights values per layer is a common practice, but I think you probably already realized that there is no consensus on how to go from A to B when we notice something weird happening. So I wouldn't rely to much on these tools to conclude that you need a different architecture or weight initialization. On overall standard weight initialization strategies like xavier or kaiming have been proven to work just fine. If you have troubles training a good model you'll have much more chance to understand what is going wrong by looking at the data and trying specific losses for unbalanced tasks.

Thanks for your answer! Yeah I guess I was too stuck up on model side.


text:
How to classify data into organised groups by using a resulting classification vector and a set of probabilities?
I am trying to figure out the best way to calculate the probability a sentence belongs to some category.
For simplicity sake, lets assume that the sentence is "yellow fruit".
Next, I use the an BERT classifier to get a classification result [-5,1,2] with the categories [apple, orange, banana].
The array simply tells us how strong is the signal that corresponds to a category. In this case -5 for apple, 1 for orange, and 2 for banana.
The answer to this classification is "banana" since 2 is the max number in the [-5,1,2] array, and it position corresponds to the category "banana".
Therefore, we know that "yellow fruit" is most-likely a banana.
Next, I have a table of likelihood of purchase for apples, oranges and bananas:
Apple: 80%
Orange: 10%
Banana: 50%
I need to calculate the likelihood of customer purchase for sentence "yellow fruit".
Therefore, I am given the sentence "yellow fruit", the classification vector [-5,1,2] the classification categories [apple, orange, banana] and the likelihood table for purchase [0.8,0.1,0.5]
How do I calculate the the likelihood of purchase for the sentence "yellow fruit" ?

To transform your logits [-5,1,2] into probabilities you can use the softmax function.  This would get you p = [0.000666195, 0.268762254, 0.730571551].


text:
ChatGPT $20/month vs. ChatGPT API
I am curious if there is any difference between (A) accessing ChatGPT4 via their web page and \$20/month subscription on the one hand, or (B) via an application like MacGPT with their paid API key.
I've used (B) and no matter how much I use it I don't get charged more than \$5 or maybe \$10 per month.
Does (A) have any advantages that anyone knows about?

This question is off-topic here because it seems to be about specific software/APIs. It has little or nothing to do with theoretical aspects of AI. For more info about our on-topic page, see https://ai.stackexchange.com/help/on-topic

I didn't realize that this was just for theoretical ai.

ChatGPT Plus (the $20/month offering) gives you access to GPT4. The API currently gives access to older models (from what I know). I have tried the davinci model via the API and to me it seems to be inferior to ChatGPT(v3.5), so it should be also inferior to GPT-4.
This is what MacGPT's FAQ says though:
"I added my API key and want to use GPT-4 but I get an error.
Currently the GPT-4 API is only available through the API after signing up for a waiting list. Please sign up through the OpenAI website and you should have access in a few days. Sign up here."
So if you join the whitelist and then get access to GPT-4 via API it should be the same as using ChatGPT Plus.


text:
Time taken to solve cartpole environment using DQN
I am trying to solve the cartpole environment (GitHub) using DQN agent. I have been building my own DQN agent by following a tutorial by Jon Krohn.
I am able to solve the environment with a maximum reward of 499. I am using the experience replay method, and I have given the number of episodes as 1000 episodes.
In the tutorial it takes very much less time to solve the environment, whereas for me it's taking almost 2 hours to solve it.
The library I am using for building my model is tensorflow (tf.keras).

Could you share your code?

Hi @hal9000 , i am sharing the github link here. This is the code I used.  https://github.com/the-deep-learners/TensorFlow-LiveLessons/blob/master/notebooks/cartpole_dqn.ipynb  . I just want to know how much time it takes to solve

From my own experience it takes around 5 min to get maximum reward on cartpole. 2 hours seems indeed a bit long.

Looking at your code I think it might be because you extract the mini batch and process each part separately. (there are nice ways to process the entire batch at once, making it go faster). Also I think I see you update the network that is used in the prediction directly instead of resyncing every m steps MSE(Q(state,action,param),reward+gamma*Q'(next_state,action,frozen_param)), while it might not cause instability for cartpole, you might want to delay resyncing the frozen parameters with the current parameters as that makes convergences more stable.

Thanks for your comment @hal9000 , I understand that this code needs improvement but shouldn't it be able to run in less than 10 minutes ? Can you please help me with the dependencies that you used when you tried to solve this environment along with the versions ? And also, I use anaconda, so does that contribute to this problem ? I am pretty sure it doesn't but I would like to know what you think about it. Thanks

I also use anaconda but I coded it in pytorch and I also run it on the gpu.

honestly there are a lot of things that can make the code run faster: making the algorithm more stable (dqn tends to overestimate the Q-values), having faster hardware, running things in parallel, running on the gpu, using the right choice of hyper parameters, maybe having a slightly bigger network, using vectorization, reduce the amount of prints (I use tqdm as loading bar for the epochs).

Try to avoid using pure python as it makes things realllyyy slow. Use gather like operations for selecting the Q values from the online network. Use masking instead of if statements like Q_target[dones] = 0.

Thanks @hal9000 , I will try to use whatever you have suggested. I don't think my hardware is a problem since my processor is i7-11800h with a nvidia rtx 3070. I did not run it on gpu since it did not have any images and I thought that running it on cpu would be enough. Anyways, thanks for your help.

Hi @hal9000 , the problem has been solved. Thanks for your help. Apparently, it was an issue with the versions of my libraries and it looks like the most recent versions were the problem. I downgraded them and as you said, it is done within a couple of minutes. Thanks again.

no problem, I'm glad I could help. :)

Will put this here as a summary of the solution.
To get to cartpole reward 500 shouldn't take a modern compute too long (few minutes).

If it runs longer than that it can be because of a lot of ways to speed it up (assuming the code works as it should):

The versions of the libraries can matter a lot, downgrading ended up fixing the problem. (as the creator of the question pointed out).
find better hyper-parameters. (For example: since DQN is quite unstable, one can play around with the syncing hyper-parameter to make it more stable as that may improve convergence (stability and/or speed))
use gather, masks and other tricks so you don't have to use very slow python loops.
use the gpu.
getting better hardware. (unless you run on an actual potato hardware shouldn't be a problem)

Suggest add 6) disable rendering the game during training. Makes a huge difference in my experience. You can still switch it on for rendering a few test episodes


text:
why cross entropy loss has to be multiplied by a batch size during an evaluation in transformer model?
I am trying to look through a code of the transformer model from Pytorch. However,
I do not understand why batch size needs to multiply with cross-entropy loss given that loss is calculated based on data at a given timestep.
This is from the line: "total_loss += batch_size * criterion(output_flat, targets).item()"
This is the section of code:
def evaluate(model: nn.Module, eval_data: Tensor) -> float:
    model.eval()  # turn on evaluation mode
    total_loss = 0.
    src_mask = generate_square_subsequent_mask(bptt).to(device)
    with torch.no_grad():
        for i in range(0, eval_data.size(0) - 1, bptt):
            data, targets = get_batch(eval_data, i)
            batch_size = data.size(0)
            if batch_size != bptt:
                src_mask = src_mask[:batch_size, :batch_size]
            output = model(data, src_mask)
            output_flat = output.view(-1, ntokens)
            total_loss += batch_size * criterion(output_flat, targets).item()
    return total_loss / (len(eval_data) - 1)

That is because the default reduction for the CEP loss calculation is mean. Hence to find the true average across all batches, you first multiply by the batch size and then divide by total number of data points. Note this only matters when all the batches are not equal, i.e. total number of data points do not perfectly divide batch size. Else you can simply add all the batch losses and divide by number of batches.


text:
Number of possible joint policies in a Dec-POMDP and the time required to evaluate each one
I was reading a book about Dec-POMDPs and came across this curious result where the author specifies the number of possible joint policies to evaluate and the time needed to evaluate a single joint policy but I can understand how he got to these results.
Can anyone please help me understand the logic used here?
Source: "Reinforcement Learning: State-of-the-Art", Editors Dr. Marco Wiering &
Dr. ir. Martijn van Otterlo Radboud, DOI 10.1007/978-3-642-27645-3, Springer-Verlag Berlin Heidelberg 2012.

What book were you reading? What don't you understand specifically and what you understand? Edit your post to include this info. Moreover, reformulate the title in the form of a question. Thanks.

Which part of the argument in the source are you having trouble with?


text:
Use of Mask in U Net for plant disease detection
I am using U-Net for plant disease detection. I am new to deep learning and computer vision.
Currently, we are feeding the masking images generated via open cv HSV format to detect colours from the images so that the background and foreground are detected.
I need to know the purpose of image masking in image segmentation using U-Net, why it is needed? What kind of masking do we need to feed to the architecture?
Do we also need to target the disease part for masking?

Here is what I understand you to be asking
Given:

Task: is plant disease detection
Solution domain: is deep learning and computer vision
Tool: is U-net
Activity: image segmentation

Questions:

Why is image masking necessary in image segmentation?
What kind of masking do we need to supply to the "architecture"
Why segment out foreground and background but not diseased section

Here is how I would engage it:
I sometimes like to see if this is a problem that has already been solved before solving it myself, especially if I am doing something for class.  You can "stand on the shoulders of giants".  When I look online I find the following:

https://link.springer.com/chapter/10.1007/978-3-030-75015-2_2
https://medium.com/analytics-vidhya/leaf-disease-detection-using-frcnn-fff55a537634
https://www.kaggle.com/datasets/saroz014/plant-disease/code
https://www.sciencedirect.com/science/article/abs/pii/S1574954122000322
https://github.com/vtalreja/Segmentation_Unet_Plants

Numbered answers:

You want the learner to only look for disease on the parts of the images that are plants and not on the parts of the images that are not plants.  Learners are both very capable and very lazy.  If they can figure out that the color of dirt is a defining difference between two pictures, then they won't look at the plants, just the dirt.  You aren't doing dirt detection, but are specifying the domain to the plant.  The learner is to lazy to figure that out.  You have to supply images where the only part you care about is taught to the learner.  As an aside, many parts of effective ML development is providing guard-rails so that the laziest solution the learner can find isn't also pathologically bad.

The masking depends on the tools.  In a way this is a paraphrase of "missing value imputation" except instead trying to do a good job of filling the hole, you want to think hard and create the right kind of a hole.  The hole-filling software that I use most is the R "mice" and some of the methods there can inform some brute-force ways to mask.  My next answer has how I would start to think about masking, though I would be open to being wrong and trying other ways, and I would also have to make a test to determine that it was detecting the disease and not the mask.

This goes into the point that you can't be supplying the answers, or the lazy learner won't become actually capable.  If the disease is always in the image in the training data, the learner will assume it is always in any image it is supplied.  There needs to be enough of a contrast that it can find the disease on its own.  That said, there is a fun smell here: you can mask out the diseased regions, and use the same basic picture in both the "true" and the "false" category.  Some care needs to be taken that it isn't learning to detect the mask, so instead of setting it to "zero" you might copy-paste some other part of the plant in various window sizes. Also you might mask good plants and have them labelled as good, so that the mask isn't the differentiator.

U-net itself is about cross-domain image reconstruction.  If I was trying to train for presence of disease then I might use a generative-adversarial method where the true/false can be used to update the weights in the U-net.  It is more of a pix2pix-style wrapper.


text:
Is there anything remotely as successful as backprop, but for training programs, not neural networks?
Backprop is used to train deep neural networks to remarkable success. Deep neural networks, on the other hands, can be seen as as a specific kind of computer function that receives inputs and produces outputs. Suppose, though, that one wanted to train functions in a programming language like Python or Haskell, that solve a specific problem. Due to the discrete nature of these objects, there is obviously no way to apply backprop to generate these. My question is: is there any algorithm that has satisfactory success in the task of training or generating these functions?
The last time I had this question, I learned about program synthesis, but the field seems to be way behind machine learning, in special due to the lack of satisfactory training procedures. What is the state of the art and where can I find the relevant literature?

As far as I know, what you call "backtracking" is more commonly termed "back propagation"

@DavidHoelzer oops, yes! No idea what happened with my brain. Fixed it. Thanks.

Can you clarify what you mean by "Due to the discrete nature of these objects, there is obviously no way to apply backprop to generate these."  You have a lot of pronouns and it is not clear what they refer to.  Maybe edit your post and give an example of what you mean.  You will get more responses with better posed questions.

@SnehalPatel since I'm not from the field (as it shows), it is hard to come up with a well-written question; sorry about that. What I mean is that neural networks are basically described as matrices of floating point numbers, so one can adjust the weights to minimize error, perform gradient descent and so on. That's doesn't apply (nor makes sense) when you're trying to optimize something like a λ-term, or a programming language's AST. The lack of such technique to "incrementally improve" is what, I believe, is making me unsure about how I could "train" programs like we train neural nets.

Like https://arxiv.org/pdf/2206.08896.pdf or https://arxiv.org/pdf/2206.08896.pdf ?


text:
Consider the following axioms-
I'm studying Artificial Intelligence and I have a question about first order logic and resolution. But I couldn't find any answer why.
I tried using Google to get the answer but nothing was found.

For first part it's extremely straightforward to encode each axiom into FOL wff's as knowledge base:

$\forall x (c(x) \to l(x, S)$
$\forall x (l(x, S) \to \forall y (d(y) \to l(x, y)))$
$d(R) ∧ r(R)$
$\forall x (r(x) → w(x) ∨ n(x))$
$¬ ∃ x (d(x) ∧ n(x))$
$∀ x (w(x) → ¬ l(G, x))$
$¬ c(G)$ (to be proved)

where $c$:=child, $l$=loves, $d$:=reindeer, $r$:=rednose, $w$:=weird, $n$:=clown, $S$:=Santa, $R$:=Rudolph, $G$:=Scrooge, respectively, in the meta language.
Then convert them to definite, unit and goal clauses via Skolemization:

$\lnot c(x) \lor l(x, S)$
$\lnot l(x, S) \lor \lnot d(y) \lor l(x, y)$
$d(R)$
$r(R)$
$\lnot r(x) \lor w(x) \lor n(x)$
$\lnot d(x) \lor \lnot n(x)$
$\lnot w(x) \lor \lnot l(G, x)$
$c(G)$ (goal clause)

From here we can further resolve clauses:

[3, 6:] $\lnot n(R)$
[4, 5:] $w(R) \lor n(R)$
[9, 10:] $w(R)$
[7, 11:] $\lnot l(G, R)$
[2, 3:] $\lnot l(x, S) \lor l(x, R)$
[12, 13:] $\lnot l(G, S)$
[1, 14:] $\lnot c(G)$
[15, 8:] $\bot$ Q.E.D.

Finally note that not any arbitrary contradictory set of clauses can be derived by resolution inference rule such as the simple case where $p \lor q$ cannot be derived from $\Gamma = \{p\}$ by resolution, though the classical FOL is semantically complete per Gödel's completeness theorem and thus the  formal system with all inference rules of FOL containing the axiom set $\Gamma$ is strongly and refutation complete (but not necessarily negation complete due to the renowned Gödel's incompleteness theorems).


text:
keras model accuracy not improving
I am trying to do multi class(16) classification, however no matter what parameters or number of layers I use my accuracy is not improving, its in 30s the max I got was 43.
I have tried early stopping to red overfilling but my testing accuracy is still low.
I have 750 images in training and 350 in testing. I am also getting high traning accuracy vs low validation accuracy.
features_train=features_train/255
features_test=features_test/255

cnn = models.Sequential([
    layers.Conv2D(filters=16, kernel_size=(3, 3), activation='relu',  strides=(2, 2), padding="same", input_shape=(224, 224, 3)),

    layers.MaxPooling2D((2,2)),
    layers.Dropout(0.25),
    layers.Conv2D(32 ,(3, 3), activation='relu'),
    layers.MaxPooling2D((2,2)),
    layers.Dropout(0.25),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.MaxPooling2D((2,2)),
    layers.Dropout(0.25),
    layers.Conv2D(128, (3, 3), activation='relu'),

    layers.MaxPooling2D((2,2)),

    layers.Flatten(),
    layers.Dense(64,activation='relu'),
    layers.Dense(16, activation='softmax')
])
cnn.compile(optimizer='adam',
             loss='categorical_crossentropy',
             metrics=['accuracy'])
cnn.fit(features_train,labels_train,epochs=20, batch_size = 4 ,validation_split = 0.25)

Because you have a relatively small dataset, you should consider performing an n-fold crossvalidation rather than a holdout validation.  This will give you a better estimate of accuracy along with error bounds.

increase batch_size to 128 or something. batch size of 4 for learning_rate 0.001 (default keras param for Adam) will not converge.

Based on having 16 classes, the naïve accuracy of a dummy classifier would be around 1/16 (6.25%) assuming class balance and the dummy classifier predicts each class according to uniform probabilities (see scikit-learn article on dummy classifiers).  Therefore, your model is likely learning something useful beyond baseline.
Depending on the complexity of your images, your training set having only 750 images may be on the low side and may be the possible/likely culprit for low accuracy.  Computer vision applications often require thousands of images for training.  For example, see CIFAR-100, CIFAR-10, MNIST digits, and MNIST fashion datasets, which have thousands of images.  For these large datasets, the achievable accuracy decreases as the complexity of the images increases (compare MNIST digits vs. MNIST fashion) and as the number of classes increases (compare CIFAR-100 vs. CIFAR-10).
Here is what I suggest for diagnosis:

Plot training and validation loss and accuracy vs. epoch to assess whether your model is overfitting / underfitting.

Determine the accuracy achievable by humans on the same dataset to assess what type of accuracy may be reasonable.  Of course, AI can outperform or underperform humans, so take this with a grain of salt.

If this a public dataset, search the literature for what accuracy other investigators have achieved.

Here is what I suggest to boost accuracy after you have run the diagnostics:

Consider generating more real data and/or using Data Augmentation.

Tune hyperparameters based on whether you are overfitting or underfitting.

Consider using a pre-trained model.


text:
How can I change the tokens BERT uses so that each digit is a separate token?
Rather than have the tokenizer generate this sort of thing:
"$1009 Dollars" => ["$", "100#", "9", "Dollars"]

I'd like to have:
"$1009 Dollars" => ["$", "1#", "0#", "0#", "9", "Dollars"]

Is that possible? I know I'd need to add all the n# tokens, but I'd need to remove a lot of tokens as well. I'm guessing I'd need to build my transformer from scratch?
For reference, here's some example code that tokenizes this string:
from transformers import AutoTokenizer, BertModel
import torch

tokenizer = AutoTokenizer.from_pretrained("bert-large-uncased")

inputs = tokenizer("$1009 Dollars", return_tensors="pt")
ids = inputs['input_ids']
for token_id in ids:
    token = tokenizer.convert_ids_to_tokens(token_id)
    print(token)

Here's the result I get:
['[CLS]', '$', '100', '##9', 'dollars', '[SEP]']

How can I control what tokens go into my model?

Also, why won't my python code highlight?


text:
Why AlphaGo didn't use Deep Q-Learning?
In the previous research, in 2015, Deep Q-Learning shows its great performance on single player Atari Games. But why do AlphaGo's researchers use CNN + MCTS instead of Deep Q-Learning? is that because Deep Q-Learning somehow is not suitable for Go?

$Q$-learning (and also its deep variant, and most of the other well-known reinforcement learning algorithms) are inherently learning approaches for single-agent environments. The entire problem setting that these algorithms are developed for (Markov decision processes, or MDPs) is always framed in terms of a single agent situated in some environment, where that agent can take actions that have some level of influence over the states that they lead to, and rewards may be observed.
If you have a problem that is, in reality, a multi-agent environment, there is a way to translate this environment to a single-agent setting; you simply have to assume that all other agents (i.e. your opponent in Go) are an inherent part of "the world" or "the environment", and that all the states in which these other agents make moves are not really states (not visible to your agent), but just intermediate steps where these part-of-the-environment-agents cause the environment to change and, as a result, create state transitions.
The primary issue with this approach is; we still need to model the decision-making of these agents in order to implement this new view of "the world", where our opponents are really a part of the world. Whatever implementation we give them, that is what our single-agent RL algorithm will learn to play against. We can just implement our opponents to be random agents, and run a single-agent RL algorithm like DQN, and then we will likely learn to play well against random agents. We'll probably still be very bad against strong opponents though. If we want to use a single-agent RL algorithm to learn to play well against strong opponents, we need to have an implementation for those strong opponents first. But if we already have that... why even bother with learning at all? We've already got the strong Go player, so we're already done and don't need to learn!
MCTS is a tree search algorithm, one that actively takes into account the fact that there is an opponent with opposing goals, and tries to model the choices that this opponent can make, and can do so better the more computation time we give it. This algorithm, and learning approaches built around it, are inherently designed to tackle the multi-agent setting (with agents having opposing goals).

It is a nice reply, but it is better to emphasize that deep Q-learning can be used for multi-agent games. TDGammon, for example play gammon and uses temporal difference learning with a neural network, a similar algorithm to deep Q-learning. Even though gammon is a two-agent game, the things you say do not apply for the algorithm of TDGammon because it uses Bellman's equation to update the parameters.

@FıratKıyak Actually, even TD-Gammon's algorithm probably "shouldn't" have worked in theory, at least in the sense that such an algorithm was not guaranteed to converge to a correct solution for a 2-player game, even given infinite computation time and an infinitely large neural network. The fact that it happened to work well in practice there was likely due to the high degree of stochasticity in the game, automatically creating variety in experience. See also: http://papers.neurips.cc/paper/1292-why-did-td-gammon-work.pdf

Deep Q Learning is a model-free algorithm. In the case of Go (and chess for that matter) the model of the game is very simple and deterministic. It's a perfect information game, so it's trivial to predict the next state given your current state and action (this is the model). They take advantage of this with MCTS to speed up training. I suppose Deep Q Learning would also work, but it would be at a huge disadvantage.


text:
How to increase accuracy of image orientation classification (Left, Right, Center)?
I am working on classifying images in "Left", "Right", "Center", "Back". Training and Validation images look like this:

The images are "Left", "Right", and "Center". I am following Pytorch transfer learning tutorial with Resnet50 architecture and have not changed anything.
The transformations I am using is as follows.
data_transforms = {"train"  : A.Compose([
                           A.Resize(256,256),
                          #  A.ShiftScaleRotate(shift_limit=0.05, scale_limit=0.05, rotate_limit=15, p=0.5),
                          #  A.RandomCrop(height=128, width=128),
                           A.RGBShift(r_shift_limit=15, g_shift_limit=15, b_shift_limit=15, p=0.5),
                           A.RandomBrightnessContrast(p=0.5),
                           A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
                           ToTensorV2(),]),
                  "val": A.Compose([
                                    A.Resize(256,256),
                                    # A.CenterCrop(height=128, width=128),
                                    A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
                                    ToTensorV2(),])
                }

Initially I had back images too, but they were getting miss-classified with center a lot with val accuracy 65%.
Without back images I got 74% accuracy with labels left, right and center.
Modified it to classify only center and not center, with only three type of images "left", "right" and "center" I have achieved 91% of val accuracy.
I am looking for ways to increase accuracy for classifying images in left, right, center.

Hi sandepp! does your dataset contain only one object?

No, there are various products which I gathered from different sources mostly include electrical products like these. One of the sites I scrapped was Eaton


text:
Recent methods for Decision Support System (DSS)
In Decision Support System (DSS), we rank items based on predetermined weighted criteria. For example, we want to rank prospective programmers based on their working experience, required salary, set of skills, age, etc. We rank using weights for each criterion that we have previously defined. The simplest method is using Simple Additive Weighting (SAW).
As far as I know, DSS is included in knowledge-based AI (it's a mandatory subject in AI specialization in most universities in my country).
My question:

With the development of AI/ML/DL today, is there another modern approach that can be used to solve similar problems?

At first, I thought it's similar with Content-Based Recommender System, but it looks different as we don't have "user" in DSS.

In my opinion, the sucess level of the Deep Learning models is perfectly suited for DSS. For example, if you have data you can built perfect Deep Reinforcement Model to rank programmers. In general AI models, specifically Deep models, are not mature enough to use without supervision of human being. But they are very suitable to build support systems.

I'm not the one who give you -1, but I think you should add more references for your answer. For example, how do you use RL to rank?


text:
Do we have to use CNN for Deep Q Learning?
I read top articles on Google Search about Deep Q-Learning:

https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8
https://skymind.ai/wiki/deep-reinforcement-learning
https://neuro.cs.ut.ee/demystifying-deep-reinforcement-learning/
Q-Learning page on wikipedia*

and then I noticed that they all use CNN as approximator. If deep learning has a broader definition than just CNN, can we use the term "Deep Q-Learning" on our model if we don't use CNN? or is there a more appropriate definition for that kind of Q-Learning model? for example, if my model only using deep fully-connected layer.
*it doesn't say explicitly Deep RL means CNN on RL, but it uses the DeepMind (that uses CNN) as an example on Deep Q-Learning

No. DQN and other deep RL methods work well with fully connected layers. Here's an implementation of DQN which doesn't use CNNs: github.com/keon/deep-q-learning/blob/master/dqn.py
DeepMind mostly use CNN because they use image as input state, and that because they tried to evaluate performance of their methods vs humans performance. Humane performance is easy to measure at games with image as input state, and that's why CNN based methods present so promptly in RL now.

I start thinking, is that because of the meaning of "deep" on DeepRL means "End-to-End without feature engineering" so most people use the term along with CNN?

@malioboro CNN's are essential for using images an inputs to a neural network. If you have image inputs to your network, then you will probably use a CNN. If you don't have images as inputs to your network, you probably won't (not getting into time-series inputs and the like).

The approximator can be any artificial neural network architecture, including deep fully-connected networks.


text:
How to estimate the cost and time to complete an AI Project
If you are a freelancer, when a client asks to create a website we can easily measure how much the total cost is needed based on the requirements of the client. (the backend, UI/UX design, features, etc.). We can even measure the estimated time of completion.
What if a client asks you to make an AI project (image recognition, speech recognition, or NLP), how do you tell the client the estimated cost and time needed to complete the project in the beginning? because the results obtained can be very different for each data used

I’m closing this question because this does not seem to be on-topic, according to https://ai.stackexchange.com/help/on-topic. Feel free to ask a meta-question, if you think this type of question should be made on-topic here. This question seems to be more suitable for https://pm.stackexchange.com/.

If you are a freelancer, when a client asks to create a website we can easily measure how much the total cost is needed based on the requirements of the client. (the backend, UI/UX design, features, etc.). We can even measure the estimated time of completion.

This is only the case when the full scope and design of the site is a well understood and relatively standard thing. With more experience, and perhaps a dedicated team, more complex sites and features can be addressed like this. It is just as common in my experience to view ongoing development of a site as a series of related projects, each of which can be estimated and costed as they become more feasible and closer to starting.
I have never seen a stand-alone site fully costed and estimated in my own work, because I work in-house on more complex features than this is possible for, but am aware this is a common approach, especially for agencies and freelancers. The difference is that the people involved are experts on producing well-defined work, and that in order to make a pre-estimated or even fixed price sale, they have put a lot of effort into risk management. Some of the risk management is contractual (e.g. "maximum of 3 major revisions" for web site design), some of it is in requirements gathering - often an initial pre-sales consultancy meeting is arranged, and/or a project proposal document generated where a lot of the possible misunderstandings between customer and developer are dealt with, and it is made clear what expectations are on both sides.
For a freelancer or agency, this de-risking is also spread over multiple projects as they will develop solutions that can be re-used including standardised contracts and templates for project planning etc.
There are also a few different project management approaches that deal with initial uncertainty in different ways. A "waterfall" approach of pre-planning as much in advance as possible requires very good knowledge of the work that will occur in later stages of a project, which of course depends heavily on earlier stages meeting all expectations of both the customer and developer. An "agile" approach will acknowledge that there are unknowns, and attempt to address them with some form of just in time planning. The problem with an agile approach is that "just in time" may be far too late if you need to budget up front (although it is still possible for instance by reducing scope to fit, provided that is agreed).

What if the client asks you to make an AI project (image recognition, speech recognition, or NLP), how do you tell the client the estimated cost and time needed to complete the project in the beginning?

There is no difference in this respect just because it is an AI project. This would apply to any software project, including web development, where one or more of the following is true:

Scope of work is not clear
There are technical hurdles with unknown resolutions. I think it is likely you are focusing on this for your imagined AI project.
Customer expectations are not clear
Project success criteria is not measurable, or not known if achievable

As any software developer, if you are faced with project work with these features, you do some pre-project tasks focused on de-risking the project before attempting it. There are a few different approaches, than can be used in combination:

Gather more requirements, spend more time up front consulting on the project
Test assumptions before starting project. E.g. if you are not sure about quality of the data, take a look at it as early as possible. If you don't know if technology X can do Y, investigate it in advance.
Work on a timesheet basis, not fixed cost. Alternatively fix cost (or have a maximum), but leave scope of work open.
Have a flexible project plan, and re-estimate remaining work routinely.
Put any assumptions about customer support for the project, such as availablilty and quality of data, into project plan and in general form into your standard contract (e.g. something legal along the lines of "Project delivery depends on customer providing resources as outlined in the project plan document. If the resources are not provided in reasonable time, then the project may over-run or have additional costs, which will be covered by the customer, and not by the freelancer")

For additional up-front work on project feasibilty and scope, if you are concerned about risk to yourself as a freelancer (because customer expects these as a pre-sale effort), you could propose to bill for that as consultancy work, prior to starting project proper. You would likely need to produce something like a project proposal document in order that the paying customer received something for spending this money. Producing such a document might even show both you and the customer that the project is not feasible without additional work - in whch case you could note the steps needed to make it feasible (e.g. customer needs to source 10,000 more training images without the flaws noticed in existing images).

authors: วันชัย แสนหวู่


text:
Is music/sound similarity comparison feasible on neural networks?
I wonder on the following concept:

A given neural network gets two audio input (preferably music) and gives a real number between 0 and 1 which describes "similarity" between the second and the first track.

As far as my understanding of neural networks go, the problem fits the concept of NNs, as pattern recognition in music can help determine similarities and discrepancies in audio, see voice recognition.
However, due to the nature of long and complex inputs, and the vague nature of learning datasets (how similar, for instance, Diana Ross "It's your move", and The Vaporwave legend "Floral Shoppe" exactly are? 0.9? 0.6? other?), such a network would be extremely slow and convoluted.
Is it possible today to build and train such a model? If yes, how would it look like?

i don't think sound recognition in humans is well-understood enough to be able to construct a model that does what you naively want to do (these two sounds are similar)

Yes, it is possible, even if the best approach could be different from  neural networks. Anyway, you should extract some significant features from the audio (energy, onsets, root frequencies, and other). Usually, more features than those really needed are extracted and afterwards the most sigificant are selected through some algorithm (e.g. PCA). In this way you will obtain an array of features (say between 10 and 100 features) with which you can train your NNs.
Note that NNs do not tell you why two audio are similar but only if they are or not. This is a big disadvantage. Instead, algorithms based on grey-box modeling such as rule or case based algorithm (maybe using fuzzy logic) could be more useful, provided that you have a deeper knowledge of the problem.
References and deepening sources: SMC Lab from University of Padua education material

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Could GA's determine fitness by "Fighting" against each other?
I am developing AI in the form of NEAT, and it has passed certain tasks like the XOR problem outlined in the NEAT Research Paper. In the XOR Problem, the fitness of a network was determined by an existing function (XOR in this case). It also passed another tests. One I developed was to determine the sine at a certain point X in radians. It also worked, but yet again, its fitness was determined by an existing function (sin (x)).
I've recently been working on training it to play Tic Tac Toe. I decided that to determine its fitness, it would play against a "dumb" AI, placing O's in random locations on the grid, and gaining fitness based on whether or not it placed X's in a valid location (losing fitness if it placed an X on top of another X or an O), and gaining a lot of fitness if it won against the "dumb" AI. This would work, but when a network got really lucky and the "dumb" AI placed O's in impractical locations, the network would win and gain a lot of fitness, making it very difficult for another network to beat that fitness. Therefore, the learning process did not work and I was not able to generate a Tic Tac Toe network that actually worked well.
I do not want the GA to learn based off an "intelligent" tic tac toe AI because the whole point of me training this GA is so that I do not have to make the AI in the first place. I want it to be able to learn rules on its own without me having to hard code an AI to be very good at it.
So, I got to thinking, and I thought it would be interesting if the fitness of a network could be determined based off how well it played against OTHER NETWORKS in its generation. This does seem similar to how humans learn to play games, as I learned to play chess by playing against other people hundreds of times, learning from my mistakes, and my friends also increased in their ability to play chess as well. If GA's were to do that, that would mean I don't have to program AI to play the game (in fact, I wouldn't have to program a "dumb" AI as well, I would only have to hard code the rules of the game, obviously).
My questions are:

Has there been any research or results from GA's determining their fitness based off competing against each other? I did some searching but I have no idea what to look for in the first place (searching 'NEAT fight against each other' did not work well :-(  )
Does this method of training a GA seem practical? It seems practical to me, but are there any potential drawbacks to this? Are GA's meant to only calculate predetermined functions that exist, or do they have the potential to learn and do some decision making?
If I were to do this, how would fitness be determined? Say, for the tic tac toe example, should fitness be determined based on whether or not a network places its X's or O's in viable locations, and add fitness if it wins and subtracts fitness if it loses? What about tying the game?
Should networks of the same species compete against each other? If they did, then it would seem impractical to have species in the first place, as networks in the same species competing against each other would not allow a successful species to rise to the top, as it would be fighting against each other.
Kind of out of topic, but with my original idea for the tic tac toe GA, would there be a better way to determine fitness? Would creating an intelligent AI be the best way to train a GA?

Thanks for your time, as this is somewhat lengthy, and for your feedback!

What you're describing is a type of what's known as a coevolutionary algorithm. Googling that should be a useful exercise.
I'd also point out that the general idea of playing against versions of yourself has a long history in the literature around learning to play games. One of the very earliest real success stories in AI was Arthur Samuel's checkers program in 1959. He trained a program to play the game at the level of a very strong human by having it play games against itself, each time improving itself via a mechanism that resembles what we would today call reinforcement learning.

Let's have it clear...try to make your question brief and kept in mind that every one understands it .fast

i'm the main developer of Neataptic, a Javascript neuro-evolution library.

Very effective! Realise that this is how real-life evolution happened as well: we kept on improving against other species, which forced them to improve as well.
Very practical, especially if you don't want to set up any 'rules' like you say, it makes the genomes find out what the rules are themselves.
Basically, you let each genome in the population play X games against other genomes, I advise you let each genome play against every other genome in the population. An example of scoring would be giving the genome 1 point for winning, and 0.25 or 0.5 for a tie. Each game should always have a result!
I'm not sure about this one, as I haven't implemented speciation.

I want to give you some examples that I have worked on:

Agar.io AI (neuro-evolved neural agents) - basically, I let neural networks evolve to get the highest score they can in agar.io, by competing against each other! It worked better than I expected.
Currently i'm working on new project, a kind of 'cops and robbers' style game.

Awesome answer!  (More links would be useful on the basic concepts, since I'm noticing a recent disconnect in that area due the growing accessibility of the methods.)  I favor this approach because I can use guided learning to give my automata some basic axioms, but let their intelligence be a factor of self evolution, which hopefully leads to axioms they themselves discover. :)

I was going to post examples from Neataptic, but then I saw your answer sir, and I am not worthy :D Nice to e-meet you!

The general notion is that of 'Competitive Coevolution' and there are many (maybe hundreds) of academic papers that describe various alternatives.
The excellent (and freely available) Essentials of Metaheuristics has a whole chapter on the subject.

Look up tournament selection
Tournament selection is a method of selecting an individual from a population of individuals in a genetic algorithm.[1] Tournament selection involves running several "tournaments" among a few individuals (or "chromosomes") chosen at random from the population. The winner of each tournament (the one with the best fitness) is selected for crossover. Selection pressure, a probabilistic measure of a chromosome's likelihood of participation in the tournament based on the participant selection pool size, is easily adjusted by changing the tournament size[why?]. If the tournament size is larger, weak individuals have a smaller chance to be selected, because, if a weak individual is selected to be in a tournament, there is a higher probability that a stronger individual is also in that tournament.


text:
How do stacked denoising autoencoders work
I've been studying a recommender system which uses a collaborative deep learning approach and Bayesian learning. It has the following NN representation :

I need to know the working of stacked denoising autoencoders.
Here is the link to the paper:http://www.wanghao.in/paper/KDD15_CDL.pdf


text:
String matching algorithm for product recognition
I'm building a web application that collects schema.org data from different webshops as Amazon, Shopify, etc. It collects data every 6h and shows the current and lowest price. It is used for monitoring products and buying at the lowest price.
My goal is to recognize products from different shops as the same product. Every shop has its own title for the same product.
Example:
Google Pixel 2 64GB Clearly White (Unlocked) Smartphone
Google Pixel 2 GSM/CDMA Google Unlocked (Clearly White, 64GB, US warranty)

Problems:

don't have a lot of data (only products chosen by the user)
needs to support every new product that app doesn't have data history

"needs to support every new product that app doesn't have data history", to obtain this, you will have to build a model that is able to generalise. To do this, you might need to use a regularisation technique. Furthermore, if you don't have a lot of data, then you likely don't want to use a very complex model. Is your data labelled?

yes and yes. I have data only of the product/site that user add to the app. I have labeled data as on shema.org: title, rating, price, etc. Any ideas?

You could try to use an SVM. You train it to predict the label of each product in the training set and validate the results on validation dataset. Given you have only a small amount of data, I would use cross-validation. Have a look at it. If SVM doesn't produce decent results on the validation dataset, then you likely will need more data or you could try to use another model.


text:
Multi Agent Sokoban Search Solvers state of the art
I also asked this question here but I'm repeating it on this SE because I feel it is more relevant. No intention to spam.
I am researching into coding a solver for a variant of the Sokoban game with multiple agents, restrictions (eg. colors of stones, goals) and relaxations (push AND pull possible, etc.)
By researching online I have found classic papers in the field, like the Rolling Stone paper (by Andreas Junghanns and Jonathan Schaeffer) and Sokoban: Enhancing general single-agent search methods using domain knowledge from the same authors.
These solutions seem to be outdated and I am currently structuring my solver per the notes of the two most performant solvers: YASS and Sokolution.
From the research, I've done these two seem to be my best bet.
It is apparent that they are not enough by themselves to solve a multi-agent environment. Those solvers are made for a single agent. So far, I have failed to find useful multi-agent proposals.
In this context, my question is:

What can be considered state-of-the-art in order to:

coordinate multiple agents with different goals, and
plug a solver's solution in and validate/edit it?

What are some search terms I can use to research this further?

Thank you very much