text:
Hyperparameters for Reproducing the Results of IRGAN on MovieLens 1M
I am trying to reproduce results reported for IRGAN (information retrieval GAN) on the MovieLens 1M dataset. The results I want to reproduce and their sources are listed in the table below.

Model
Precision@5
NDCG@5
Source

IRGAN
26.30%
26.40%
CFGAN

IRGAN
30.98%
31.59%
CoFiGAN

IRGAN
31.82%
33.72%
BiGAN

While my implementation of IRGAN is able to reproduce the results on the MovieLens 100k dataset, I am having problems discovering the hyperparameters for reproducing the results on the MovieLens 1M dataset; currently my IRGAN implementation is achieving a precision@5 score of 21.7%.
Unfortunately, the authors of the aforementioned papers do not share the hyperparameters used for training their version of IRGAN.
Thus, I want to ask if there is a repository with the used hyperparameters?
Furthermore, I would be most grateful if you could provide me information on how to contact the authors?


text:
A neural network to learn the connection between two totally different type of images
I have a dataset of two different type of images. Say, I have images of a person and his all 10 fingerprints. I want to create a relation between them to predict one from another. How I can do that and which architecture is suitable for this problem or similar type of problem.

I doubt you can predict a fingerprint image from a person's face, but this is still a valid question for some other datasets.

I would try a pair of separate deep image embeddings with a contrastive loss. The idea is similar to the Siamese network architecture. In Siamese networks the pairs of images are of the same type - so both input images are fed through the copy of the same network. In your case the images are of different kinds, so I would just have separate nets for person images and fingerprints.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Sutton and Barto 2nd Edition Exercise 13.1
I'm attempting exercise 13.1 in the Sutton and Barto textbook. It asks for an optimal probability for selecting action right in the short corridor scenario (see first 6 lines of the image below for the scenario).
Exercise 13.1: Use your knowledge of the gridworld and its dynamics to determine an
exact symbolic expression for the optimal probability of selecting the right action in
Example 13.1.
My attempt:
Letting $p$ denote the probability of choosing right, I understand that using the Bellman equations, we can solve for the value of $s_1, s_2, s_3$ where the states are numbered from left to right in terms of $p$. We have $v(s_1) = \frac{2-p}{p-1}$, $v(s_2) = \frac{1}{(p-1)p}$, $v(s_3) = -\frac{p+1}{p}$.
I can see how we can find the max of each of these functions to get the best optimal policy, given the state we're currently in.
However, how do you find the optimal policy generally (irrespective of starting state)? I found solutions here, which magically arrives at
$\frac{p^2-2p+2}{p(1-p)}$. Can someone explain this part?
https://github.com/brynhayder/reinforcement_learning_an_introduction/blob/master/exercises/exercises.pdf

The expression $(p^2-2p+2)/(p(1-p))$ is positive for all, so I wouldn't rely on it...

The other answers are cool, here I give a straightforward proof, it might be more clearly for understanding the whole process.
The core idea here is that since each state can rollback to the previous state (the state before State S is itself), and each time it rollbacks to the previous state, the expected value of steps should increase the expectation of the previous state. The number of steps, which is ultimately expressed in the form of an infinite series.
Assuming that the action Left is selected in the first $t-1$ steps, and the action Right is selected in the $t$-th step, a total of $t$ steps are required to reach the State 2, where $ t \in [1, \infty) $:
$ E_{S\rightarrow 2}=1\cdot p^1+2\cdot p^1 (1-p)^1+3\cdot p^1 (1-p)^2+\cdots=p\sum_{t=1}^{\infty}t(1-p)^{t-1} $
$ =\frac{p}{1-p}\sum_{t=1}^{\infty}t(1-p)^t=\frac{p}{1-p}\cdot\frac{1-p}{p^2} = \frac{1}{p} $
The transition from State 2 to State 3 is divided into two cases, namely 2→3, 2→S→2→3, and if rollbacking from 2 to S, it is necessary to recursively calculate the expected number of steps from 2→S→2: $ E_{s\rightarrow 2} $, where $t$ is the total number of times to return to state S:
$ E_{2\rightarrow 3}=(1-p)+(2+E_{S\rightarrow2})p(1-p)+(3+2E_{S\rightarrow2})p^2(1-p)+\cdots $
$ =(1-p)+(1-p)\sum_{t=1}^{\infty}\left[1+t(1+E_{S\rightarrow2})\right]p^t$
$ =(1-p)+p+\frac{1-p^2}{p}\frac{p}{(1-p)^2} = 1+\frac{1+p}{1-p} = \frac{2}{1-p}$
$ E_{3\rightarrow G} $ can be obtained in the same way:
$ E_{3\rightarrow G}=p+(2+E_{2\rightarrow3})p(1-p)+(3+2E_{2\rightarrow3})p(1-p)^2+\cdots $
$ =p+p\sum_{t=1}^{\infty}\left[1+t(1+E_{2\rightarrow3})\right](1-p)^t $
$ = p+ p \left[ \frac{1-p}{p} + \frac{3-p}{1-p} \frac{1-p}{p^2} \right] = p+(1-p) + \frac{3-p}{p} = \frac{3}{p} $
Above we have calculated the expected number of steps for the three state transitions $ E_{S\rightarrow 2}, E_{2\rightarrow 3}, E_{3\rightarrow G} $, so the total expected value is:
$ E = E_{S\rightarrow 2} + E_{2\rightarrow 3} + E_{3\rightarrow G} = \frac{4}{p} + \frac{2}{1-p} $
To maximize the expectation, then
$ \left[\frac{4}{p} + \frac{2}{1-p}\right]^{\prime} = -\frac{4}{p^2}+\frac{2}{(1-p)^2} = 0 $
Get a solution that satisfies the condition: $ p=2-\sqrt{2}\approx 0.5858, E=6+4\sqrt{2} \approx 11.6568 $.

There are problems with both the approach and the expressions that you have. I don't want to just give the correct solution, though, that's an exercise for you to go through and learn from your own experience trying to accomplish it. Instead, let me illustrate that your expressions for $v(s_i)$ are wrong. To do that we'll just do a Monte-Carlo estimate for a range of values of $p$ and compare to your expression.
Here's the Python code that runs a single payout starting from state state and following the policy with probability p of choosing "right". It returns the collected reward:
import numpy as np

def playout(state, p):
   reward = 0
   while state != 0:
       action_right = np.random.rand() < p
       move_right = action_right if state != 2 else (not action_right)
       state = state - 1 if move_right else state + 1
       state = 3 if state > 3 else state
       reward -= 1
   return reward

Then we make a simulation code that runs the playout multiple times, collects the reward counts, and returns the average reward (so, essentially estimates $v^\pi(s_i)$):
from collections import defaultdict

def simulate(state, p, n):
    rewards = defaultdict(lambda : 0, {})
    for _ in range(n):
        rewards[playout(state,p)] += 1
    results = np.array([[a,b] for a,b in rewards.items()]).T
    reward , nplayouts = results[0] , results[1]
    value = (reward * nplayouts).sum() / nplayouts.sum()
    return value

Finally, I make a grid in p and run each playout 10000 times:
p = np.linspace(0,1,51)[1:-1]
v3 = [simulate(3,p,10000) for p in p]
v2 = [simulate(2,p,10000) for p in p]
v1 = [simulate(1,p,10000) for p in p]

I've plotted the resulting value estimates for each state. Together with your expression for them (blue curves). And the correct expression (red curve) that I've obtained by actually writing down and solving the equations:

As you can see, the expressions you've presented are all too far off from the results that the simulation returns. More than that - the asymptotic behavior of your solutions at $p\to0$ and $p\to1$ doesn't make much sense.
I've obtained the expressions for the red curves above by solving the system for $v(s_i)$ on my own account - without relying on weird and wrong solutions that I googled on the internet. Which I'd recommend you do as well.
Finally, the question of the exercise is to find an optimal $p$ for a policy that starts at $s_3$ - not "irrespective of starting state" as you've thought is should be. Unlike your expression the correct expression for  $v(s_3)$ has a maximum, which can be found analytically and it is
$$\max_p v(s_3) = -6-4\sqrt2 \simeq -11.6$$
$$ \text{at}\quad p = ??? \simeq 0.59$$

I reread the question, it is word for word what I wrote in the question, where does it say to find an optimal policy that starts at state $s_3$?

@Snowball "S" and "G" is a standard notation for start and end states across the book. See e.g. examples 6.5, 6.6, 8.1

Here is a simple way to solve the problem.
With probability p of moving right in normal grids, we can write down the transition matrix.
$$
Q=\begin{bmatrix} 1-p & p & 0 \\
p & 0 & 1-p \\
0 & 1-p & 0 \end{bmatrix}
$$
From this answer we have the method to calculate the expected steps between two states.
https://math.stackexchange.com/questions/691494/expected-number-of-steps-between-states-in-a-markov-chain
which is, the sum of the first row of the matrix $(I-Q)^{-1}$
$$
M=(I-Q)^{-1}=\begin{bmatrix} \frac{2}{p}+\frac{1}{1-p} & \frac{1}{p}+\frac{1}{1-p} & \frac{1}{p} \\
\frac{1}{p}+\frac{1}{1-p} & \frac{1}{p}+\frac{1}{1-p} & \frac{1}{p} \\
\frac{1}{p} & \frac{1}{p} & \frac{1}{p} \end{bmatrix}
$$
So the expectation of the number of steps is
$$
E[t]=\frac{4}{p}+\frac{2}{1-p}
$$
And since the reward is -1 per step
$$J(p)=-E[t]=-\frac{4}{p}-\frac{2}{1-p}$$
$$
\frac{dJ}{dp}=\frac{4}{p^2}-\frac{2}{(1-p)^2}=0
$$
$$ 2p^2=4(1-p)^2 $$
$$ p^2-4p+2=0 $$
We then have solutions $p=2\pm\sqrt{2}$, and since $p$ is a probability in $[0,1]$
$$ \hat{p} = 2-\sqrt{2} \approx 0.586 $$

As Alex mentioned, this is a typical mean time to absorption type of problem in a Markov chain except each step incurs a negative reward. So, yet another way to solve this problem is to set up a system of equations. Let $v_1, v_2, v_3$ denote the mean reward starting from states 1, 2, and 3, respectively. Then we have:
$$
\begin{cases}
v_1 =& (1-p)v_1 + pv_2 -1 \\
v_2 =& (1-p)v_3 + pv_1 -1 \\
v_3 =& (1-p)(v_2-1)
\end{cases}
$$
Solving this gives $v_1 = \frac{p^2-3p+4}{p(p-1)}$. Now, taking the derivative and setting it to 0 yields $p^* = 2-\sqrt{2}$

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Does it make sense to apply batch normalization to a batch size of 1?
I am interested in your opinion on the topic if you think that it makes sense to use batch normalization layer in a network that is trained with a batch size of 1. This is a special case as part of an experiment. What effects can be expected?

It's not possible to use batch normalization with a batch size of 1. Batch normalization requires you to calculate the variance of activation values in the current batch, and variance is undefined for a sample of size 1.

Why it is undefined? It is zero.

The sample variance formula has a denominator of N - 1. When N = 1, N - 1 = 0, and division by 0 is undefined. @adosar


text:
Current extensions of the "Turing Test"?
In 2014 it was widely reported that the Turing Test had been passed, and that this was a major AI milestone.
See: Computer AI passes Turing test in 'world first [BBC]; Turing Test Success Marks Milestone in Computing History [reading.ac.uk]; What is the Turing test? And are we all doomed now? [The Guardian]
Never mind that the "Imitation Game" is subjective, and that porn bots have been passing it since there were porn bots—University of Reading was clear about their metrics.
But I understand that it did lead to revised tests, and extension of thinking on what constitutes passing such a threshold.

How have Turing Tests been extended since 2014?

How strong was the 2014 test?  What have been the criticisms of the 2014 determinations?

Anyone can read the wiki, but here I'm looking for expert answers that synopsize, ideally with links, and can comment on any limitations, or potential weaknesses in methology or theory of the widely publicized 2014 experiment.


text:
How does policy network learn in AlphaZero?
I'm currently trying to understand how AlphaZero works. There is one thing with the training of the AlphaZero's policy head that confuses me. Basically, in AlphaGo Zero's paper (where the major part of AlphaZero algorithm is explained) a combined neural network is used to estimate both, the value of the position and a good policy. More precisely, the loss function used is:
$$L = (z-v) - \pi^t \log(\textbf{p}) + c \Vert \theta \Vert$$
where $z$ is the outcome of the game, $v$ is the value estimated by the neural network, $\pi$ is the policy calculated by the MCTS and $\textbf{p}$ is the policy predicted by the neural network.
I would like to focus on the policy head loss. Basically, we are trying to minimize the difference between the policy calculated by the MCTS and the policy predicted by the neural network. That makes sense when the player has won the game, but it doesn't (at least from my point of view) when the player has lost it. You would be teaching your neural network a policy that has lost. Maybe the loss was unavoidable, but if it wasn't that's definitely not the policy we want to learn.
I have programmed a slightly simplified version that works well with Tic Tac Toe. But for Connect 4 some problems related to this arise. Basically, it learns a bad policy. At the beginning of the training, the values estimated for each board are quite random, and that makes the policy shift to a random (and wrong) direction. At the same time, that makes the value function to be wrong (because we are losing games that we could have easily won), worsening even more the policy.
I suppose that with enough training this problem disappears. The correct value and policy should backpropagate from the leaf nodes of our simulation. Even if the neural network policy gives a probability of 0 to the optimal action, thanks to the Dirichlet noise added to the probabilities the MCTS can find that optimal action and learn it.
However, several things confuse me:

In AlphaGo's paper, they take into account whether if the outcome
of the game has been a win or a loss when training the policy
network with reinforcement learning. More precisely, the
optimization made is
$$\Delta p \propto \frac{\delta \log{p(a_t|s_t)}}{\delta p} z_t  $$
where $z_t = 1$ if we have won or $z_t = -1$ if we have lost. So DeepMind's take into account if the action was good or not and change the direction to optimize.

I haven't found anywhere in AlphaGo Zero's paper that we are training just with the examples where the player has won, so they might be using all the data gathered, including also the examples where the player has lost. As far as I know, they don't mention anything related to this problem.

$\pi$ (the policy provided by the MCTS) is calculated as the exponentiated visit count of each action
$$\pi(a|s_0) = \frac{N(s_0,a)^{1/\tau}}{\sum_b N(s_0,b)^{1/\tau}}$$
where $\tau$ is a parameter that controls the ``temperature''.
DeepMind's team sets $\tau = 1$ during the 30 first movements to
ensure exploration. After that, they set it to $\tau \approx 0$ to
ensure that the action that is considered the best one (and thus has
been simulated more times) is the one played. However, that means
$\pi$ is something like
$$[0,0,\dots,0,1,0, \dots, 0]$$
making the policy changes a bit agressive and especially harmful if
the movement is not the good one (making it even harder to recover from a bad action).

Am I missing something? Is this the intended way of working of the algorithm? Is there any way to overcome the learning of bad policies?

Maybe this will help with your last point https://ai.stackexchange.com/questions/32696/what-is-a-policy-training-target-in-alphazero.

I'll first address the big-picture intuition and then address each point separately.
First of all, the tree search tries to find the best policy at each turn. Losing the game doesn't necessarily mean the policy is bad, often it's just caused by us picking a bad move at least somewhere in the game (thanks to the policy temperature $\tau$), this could even be long before or after the current turn. Training can progress as long as the tree-derived policy is (at least slightly) better than the raw output of the neural net.
You're right that a bad value head can cause tree search to find a bad policy. However near the end of the game we are always getting some true signals:

the tree search encounters some terminal states which can directly influence the policy to be better
the value head can at least learn the right outcome for states close to the end

These two mechanisms cause both value and policy head to become better at the endgame, this means tree search and its policy can become better at near-endgame positions, which means we actually pick better moves to play which means the value head gets to learn better values on average.
So yes, early on during training the network can lean some wrong things for the early- and mid-game. There is always some pressure towards correctness coming from the endgame though, and with enough training and the right hyperparameters this can propagate all the way back to the start of the game.
To address your individual questions:

Yes, AlphaGo was doing something policy gradient-like where they used the game outcome to directly change the policy. AlphaZero works very differently, it's best to view them separate.

Indeed, the tree-derived policy of all moves is used for training, regardless of whether the current player ended up winning or losing the game.

This temperature is only used to pick an action during selfplay, not for the training target which still just uses $\tau = 1$. (as a side note, it happens to still be a good idea to use a temperature for the training target for regularization, LeelaChessZero and KataGo do this, see here for an explanation)


text:
How to detect the description of spine segments in short text using a neural network?
The input data is a set of text chunks containing the description of the pathology or the surgical procedure:
For instance:

Tere is a lumbar stenosis L3/4
Patient ist suffering from [...], MRI and X ray showed lumbar stenosis L3/4, segmental instability L3-5, foraminal stenosis L5/S1 both sides
The patient [...] underwent an MRI showing cervical stenosis C4-7 with myelopathy
[...] showed lumbar adult scoliosis L2-S1 with Cobb angle of 42°
Patient fell from the chair [...] showed osteoporotic fracture L3

Now, the ideal classificator would give me:

Segments: L3,L4; typeofpathology: degenerative; subtypepathology: stenosis
Segments: L3,L4,L5,S1; type of pathology: degenerative; subtypepathology: stenosis, instability
Segments C4, C5, C6, C7;type of pathology: degenerative; subtypepathology: myelopathy
Segments L2,L3,L4,L5,S1;type of pathology: deformity; subtypepathology: de novo scoliosis
Segments L3; type of pathology: pathological fracture; subtypepathology: -

I think that this cannot be reasonably achieved by a pre-programmed algorithm, because the amount of the text before the description can vary, and the choice of words can vary too. Is there an approach using neural networks or NLP tools that would have chance at reaching such classification? How large would the dataset used for training have to be (approximately)?
Maybe it would be reasonable to separate the two problems: detection of the segments AND detection of the pathology. For the segments, one could search for a pattern of C? T? L? or S? with ? being a number and then include all such segment descriptions in the next 20-30 characters, then use an algorithm to mark the continuous segments from the upper to the lower vertebra.
Done this, do neural networks offer any significant advantages over simple keyword matching classification? Most importantly, which NLP neural network tools would be the ones you would start trying with?


text:
Forecasting of spatio-temporal event data
I’m currently working on my dissertation which is centred around forecasting social conflict events. I’m using data from GDELT (Global Database of Events, Tone, and Language) to develop my forecasting model. For the sake of conveying the problem and limiting the length of this post, I have simplified the features used in my investigation. These can be summarised as follows:
(please feel free to skip the feature description to the end of this post indicated by "The Question" marked in bold if TL:DR)
Temporal Attribute:

FractionDate: Date of event [numerical].

Actor Attributes:

Actor1Type: The type of actor who performed the action [factor]. (e.g. Government, Rebels, Civilians, etc.)
Actor2Type: The type of actor who received the action [factor]. (e.g. Government, Rebels, Civilians, etc.)

Event Action Attributes:

EventClass: Verbal cooperation, material cooperation, verbal conflict, and material conflict encoded as 1,2,3,4 respectively [factor].
EventImpact: A numeric score from [-10,10] capturing the potential impact that type of event may have on the stability of a country [numerical].

Spatial Attributes:

ActionGeoLong: Longitude where the action took place [numerical].
ActionGeoLat: Latitude where the action took place [numerical].

The database is updated on a daily schedule and is roughly 50 MB on average for single days data. The data is filtered to include only events that took place in a single country, which decreases the file size to about 1-2 MB. These events are then aggregated on a weekly basis.
One notable modeling method to predict spatio-temporal data is by means of ConvLSTM models. These models have been successfully implemented in, for example, predicting precipitation or traffic flow. So the strategy that I have so far is:

Aggregate the spatial data to generate weekly geographical heatmaps, showing the intensity (for the sake of simplicity, can be thought as a weighted product of frequency and EventImpact) of events for each EventClass. That is, you are left with a time series of 4 heatmaps similar to the ones below.

Aggregate the actor data to generate weekly actor "Interaction" matrices [I]. These matrices show the intensity (yet again, can be thought as a weighted product of frequency and Eventimpact) of interaction between each actor for each EventClass. Actor 1 (performer) are on the rows and Actor 2 (receiver) are on the columns, therefore, [I]_{n,m} would mean the intensity of Actor n doing something to Actor m. (Note that these matrices won't be symmetrical, the intensity of actor n doing something to actor m, is different from actor m doing something to actor n). Then you are left with a time series of 4 matrices similar to the ones below:

The two above (geographical heatmaps and interaction matrices) will be the "input" to my model, and it should be able to predict the next weeks heatmap and interaction matrix given the history of events. In theory I should be able to construct ConvLSTM model for the geographical heatmaps or the interaction matrices separately. Therefore, the problem I am faced with is building a sort of ensemble of ConvLSTM which is able to learn from both input sources simultaneously.
The Question:
Is there a way to construct a ConvLSTM that can learn from two different "types" of input tensors? The first being a sequence of geographical heatmaps (with 4 channels), and the second being a matrix (also with 4 channels). If so, how would you implement this in Keras? It is very important that the model considers both sources in order to learn underlying mechanisms of the system. An example of the model Input and Output is provided below.

Thank you for taking the time to read. I would appreciate additional opinion or other applicable modeling methods very much.


text:
Split on dataset with some shared features?
I have a dataset with financial stock data, some of the features are shared, for example daily gold prices, while the stock price for each individual stock is different, the gold price would be the same for everybody that day.
When I split 80/10/10 randomly, it's "cheating" and while the result accuracy is great the actual real world live result is bad.
When I split sequentially, ie first 8 years of data in training, next year in validation, last year in testing.  The result accuracy is bad, and live testing is also bad.
What I want to ask is, should I do random split between just training and validation on first 9 years of data, then do testing on last year of data separately?
OR is sequentially as good as it's gonna get and I simply can't predict the future?

Welcome to SE:AI!  I think this has gotten close voted per the solo "datasets" tag.  My sense is this is, in fact, a question about AI theory, and the better method of splitting such data?  Is this correct?  Are there other tag you could potentially apply?

It is about theory and I am putting it in practice, at the end, the question boil down to: can training set and validation set share information?  OR is that consider cheating?  It is well known in our industry already that sharing information between training and testing is a big no no, but what about training and validation?  I am not sure what other tag to put this under.

My advice is to edit your question to reinforce the on-topic nature and add a couple more tags.  (We get a lot of pure data set questions, not related to theory, which are off topic.)


text:
Neural Network Regression Experiment Going Wrong
I've been trying to get a simple regression experiment going with a neural network and I would like some help interpreting what is going wrong.
My goal is to see what level of regression accuracy I can achieve with a feed forward neural network. I have N pairs of inputs, x, and outputs, y.
As an example:
X is made up of serial integers starting at 0: e.x [0 1 2 3 ... N]
Y is made up of pairs of random floats between 0 and 1: e.x [[.263 .548] [.157 .014] [.988 .478] ... Nth [.356 .245]].

So my neural network's structure has 1 input neuron and 2 output neurons and some hidden layers in between whose properties are part of this experiment.
These are the questions I seek to answer:

Can this neural network with some configuration of hidden layers map these inputs and outputs with perfect accuracy?
If not, what is the best accuracy that can be achieved with a reasonably sized network?
Is there a limit on the value of N such that the accuracy of the mapping deteriorates past an accuracy threshold t for each value a and b of an output pair, [a, b], of +- z where z is fairly small? In other words can the prediction z stay withing a - t and a + t
and the same for b?

Here is my model and some relevant functions as it stands:
# Custom Loss
def absoluteDifference(y_true, y_pred):
  return abs(y_pred - y_true)

model = Sequential()
model.add(Dense(1, activation='relu', input_dim=1, kernel_initializer='he_uniform'))
model.add(Dense(10, activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(magnitudes))

model.compile(loss=absoluteDifference, optimizer='adam', metrics=['accuracy'])

The results of my experiments are confusing. It always seems to bottom out at some accuracy threshold and never progress. When I compare the outputs of the network to the y value pairs they are either  wildly different than y or the predictions seem to be flattening at one value:
Here are some actual results:
Results 1: Flattening:
Y:
[[0.2890625  0.43554688]
 [0.1171875  0.02734375]
 [0.44921875 0.11328125]
 [0.04296875 0.25585938]
 [0.4921875  0.42578125]
 [0.09960938 0.04101562]
 [0.265625   0.05273438]
 [0.421875   0.26757812]
 [0.40625    0.0859375 ]
 [0.25976562 0.1328125 ]]

Predictions:
[[0.27030337 0.11708295]
 [0.27030337 0.11708295]
 [0.27030337 0.11708295]
 [0.27030337 0.11708295]
 [0.27030337 0.11708295]
 [0.27030337 0.11708295]
 [0.27030337 0.11708295]
 [0.27030337 0.11708295]
 [0.27030337 0.11708295]
 [0.27030337 0.11708295]]

Results 2: Different:
Y:
[[0.3515625  0.47851562]
 [0.16796875 0.28320312]
 [0.140625   0.453125  ]
 [0.21679688 0.44726562]
 [0.21484375 0.0859375 ]
 [0.47265625 0.15429688]
 [0.37304688 0.30078125]
 [0.06054688 0.04492188]
 [0.49609375 0.41992188]
 [0.4453125  0.40820312]]

Predictions:
[[0.18319808 0.33377975]
 [0.16718353 0.33032405]
 [0.19876595 0.32554907]
 [0.23596771 0.3200497 ]
 [0.2726316  0.31566542]
 [0.30836326 0.31124872]
 [0.34609824 0.30569595]
 [0.38202912 0.30291694]
 [0.4184485  0.2967524 ]
 [0.45500547 0.29231113]]

I would have expected Y and Predictions to match after training. What am I doing wrong here?


text:
Loss function to minimize the distance between sets
Are there references or links to examples about loss functions "Distance Metrics" which could be used to minimize the distance between two sets for a neural network. More precisely, this distance metric must depend on the whole set in calculation and not only a single point as the Euclidean distance.
It is known that Hausdorff distance is used to find the distance between two sets and it is well used for images comparison but it depends additionally on a point in calculation. For my case, I can't depend on a single point for the distance metric but I must consider the whole set to compare it with the other set! Is there any recommendation?

Can you please provide more info about why you need to compute this distance based on all elements of the set? What are the sets in your case? So, what task are you exactly trying to solve with neural networks? What are the inputs and outputs of this neural network? How does your dataset look like?

@nbro In fact I don't have a data set in hand but I am thinking about transforming and modifying a numerical dataset to a new dataset then to use a distance metric to minimize the difference between them. Consequently, I am looking for a metric which calculates the distance based on all elements not only one!


text:
How do autoregressive attention mechanism work in multi-headed attention?
[LONG POST!!] I am working on a DNN model that works as an improviser to generate music sequences. The idea of generating music is based on taking a sequence of music nodes (their index representation) and generating sequences that are distinctive with more context and coherent structure as well as capturing syntactic and structural information from the original sequences. Therefore I am dealing with a time series dataset. Similar work was reported in "Attentional Networks for music generation" but in our case, we have a different model architecture and different dataset.
It has been known that Transformer (attention) suffers in multivariate time series dataset (Source: Attention for time series forecasting and classification). But given these problems were reported two years ago, the SOTA should be better by now. For that reason, my target is to use the attention mechanism in a way to overcome these challenges.
Recently I have been using the multiheaded attention layer from TF and testing with head size between 128 and 3074 and head number from 1 to 10 and dropout from 0.1 to 0.5. Based on the results there was no noticeable improvement in the model performance, it seems that the multi-headed attention layer didn't have contribution during training.
Therefore and after carefully reading the literature I found that autoregressive attention is the best option for this types of problem. Basically, by making the attention autoregressive, it will compute the attention over the previous (decoder) outputs in such a way as to avoid using future information to make current predictions (to preserve the notion of causality). So the attention has to designed so that at each time step it needs to be autoregressive, for example, use previously generated sequences as extra input while generating the next symbol.
In "Autoregressive Attention for Parallel Sequence Modeling" paper they introduced the autoregressive attention mechanism in order to maintain the causality in the decoder. I didn't understand what they mean in Section 3.3 which describe the implementation of autoregressive attention. My problem is in the autoregressive implementation, in the paper they stated that autoregressive mechanics was implemented using the masking technique which changes all of the elements in the upper-right triangle and the diagonal
to −∞ 3 to ensure that all the scores that would introduce future information into the attention calculation are equal to 0 after the softmax. I was hoping to see how it was implemented in the code to get a better idea of how it works.
Here is how the attention is implemented in tensorflow:
def multiHeadedAttentionLayer(cell_input):
    cell_state = None

    if cell_state is None:
        cell_state = cell_input

    mha = tfa.layers.MultiHeadAttention(head_size=128, num_heads=5, dropout = 0.5)
    cell_output = mha([cell_input, cell_state])
    cell_state = cell_input

    return cell_output

Then the function is recalled in the model architecture with the rest of the layers (below is a section of the model architecture only):
x = MaxPooling1D(pool_size=2)(x) # previous layer
x = multiHeadedAttentionLayer(x) # attention layer
x = LSTM(lstmneurons, kernel_regularizer=regularizers.l2(kreg3_rate), dropout=dropout3_rate, recurrent_dropout=dropout4_rate)(x) # following layer
x = BatchNormalization()(x) # following layer

etc....
Based on my intuition the autoregression should take the output results and feed them back to the input at every time step, so my questions are:
Why do we need the masking technique?
How to implement the masking technique in this case?
Is there is a code for the autoregressive attention that I have a look at for reference?
Is my current intuition about autoregressive attention correct as shown in the diagram?

https://github.com/neqkir/attention-mechanism

Thanks for your comment. if you post it as an answer to the post then I will make it the correct one


text:
How to reject boxes inside each other with Non Max Suppression
I’m working on an object detection cnn, and having some issues with non max suppression. When I have a small box inside a large box, NMS is not rejecting the smaller, incorrect box, because its IOU is small (large union, small intersection). How is this scenario typically dealt with? When using out of the box pretrained models for object detection I don’t seem to get boxes completely inside other boxes. Example here:  green is ground truth, blue is prediction. The center box has a tiny blue box inside that’s not getting rejected by NMS

IOU makes sense for determining accuracy against ground truth, but for non max suppression have you tried intersection over minimum size?

Possibly but this isn’t standard practice. Standard COCO object detectors like SSD and FasterRCNN don’t seem to have such issues

then maybe there's something going on with your training data that you're getting these boxes. That, or should boxes be this small for your use case even? Maybe there's some class confusion going on.


text:
Text matching: fuzzy names matching with learning
I'm new to AI/ML and I want to research and learn about techniques that could help me to solve this complex task. Any hint would be appreciated.
Let me explain it with an example:
Let's look at two columns PUR.SUPPLY.MTL_REQ_HDR_ID and MTL.PO_REQUISITION_HEADERS_TAB.ID. It is likely they are related (one is the FK and the other one is PK).

As a human I was able to do it by doing the following:

I decoded abbreviations,
I identified context (MTL module),
I identified the subject (requisition header),
I identified ID keyword,
I identified irrelevant information (TAB postfix),
I matched words that are not in the exact order,
I estimated which elements/words do not have to match/occur,

I would like to match millions of columns relatively quickly (seconds). I would like algorithm to learn:

what words are likely context,
what are irrelevant,
what are subjects,
ideally learn some patterns (prefixes, postfixes, name formats, etc),
based on user responses - approve/reject match,
build dictionary of abbreviations,
estimate probability...

I know this is a complex task, but maybe you know a technique, tool, library, article, example ... anything that could be helpful? Any help would be appreciated.
Thanks.

In general Bayesian networks/Naive Bayes do well at text processing. As a library sci-kit has an implementation.

Thanks @A_Arnold, Naive Bayes seems to be a classification function. I don't think my problem is a classificaiton problem.

If you already have some of these matches you can try and scale it to new data. i.e. your pk & fk cols would be x1, x2, match would be y1=1 no match y1=0. Do this for some data, then let it predict some data, update that data if the classification is wrong etc. ALT: you may want to look into ICA which can predict topics. https://www.cc.gatech.edu/~isbell/papers/isbell-ica-nips-1999.pdf

thanks @A_Arnold. We will need to test the bayes and research the ICA. That might be a good start.


text:
CFD Reinforcement Learning Topology optimization wind tunnel
I want to create a reinforcement learning environment, designed for win tunnel simulations, where for each iteration a deep convolutional model could receive the 3D vector/scalar fields from the past simulation and output a better shape that maximizes the reward function (e.g. minimize drag, maximize lift, etc.)
The observation and action space for the neural network is the same, the inputs of the model will be 3D arrays representing velocity field, pressure field, etc. and the output will be a 3D array (created using Conv3DTranspose) with values [0, 1] which represents the mesh. I'm thinking that the architecture of the model could be something similar to an auto-encoder.
My plan is to use the algorithm of Marching Cubes in order to create the mesh from those points and openFoam for the CFD simulations.
This is a small diagram showing the workflow

The goal will be to have multiple trained models specialized in optimizing a particular reward function, like minimizing drag or maximizing lift, for any object/shape given as input.
What are your thoughts on this? Do you think it makes sense?


text:
LSTM Forecast Evolution
I have a confusion about the way the LSTM networks work when forecasting with an horizon that is not finite but I'm rather searching for a prediction in whatever time in future. In physical terms I would call it the evolution of the system.
Suppose I have a time series $y(t)$ (output) I want to forecast, and some external inputs $u_1(t), u_2(t),\cdots u_N(t)$ on which the series $y(t)$ depends.
It's common to use the lagged value of the output $y(t)$ as input for the network, such that I schematically have something like (let's consider for simplicity just lag 1 for the output and no lag for the external input):
$$
[y(t-1), u_1(t), u_2(t),\cdots u_N(t)] \to y(t)
$$
In this way of thinking the network, when one wants to do recursive forecast it is forced to use the predicted value at the previous step as input for the next step. In this way we have an effect of propagation of error that makes the long term forecast badly behaving.
Now, my confusion is, I'm thinking as a RNN as a kind of an (simple version) implementation of a state space model where I have the inputs, my output and one or more state variable responsible for the memory of the system. These variables are hidden and not observed.
So now the question, if there is this kind of variable taking already into account previous states of the system why would I need to use the lagged output value as input of my network/model ?
Getting rid of this does my long term forecast would be better, since I'm not expecting anymore the propagation of the error of the forecasted output. (I guess there will be anyway an error in the internal state propagating)
Thanks !


text:
What to do when you have massive amount of data but you don't have enough computation power for training a machine learning model?
For example, I have a massive amount of data, but I have limited computational resources and time to train on the full data. Other cases may include, I have huge amounts of 360-degree images, where I need to train on full-size images (without cropping down), but I have limited computation power (GPU, RAM, etc.), what can I do in those cases?

It's hard to answer this question without knowing what your goal is, but if your data is extensive, high quality, especially if it is labelled, and no similar dataset is publicly available, then publishing it freely with some kind of challenge could be very helpful if that's an option. Many organisations have the opposite problem: available computing resources but lack of data. If your data are new and interesting, I could imagine researchers wanting to use it. If people get good results, they may publish them, good for AI generally, and presumably also useful to you. If you do this of course you would need to publicise it a bit so people become aware of it.


text:
How to ensure that the ES-HyperNEAT algorithm generates an ANN in the substrate?
I'm trying to implement the ES-HyperNEAT algorithm using the original paper, as well as the pseudocode provided in the official user page. Occasionally, the algorithm would be unable to generate a network in the substrate. This happens when it finds no valid nodes that could connect a path between the input and output neurons.
I've noticed that this is highly dependent on how the hyperparameters (e.g., variance threshold and band threshold) were tuned.
Is my implementation correct, i.e., is this normal behavior? If so, is there a good way to ensure that a network is always generated (aside from directly connecting the input and output neurons)?

You can modify your fitness-funciton in a kind of "dirty" way:
Check while generating the fitness if all outputs are the same and than set the fitness to -10000 (or another very bad value depending on your normal fitness).
This way ES-HyperNEAT could/should learn to not generate this kind of networks.
PS: I do not have enough reputation to make this a comment.

that behavior can be expected depending upon how you set your thresholds as those are what determines whether a node is explored and assigned a connection. Assuming empty networks don't perform well in your environment genomes that don't find any connections will likely go extinct early on.


text:
How is complex systems research interacting with AI research?
Because future AI may produce emergent phenomena, and because these are probably gaps in our current understanding of this, it feels like complex systems may be an increasingly important research field.
Unless there is some kind of commonality of emergent behaviour (or aspects of complex systems more generally) it seems that future AI systems may behave in ways very difficult to predict. Potentially complex system research of the brain could help in artificial neural network understanding but because of the diversity of the latter this may be a loose similarity. Further, this may or may not hold for other types of AI.
The main question is from a basic educational level, how does complex systems research affect AI and vice vera. Possible subquestions (whatever helps understanding of this topic better): Is there much research at the intersection of AI and complex systems (so someone like Melanie Mitchell)? What kinds of things may complex systems research inform AI? Is AI now helping us understand AI better?

This seems like more of a discussion question, so I'm not sure it's a good fit for ai.stackexchange.

I don't really see the connection between complex systems and machine learning except in a broad sense. Anything with multiple parts that interact in nontrivial ways can be said to be a complex system, so things such as GAN, autonomous vehicles can certainly be called complex systems. However complex systems research deals with topics like swarms/collective behavior, which aren't really relevant to machine learning.

The question in this post is not really clear. You mention Melanie Mitchell - can you give an example of one of her papers that you think is at the intersection of AI and complex systems.

Ballpark would be something like the 1993 Revisiting the edge of chaos: Evolving cellular automata to perform computations.

@Taw: Swarms and collective behavior are topics of AI. For examples see papers presented at the International Conference on Intelligent Systems, Metaheuristics & Swarm Intelligence (ISMSI) http://www.ismsi.org/.

Revisiting the Edge of Chaos: Evolving Cellular Automata to Perform Computations,
Melanie Mitchell, Peter T. Hraber, and James P. Crutchfield, https://arxiv.org/pdf/adap-org/9303003.pdf


text:
How to enforce action bounds between 0 & 1 in soft actor-critic algorithm?
In the paper "Soft Actor-Critic Algorithms and Applications", appendix C shows enforcing action bounds using the tanh squashing function which is in (-1, 1). I have action bounds in (0, 1), so can I just modify the tanh output by applying the following transformation:
output = 0.5 * (tanh_output + 1). If so, do I need to change logprob formula too?
I have not seen any SAC implementation with different action bounds other than the paper's (-1, 1).

Yes you can map the output onto [0,1] as you indicate. You should treat this as a modification to the environment. I.e. imagine that the environment takes actions in [-1, 1] instead of [0,1]. No you don't need to change any equations.


text:
Feeding CNN FFT of an image, a dumb idea?
My dataset consists of about 40,000 200x200px grayscale images of centered blobs bathed in noise and occasional artifacts like stripes other blobs of different shapes and sizes, fuzzy speckles and so on in their neighborhood.
They are used in a binary classification problem, with emphasis on recall.
I read that using FFT of image and FFT of the convolutional kernel and multiplying the two, produces a similar result as convolutions would but at a way lower resource expense. This is probably the most straightforward article I found if you need a more detailed description(https://medium.com/analytics-vidhya/fast-cnn-substitution-of-convolution-layers-with-fft-layers-a9ed3bfdc99a)
What I want to do however is simply feed the FFT of images to the standard CNN. The reasoning being, maybe it would be easier for the network to catch on to features that it would miss or tend to weigh less. Or in other words, FFT as a feature engineering technique.
Would this be an idea worth trying to pursue?
If so, any suggestion on which FFT components to extract (Amplitude/Phase, Real/Imaginary)?

Lots of people do this for dimensionality reduction for somewhat mixed results.

You may be interested in checking out the discussions in this Kaggle competition. It's an audio classification challenge actually, but there the approach is to use FT to produce a spectrogram. You may get some useful pointers to help with what you're trying to do.

I don't think that is a dumb idea. This paper fed High-frequency components of the image and found interesting results.

 FFT is in essence linear transformation of the input image and can be represented by application of convolutional filter of the same size as image on the input.
Provided, the convolutinoal neural network is deep enough with sufficient number of parameters and there are skip connections (in order to have a path of purely linear transformations on the input), FFT can be represented by the learned filters. If FFT of the image is relevant for the classification problem, NN most probably would learn to produce them in a certain way.
For image classification problems - when the goal is identify an instance of something, local information is crucial, and this problem is better solved in the spatial, not frequency domain.
However, for your case it seems, like the semantic is rather trivial, and the goal is to get rid of some frequencies. Hence, working in the frequency domain is a sensible option. Possibly, you can combine the spatial and frequency representation in some way.
I think, it would be simpler to work with the real and imaginary part, that with the complex abs and phase, since you need to account for periodicity of the phase in a certain way, and then in the end transform phase to $e^{i \phi}$.


text:
Can Neural Networks using ReLU activation work without using the bias term in their neurons?
I created a super simple NN of 1 input, 2 hidden layers of 2 neurons each and 1 output neuron as shown below.

All activations are ReLUs and neurons doesn't use the bias term. What I found is that the output graph is a combination of two linear functions (one when the input is negative and another when the input is positive) kind of like this.

I think without the bias term, the output will be a linear function (for negative and positive inputs separately) no matter how big the network is. My question is, is this useful at all as an architecture? I assume it might be - if multiple output nodes are available, or is it? Does any of this mean that the a bias term is mandatory? Just trying to get my intuition right here...


text:
DeepLabV3: Why use global average pooling in the ASPP module?
I'm trying to understand the rationale of the various modifications the authors of the DeepLab models have made to their third version, DeepLabV3. In the paper, the following is written:

ASPP  with  different  atrous  rates  effectively  captures
multi-scale information. However, we discover that as the sampling
rate  becomes  larger,  the  number  of  valid  filter weights (i.e.,
the weights that are applied to the valid feature region, instead of
padded zeros) becomes smaller. This effect is illustrated in Fig. 4
when applying a 3×3 filter to a 65×65 feature map with different atrous
rates. In the extreme case where the rate value is close to the
feature map size, the 3×3 filter, instead of capturing the whole image
context, degenerates to a simple 1×1 filter since only the center filter
weight is effective. To overcome this problem and incorporate global
context information to the model, we adopt image-level
features, similar to [58,95]. Specifically, we apply global average
pooling on the last feature map of the model, feed the resulting
image-level features to a 1×1 convolution with 256 filters (and batch
normalization [38]), and then bilinearly upsample the feature to the
desired spatial dimension.

I do not understand how global pooling solves this problem. Is it simply because it does not suffer from the same issue of ASPP (the degeneration of the weights), and serves as an alternative?
From: Chen, L. C., Papandreou, G., Schroff, F., & Adam, H. (2017). Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587.


text:
Can we modelize an RNN by an ANN that takes precedent output as a part of input?
Is it possible to consider an RNN as a classical feedforward neural network that just take the precedent output as a part of the input ?

Almost. I think that to match the common interpretation of an RNN you need to also have a new input at each time-step (whereas you used the word "just" suggesting otherwise).
What you're describing is some function $f_1: \mathbb{R}^n \to  \mathbb{R}^n$ whereas the "conventional" RNN is more like $f_2: \mathbb{R}^m \times \mathbb{R}^n \to  \mathbb{R}^n$, where $\mathbb{R}^m$ refers to the new information at each time step.
Of course, this is all an exercise in interpreting semantics. So the other answer to your question is: "if you want it to"


text:
How to generate text descriptions from keywords?
I wonder how can I build a neural network which will generate text description from given tag/tags. Let's assume I have created such data structure:
{
 'tag1': ['some description1', 'some description2', 'some description3'],
 'tag2': ['some description4', 'some description5', 'some description6'],
 'tag3': ['some description7', 'some description8', 'some description9']
}

Then I would like to create a neural network which will generate randomly generated description based on given tags. For example:
INPUT: ['TAG1', 'TAG2', 'TAG3'] => OUTPUT: 'some description1. some description5 some description9'

Then I thought that it can be a good idea to implement a LSTM and doing text generation, but here I have a problem I know how I can do it for one tag. I can create one corpus of text contains different sentences for tag, then do the training and generate a sentence for given tag, but what If I have multiple tags should I create a corpus for each tag or maybe there is a better way to do that? If you know any articles which covers this problem, I would appreciate if you share them with me. If you have a neural network proposition which will solve this problem, I am also open for proposals.

PS. I know, I can solve this problem with easy Map, for example: ['tag1', 'tag2', 'tag3'].map(tag => tagSentenceMap.get(tag).randomChoice()).join('. ') but this is not the case for me.


text:
Finetuning solver for Caffe neural network
We're working on object detection in thermal images using neural network with Caffe framework. We use SSD ResNet-10 network available in OpenCV repository as it seems to provide the best performance on Raspberry Pi for our needs (in comparison to MobileNet etc.)
https://github.com/opencv/opencv/blob/master/samples/dnn/face_detector/solver.prototxt
train_net: "train.prototxt"
test_net: "test.prototxt"

test_iter: 2312
test_interval: 5000
test_initialization: true

base_lr: 0.01
display: 10
lr_policy: "multistep"
max_iter: 140000
stepvalue: 80000
stepvalue: 120000
gamma: 0.1
momentum: 0.9
weight_decay: 0.0005
average_loss: 500
iter_size: 1
type: "SGD"

solver_mode: GPU
random_seed: 0
debug_info: false
snapshot: 1000
snapshot_prefix: "snapshot/res10_300x300_ssd"

eval_type: "detection"
ap_version: "11point"

Train batch size is 16. Test batch size is 1.
The training process starts at loss 23.4728 and reaches plateau around loss 1.2, learning rate is decreased at iteration 80000 and loss falls down to 0.89. Further decrease continues very slowly to iteration 120000 where loss is around 0.85. Then LR is decreased. Process ends at iteraion 140000 with loss around 0.80 and test evaluation around 0.90.
I noticed that selecting different optimizer gives different results. I tried Nesterov, Adam (with fixed LR) and SGD with different base_lr (0.05) and step size (100000). Are there any recommendation that I could try except of trial&error and waiting 12 hours to compare the results? Reduce/increase batch size? More iterations? Different step sizes?
Adam provides the worst test evaluation. SGD with base_lr reduced to 0.05 and step size 100000 seems to provides the best result now (test eval = 0.94)


text:
Too slow search using MCTS in OpenAI Atari games
I'm recently using Monte Carlo Tree Search in OpenAi Gym Atari, but the result isn't satisfying.
Without render, the game lasts about 180 steps ( env.step() was called this much time )  with random agent. However, my MCTS agent only made the game last 12 steps. And it took pretty much time to give a next step.
I guess it's the problem of rollout. I build the MCTS tree using nodes containing AtariEnv objects, and deepcopy it each time I rollout, add the reward.
So it takes about 1 second to expand nodes and rollout, if I do 100 iterations, that would be massive waiting time.
My code of rollout is shown below:
def rollout_(current,if_render):
        '''
        current is going to be a Node object
        '''
        sandBox = deepcopy(current.state)
        endReward = 0
        done = False
        while done != True:
            action=sandBox.action_space.sample()
            _,reward,done,info = sandBox.step(action)# wierd return obs_next
            if reward > 0:
                reward *= 2
            endReward += reward-0.008
        return endReward

Anyone can help?


text:
Can an animal-level artificial general intelligence kickstart the Singularity?
Most people seem to assume that we need a human-level AI as a starting point for the Singularity.
Let's say someone invents a general intelligence that is not quite on the scale of a human brain, but comparable to a rat. This AI can think on its own, learn and solve a wide range of problems, and basically demonstrates rat-level cognitive behavior. It's just not as smart as a human.
Is this enough to kickstart the exponential intelligence explosion that is the Singularity?
In other words, do we really need a human-level AI to start the Singularity, or will an AI with animal-level intelligence suffice?

This is an interesting question, but it's difficult to answer it without just giving an opinion. So, what kind of answer are you expecting?

do we really need a human-level AI to start the Singularity, or will an AI with animal-level intelligence suffice?

The requirement from "theory" of the singularity is that:

The AI is able to design and implement a better AI than itself.

The trait of being able to design better than itself continues to apply in each iteration.

If both these things hold, then each generation of AIs will continue to improve. This is often assumed by singularity pundits to be an exponential growth curve e.g. each iteration makes +50% compound improvement on whatever measure is being made of intelligence. (Aside: Personally I find this a major weakness in the argument for the singularity being meaningfully possible, that it assumes this growth)
The first item of these two is important to your question. For the singularity to work, there is a baseline capability required - the AI needs to be able to design and build other AIs. A general intelligence at the level of an animal - at least any animal intelligence that we are aware of - does not seem capable of this task. It is not clear that humans are even capable of this task when the AI being built has to possess at least some general intelligence.
The term "animal-level intelligence" is tricky. The narrow AIs that we currently build can outperform animals and humans on specific tasks, but in terms of general intelligence they do not score highly (or at all). If we could build one that can outperform humans on a "building an AI" task, it might still have the general intelligence of an animal whilst having the capability to bootstrap an iterative process of self-improvement. This does seem like a very dangerous experiment to try though, with idiot-savant-deity AI and paperclip maximiser scenarios as possible outcomes because the AI's general intelligence lags behind its raw capabilities.

I like this answer, but do you really need the ability to design another more capable general intelligence? Isn't it enough to be able to reproduce/mate and create children that are more intelligent than you and that this holds throughout generations? If this was true, then animals could also evolve into more intelligent ones...

@nbro: A more pure evolutionary approach would lead to better fit to whatever environment the generations of child AIs find themselves in, but there may be no reason for them to gain greater intelligence. I believe that the idea behind the Singularity is that the improvements in intelligence are somehow directed, in the same way we direct machines to compute. In general we could not rule out an evolutionary approach, but I think that would also have to be directed in some way e.g. through designing an environment which would select based on intelligence.

If you assume that Darwinism is an accurate theory of why we exist, then we were created by evolution, so it's possible that evolution can still play a role, although it's also true that humans have been trying to create "intelligence" by designing it directly, as you put it.

Consider scientists creating a worm-level AI (which supposedly has happened, scientists have fully simulated a worm's brain, as far as I am aware); Now what? Is that simulation of 302 neurons going to rapidly explode and take over the world? Of course not! You need more than just the baseline intelligence, you require an AI not only with the intelligence to learn, but the infrastructure/capacity to advance to a point where it can then create more infrastructure/capacity for itself. That is when it will be able to explode in a singularity kind of event. This is all speculation of course, so take it as you will.


text:
Rotationally independent distributions
Maxwell's theorem states that multivariate normal distribution $\mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I})$ is the only distribution of a random vector that is invariant and have independent components after random rotations (orthogonal transformation).
Generally, many distributions are rotationally invariant and it's called spherically symmetric distributions. However, any non-normal spherically symmetric distribution has uncorrelated but dependent components.
Consequently, I am wondering if there are some distributions of vectors which are not neccessary to be rotationally invariant but have rotationally independent components.


text:
Taking a machine learning model to production\deployment
I've designed a machine learning model for the predictive maintenance of machines. The data used for training and testing the ML model is the data from various sensors connected to various parts of the machines. Now, I'm searching for a good approach for deploying the model in the real-time environment as explained here. I did some research and found some information about using real-time data for prediction such as using Kafka. I have some questions unanswered regarding the deployment of the ML model. Following are the details of my system:

The sensors (pressure, temperature, flow, vibration, etc) are deployed across the parts of the machines.
The ML model is trained with historical data.
For predictive maintenance (anomaly detection), streams of data will be available via MQTT. As there are 3000 machines, the volume of data will be very high.

My questions are:

Where will be the best place to perform prediction operation, at the factory premice where machines are located (edge computing), at our office (that designs ML model), or at cloud server? I want to know it in regard to operational cost.
Is there any way to estimate the effectiveness of the complete system (full-stack ML architecture)?

These seem to be more-so questions of machine learning engineering than machine learning theory. ML engineering is a cross between software engineering and data science, and the role (of the so-called 'machine learning engineer') is more-so one of implementing machine learning models (than conceiving of statistical models to be used for some purpose, as the data scientist does). I doubt you'll get answers to such questions on this website. ML engineering itself is relatively new and does not have much formal theory (compared to, say, pure software engineering). [...]

[...] This http://www.mlebook.com/wiki/doku.php is a recently published textbook on ML engineering (that I have purchased but not read) by Andriy Burkov; I suggest you read this as a starting point.


text:
Is it possible to create a simple face-tracking app that can monitor how much time one spends at their desk?
Context: I'm an experienced programmer with a graduate education in AI and previous CUDA programming experience. I'm versed in Machine Learning but am out of the loop -- I've not used any of the modern software packages of the last 10 years.
Question: Is it possible using modern AI software to easily create a face-tracking application that can use a webcam to track the amount of time spent at one's desk.
My environment is Fedora Linux. I also have an NVidia GTX 1660 for acceleration.
To make this question and answer precise, I've narrowed it to the following sub-questions:
As of June 2021,

Is there existing software that one can simply "set up" with a small amount of programming work (or none at all) that would facilitate training a video classifier from webcam recordings?

How does one provide training examples to this software, or how is data labeled? Does it provide some sort of GUI or accessory tool to label still frames or video sequences?

Does said software provide "hooks" or an event API so that one may invoke code on the event of e.g. a classifier edge?

Finally (and consider this optional), would it be realistic for a seasoned programmer to accomplish such a project using said software in about 30 hours? I understand that this is subjective -- just assume a graduate student in the AI field and ballpark terms. Or, answer in terms of the software's intended audience.

First posting in this community, so just offer guidance if you'd like to see this question refined.


text:
Vector input to CNN for object detection
I am training a 3D object detection network (Retinanet-based as of the moment) for re-detecting tracked objects. I would like to be able to add the velocity vector of the tracked object as an input to the detection network, as the velocity directly informs the direction along which the principal axis of the 3D bounding box should lie. I would like to include this information as early as possible (i.e. pass in as an input feature map) rather than simply adding it at the end with a few fully connected layers.
Is there a good or established way to encode such a vector in a feature map?


text:
What is a Silhouette Neural Network
I was going through a study in which I found something called a dilated Silhouette Neural Network. I want to know what it is, what it can do, and how it is better from a CNN?
Link to the journal: Link


text:
(explore-exploit + supervised learning ) vs contextual bandits
Lets take an ad recommendation problem for 1 slot. Feedback is click/no click. I can solve this by contextual bandits. But I can also introduce exploration in supervised learning, I learn my model from collected data every k hours.
What can contextual bandits give me in this example which supervised learning + exploration cannot do?

Hello. Can you please clarify what you mean by "I can also introduce exploration in supervised learning"? What's your idea more specifically?

I will learn a supervised model from the logging policy/model 's data and I will use it in a stochastic way which will promote exploration. If I do this iteratively(i.e. learn model with IPW supervised loss and collect data with exploration), i hope i can achieve what a batch learning with bandit feedback setup will also achieve. I will use inverse propensity weights in bandits and supervised learning loss as well.
then my question is that for this specific problem is, will bandit give me anything i cannot achieve with a IPW supervised loss.

the whole point is working in a practical setup and trying to learn from partial observations i.e. in recommendation you dont have data for what you dont recommend.
learned stuff from this ( http://www.cs.cornell.edu/courses/cs7792/2018fa/ ) . batch learning with bandit feedback seems exciting to me. But how do i convince myself that for simple problem as explained in the question exploration plus inverse propensity weighted supervised loss is equal to a batch learned bandit. IPW supervised loss will obviously require less iterations over data.

In the example in question description, i can have a softmax over candidates for that slot. exploration will let me try different candidates and hence collecting data by exploration.

I'm wondering the same question. Bandit is not "an algorithm", it is a modeling setup. Your bandit will follow a policy π.
You say your policy π is based on a supervised model (disregard exploration for now). I believe that your supervision signal (click) would directly translate to a reward that you get at time t. In non-bandit agents, RL would optimize for long term future rewards. Bandit is a one-episode game, so I think supervised learning is equivalent to RL.  The benefits you get by posing problem as bandit is you can reason about and control your exploration

No. If I am using IPW weighted supervised loss then actually cross-entropy is my negative reward and click is consumed in the loss.


text:
How to interpret the training loss curves in Soft-Actor-Critic (SAC)?
I am using stable-baseline3 implementation of the Soft-Actor-Critic (SAC) algorithm. The plotted training curves look promising. However, I am not fully sure how to interpret the actor and critic losses. The entropy coefficient $\alpha$ is automatically learned during training. As the entropy decreases, the critic loss and actor loss decrease as well.

How does the entropy coefficient affect the losses?
Can this be interpreted as the estimations becoming more accurate as the focus is shifted from exploration to exploitation?
How can negative actor losses be interpreted, what do actor losses tell in general?

Thanks a lot in advance

I am getting negative actor loss in SAC implementation from stable-baseline3. Anything on this?

@devilinthedetail that's expected and good, if expected return is positive :)


text:
Is there any dataset to convert text to sign language?
I'm going to start working on one university project and I would like to ask a question regarding it. My project is about "Sign language synthesis from NLP" and I need to develop an application where:

Take spoken language from user microphone
Recognize a word with an algorithm and convert words to sign language

Output should be images with sign language.
For instance, if we say "I go home", we should have images of those words in sign language.
My question is that is there any dataset you would recommend to get the images for the sign language?


text:
Book(s) on generative models
Generative models in artificial intelligence span from simple models like Naive Bayes to the advanced deep generative models like current day GANs. This question is not about coding and involves only science and theoretical part only.
Are there any standard textbooks that covers topics from scratch to the advanced?

From the theoretical foundations one can look into the Chapter 20: Deep Generative Models of the classic DL book by Goodfellow, Bengio https://amzn.to/2MmZNbH. Not the most recent reference, but written by the professionals in simple and accessible way.
There is a nice book Generative Deep Learning by D.Foster with some simple heuristics and probability theory motivations and examples https://www.google.ru/books/edition/Generative_Deep_Learning/RqegDwAAQBAJ?hl=en&gbpv=1&printsec=frontcover.
Finally, there is a book from Jason Brownlee (author of many nicely written articles on machinelearningmastery.com) https://machinelearningmastery.com/generative_adversarial_networks/


text:
Semantic-based evaluation of translations instead of BLEU
I have a text generation model and I want to evaluate its output by comparing it to a set of gold human-annotated references.
I went through machine-translation metrics and I found that BLEU is used as the main metric usually.
I didn't like using it because it's shallow as it uses ngrams comparison; the semantics of the translation is missed.
Is there any other metric to do a semantic-based evaluation?
I've thought of using a text similarity model to evaluate the output or even an NLI (Natural language inference) system. I am not sure how precise the evaluation will be because SOTA systems are not really accurate.

There are other possible metrics, e.g. meteor and BLEURT.
They compensate some of the basic problems most researchers would like to avoid BLEU for. The downside of not using known metrics is, that your model is even harder to evaluate against other candidates. If you compare to human gold standard corpus, you should not count on BLEURT too much since it is actually intended to evaluate two systems against a gold corpus and tell you which is better.


text:
Computational complexity of a CNN network
In the following network, the convolution operations of convolutional blocks are performed by three 1-D kernels with the sizes 8, 5, and 3 respectively along with stride equal to 1. The final network is constructed by stacking three convolution blocks with the filters of sizes 128, 256, and 128 in each block. Pooling operation is excluded from the network. I wanted to find the computational complexity of the following network. I was wondering if you could give me some hints to compute the computational complexity of this network. I appreciate your time! Thanks!

By "computational complexity", do you mean the time complexity of the forward pass?

@nbro I am curious to know how expensive this network is! Is it possible to know the overall complexity of this network? It would be nice to know the time complexity of the overall network as well.

There's a very similar question here: https://ai.stackexchange.com/q/22913/2444, to which I gave an answer where I give some hints. Let me know if your question is a duplicate of that one.


text:
How to handle cycles in minimax algorithm
For example, I am implementing AI for turn based game and have enough computational resources for build full game tree. My problem is the game can be infinite if both players will repeat moves and my minimax implementation stucks because game tree is infinite respectively.
For example, my game is in state S1, player 1 do action A1, player 2 do action A2 and we are again in state S1. I can't evaluate S1 node because I need to evaluate all subnodes.
I have no idea how to handle this.

Did you find a solution? I'm trying to do the exact same thing.

Indeed, if your game allows cycles, your tree is more of a graph.
A simple approach to avoiding infinite loops is to have a hard cap on how far down the tree you explore. If you reach it before the end of the tree, you can use a heuristic to compute your leaf value. But I understand you'd like to compute the full game tree.
First things first, how do you detect cycles? You can store explored game states and check whether you've encountered them. A way to do this relatively efficiently is to use Zobrist hashing to represent each game state.
What happens when you reach a cycle?

One simple approach is declaring a draw when you end up at a cycle and end the game there. After all, your "ideal" player should not make many moves to end up in the same position, so marking a cycle as a draw makes sense.
If you've stored your previous positions and already have a value for that position, you can simply use it as your leaf value.
You could try to discourage moves leading to cycles altogether. If you detect a cycle, you could penalize that move so your player doesn't go through it.


text:
Upscaling a low-res IR image with a high res webcam image
I have a low resolution thermal/IR image (for example 32x32 or 80x64) and a high resolution webcam image. I would like to combine the two to "fake" a high resolution thermal image (I can already map them together via homography). One could probably just apply a FLIR-like palette to the IR image, scale it up, and combine it with the brightness channel of the visible spectrum image. But that would of cause visible artifacts at the pixel edges of the IR picture.
I wonder if there is an AI based approach to colorize the webcam image with the IR data. When a warm IR pixel partially covers a person and partially the background, it would only color the "person" warm, and take the "background" color from the neighboring IR pixel. For this it would have to consider a small vicinity of either picture at a time.
Although I'm familiar with machine learning in the context of multivariate analysis and classification, I have no experience with modern deep learning or AI based image processing. I would guess that something like style transfer would be a starting point for what I'm trying to achive. One would need 1) a way to identify features (like foreground/background, person/wall) and 2) a way to combine these features with the IR truth to result in a colorized bitmap, I assume.
What would be the best approach to do this? Maybe this is already a solved problem - I have a feeling this might already be a solved problem. In any case I would be grateful for literature pointers.

This sounds like a segmentation task. Take a look at U-Net. If you have one-to-one IR and webcam images, you can then upscale the resulting map to the webcam image. This also could be done with the Pix2Pix network (which is also based on U-Net)

Thanks, U-Net and Pix2Pix look pretty promising. The only issue is that they are focused on segmentation, whereas I don't think I always have clear boundaries. In my case there can also be "gradients" (think a wall diagonally to the camera) or soft bodies. But it is probably a good starting point.


text:
Conditional input deep neural network
I need to input data conditionally to my deep network. In order to explain cases, I'd like to give an example. Assume that I have a 50-attribute dataset. For some attributes, a specific part of hidden layers is responsible, and for others, a different part is responsible. Also, for some cases, the same parts of the hidden layers might intersect. I think I can decide which attributes must go which hidden neurons in the input layer by using some kind of if-else block. However, I could not figure out how.

My current idea

I can enter an identity element for some attributes. For example, I have att1, att2, att3, etc. I have ins1, ins2, etc.
For ins1 -> att1 = 0.5, att2 = 0.2, att3 = None
For ins2 -> att1 = 0.1, att2 = None, att3 = None
But, if I do this approach, the number of attributes for an instance becomes bigger unnecessarily.

End of my current idea

Are there any opinions on this? Should I rearrange my excel file or is there any way to use if-else conditions?
Regards,

This is very confusing, so to clarify: You have (or want?) a neural network that has that capacity to accept all 50 attributes, BUT is able to only accept a subset of them at a time? So you can input attributes [2,5,7] and nothing else? Or am I misunderstanding

Hello. Welcome to AI SE. Could you please put your specific question in the title (to give an idea of what your question is to people that scroll down the list of questions)?


text:
Using a rectified Tanh to train a CNN?
I have been experimenting with activation functions on CNN, and it occurred to me to use a rectified tanh function. So that is basically if z > 0 tanh(z) else 0. I have implemented it and I compared with ReLu on odd MNIST. They both achieved about 94% success rate in 10 epochs. My logic was that usually humans tend to stop feeling more confident once they have learnt something. Similarly, I thought a Convolutional layer neuron should not feel much more confident (higher activation) with growing evidence (higher weighted input). So is there any evidence of perhaps such a rectified tanh being more successful ?

I don't have any evidence, but I would say probably not. Ultimately you're trying to fit a function, and a ReLU does it so fast because of how easy it is to combine ReLU functions and also how simple it is to compute (in the forward and backward pass). I don't think you'll have much luck getting much more accuracy using a rectified tanh, but if you compare convergence time or computation time I think you'll find that a simple ReLU outperforms by quite a margin.

Yes I agree. I should mention that mathematically, rectified tanh would make it harder for the neuron to change its parameters as they grow larger (if they start out positive). This appears a useful characteristics as long you have small parameter initialization. I think it would allow you to get the extra percent or two in accuracy. For computation, perhaps a rectified arctan would perform better since its derivative is simpler than tanh.


text:
Doubt in Sutton & Barto's off-policy Monte Carlo control algorithm
The algorithm is described as below:

My understanding: In the third last step, we act greedily w.r.t $Q$. Since we use importance sampling, this $Q \approx Q_\pi$. However, in the next step, whenever $A_t \neq \pi(S_t)$, it means the behavior policy isn't aligned with the target (greedy) policy. Hence, we can't use importance sampling and for such $(S_t, A_t)$ we simply take the average of $Q(S_t, A_t)$. Which means these $Q$ values aren't estimates of $Q_\pi$ but rather $Q_b$.
What's been bothering me is when the behavior and target policy eventually align for state $S_t$, won't that alignment be incorrect? Because in the previous step, we would be doing:

$\pi(S_t) = \arg \max [Q_\pi(S_t, a_1), Q_b(S_t, a_2), Q_b(S_t, a_3)]$

assuming $A(S_t) = \{a_1, a_2, a_3\}$ and the true greedy action is $a_1$.

Could you clarify "We simply take the average of $Q(S_t,A_t)$."? What is this average taken over, and where is it used? AFAICS, the only Q update step uses weighed importance sampling. I guess you are making some interpretation of this update function in the case you describe ($\pi(S_t) \neq A_t$ after the update step), but I am not sure what you are thinking

If $\pi(S_t) \neq A_t$, $W$ will always be 1. Hence, when I revisit the $(S_t, A_t)$ in a new episode, my $C(S_t, A_t)$ will increment by 1. And my $Q(S_t, A_t)$ will be the running average. But this $Q$ for $(S_t, A_t)$ is an estimate of $Q_b$ as the behavior policy isn't aligned with target policy.

OK, yes. That's actually covered in the book under weighted importance sampling. The initial estimates in weighted importance sample are biased towards action values of $Q_b$ but will converge on $Q_{\pi}$ - and specifically the first sample will also be an unbiased estimate of $Q_b$. The bias is unwanted but usually preferable to the potentially unbound variance of basic importance sampling.


text:
KL divergence coefficient update doesn't make sense in RLlib's PPO implementation
I am using RLlib (Ray 1.4.0)'s implementation of PPO for a multi-agent scenario with continuous actions, and I find that the loss includes the KL divergence penalty term, apart from the surrogate loss, value function loss, and entropy.
The KL coefficient is updated in the update_kl() function as follows:
    if sampled_kl > 2.0 * self.kl_target:
        self.kl_coeff_val *= 1.5
    # Decrease.
    elif sampled_kl < 0.5 * self.kl_target:
        self.kl_coeff_val *= 0.5
    # No change.
    else:
        return self.kl_coeff_val

I don't understand the reasoning behind this. If the point of the KL "target" is to reach the target, then why do the conditions above imply that the KL coefficient is getting larger (multiplied by 1.5 when the sampled KL is already found to be larger than the target?) when it is supposed to be made smaller instead? I feel like I am missing something here, but I am not able to get my head around it.
I would appreciate any insights on this. Thank you.

The loss in PPO has a penalty
$$-\text{kl_coeff_val}\cdot\mathrm{KL}(\pi^{\text{NN}}||\pi^{\text{Target}}).$$
Increasing this coefficient means a lower reward, so it incentivizes a smaller KL divergence. Likewise, decreasing it allows a higher KL divergence.

Please note that the official paper introduces two different algorithms for proximal policy optimization: PPO-CLIP (Sec. 3) and PPO-Penalty (Sec. 4)
The algorithm that you are describing is PPO-Penalty and the objective function is given by the following equation:
$$ L^{KL} = \mathbb{E} \bigg[ \frac{\pi_\theta (a|s)}{\pi_{\theta_{old}} (a|s)} A(a,s) - \beta \text{KL}(\pi_\theta, \pi_{\theta_{old}}) \bigg]$$
Here $\beta$ is not a fixed hyperparameter, but is a Lagrangian multiplier for the constraint. In this case the constraint is that $\text{KL}(\pi_\theta, \pi_{\theta_{old}}) \leq KL_{tgt}$.
During optimization you want to increase the multiplier if the constraint is violated, and lower it otherwise.
Why do we want to do that?
If your constraint is violated too much, you want to increase the multiplier $\beta$ so that the next update step will pay more attention to this term and will try to take smaller update steps on the policy network. If the constraint is not violated, then you decrease the multiplier $\beta$ so that we can take larger update steps on the policy network.
In theory you should converge to an optimal coefficient $\beta$ such that you take the largest update step on the policy network, such that the constraint is not violated. In this case the constraint will be fulfilled exactly.


text:
How can I use a prediction model (e.g., ARMA model or LSTM) for multi-variate data?
I have a question
I have had a dataset below
             sensor1   sensor2  sensor3 ...
2021-01-01    1.32       2.2      1.0
2021-01-02    4.3        2.0      0.8 ...
...

I know ARMA model is useful for time-series forecasting

However, how can I use ARMA model for data with multiple attributes ?
If data with a single attribute can be a input for ARMA model, should I aggregate the attribute set?
(for example, after normalizing each attribute, I add up all values every rows to transform all attributes to a single attribute)

I don't think you need to go for aggregation -- this looks like a job for VARIMA, the vector-version of ARIMA. In ARIMA, the output of the sequence at time $t$, which can be notated $X_t$, is a function of the past inputs $\{X_1, X_2, \dots, X_{t-1}\}$. For a univariate $AR(k)$ process, the corresponding ARIMA model is given by
$$ X_t - \sum_{i=1}^k \alpha_i X_{t-i} = \varepsilon_t + \sum_{i=1}^k \theta_{i}\varepsilon_{t-i}$$
with parameters $\alpha_i, \theta_i$, inputs $X_i$, and i.i.d. zero-mean Gaussian error terms $\varepsilon_i$. The generalization of this to multiple variables is thus simply
$$\mathbf{x}_t - \sum_{i=1}^k \mathbf{A}_i \mathbf{x}_{t-1} = \mathbf{e}_t + \sum_{i=1}^k \mathbf{\Theta}_i \mathbf{e}_{t-i}$$
for $\mathbf{x}_i, \mathbf{e}_i \in \mathbb{R}^n$. Note that before, the parameters were scalars. Now, $\mathbf{A}_i, \mathbf{\Theta}_i \in \mathbb{R}^{n\times n}$ -- size-$n$ square matrices.
It looks like there's a Github implementation here as well, though I haven't looked closely at this.
If you're going for an LSTM-based sequence modeling approach, this is even easier -- an LSTM cell can take in input of arbitrary dimensions, so you shouldn't have to make any changes.
If you'd like to see the math, concretely, the LSTM forward pass equations have the form
$$(\cdot)^{(t)} = g(\mathbf{W}^{(\cdot)} x^{(t)} + \mathbf{U}^{(\cdot)}x^{(t)} + \mathbf{b}^{(\cdot)})$$
where $\mathbf{W}, \mathbf{U}$ are the parameter matrices associated with the inputs and hidden states, respectively, for each gate, and $\mathbf{b}$ is a bias term. So in the single-variable case, $\mathbf{W}, \mathbf{U}, \mathbf{b}$ are all scalars; in the multivariate case, $\mathbf{W}, \mathbf{U}$ are now size-$n$ square matrices, and $\mathbf{b}$ is a vector of length $n$. No further modification is needed, and you should be able to just plug-and-play with most deep learning libraries.
So your inputs would just be the vector of attributes at a particular time step.


text:
What does it mean by "zeros the networks parameters gradients" in the context of training a neural network?
Consider the following PyTorch code
# Run a sample training loop that "teaches" the network
# to output the constant zero function
for _ in range(10000):
  input = torch.randn(4)
  output = net(input)
  loss = torch.abs(output)
  net.zero_grad()
  loss.backward()
  optimizer.step()

and its corresponding explanation on training a neural network
A training loop…

acquires an input,
runs the network,
computes a loss,
zeros the network’s parameters’ gradients,
calls loss.backward() to update the parameters’ gradients,
calls optimizer.step() to apply the gradients to the parameters.

Code contains net.zero_grad() which has been explained as zeros the network’s parameters’ gradients.
What does it mean by zeros the networks parameters gradients? In general, loss is back propagated by calculating the gradients of loss wrt parameters. But, I didn't understand the phrase "zeros of networks parameters gradient". What does that particular step do?

In the automatic differentiation procedure after backward pass
the gradient with respect to the scalar is added to the current gradient. Without calling zero_grad you will have the sum of all gradients, calcluated before, with the current gradient.
Therefore, optimizer.step() will do not this:
w = w - eta * grad L[i] # L[i] - loss function for the i-th sample

But rather:
w = w - eta * sum_i(grad L[i]) # sum of gradient with respect to all samples

Which is not the desired behavior.

Worth noting that this is an implementation detail in PyTorch, and other frameworks may do the same implicitly. It is a nice feature when there's more than one source of gradients, but for simple NNs it's just some boilerplate code that needs to be written by the framework user

So, if we don't use net.zero_grad() then the updation of weights will go wrong because of issue in updation of gradients. Updation of weight takes the sum of all the gradients calculated so far instead of the current gradient.

@hanugm, yes, precisely, it can be beneficial to use gradient information from the previous timesteps, like in the momentum optimizers - Nesterov, Adam, but not in tihs way. With the decay of the importance of gradients from previous timestamps with time.

I agree with the answer from @spiridon_the_sun_rotator .
If you don't zerorize the gradient, before calling loss.backward()
then gradient will be accumulated which is not what you want. you want to calculate the gradient associated with the current step.

That answer has already been given.


text:
Would the reward normalization be wrong in early episodes?
It's confusing me that how can we normalize the reward without actually knowing the true mean and variance of the reward distribution, specifically, at the early steps and episodes. This may cause problem for the RL algorithms that use the replay buffer such as DDPG, because this wrongly calculated rewards can stay in buffer for too long and the network will adapt with them. Is there something that I am missing or misunderstood? For algorithms with replay buffer, using standardization is better that normalization?


text:
What to do when model stops learning after some epochs
I am training a segmentation model on 3D data, after around 170 epochs which took around 4 days, I notice the model is no more learning and the dice score is at 0.51. What is the best approach at this point to keep model learning?
Learning rate: 1e-4
Batch size: 6
optimizer: adamw
loss: generalized dice loss
PS: there are more augmentation in training data than validation data, this might be the reason, the validation loss curve is below training loss curve

There are hundreds of things you could try: Muck around with hyper-parameters, use a learning rate scheduler, change the model structure, more augmentation, less augmentation. It's just a matter of experimentation. I'd suggest using a grid search on a smaller sample that doesn't take 4 days to train, then use those values on the real training set.

Additional to @Recessive's response, very specifically, go for smaller learning rate. As he suggested, you can use learning rate scheduler for that, which will reduce learning rate as epochs pass.

i already tried reduce on plateau but tht did not helped much


text:
Positional Encoding in Transformer on multi-variate time series data hurts performance
I set up a transformer model that embeds positional encodings in the encoder. The data is multi-variate time series-based data.
As I just experiment with the positional encoding portion of the code I set up a toy model: I generated a time series that contains the log changes of a sine function and run a classification model that predicts whether the subsequent value is positive or negative. Simple enough. I also added a few time series with random walks to try to throw off the model.
Predictably, the model very quickly reaches a categorical accuracy of around 99%. Without positional encoding that happens already in the 3rd epoch. However, with positional encoding (I use the same implementation as proposed in the "Attention is all you need" paper), it takes over 100 epochs to reach a similar accuracy level.
So, clearly, all else being equal, learning with positional encoding takes much longer to reach an equal accuracy level than without positional encoding.
Has anyone witnessed similar observations? Apparently adding the positional encodings to the actual values seems to confuse the model. I have not tried concatenations yet. Any advice?
Edit: Or does it simply mean that learned positional encodings perform better than sin/cos encodings? I have not made any special provisions to encourage learned positional encodings, I simply either added the positional encodings to the actual values or I did not.

I wonder what kind of positional encoding you used. If you just add, like I think the Transformer paper did, then it makes perfect sense that it's harder to train because that basically scrambles all the word vectors based on position. Perhaps concatenating could work better, but you'd have more weights to learn.

@user253751, which then begs the question of the need for postional encodings. I definitely reach higher accuracy levels and that a lot earlier without positional encodings as applied to multivariate time series


text:
How to implement a rule-based decision maker for an agent-based model?
I had no idea that there is a stack exchange community for A.I. :-/ So I repost this question here in hope of some guidelines. I tried to delve into the materials discussed in AI: A Modern Approach course book, I am struggling to wrap my head around the model I'm trying to build without some code examples to aid me fill some gaps.
I have a hard time understanding how to combine a rule-based decision making approach for an agent in an agent-based model I try to develop.
The interface of the agent is a very simple one.
public interface IAgent
{
   string ID { get; }

   void Execute(IAgentMessage message,
                IActionScheduler actionScheduler);
}

For the sake of the example, let's assume that the agents represent Vehicles which traverse roads inside a large warehouse, in order to load and unload their cargo. Their route (sequence of roads, from the start point until the agent's destination) is assigned by another agent, the Supervisor. The goal of a vehicle agent is to traverse its assigned route, unload the cargo, load a new one, receive another assigned route by the Supervisor and repeat the process.
The vehicles must also be aware of potential collisions, for example at intersection points, and give priority based on some rules (for example, the one carrying the heaviest cargo has priority).
As far as I can understand, this is the internal structure of the agents I want to build:

So the Vehicle Agent can be something like:
public class Vehicle : IAgent
{
  public VehicleStateUpdater { get; set; }

  public RuleSet RuleSet { get; set; }

  public VehicleState State { get; set; }

  public void Execute(IAgentMessage message, IActionScheduler actionScheduler)
  {
    VehicleStateUpdater.UpdateState(VehicleState, message);
    Rule validRule = RuleSet.Match(VehicleState);
    VehicleStateUpdater.UpdateState(VehicleState, validRule);
    validRule.Fire(this, VehicleState, actionScheduler);
  }
}

For the Vehicle agent's internal state I was considering something like:
public class VehicleState
{
  public Route Route { get; set; }

  public Cargo Cargo { get; set; }

  public Location CurrentLocation { get; set; }
}

For this example, 3 rules must be implemented for the Vehicle Agent.

If another vehicle is near the agent (e.g. less than 50 meters), then the one with the heaviest cargo has priority, and the other agents must hold their position.
When an agent reaches their destination, they unload the cargo, load a new one and wait for the Supervisor to assign a new route.
At any given moment, the Supervisor, for whatever reason, might send a command, which the recipient vehicle must obey (Hold Position or Continue).

The VehicleStateUpdater must take into consideration the current state of the agent, the type of received percept and change the state accordingly. So, in order for the state to reflect that e.g. a command was received by the Supervisor, one can modify it as follows:
public class VehicleState
{
  public Route Route { get; set; }

  public Cargo Cargo { get; set; }

  public Location CurrentLocation { get; set; }

  // Additional Property
  public RadioCommand ActiveCommand { get; set; }
}

Where RadioCommand can be an enumeration with values None, Hold, Continue.
But now I must also register in the agent's state if another vehicle is approaching. So I must add another property to the VehicleState.
public class VehicleState
{
  public Route Route { get; set; }

  public Cargo Cargo { get; set; }

  public Location CurrentLocation { get; set; }

  public RadioCommand ActiveCommand { get; set; }

  // Additional properties
  public bool IsAnotherVehicleApproaching { get; set; }

  public Location ApproachingVehicleLocation { get; set; }
}

This is where I have a huge trouble understanding how to proceed and I get a feeling that I do not really follow the correct approach. First, I am not sure how to make the VehicleState class more modular and extensible. Second, I am not sure how to implement the rule-based part that defines the decision making process. Should I create mutually exclusive rules (which means every possible state must correspond to no more than one rule)? Is there a design approach that will allow me to add additional rules without having to go back-and-forth the VehicleState class and add/modify properties in order to make sure that every possible type of Percept can be handled by the agent's internal state?
The examples I've seen in the Artificial Intelligence: A Modern Approach course book and in other sources are too simple for me to "grasp" the concept in question when a more complex model must be designed.
I would be grateful if someone can point me in the right direction concerning the implementation of the rule-based part.
I am writing in C# but as far as I can tell it is not really relevant to the broader issue I am trying to solve.
An example of a rule I tried to incorporate:
public class HoldPositionCommandRule : IAgentRule<VehicleState>
{
    public int Priority { get; } = 0;

    public bool ConcludesTurn { get; } = false;

    public void Fire(IAgent agent, VehicleState state, IActionScheduler actionScheduler)
    {
        state.Navigator.IsMoving = false;
        //Use action scheduler to schedule subsequent actions...
    }

    public bool IsValid(VehicleState state)
    {
        bool isValid = state.RadioCommandHandler.HasBeenOrderedToHoldPosition;
        return isValid;
    }
}

A sample of the agent decision maker that I also tried to implement.
public void Execute(IAgentMessage message,
                    IActionScheduler actionScheduler)
{
    _agentStateUpdater.Update(_state, message);
    Option<IAgentRule<TState>> validRule = _ruleMatcher.Match(_state);
    validRule.MatchSome(rule => rule.Fire(this, _state, actionScheduler));
}

For this question:
Is there a design approach that will allow me to add additional rules without having to go back-and-forth the VehicleState class and add/modify properties in order to make sure that every possible type of Percept can be handled by the agent's internal state?,
I think that you can implement a class called  policy or stragegy, with different subclasses, and each object could have a different strategy.
To construct a kind of mindset refering to multi-agent simulations I think that you can find some inspiration in http://cormas.cirad.fr/.
I have worked in some related problems, and it should be useful to look at https://agritrop.cirad.fr/541188/