text:
Difference in average rewards between taking random actions and following random policies
I wrote two programs that simulated 10000 episodes in gym environment CartPole-v0.
The first program takes random moves in every steps in each episode. The average reward over 10000 episodes is 22.1582.
The second program uses a random policy in each episode. For each episode, initialize a 4 by 2 matrix $M$ with random numbers from a uniform distribution on $[0,1)$ that maps state observations to action values. Then choose the action with higher value in each step. The average reward over 10000 episodes is 46.8291.
The linear mapping given by $M$ covers only a portion of the search space so it seems the way actions are selected in the first program is more "random" than the second program. How can we go about explaining the huge discrepancy in the average rewards obtained by the two methods?

Cartpole has far more than 4 by 2 state/actions. How are you mapping from state to lookup of action value in your 4 by 2 table? I would guess that the 2 corresponds to "move left, move right" evaluation. But how are you altering the state in order to look it up in the table?

Matrix $M$ is not a lookup table. It is a matrix that maps a vector of size 4 (representing the state) to a vector of size 2 (representing the action values)

So you are using the matrix to multiply the state vector? Essentially that is a linear value approximation. Except the values will be meaningless/wrong, so essentially it's a randomised policy with a few inflection points between left/right actions

Yes they are randomly generated policies. Basically, I am just wondering why randomly generated policies, on average, perform better than taking random actions in terms of the average returns from each episode.


text:
How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?
I've seen plenty of examples of people doing Sigmoid + MSE backpropagation implementations, yet I do not seem to understand how to implement backpropagation as stated in the title in the case of multi-class classifications.
What confuses me mainly are the matrix-vector shapes and their multiplications, and their implementations in code.

There's no shortage of tutorials on the matter. You need to be more specific about what gaps they didn't bridge

@Alex what confuses me mainly is how the matrix shapes are matched, during backpropagation I saw the derivative of softmax being multiplied element wise with the gradient of the loss function. However since softmax's gradient is a Jacobian - a 2D matrix, I am confused as to how this is multiplied with a 1D vector

I don't know if this will really be useful, but maybe take a look at Notes on Backpropagation by Peter Sadowski.

Alright. Consider an ordinary neural network, so, in the last layer, we have, $z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]}$, where $a^{[L]} = \sigma(z^{[L]})$, where $\sigma$ is the softmax activation:
$$
\sigma(\mathbf z)_{i} = \frac{e^{z_i}}{\sum_k e^{z_k}}
$$
I think, one of the most effective ways of not to get confused about all these matrices with different dimensions, is to simply get rid of the matrices and do all calculations componentwise. So, calculating the $ij$ component of the jacobian matrix, we have:
$$
J_{ij} = \frac{\partial\sigma_i}{\partial z_j} = \frac{e^{z_i}}{\sum_k e^{z_k}}\delta_{ij} - \frac{e^{z_i}e^{z_j}}{(\sum_k e^{z_k})^2}
$$
Consider a cost $C$, and we wish to calculate the derivative with respect to the weights of the last layer, that is, we are running the first step of backpropagation. Applying chain rule to the $ij$-component of the weight:
$$
\frac{\partial C}{\partial W^{[L]}_{ij}}
= \sum_k\frac{\partial C}{\partial a_{k}}\frac{\partial a^{[L]}_{k}}{\partial W^{[L]}_{ij}}
$$
And, with one more chain rule:
$$
\frac{\partial a^{[L]}_{k}}{\partial W^{[L]}_{ij}} =
\sum_s\frac{\partial a^{[L]}_{k}}{\partial z^{[L]}_s}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
$$
Now we can identify exactly where the jacobian matrix is:
$$
J_{ij} = \frac{\partial\sigma_i}{\partial z_j} = \frac{\partial a^{[L]}_{i}}{\partial z^{[L]}_j}
$$
Thus, the back-propagation step is just:
$$
\frac{\partial C}{\partial W^{[L]}_{ij}}
= \sum_k\sum_s\frac{\partial C}{\partial a_{k}}\frac{\partial a^{[L]}_{k}}{\partial z^{[L]}_s}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
= \sum_k\sum_s\frac{\partial C}{\partial a_{k}}J_{ks}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
$$
Now, in component language, it is much simpler to identify where are all the matrix multiplications: The transpose of the gradient with respect to the cost function, multiplied by the jacobian matrix of the softmax activation (or whatever activation), and finally, the last derivative, which will evaluate to something depending on the activation of the previous layer.
=].

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What algorithm would you advise me to use for my task?
I have an image and a mask. I want the image to be the same, but rotated, scaled and positioned like mask. What can I use?

Assuming that the image is blank everywhere but where the face is drawn...
The first step is to scale the image to the mask.  That doesn't require a detailed explanation here as it is too trivial a problem.
Second, rotate the image by 90 degrees three times and save each one.
Third, for the four versions of the image (the original and three rotations), do the image addition.  The only one with information will be the one that is rotationally aligned with the mask.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Can DQN outperform DoubleDQN?
I found a similar post about this issue, but unfortunately I did not find a proper answer. Are there any references where DQN is better than DoubleDQN, that is DoubleDQN does not improve DQN ?

This indeed seems to be a duplicate of the other question, although maybe not an exact duplicate because you're also asking for references, so I will leave it open.


text:
Predicting continous value with CNN (prediction of fruit maturity)
I want to train some IA algorithm to be able to evaluate the maturity of a fruit (say, measured in numbers of days before rotten) based on an image of the fruit.
My first instinct is to go with convolutional neural network (CNN), since those have proven very efficient for recognizing images. However, I am not sure what the output layer should look like in this case.
I could separate the data into a bunch of classes (1 day left, 2 days left, 3 days left, etc.) and use one output node for each of these classes, as in an usual classification task, but in doing so I completely lose the continuous nature of the output, which makes me think it might not be the optimal way to proceed.
Another option would be to just have a unique output node, whose activation would correspond to the continuous value to predict, here the number of days left (normalized appropriately to lie between 0 and 1). This would have the advantage of taking the continuity into account, but I have been told that neural networks aren't made to predict values in that way, they really are best suited for classification into discrete classes.
What do you think would be the best way to proceed? Is there another way to nudge a neural network so that its output is continuous? Or maybe CNN just aren't suited for this task? If you have any suggestions of other algorithms that would be efficient for this kind of task, I would be happy to know them.

Just to make sure. Do you actually have continuous-valued labels corresponding to the input training data? Also, I am not sure what would per-se be bad about generating a probability distribution over a discrete set of classes represented by the output nodes. As long as you set granularity of the discrete classes to an appropriate level and train the classifier in that way, you would achieve your goal as well. Or is there a reason that you really need continuous outputs (except for subjectively preferring it)?

My labels are integers corresponding to the number of days. I just feel like we're losing some information by representing the maturity as a discrete unordered set of classes, I feel like somehow the NN should know that if the answer is 2 days then predicting 3 days is a much better prediction than 10 days. But maybe I am wrong and the NN will just learn this by itself (I don't have any specific reason for preferring continuous output beside that).

but I have been told that neural networks aren't made to predict values in that way, they really are best suited for classification into discrete classes

I don't agree with this statement. I already trained many CNNs for regressions tasks where a continous output is trained and they generally perform very well.
I think the general "advantage" for a classification approach over a regression approach is that there is usually some margin where the output of the NN can still be clipped to a specific class, which might not be the case in regression tasks. Furthermore a lot of the "better classification performance" is due to the fact that the cross-entropy loss of classification tasks always tries to increase probability of one class while reducing all others. But this advantage might not be really important in your case where a specific range of days (e.g 3-5 days) can be probable instead of just one single day (e.g. exactly day 4).
Thats why I think you should defenitly try the regression approach. For this just normalize/clip your output layer to the min/max range of days. You should also defenitley decrease the learning rate at the end of training, I found this much further improves regression predictions to be more accurate.
And lastly if you want to achieve higher performance and more robustness, in your case I highly recommend a bayesian approach where you average over an ensemble of NNs (this ensemble can be your single NN with different parts truned off via Dropout layers during inferencing). With this average you can get an estimate of the uncertainty (the variance over different predictions) which gives you a much better idea of the probability distribution over the different days for a specifiy input.

Thanks, this is very interesting. I'll wait a bit to see if there are other answers and if not I'll award you the bounty. I'll try to read on the bayesian approach, I don't know much about it (if you have any reference I'd be happy to see it).


text:
Are monotonically increasing functions easier to learn?
A monotonically increasing function is a function that as x gets bigger so does its output. So, if plotted, it will never go down. Although the outputs might stay constant.
Logically this seems like an easier function to learn when compared to something that can, when plotted, go up or down.
Wikipedia has some example diagrams on monotonic functions.
If I were to say that it is easier for a neural network to learn a monotonic function compared to a non-monotonic function would the statement be correct? If so, is there any reason to it other than 'it only goes one way'?

This paper Monotonic Networks (1997), by Joseph Sill, may be of interest to you.


text:
How to make SAC (Soft-Actor-Critic) learn a policy?
I cannot make SAC learn a task in a certain environment. The point is that it actually sometimes finds a very good policy, but it never learns the policy in the end. I am using the SAC implementation from stable-baselines3, which is correct as far as I have seen.
I have an environment driven by complex dynamics. I have to control a variable to be in a certain range. Every time the variable goes out of minimum or maximum range the environment is done. The action is continuous (between 0 and 30). My goal is to keep the variable (1D) in the range for as long as possible (millions of steps per episode would be ideal). There are certain characteristics of the environment that may make it particular:

The action can only drive the variable to lower values. The variable can go up as a result of the environment dynamics (not controlled) and as a consequence of certain events (not controlled) that occur at random intervals.
The observation is a noisy sample of the variable. The observation is just a real number.
The effect of actions in the variable is usually delayed. That is, applying an action does not immediately lower the value of the variable.

I have tried SAC with many different hyperparameters. It sometimes find very good policies, policies that last for thousands and even millions of steps in evaluation or rollout. But it never learns such policies. Even saving the policy in those cases, they are not able to produce a lone episode later. In the attached image, it can be seen that during the training (in some evaluations) the policy is able to run for thousand of steps. But then it never learns that. I only show 500K here steps but I have run test for 1.5 million training timesteps.

So, my question is (I have several ones actually):

Is SAC not suitable for this problem? I have also run TD3 and PPO but without better results and SAC is the only one actually able to find those  policies that make very long episodes. Any other algorithm?
I have tried several reward functions, and, in the end, a simple one that gives 1 for every step and 0 when done is the one that seems to give better results. In the image, the reward is one for every step and -100 when done.
Since the values of the variable are time correlated due to the dynamics, I have also tried with RNN actors (with TF Agents), but results do not improve.
I cannot see any relationship between the actor loss and critic loss and the results (maybe that is my problem). The loss seem to be larger when the episodes are longer (which is what I want).

Any advice is highly appreciated. Thanks


text:
Is it possible to train one part of the network with a particular learning rate and the other part with a different one?
I have a combined network consisting of two parts: one is for images and the other is for numerical data. Each sample is matched with a numerical case by an ID. For this combined network, a lr of 0.01 was found to be best working via hyperparameter tunings
However, when I trained them as a separate task (binary classification), a lr of 0.001 for images and 0.01 for numerical data were best.
As for an AUC metric, the combined network (0.818) is performing on average of image and numerical networks (0.799 and 0.821, respectively).
Here I thought maybe the combined network's lr is too high for image part and should apply lower lr for that part. But, I don't know if it is possible.
If anyone has any idea of what is what, let me know

Is this a programming issue/question or a conceptual/theoretical one? You tagged your question with PyTorch, so maybe you're looking for a solution in PyTorch and not really for a conceptual answer. If yes, then this question is off-topic here, and should be asked on Stack Overflow, because it would be a programming issue, which is off-topic here. Please, let me know, because this post was voted to be closed as off-topic, probably because of that.

it is a conceptual one, I tagged pytorch just in case someone has done something similar in pytorch

Create two different optimizers and split the subnets' parameters into either with different lrs. You will have to call optimizer1.step(), optimizer2.step() with a single backward() call


text:
How to implement very simple move-ordering for alpha-beta pruning
I've done implementing alpha-beta, and transpositional table on my search tree algorithm so I decided to implement move-ordering next. But once I implemented it, it's way more longer to respond than before?
Here's my code so far:
function sortMoves(chess)
{
    const listA = [], listB = chess.moves({ verbose: true });
    const scores = [];

    // calc best moves
    const moves = [...listB];
    for (const move of moves)
    {
        const state = chess.move(move, {
            promotion: 'q'
        });
        scores.push(evaluate(chess.board()));
        chess.undo();
    }

    // sort move
    for (var i = 0; i < Math.min(5, moves.length); i++)
    {
        let maxEval = -Infinity;
        let maxIndex = 0;

        for (var j = 0; j < scores.length; j++)
        {
            if (scores[j] > maxEval)
            {
                maxEval = scores[j];
                maxIndex = j;
            }
        }

        scores[maxIndex] = -Infinity;
        listA.push(moves[maxIndex]);
        listB.splice(maxIndex, 1);
    }

    const newList = listA.concat(listB);
    return newList;
}

I am expecting for this to respond quicker than before but it turns out it's not. So my question is am I actually sorting the moves correctly? or should I write some code for the alpha-beta pruning related to the sorted moves?
Here's my negamax function:
function negamax(chess, depth, depthOrig, alpha, beta, color)
{
    // transposition table look up
    const alphaOrig = alpha;
    const hashKey = zobrist.hash(chess);
    const lookup = transposition.get(hashKey);
    if (lookup)
    {
        if (lookup.depth >= depth)
        {
            if (lookup.flag === EXACT)
                return lookup.score;
            else if (lookup.flag === LOWERBOUND)
                alpha = Math.max(alpha, lookup.score);
            else if (lookup.flag === UPPERBOUND)
                beta = Math.min(beta, lookup.score);

            if (alpha >= beta)
                return lookup.score;
        }
    }

    if (depth === 0 || chess.game_over())
    {
        // if current turn is checkmated,
        // remove the king on the board
        // so the AI knows if the move
        // will lead to checkmate or not, if
        // it's remove on the board,
        // the checkmated team will
        // reduce the king's value leading
        // the AI to move in checkmate
        const kingPos = getPiecePos(chess, 'k', chess.turn());
        if (chess.in_checkmate())
            chess.remove(kingPos);

        const evaluation = evaluate(chess.board());
        chess.put({ type: chess.KING, color: chess.turn() }, kingPos);

        return color * evaluation;
    }

    /* let moves = chess.moves();
    if (lookup)
    {
        console.log(moves, depth)
        const bestMove = lookup.move;
        const moveIndex = moves.indexOf(bestMove);
        const arr = moves.splice(moveIndex, 1);
        moves = arr.concat(moves);
        console.log(moves, depth)
    } */
    const moves = sortMoves(chess);
    /* const moves = chess.moves({ verbose: true }); */

    let count = 0;
    let score = -Infinity;
    let bestMove = null;
    if (lookup)
        bestMove = lookup.move;

    for (const move of moves)
    {
        const state = chess.move(move, {
            promotion: 'q'
        });
        searchedMoves++;
        if (count === 0)
            score = -negamax(chess, depth-1, depthOrig, -beta, -alpha, -color);
        else
        {
            score = -negamax(chess, depth-1, depthOrig, -alpha-1, -alpha, -color);
            if (alpha < score < beta)
                score = -negamax(chess, depth-1, depthOrig, -beta, -score, -color);
        }
        chess.undo();

        if (score > alpha)
        {
            alpha = score;
            bestMove = move;
        }

        // do I add something on this part?
        count++;
        if (alpha >= beta)
            break;
    }

    // transposition table store
    const key = zobrist.hash(chess);
    keyArr.push(key);
    const entry = new Transposition();
    entry.score = score;
    entry.depth = depth;
    entry.move = bestMove;
    if (score <= alphaOrig)
        entry.flag = UPPERBOUND;
    else if (score >= beta)
        entry.flag = LOWERBOUND;
    else
        entry.flag = EXACT;
    transposition.set(key, entry);

    return alpha;
}
```

Are you saying that it's taking too long to sort the moves?

Yes, when I sort moves, it takes longer than before (which has no sorted moves).

No, my question was: what is taking too long: 1. sorting the moves, 2. minimax after sorting the moves? Also, I didn't look at your code and it's been a while I had to deal with minimax, but what would it mean to apply minimax after having sorted the moves? Can you explain a little bit what your idea is? Also, how long does it take?


text:
Is it possible to ensure the convergence when training a RNN weight on its SVD decomposition?
I'm reading the following paper in which the author seems to do 2 things interesting:

The hidden-to-hidden weight matrix of the RNN is SVD decomposed and train separately.
Each orthogonal part of the decomposition is optimized multiplicatively according to Cayley Transformation to maintain its orthogonal properties.

Now, I'm not so strong with the math behind the technique, but I could be hand-waving and say that albeit being multiplicative, it is just another method of gradient descent, and each orthogonal part is still minimizing the Loss function. So far so good.
But what they are doing is actually split the original optimization problem into multiple sub-optimization (2 for orthogonal matrices and n for the number of singular values), and then multiplied the result together. How can we be sure about the convergence and the optimality of such method? Or is this the case where we can say nothing and let the experiment speak for themselves?


text:
For binary classification learning problems, how should I label instances where I'm only 60% sure?
I've come across a few binary classification problems lately where the labelling was challenging even for an expert. I'm wondering what I should do with this. Here are some of my suggestions to get the ball rolling:

Make a third category called "unsure" then make it a three-class classification problem instead.
Make a third category called "unsure" and just remove these from your training set.
Make a third category called "unsure" and during training model this as a 0.5 such that the binary cross entropy loss looks like $-0.5\log(\hat{y})-0.5\log(1-\hat{y})$
Allow the labeller to pick a percentage on a sliding scale (or maybe multiple choice: (0%, 25%, 50%, 75%, 100%), and take that into account when calculating cross entropy (as in my point above).

I recently saw a paper which goes for option 2, although that's not enough to convince me. Here's the relevant quote:

In case of a high-level risk, collision is imminent and
the driver must react in less than 0.5 s (TTC < 0.5s). For
low-level risk, the TTC is more than 2.0 s (TTC > 2.0s).
Videos that show intermediate-level risk (0.5 s ≤ TTC
≤ 2.0 s), which is a mixture of high- and low-level risks,
were not included in the NIDB because when training
a convnet, it must be possible to make a clear visual
distinction of risk.

Your problem seems to be related to weakly supervised learning (WSL). Maybe you get an answer to your question by reading about WSL.

Duplicates:   https://stats.stackexchange.com/questions/74042/classifier-for-uncertain-class-labels and  https://stats.stackexchange.com/questions/218656/classification-with-noisy-labels


text:
How to build a test set for a model in industry?
Most of the tutorials only teach us to split the whole dataset into three parts: training set, develop set, and test set. But in the industry, we are kind of doing test-driven development, and what comes most important is the building of our test set. We are not given a large corpus first, but we are discussing how to build the test set first.
The most resource-saving method is just sampling(simple random sampling) cases from the log and then having them labeled, which represents the population. Perhaps we are concerning that some groups should be more important than others, then we do stratified sampling.
Are there any better sampling methods?
What to do when we are releasing a new feature and we cannot find any cases of that feature from the user log?

I am not sure whether that solves your problem at hand, but one approach you could look into is k-fold Cross Validation (CV).
In this approach, you split your combined train, development, and test data into $k$ randomized and equally-sized partitions.
Afterwards, you train and evaluate your model $k$ times. In the $i^{th}$ iteration, you train your model on all but the $i^{th}$ partition. After training on the $k-1$ partitions is done, you evaluate your model on the $i^{th}$ partition. You repeat this process for all $i \in {1, 2, ..., k}$.
To be clear, you keep your initially randomized partitions fixed during k-fold CV.
Then, you take the average performance over all $k$ performed test runs to assess the quality of your model based on the averaged test performance. Afterwards, you could train your model on all train, development, and test data and deliver the resulting model as your final one.
In the most extreme case, you would perform Leave-One-Out-CV, where you set $k$ equal to the number of all the data points at your disposal. That is the most expensive approach, but yields the most accurate performance estimates.
For more information, see this website.
Generally, using that approach, you don't waste any data by reserving it exclusively for development/testing. Also, it might be important to mention that this approach is compatible with other sampling techniques as well. For example, in the $i^{th}$ iteration of your CV algorithm, you could apply stratified sampling to your $k-1$ partitions used for training during that ($i^{th}$) iteration.
I am not entirely sure whether I get the second part of your question right.
If it is about how to later introduce the new feature to a given model, I would say the following.
When it comes to introducing new features, I think you are pretty much out of luck with respect to recycling your old model. Of course, (assuming that the introduction of new features to the existing model is technically possible) there might be types of models which allow continual learning under certain circumstances, but in the worst case that might cause catastrophic forgetting since you change the distribution of your underlying training data when adding new features, which not all models might be able to deal with.
An example of this case is when you add more diverse training images for a given Convolutional Neural Net (CNN), which the CNN then has to learn to map to an already existing set of classes.
In other cases, introducing new features might not even be technically possible if their introduction would require adding new input (or output) nodes to an existing model.
However, when your second part of the question asks for how to fill gaps in your older data, caused by missing data, there are different strategies you could try for imputing the missing data, some of which are briefly mentioned here.

Your description of k-fold CV seems good to me, apart from one thing that is not clear to me from your description: in k-fold cross-validation, as far as I remember, you have only 2 datasets (training dataset and test dataset), so I guess that the validation dataset doesn't come into play here. Given that you read that article, maybe you're currently more familiar with the details of the k-fold CV and tell me if I'm wrong or not. Anyway, I'm not sure the question was about assessing the quality (e.g. the generalization) of the model or how to train the model given a fixed dataset.

The question seems to be more about "Which sampling techniques should I use to build a test dataset". I guess the question arises because the OP may not know how to build the test dataset (to test, I suppose, the generalization ability of the model) given a stream of data or something like that (they mention a "log"). It seems that they are also concerned with the work that needs to be done to label the data from the log, so, in this regard, your second paragraph seems to address that. So, maybe you could edit your answer to address the question in the title more directly.

In any case, I could be wrong regarding what the main question here is or what are the main issues of the OP.

If Lerner clarifies what he exactly means, I could of course delete the second part if it's not relevant. Regarding the first part, I only mentioned the dev/validation dataset, since it was mentioned in the original question as well. I just wanted to state that actually all available data is used for performing k-fold CV.

Ok, let's then wait for some clarification.


text:
Estimating dimensions to reduce input image size to in CNNs
Considering input images to a CNN that have a large dimension (e.g. 256X256), what are some possible methods to estimate the exact dimensions (e.g. 16X16 or 32X32) to which it can be condensed in the final pooling layer within the CNN network such that the important features are retained? I have found references to using linear dimensionality estimates (such as PCA) and the Riemannian Metric for non-linear estimation, but am not confident of how accurate the predicted dimensions may be.
One paper that explores this issue in Deep Neural Networks in a better way can be found here. Answers specifically pertaining to processing of SAR images would be more helpful.


text:
How to predict multiple set of coordinates (of bounding boxes) for signboards text localization through neural network?
I am creating a signboard translation application from scratch. I have images of signboards where there are multiple texts and I have the corresponding set of coordinates of bounding boxes for multiple texts. I want to create a regression model which will try to predict the coordinates if there is some text in the image. I am really stuck at a place. In some cases, I have multiple words in the image, so each word will have its own set of coordinates. So, how can I make a model such that if there is a single word then it will output single set of coordinates, but if there are 5 words then it should give me 5 set of coordinates? The number of output may vary with each image. What kind of neural net should I use? I don't want to use sliding window approach. Please help me out.


text:
How are Ground truth provided to each Pyramid map in RetinaNet or YOLOv3 Paper? How is the mapping of Feature Pyramids done to Ground Truth
SO the YOLO V3 and RetinaNet both uses the Feature pyramids which look something like this:
(except b and e which have one output)
I'm just confuse how the predictions and training is done?
Do we have to give EACH feature map a different Y label? IF yes, how is that possible? We need to have N different ground truth in my opinion. (Also ther'll be 3 different losses I think?)
If not, then how are these done at once?
There is a lot of confusion on these networks because I am not able to get my head around How are y-labels provided, trained and predicted in YOLOv3 and RetinaNet . Everything will make sense about loss, multioutputs and all if I know this one thing.

great question man, in short they do a heck of a complicated things to map boxes to anchors and then to tensors. Moreover each of the different approaches use a different strategy to map anchors, so the answer to your question is not short.

Long would do too ;)


text:
Single-value loss/training in a CNN with a tensor output
I am playing around with an idea of using using Q-learning with a DQN (Deep Q-Network), to determine the optimal position of a number of 'units' on a grid of allowed locations, according to some reward-metric that is only calculated after placing all of the units. I am following much of Deep Convolutional Q-Learning with Python and TensorFlow 2.0, but I diverge with the grid output.
To do this, I use a CNN that takes in a one-hot encoded grid of the units (0=no units here, 1=one unit here), and outputs a grid with the same shape as the input with expected rewards of placing a unit at this location.
I can use the following Tensorflow/Keras code to get the expected rewards, where units can be placed in 3 dimensions and channels determining the different unit-styles:
from tensorflow.keras import layers, models

model = models.Sequential(name="DQN")
model.add(
    layers.Conv3D(
        filters=10,
        kernel_size=3,
        activation="relu",
        padding="same",
        input_shape=input_shape,
        bias_initializer="random_normal"
    )
)
model.add(layers.Conv3D(filters=10, kernel_size=3, activation="relu", padding="same"))
model.add(layers.Conv3D(filters=input_shape[-1], kernel_size=3, activation="relu", padding="same"))

model.compile(optimizer="adam", loss=tf.keras.losses.Huber(), metrics=["accuracy"])

Currently I am using a very simple training scheme, where Q-values are first generated from the current state. At the position where the agent earlier placed a unit, the calculated true reward is given and trained against.
If the following state was a terminated state, the calculated reward is used directly, while the discounted reward is used in non-terminated states.
for state, action, next_state, reward, terminated in minibatch:
    # state: one-hot grid
    # action: index in the state where a unit is placed by the agent
    # terminated: True/False whether the 'next_state' is the terminated state

    q_target = model.predict(state)

    if terminated:
        q_target[0][action] = reward
    else:
        following_target = model.predict(next_state)
        q_target[0][action] = reward + gamma * np.amax(following_target)

    model.fit(state, q_target, epochs=1, verbose=0)

This means, that only a single value in the entire training tensor is the true reward - all other are approximated by the CNN.
However, all of the expected rewards are used in training, instead of this singular value. So I was considering whether it would be possible to train the CNN towards this single value only, and whether it would make any sense at all?
I thought of creating a custom loss function that would calculate the loss function for this single action, so training is done against this. However, I can't really figure out how I would go about doing this. I've looked at something like Custom training with tf.distribute.Strategy, but I wasn't successful at it..


text:
How to go about classifying 1000 classes?
I am trying to find research paper with theory(preferably implementation) that is about classifying 1000 (or more) classes. I have heard of an implementation, that initially clustering needs to be done then classification with something like softmax. Does anyone know of any research paper that implements 1000+ class classification.

If you are asking for arbitrary ML task dealing with 1000+ classes the most straigtforward thing that comes to mind is the ImageNet - https://en.wikipedia.org/wiki/ImageNet#cite_note-nytimes_2012-2. It has more the 20k categories in the present time.
In order to perform such classification task you need

A large enough dataset, such that each class occurs frequently enough or efficient data augmentation
A powerful model, such that can adapt to variety of different problems

The first neural network to tackle this problem was AlexNet and since then plethora of architectures : VGG's, ResNet's, EfficientNet got better and better quality.
They have a lot of filters that react on different patterns.


text:
How to scrape product data on supplier websites?
I'm currently trying to build a semantic scraper that can extract product information from different company websites of suppliers in the packaging industry (with as little manual customization per supplier/website as possible).
The current approach that I'm thinking of is the following:

Get all the text data via scrapy (so basically a HTML-tag search). This data would hopefully be already semi-structured with for example: name, description, product image, etc.
Fine-tune a pre-trained NLP model (such as BERT) on a domain specific dataset for packaging to extract more information about the product. For example: weight and size of the product

What do you think about the approach? What would you do differently?
One challenge I already encountered is the following:

Not all of the websites of the suppliers are as structured as for example e-commerce sites are → So small customisations of the XPath for all websites is needed. How can you scale this?

Also does anyone know an open-source project as a good starting point for this?


text:
What machine learning model should I use for a random dice-based game?
Consider a game like Pig (https://en.wikipedia.org/wiki/Pig_(dice_game)), but with a few additions: namely functions of both player's score and turn number that have unique impacts on scoring.
What machine learning model should I use to try and get the optimal number of dice roles per turn (say number of dice roles are bounded between 1 and 10)?
I was reading this tutorial: https://towardsdatascience.com/playing-cards-with-reinforcement-learning-1-3-c2dbabcf1df0, and they suggested reinforcement learning with a Q value function. I don't know how this work though, because turn number isn't bounded, but also needs to be a parameter to the Q value function. Multiplying the range of all parameters suggest this Q value function needs 2,000,000 states. Is this too many? - I have no idea how to judge this.
Is there a better model I should use to try and solve this problem which at its core takes the parameters of (my_score, opponent_score, turn_number) and should return a number 0 - 10 representing how many dice to roll.

bump, still curious on what models would be useful


text:
Beating iterative alpha beta search in Isolation Game
I'm having trouble beating an AI in Isolation game: https://en.wikipedia.org/wiki/Isolation_(board_game) with 3 queens on a 7x7 board. I tried applying alpha beta iteration with a scoring function on the state. I tried 3 scoring functions:

(0.75/# moves taken) * number of legal moves - (# moves taken/0.75) * number of opponent legal moves
number of legal moves - 3 * number of opponent legal moves
3 * number of legal moves - number of opponent legal moves

The first is an annealed aggressive strategy so the agent gets more aggressive as the game goes longer. The 2nd is a pure aggression strat and the last is a pure defensive strat. None of them consistently beat standard alpha beta iteration with the state scoring function: number of legal moves -  number of opponent legal moves. They all broke roughly even.
Any suggestions of scoring state functions or search algorithms are appreciated. The search has a limit of 6000 seconds per turn though.

What is your main specific question here? Please, choose one specific question and put it in the body and title of your post to clarify what you're asking here.

I'm simply asking for a search algorithm with a cost function that can do better than iterative alpha beta

Is your question somehow related to this: https://ai.stackexchange.com/q/5174/2444?


text:
Multi class text classification when having only one sample for classes
I have a dataset of texts, each text was identified with an ID number. I would like to do a prediction by finding the best match ID number for upcoming new texts. To use multi text classification, I am not sure if this is the right approach since there is only one text for most of ID numbers. In this case, I wouldn't have any test set. Can up-sampling help? Or is there any other approach than classification for such a problem?
The data set looks like this:
id1 'text1', id2 'text2', id3 'text3', id3 'text4', id3 'text5', id4 'text6', . . id200 'text170'
I would appreciate any guidance to find the best approach for this problem.

won't be able to classify if there are too many kinds of ids

and theoretically, won't be able to classify, if the data are singlesample --to--> multiple ids; it must be manysamples --to--> single id

If your texts are simple (have same words/phrases/subparts) then approaches like edit distance might work (levenshtein distance or any similar thing). If they are more complex, such as different words but similar meaning, then you can use pretrained models like Bert and use them to get embeddings and classify based on distance of embeddings

Siamese networks may be useful in your case.
http://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf
https://en.wikipedia.org/wiki/Siamese_neural_network
https://link.springer.com/protocol/10.1007%2F978-1-0716-0826-5_3

Thank you very much! It seems really useful. I hope it works on my text data as well considering the texts in my datasets are kind of messy complaint data.


text:
Embedding Isolation game states into key values for RL
I'm trying to think of how I can embed a game's state into a unique key value. The game I'm specifically working with is Isolation: https://en.wikipedia.org/wiki/Isolation_(board_game). The game state has the coordinates of player 1's pawn, coordinates of player 2's pawn, coordinates of free spaces and coordinates of already used spaces. Is there a way to embed this into a unique key value? My plan is to generate a dict and use that for value iteration with RL to learn the optimal value function for every state.

I think that there are too many game states in that game for you to use value iteration. The upper bound for a simple concise representation would be $49^2 \times 2^{49}$. That is:

$49^2$ covering all possible locations of the two players.

$2^{49}$ covering whether each square exists or has been removed.

In this scheme, many combinations are not feasible, as pieces do have to be on existing squares and cannot share a space. However, this doesn't drop the number of allowed states by a significant amount (several or more orders of magnitude) that would make it worth a more complex representation, or put the game into reach of dynamic programming solutions.
Potentially, a version on a 4x5 grid would be small enough to fit in memory and be solvable with dynamic programming. That would have a few million valid states.
In terms of an id code, you could use a 64-bit unsigned integer, reserving 49 bits for the existence of the squares, and for simplicity giving 6 bits each to the locations. This would also be a valid state representation for actually playing the game efficiently, most programming languages support the bit manipulations that you would need in order to maintain the representation. However, it would need a separate expanded representation if you were to create neural network features for approximate value functions.
In terms of writing a solver for the 7x7 version, I would recommend combining a look-ahead planner such as negamax, with an approximate neural network-based reinforcement method, perhaps DQN for simplicity. The neural network would provide a backup "best guess" solution when you could not look far enough ahead in the early stages of the game.
On a quick search for the true number of states in this game, I found an introduction to using minimax on a smaller version that you may find useful.

How about Monte Carlo tree search? Either way no matter the method I need a way to embed the state into a key. Any suggestions? Also I believe representation is 49482^47

Also why do you think Value Iteration will work? It should be able to handle this? It'll just take a long time.

@user8714896 Regarding 48*49 yes that's a better upper bound for that part of the state  space, but it is awkward to encode and not worth the effort for the space it saves (as per my second paragraph). MCTS would be fine, as would many other approaches, I just suggested a couple. Regarding value iteration "it'll just take a long time" - do the maths and you will see you need memory space of thousands of terrabytes. If you can arrange that, then yes it will just take a long time (years on current CPUs)

actually what stops MCTS from not requiring more space than in computationally feasible? Is there just a depth cut off? If you don't have a horizon it can take as much space or more than value it because you're also storing # of visits and relations between child states and parent.

@user8714896 MCTS typically only stores a small sub-tree out of the whole game tree, even with the evaluation horizon being the end of the game.

but it's not guaranteed to be of a reasonable computational size cause part of the learning is expanding nodes unless there's a guaranteed with UCB that it'll stop at some point due inadvertently pruning states not worth remembering.

@user8714896: You have a lot more control in MCTS over the size of the space explored. It's definitley more feasible than value iteration for your problem. I see you have asked another question about MCTS, so perhaps try to clarify there or in a new question.


text:
Using one-class classification first to find anomalies then apply multi-class classification
I'm new to machine learning and trying to apply it for fault detection, an idea came to mind which is using only anomaly detection after which if the results after a while come up as positive, a multi-class classification algorithm (using 7 different classes) is used to classify the fault type. would that be efficient and saves on resources power?


text:
What is the reverse of passing a Turing test by a human pretending to be a robot that can't be identified?
A Turing Test is a method of inquiry for determining whether or not a computer is capable of thinking like a human being. In an ideal Turing test, it would be clear to differentiate between a real human being and a robot or AI with human characteristics.
However, it is also possible in a Turing test that a human tries to mimic the behaviour of a computer so that the person applying the test cannot distinguish between a human being and a robot/AI.
Is this a concept that is explored much in computer science? As in research into the variations of Turing tests that can be used to identify whether a human is trying to mimic or impersonate as a robot or AI.

@DuttaA isn't psychology our attempt to define the 'rules' of human behaviour? I ask this question because there are also psychological conditions (some of which are physical and affect areas of the brain) that can cause people to lose particular cognitive functions. As a result, they may not be able to recognize faces, have triggers that cause either verbal or non-verbal reactions that are less controllable, or other behaviours that even normal people consider as being 'robotic' or 'abnormal'. As for your point, even using a test of speed would still depend on the type of question right?


text:
Confusion about computing policy gradient with automatic differentiation ( material from Berkeley CS285)
I am taking Berkeley’s CS285 via self-study. On this particular lecture regarding Policy Gradient, I am very confused about the inconsistency between the concept explanation and the demonstration of code snippet. I am new to RL and hope someone could clarify this for me.
Context
1.The lecture defines policy gradient as follow:

log(pi_theta(a | s)) denotes the log probability of action given state under policy parameterized by theta
gradient log(pi_theta(a |s)) denotes the gradient of parameter theta with respect to the predicted log probability of action
2.The lecture defines a pseudo-loss.By auto differentiate the pseudo-loss, we recovery the policy gradient.

Here Q_hat is short hand of sum of r(s_i_t, a_i_t) in the Policy gradient equation under 1)

The lecture then proceeds to gives a pseudo-code implementation of 2)

My confusion

From 1) above,
gradient log(pi_theta(a |s)) denotes the gradient of parameter theta with respect to the predicted log probability of action , not a loss value calculated from a label action and predicted action.
Why does the below in 2) implies that gradient log(pi_theta(a |s)) just morph into output of loss function instead of just predicted action probability as defined in 1) ?

2.
In this pseudo-code implementation,

Particularly, this line below.
negative_likelihoods = tf.nn.softmax_cross_entrophy_with_logis(labels=actions, logits=logits)

Where does the actions even coming from ? If it comes from collected trajectory, aren’t the actions result of logits = policy.predictions(states) to begin with ?
Then won’t tf.nn.softmax_cross_entrophy_with_logis(labels=actions, logits=logits) always return 0 ?

Based on the definition of policy gradient in 1), shouldn’t the implementation of pseudo-loss be like below ?

# Given:
# actions - (N*T) x Da tensor of actions
# states - (N*T) x Ds tensor of states
# q_values – (N*T) x 1 tensor of estimated state-action values
# Build the graph:
logits = policy.predictions(states) # This should return (N*T) x Da tensor of action logits
weighted_predicted_probability = tf.multiply(torch.softmax(logits), q_values)
loss = tf.reduce_mean(weighted_predicted_probability )
gradients = loss.gradients(loss, variables)

Hello. Welcome to Artificial Intelligence Stack Exchange. I would suggest that, rather than saying in the title that you are confused, you should put in the title your main specific question that you have. An example of a question is "Why is this equation X inconsistent with this implementation?" or something like that. That's what I mean by question.


text:
Is (log-)standard deviation learned in TRPO and PPO or fixed instead?
After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, I assumed that this would be common practice nowadays in the domain of Deep Reinforcement Learning (DRL).
In the supplementary material associated with the paper introducing Trust Region Policy Optimization (TRPO), however, it is stated that:

A neural network with several fully-connected
(dense) layers maps from the input features to the mean of a Gaussian distribution. A separate set of parameters specifies the log standard deviation of each element. More concretely, the parameters include a set of weights and biases for the neural network computing the mean, $\{W_i , b_i\}_{i=1}^L$ , and a vector $r$ (log standard deviation) with the same dimension as $a$. Then, the policy is defined by the normal distribution $\mathcal{N}(\text{mean}=\text{NeuralNet}(s; \{W_i , b_i\}_{i=1}^L), \text{stdev}=\text{exp}(r))$.

where $s$ refers to a state and $a$ to a predicted action (respectively a vector of actions if multiple outputs are generated concurrently).
To me this suggests that the standard deviation stdev (being a function of $r$) is actually not learned when training a TRPO agent, but that it is solely determined by some possibly constant vector $r$.
Since I found the idea of adjusting both the mean and standard deviation together when training a REINFORCE agent quite reasonable, I got wondering whether it is actually true that TRPO agents do not treat the standard deviation for sampling output values as a trainable parameter, but just as a function of the state-independent vector $r$.
(Pretty much the same shall then apply to Proximal Policy Optimization (PPO) agents as well, since they are reported to follow TRPO's model architecture in the continuous output case.)
In search for an answer, I browsed OpenAI's baselines repository containing reference implementations of both TRPO and PPO.
In my understanding of their code, the code seems to confirm my assumption that standard deviation is a non-trainable parameter and that it is, instead of being trainable, taken to be a constant.
Now, I was wondering whether my understanding of the procedure how TRPO (and PPO) computes standard deviation(s) is correct or whether I misunderstood or overlooked something important here.

Hi, have found a confirmation? I have the same question as yours.

It’s a modelling choice. You can learn it, or fix it to a value of your choosing.

Using PPO from stable baselines 3, I see clearly that the variance of the actions are reducing in the environment I am using, where the optimal actions are static and not dependent on the state. This seems to suggest that the log_std parameter is being learned for the SBL3 implementation.
In the docs for the SBL3-PPO, the description says log_std_init is the initial value which also suggests it is a learned parameter.
log_std_init (float) – Initial value for the log standard deviation
Of course, it is ultimately a design choice whether you want to make this parameter learnable.
Update: Upon inspecting the PPO paper again, it is stated that they using linear annealing of the log_std parameter. However, when I inspect the log_std parameter in my experiments which start at 0 and has 3 dimensions, the values are not the same. This leads me to think there is both annealing and learning at the same time.


text:
What is the consensus on the "correct" temperature settings for the AlphaZero algorithm?
In the AlphaZero learning algorithm, during self-play to generate training games, the move played is chosen with probability proportional to the MCTS visits raised to the $\tau$-th power, where $\tau$ is the so called temperature. Higher temperatures correspond to more exploration. It seems that in deepmind's original paper (on AlphaGo Zero if I'm not mistaken) it is mentioned that temperature is decayed to zero after move 30 in Go/Baduk, then this is contradicted in the AlphaZero with it saying that temperature is not decayed at all, and finally in AlphaZero's pseudocode I believe it is implied that the temperature is decayed after some number of moves. Specifically I believe that lczero concluded that they decayed after 15 moves for chess. It's not clear to me after searching what the current training regime for lczero is with regards to temperature. Also, I believe that ELF openGo efforts used $\tau=1$ for the entire game.
Question: Is there a consensus on what $\tau$ should be? Does it matter if the training is in early phases or not (i.e. if the AI is not advanced yet is it beneficial to explore seemingly "worse" moves?) How dependent on the game is this optimal $\tau$? If I have a game which lasts 50 moves average, and I want to decay $\tau$, is there a best practice?


text:
Error in MobileNet V1 Architecture?
From the architecture table of the first MobileNet paper, a depthwise convolution with stride 2 and an input of 7x7x1024 is followed by a pointwise convolution with the same input dimensions, 7x7x1024.
Shouldn't the pointwise layer's input be 4x4x1024 if the depthwise conv. layer was stride 2? (Assumming padding of 1)
Is this an error on the author's side? Or are there something that I've missed between these layers?
I've checked implementations of MobileNet V1 and it seems that everyone just treated this depthwise layer's stride as 1.

It depends if there is zero padding or not. There was probably some padding.

@Brale from the source code it seems that the padding is always of form valid - https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.py

@spiridon_the_sun_rotator I see. Then I assume there would be padding of 4 on each side, making the input at least 15x15. This does mean that roughly 30% of output values are zeros... I don't think I understand why they would have made such design choice.


text:
Is there any difference between ConvNet and CNN?
ConvNet stands for Convolutional Networks and CNN stands for Convolutional Neural Networks.
Is there any difference between both?
If yes, then what is it?
If no, is there any reason behind using ConvNet at some places and CNN at some other places in literature?

it's the same, networks are almost always neural networks in DL

Both terms just mean convolutional neural network. I don't believe there is any particular reason to choose one over the other: ConvNet is slightly easier to say out loud and CNN is slightly shorter to write, but there is absolutely no difference in meaning.
For some contrasting examples in the literature, the EfficientNet paper chooses the term ConvNet and this paper on AlexNet chooses CNN. They are talking about the same sorts of neural networks though!


text:
Late Onset Augmentation
If I train a U-Net model for image segmentation (e.g. medical images) and start training until it converges and then add augmentation - can i expect similar results as if i train with augmentation from the beginning ?

It would be better to run experiments on this. But, here's answer analytically.
The models will be different.
With augmentation, the network starts to learn to combat the noise too.
With late onset augmentation, the network will start to deviate from its original solution to combat noise.
Comparing this to the ball rolling down the hill, in case of augmentation, the ball will roll but will face a bit of friction (the noise due to augmentation). On the other hand, with late onset augmentation, it would be like the ball reached the bottom of hill, and now it has to face an uphill battle to model the noise. The ball will need to go a bit up the hill against gravity and friction due to noise.
I don't think it would be a good thing to do. But, running experiments is the only way we can confirm.

There is no guarantee that the two approaches will converge to the same minimum loss or the same set of model weights.  All  of that depends on how the training is performed.  The order in which training images are presented is a part of the training process, as is the optimizer that is used.
However, I do believe that the loss function landscape (and global minimum) should be the same (or at least similar) in both scenarios.  The loss landscape is dependent on the model architecture and the dataset, which are the same in both scenarios.  What is different is how the training images are presented, but I don't think that effects the actual shape of the loss landscape as much (just how we move over that landscape).  This, of course, will dictate the final loss and state of the model weights, which may or may not be the same.


text:
Classifying generated samples with Wasserstein-GAN as real or fake
I'm quite new to GANs and I am trying to use a Wasserstein GAN as an augmentation technique. I found this article
https://www.sciencedirect.com/science/article/pii/S2095809918301127,
and would like to replicate their method of evaluating the GAN. The method is shown in the figure.
In the article they write that they extract the generated samples that fooled the discriminator and use these to train a classifier. They also say that they use a Wasserstein GAN. Does anyone know how it is possible to extract samples that fooled the discriminator, since for a Wasserstein GAN the critic (discriminator) only puts a rating and not a label on the generated data?

My suggestion is to compare the critics rating for the real data and the generated data. Compute the mean of the ratings on real data to get some kind of threshold. If your critic is designed so that a high rating indicates a real sample, then generated data with ratings greater than the mean of the ratings of the real data could be viewed as good enough to fool the critic at that point in training.
This might be a too harsh criteria, but you will at least not accept poorly generated data.


text:
Should I use batch gradient descent when I have a small sample size?
I have a dataset with an input size of 155x155, with the output being 155 x 1 with a 3-4 layer neural net being used for regression. With such a small sample size, should I use full batch gradient descent (so all 155 samples) or use mini batch/stochastic gradient descent. I have read that using smaller mini batch sizes allows better generalisation, but as the batch size is very very small computationally it shouldn't be a burden to use BGD.

Input size of 155x155 is rather ambiguous i feel, it could mean a input with dimension 155x155 and not the size of the training set. If the number of samples is small as in your case, I believe that a BGD should converge faster than a SGD. However, in your case where you have so few samples, the time to converge shouldn't be that much significant. FYI: Some Mini batch sizes range up to 256 samples

Thanks for your answer! To clarify, the training set size is 155x155 not the dimension.

You mean (n_samples, n_features) = (155, 155) right?

Yes, exactly right!


text:
How should I simulate this Markov Decision Process?
I am working on solving a problem on nodes in a graph communicating with each other. They try to estimate a central state using Kalman consensus filter, with the connections described by the graph's adjacency matrix. Considering time to be discrete, the adjacency matrix changes at each time instant with a transition probability matrix (unknown) to some other matrix (in some finite set of matrices). I want to simulate this in python to solve an MDP with some cost/reward function. (An external agent is concerned with this cost/reward and takes actions accordingly)
Since the state space can be large, my advisor suggested using deep-RL techniques. However I have only studied (formally) basic RL (Q learning, stochastic approximation, etc. with finite states and finite actions at every instant). I tried looking at RL libraries but I can't figure out which one to pick. And even before that I am very confused by how to simulate KCF between nodes in python (from scratch?). How should I proceed?


text:
GAN Generator Output w/ Periodic Noise
I am training a Semi-Supervised GAN, using multivariate time-series with window of shape (180*80) with the generator and discriminator architecture below. My data is scaled using Robust Scaler, so I kept linear activation for the generator output.
During the training I get noise in the generated signals and I can't understand the reason why whereas the original data is smooth. What can be the reason for this noise?

def make_generator_model(noise):
    w_init = tf.random_normal_initializer(stddev=0.02)
    gamma_init = tf.random_normal_initializer(1., 0.02)

    def residual_layer(layer_input):

        res_block = Conv1D(128, 3, strides=1, padding='same')(layer_input)
        res_block = BatchNormalization(gamma_initializer=gamma_init)(res_block)
        res_block = LeakyReLU()(res_block)
        res_block = Conv1D(128, 3, strides=1, padding='same')(res_block)
        res_block = BatchNormalization(gamma_initializer=gamma_init)(res_block)
        res_block = LeakyReLU()(res_block)
        res_add = Add()([res_block, layer_input])

        return res_add

    in_noise = Input(shape=(100,))

    gen = Dense(180*65, kernel_initializer=w_init, use_bias=None)(in_noise)
    gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
    gen = LeakyReLU()(gen)

    gen = Reshape((180, 65))(gen)
    #assert model.output_shape == (None, 45, 256) # Note: None is the batch size

    gen = Conv1D(64, 7, strides=1, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
    #assert model.output_shape == (None, 45, 128)
    gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
    gen = LeakyReLU()(gen)

    gen = Conv1D(64, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
    #assert model.output_shape == (None, 45, 128)
    gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
    gen = LeakyReLU()(gen)

    gen = Conv1D(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
    #assert model.output_shape == (None, 45, 128)
    gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
    gen = LeakyReLU()(gen)

    for i in range(6):
        gen = residual_layer(gen)

    gen = Conv1DTranspose(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
    #assert model.output_shape == (None, 90, 64)
    gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
    gen = LeakyReLU()(gen)

    gen = Conv1DTranspose(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
    #assert model.output_shape == (None, 90, 64)
    gen = BatchNormalization(gamma_initializer=gamma_init)(gen)
    gen = LeakyReLU()(gen)

    out_layer = Conv1D(65, 7, strides=1, padding='same', kernel_initializer=w_init, use_bias=None)(gen)
    #assert model.output_shape == (None, 180, 65)

    model = Model(in_noise, out_layer)

    return model

def make_discriminator_model(n_classes=8):
    w_init = tf.random_normal_initializer(stddev=0.02)
    gamma_init = tf.random_normal_initializer(1., 0.02)

    in_window = Input(shape=(180, 65))

    disc = Conv1D(64, 4, strides=1, padding='same', kernel_initializer=w_init)(in_window)
    disc = LeakyReLU()(disc)
    disc = Dropout(0.3)(disc)

    disc = Conv1D(64*2, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
    disc = LeakyReLU()(disc)
    disc = Dropout(0.3)(disc)

    disc = Conv1D(64*4, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
    disc = LeakyReLU()(disc)
    disc = Dropout(0.3)(disc)

    disc = Conv1D(64*8, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
    disc = LeakyReLU()(disc)
    disc = Dropout(0.3)(disc)

    disc = Conv1D(64*16, 4, strides=1, padding='same', kernel_initializer=w_init)(disc)
    disc = LeakyReLU()(disc)
    disc = Dropout(0.3)(disc)

    disc = Flatten()(disc)

    disc = Dense(128)(disc)
    disc = Dense(128)(disc)

    out_layer = Dense(1)(disc)

    c_out_layer = Dense(8, activation='softmax')(disc)

    model = Model(in_window, out_layer)
    c_model = Model(in_window, c_out_layer)

    return model, c_model

I think at least some of the noise may be because you are using Conv1DTranspose to scale up directly. You might prefer to scale up using simple interpolation then run a normal convolution instead. Hopefully someone can provide a proper answer to this, but intuitively it is related to image GANs' fondness for grid patterns, which is similar high frequency noise.

Sorry cannot directly reply to your comment as I posted without an account, and you were right! I replaced transposed layers with Upscale1D+Conv1D and that solved the issue.
gen = Conv1DTranspose(128, 4, strides=2, padding='same', kernel_initializer=w_init, use_bias=None)(gen)

should become (notice that strides=2 becomes strides=1):
gen = Upscale1D()(gen)
gen = Conv1D(128, 4, strides=1, padding='same', kernel_initializer=w_init, use_bias=None)(gen)

I am glad that worked. If you have time to write a self-answer here with abit more detail, I will definitely upvote it. For instance if you described the working architecture in this answer with the same level of detail as in the question

Please, follow the steps here in order to merge your just created account with the unregistered one.


text:
Theoretical limits on correlation between classification algorithm performances
Are there any known theoretical bounds, or at least heuristic approaches, regarding the relation or correlation between the performances of any two different classification algorithms?
For example, would there exist binary classification datasets for which, say, $k$-nearest-neighbour classifiers would perform with say >90% accuracy, whereas say decision tree classifiers would do no better than 50-60%? (Accuracy here is measured by say $k$-fold cross-validation.)
It seems to me, at first glance, that a dataset which is able to achieve a very high accuracy on some classification algorithm would necessarily have some structure that would make it highly improbable that some other general classification algorithm would be able to perform very poorly. Yet it's also not impossible that there might be some 'exotic' type of dataset that does exhibit such a phenomenon.

Did you look at the regret bound?


text:
What are the metrics to be used for unsupervised monocular depth estimation in computer vision?
I am currently replicating the results of this paper. In this paper they have not mentioned how they are evaluating the results as no ground truth is available for comparison. Same goes for other papers of this topic (unsupervised depth estimation). So I am very much confused about how to evaluate the model for overfitting, underfitting for this.

Also if the suggestion is lidar scans then how can we use lidar scans as their output is very sparse and many objects are missing from it.


text:
Adding corpus to BERT for QA
I was wondering about SciBERT's QA abilities using SQuAD. I have a scarce textual dataset consisting of less than 100 files where doctors are discussing cancer in dialogues. I want to add it to SciBERT to see if the QA abilities will improve in the cancer disease domain.
After concatenating them into one large file which will be our vocab, I then clean the file (all char to lower, white space splitting, char filtering, punctuation, stopword filtering, short tokens and etc) which leaves me with a list of 3000 unique tokens
If I wanted to add these tokens, do I just do scibert_tokenizer.add_tokens(myList) where myList is the 3k tokens?
I can confirm that more tokens are added doing print(len(scibert_tokenizer)) and I can see that embeddings do change such as corona and ##virus changes to coronavirus and ##virus.
Does the model need to be trained from scratch again?

Is this a programming question? Programming issues are off-topic here, but maybe this is just a conceptual question. Can you clarify whether your question is about a programming issue or not? Could you also put your main specific question in the title? It seems that you have 2 questions here, but, ideally, every post contains only 1 question.


text:
$\epsilon$-greedy policy in environments where actions are performed in a long term. Does it has influence?
I'm working in an environment where once an action $a \in A$ is performed, it must hold this action selection for a while. To clarify this, assumes a horizon length $h$ and the set of actions: $\{a_{1}, a_{2}, a_{3}\} \in A$. Assumes now that the length $h$ to converge for an optimal solution must be split by 3 where each action is applied distinctly because if one of these actions is selected twice during an episode, the reward penalizes it severely in such way that there's no possible policy that can get a better return than selects each action just once. Thus, the length of each of "sub-horizon" $h_{i}$ is $h_{i} > 0$ and $h_{i} < h - 2$.
Now seeing the DQN setting in the naive approach for exploration using $\epsilon$-greedy policy that selects at each time-step an action following:
n = random value between 0 and 1
if n < epsilon then:
   a = random action in A
else:
   a = maxQ(S, a)

For my particular problem (and probably many with similar settings), this way of selects an action seems oppositely and hard to converge for the optimal solution, shifting the distribution of samples for far from that. Does it make sense?
Actually, I believe that sticky actions, could be way beneficial for many similar settings, beyond the scope of his original approach.


text:
AI approach for layout mapping
I am researching different AI approaches and was curious what approach would be useful in my scenario.
Assume you are tiling a room. The tiles, and the room itself, can be any shape. In this room you could encounter N number of obstacles, such as a wall, or built-in. The goal is to layout the tiles, taking into account cutting into the obstacles mentioned above, along with the shape and dimensions of the destination room. This would have to account the shape, and measurements of said tile being placed onto the room.
Which AI approach would prove useful in this scenario?

Hi, that is an interesting question. It seems to be an optimisation question, but it is not clear what you are trying to optimise. For example do you want to minumise the number of tiles used (and if so, do you allow re-use of cut tiles if pieces can be re-used), or minimise the number of partial tiles, or have some more complex cost function? Also, if the tiles can be "any shape", do they need to be a tesselating shape, or even a single shape?

Hey Neil, yes I think reuse would nice however could be allocated as a waste ratio too, like always assume a throwaway for shapes manipulated to fit into the larger shape, which would in turn increase the tally of supply accordingly.

To be clear, your loss metric is the waste ratio (total area of whole tiles used, divided by area of room), with an ideal score of 1.0? Could you perhaps state that in the question, and ideally mathematically (give an actual formula), although probably the formula is not an important part of any answer, just it is important that you are able to express a cost function in order to clearly make this an optimisation problem. Use [edit] to adjust the question


text:
Why Autoencoder Weights Are Not Always Tied
To me, tying weights in an autoencoder makes sense if we think of the auto encoder as doing PCA. Why in any situation would it make sense to not tie the weights? If we don't tie the weights, would it not try to learn something that is PCA anyway or rather something that might not be as optimal as PCA?
Also, if weights are not tied, it doesn't make sense to me that the auto-encoder is invertible i.e. if the decoder is looking for an inverse operation because it's a mapping between spaces of different dimension which should not invertible.
So, if the weights are not tied then why do we expect the decoder to learn anything meaningful i.e neither PCA nor an inverse operation?

What do you mean by "tying weights" or "tie the weights"?

We expect the decoder to learn anything meaningful without tying the weights because the loss function is calculated between the input and reconstructed output and training will minimize that loss. The untied autoencoder's decoder will learn to transform the embedding back into the input. Not tying the weights gives more representational power and also increases chances of overfitting.
If you are running into overfitting while training your autoencoder tying your weights is an option. It is a regularization method. Not tying them is fine if you are getting any overfitting or want to solve overfitting by an alternative method.


text:
How to restrain a model's outputs to a certain range without affecting its representative capacity?
CONTEXT
I am trying to build a regression model that finds the optimal parameters for a given input. The data I am using are point clouds, with N points and 3 coordinates (x,y,z) each. Each point cloud is divided into neighborhoods of constant size and, during inference, a batch of these neighborhoods are fed into the model which outputs a set of parameters. The parameters represent a family of surfaces and the goal is to find parameters such that the surface fits the neighborhood of points as tightly as possible (in the least squares sense).
THE ISSUE
The problem is that each type of parameter must fall into a specific range, otherwise it has no meaning. For example the first two parameters must lie inside [0.1, 1.9], the next three must be strictly positive etc.. I have tried restraining the outputs by adding a scaled sigmoid activation or simply clamping the output to the range that I want. However, it seems that such hacks result in saturation, the model outputs negative values and all the outputs become 0 from clamping.
I can't imagine I'm the first one to encounter such a problem, but I haven't been able to find out a way to solve it. Is there a defacto way of dealing with this situation?
P.S. I am not including details of the model architecture to keep this question general interest, but I will include them upon request, if it helps.

Hi @VlassisFo, I think you could try making a custom loss function, which would add extra punishment when the parameters go over the limit - this might help to discourage your model from spitting out large/wrong values.

@markmark That's an interesting idea! How would i go about making the loss as strict as possible? This constrain needs to be enforced in as little iterations as possible, as the rest of the training is based on the correctness of these parameters.

For numbers within the acceptable range($[0.1, 1.9]$) your loss could work as normal. If a value, say,  $5.9$, you could add  normal loss, plus squared distance from the upper bound(in this case it would be $normal loss + (5.9-1.9)^2$). The further away the parameter is, the larger the loss will be. The neural net should be able to pick that up and learn quicker, in theory.

Beware of the numbers that are below 1 ($0.5^2 = 0.25$), as they become smaller when squared. You could add 1 to all the distances to ensure that they always become larger.


text:
Using numerical/categorical data and image data to detect objects
Let's say that I want to create a program capable of detecting lamps on some pictures. Those pictures can be, for instance, of a room, a street, etc.
I would like to know if the following is possible:

Create a program that is trained using both pictures of lamps (so a dataset of images) and numerical/categorical data of lamps (so that would be a .csv file which contains features ranging from type, height, etc.)
The idea is to combine both types of data in order to detect on another unseen picture if it contains a lamp.

I'm not entirely sure if the resulting algorithm will be performant, but I would like to try.
If what I described above is possible, could you please point me in the right direction as to which literature to consult ? My research have lead me nowhere.

Welcome to Ai Community,however..,I kindly request you to first go through our community guidelines - help. For effective feedback.

i dont think those categorical data are useful, they might be actually labels like after object detection of lamps, classify them into 'types'


text:
What are the state-of-the-art Person-Detektion / Human-Segmentation?
I would like to use a deep learning approach to detect people in videos. I have found some freely accessible implementations like Human Segementation with Pytorch or BodyPix / DeepLab / Pixellib with Tensorflow. They all work well, but with many it happens that, for example, half hand is not detected or if a person is sitting in the picture only the legs and the head are detected. Are there other approaches to detect people who are freely accessible or is that state-of-the-art?
I had imagined such problems have been solved, but I don't know so much about it. Thanks for your answers.

Person detection and human segmentation should not be the same things. So, please, edit your post to clarify if you're looking for one or the other. Moreover, it may be a good idea to provide the links to the implementations/models you have found so far. In any case, note that asking for "software libraries" is off-topic. If you're looking for references (e.g. papers), that's ok, but clarify that in your post, please.

You can use any state of the art object detection such as DETR, EfficientDET, YOLO etc. since they have an object 'person' in them. But, if the density of people is high in the images, then, you should look at 'Finding Tiny Faces' which is good for that problem (this will help you detect faces only, think group photos). If you want to find the whole person even if they are occluded, then, it will be difficult (you can then think of segmentation techniques and then, try to find out the people but it will be more taxing).


text:
Why are CNN binary classifier output probability distributions often skewed?
I've been working on a lot of simple resnet18 binary classifiers lately and I've started to notice that the probability distributions are often skewed one way or the other. This figure shows one such example. The red and blue color code the negative and positive ground truths respectively. And the bottom axis is the output prediction of the binary classifier (sigmoid activated output neuron). Notice how the red is more bunched towards 0, but the blue has quite some spread.

At first I began to reason this to myself with arguments like "well the positive clues in the image have a small footprint, so they are hard to find, therefore the model should be unsure about positives more of the time."
Later I found oppositely skewed distributions and tried to say "well the positive clues in the image have a small footprint, so it might be easy to confuse some other things for the positive clues, therefore the model should be unsure about negatives more of the time"
You can see where I'm going with this. It took me training up quite a few models like this in a short amount of time to realise I was kidding myself. Even the exact same architecture and similar dataset may produce a different skew over different training runs. And if you think about it, negative probability is just the complement of positive probability, so any argument you make in favor of one over the other can be easily reversed.
So what's influencing this skew? Why is there a skew at all? If there's a skew, is it because of something "real", or is it just random?
These all may seem like philosophical questions, but they have great practical significance. Because that skew basically tells me where I should put my decision threshold in production level inference!

Yes, due to this issue, you should use temperature scaling after training your model. It will calibrate your probability and you will start to get the same kind of distributions. Here are a good article along with implementation on it.

Thanks! This is a useful answer and checks the "practical" box. I'm also wondering a lot about the data science behind why this is the case. I had previously thought that logistic regression (which is essentially what's happening on top of the CNN) is supposed to give calibrated probabilities. In fact, that's what Platt Scaling is really doing.

The reason each time, you get a different model is because of the entropy involved in creating the new model (initialization). Hence, reproducibility has come into scene strongly in DL. But, even if we set the random seed at every possible level, some amount of entropy is still retained. It is that entropy which leads to a different solution every single time. A lot of regularization may help with that, one should experiment. So, every time, you end up with a different minima in the loss landscape.


text:
Why identity mapping is so hard for deeper neural network as suggested by Resnet paper?
In resnet paper they said that a deeper network should not produce more error than its shallow counterpart since it can learn the identity map for the extra added layer. But empirical result shown that deep neural networks have a hard time finding the identity map. But the solver can easily push all the weights towards zero and get an identity map in case of residual function($\mathcal{H}(x) = \mathcal{F}(x)+x$). My question is why it is harder for the solver to learn identity maps in the case of deep nets?
Generally, people say that neural nets are good at pushing the weights towards zero. So it is easy for the solver to find identity maps for residual function. But for ordinary function ($\mathcal{H}(x) = \mathcal{F}(x)$) it have to learn the identity like any other function. But I do not understand the reason behind this logic. Why neural nets are good to learn zero weights ?


text:
What model structure I should use to train on low res and blurry images?
I am looking for advice or suggestion.
I have photos like these: photo_1 and photo_2 and many more similar to that. The average shape of these photos is about 160 x 100. What we are doing is we are trying to find wheather or not person in a photo is wearing safety vest and helmet (if person is wearing both it is 1, if something is missing or both are missing it is 0). Training data consists of about 5k almost equally distributed image sets. I have tried to use augmentation techniques (flipping, adding noise, brighness correction) but results didn't improove.
I tried to train on many pretrained popular models: resnet101, mobilenet_v2, efficientneyb3, efficientneyb0, DenseNet121, InceptionResNetV2, InceptionV3, ResNet152V2, ResNet50V2, but results are not eyepleasing. I have tried different input sizes ranging from 224x224 to 112x112 but result didn't improve as much as I would have liked it to be. And the weird thing is that the image shape does not correlate to wheather or not there are more wrong predictions using bigger or smaller images.
As a side not I would lik to ask couple questions:

Should I use my own written small net?
Are the models that I use too big for this problem?

Any advice will be appreciated.

What is happening with your training and validation accuracy? Is training accuracy high but validation low (suggestive of overfitting)? Both low (possibly underfitting or a poor model)?

It is overfitting by a bit.

Yes, you may use the small model you have but you need to tune it.
The models you have used are not too big if any overfitting happens you can just use dropout to alleviate that.
You need to improve the quality of your images. One way is to reconstruct them using autoencoders or GANs (Super-resolution GANs). Then, you can classify them.

I would first suggest to cluster your data using T-SNE visualisation technique. Try to see if different classes are separable. If not, try to perform different image enhancement filters (e.g. white balance, sharpening) and see how does the separation change. This is how you design your image preprocessing pipeline.
Since your dataset is relatively small, it is indeed a good idea to use a pretrained network (e.g. MobileNet V3) and perform transfer learning (by retraining only the final layer).
Finally, for your task image size might be not crucial indeed.


text:
In VQ-VAE code what does this line of code signify?
The VQ-VAE implimentation:https://colab.research.google.com/github/zalandoresearch/pytorch-vq-vae/blob/master/vq-vae.ipynb
quantized = inputs + (quantized - inputs).detach()

Why are we subtracting and adding input to quantized result?

Numerical stability.

@AbhishekVerma it would be really helpful if you could please explain this with a little detail....kind of a newbie here.

This is to get the gradient to "skip" the quantization part.
The trick implements the red arrow in the original paper's diagram:

Simplified example: Rounding
Let's simplify this a bit and imagine we want to use rounding in our architecture:
import torch
x = torch.tensor([1.1, 2.1], requires_grad=True)
y = 2*x
z = torch.round(y)
r = z.sum()

The graph (torchviz.make_dot) looks like this:

We can look at the output:
r
# tensor(6., grad_fn=<SumBackward0>)

All is looking good. However, when we try to compute the gradient, we get a tensor of zeroes:
r.backward()
x.grad
# tensor([0., 0.])

This makes sense: the rounding function has derivative zero almost everywhere:

However, it also means we cannot train our network.
To circumvent that, we could simply tell the gradient to skip the rounding network. To do that, we use detach, a function that tells PyTorch to detach a vector from the computational graph.
x = torch.tensor([1.1, 2.1], requires_grad=True)
y = 2*x
z = torch.round(y)
z = y + (z - y).detach() # Detach everything between z and y, including z
r = z.sum()

We still get the same answer for $r$
r
# tensor(6., grad_fn=<SumBackward0>)

But we now also get reasonable gradients
r.backward()
x.grad
# tensor([2., 2.])

Thanks to the fact that the gradient now "skips" the rounding part:

Back to VQ-VAE
In VQ-VAE, we are replacing the output of the encoder (the input to the vector quantization layer) with the code vector. In some sense, this similar to "rounding", only this time we're "rounding" the encoder's output to the nearest code vector.

We therefore run into the same problem. Here's what the authors say about it

Note that there is no real gradient defined for [the quantization], however we approximate the gradient similar to the straight-through estimator and just copy gradients from decoder input $z_q(x)$ to encoder output $z_e(x)$

which is precisely what
quantized = inputs + (quantized - inputs).detach()

does in the notebook.


text:
Why do popular object detecting models output heatmaps instead of coordinators of object directly?
I think heatmap outputs of architectures like CenterNet, OpenPose, etc. can be changed to coordinator outputs, and loss functions like focal loss can be modified so they can deal with coordinators instead of heatmaps. Is there any particular reasons that researchers use heatmaps instead of coordinators?

To have a more powerful representational output. You can derive a bounding box from heatmap but not vice-versa. Also, in case of dense object detection it is hard to create bounding boxes for each object (people standing in front of each other).
That being the case, it is better to run a segmentation loss for these networks. It also leads to less confusion for the network. Also, it is a single shot creation of output. On the other hand, the bounding box approach will have the creation of a lot of bounding boxes then finding their objectness score and finally applying NMS (non-max suppression).
In regards to focal loss and use of bounding boxes, it is something that is a consequence of the heatmaps/activations that neural networks learns. They are built upon that and then regressed.


text:
Can you correlate decision boundary of final layer of a neural network to predictive distribution?
I was reading in a On the Decision Boundary of Deep Neural Networks
 that the final layer of a MLP can be equated to an SVM and can generate decision boundaries similar to methods with SVM. I was wondering if using this boundary detection method or another can you quantify how much probability a model assigns to each bin where a bin is a class. So for example, if a project of an input before the final layer has a SVM margin of let's say 4 from the boundaries and classification 1 and 2, can we determine how much probability it'll give class 1 or 2 after the final layer?


text:
Finding Specific Patterns in Data
I'm trying to research modeling that can help me find very specific patterns in data.  I've done a fair amount of work about generalized predictions with machine learning, but I'm very confused about how to approach something that gets into very specific predictions.
As an example, for an IT infrastructure that has 1000 Cisco Router across the world.  I'm trying to find patterns in this data when outages occur.  Outages typically are either power related on transport circuit related.  I have historical data for over a year.  I'm trying build a model to help predict the outage type.  But, I want it to have very specific knowledge of previous outages.  Maybe there is a pattern when Routers 17,245 and 813 fail it is always a power problem at each of the sites.
I will have input data about geospacial, diagnostics, and other IT type information.
I know a lot of modeling can generalize this type of scenario, but I'm trying to see if there are options to remember more specific patterns within a large dataset.

It's not clear to me whether you have "labels" or not in your training data, i.e. is your data labelled with "this input $i$ is of type X", "this input $j$ is of type $Y$", etc., or you have only inputs without labels? In the second case, you may consider unsupervised learning? If that's not what you're looking for, can you please clarify why typical clustering algorithms are not suitable for your use case?


text:
What is the input to the left most LSTM cell c(t-1) and h(t-1)?
Given an LSTM model with 3 cells shown below, what would be the input to the left most cell c(t-1) and h(t-1)?

Most commonly these states are set to zero, this usually works, but it can have a negative influence on the performance.
Another option is to initialize randomly, but this is not straight forward and the performance highly depends on the noise level. In this work the noise level is selected in proportion to the prediction error of the first timestep in the series.
A third option is to use learnable variables. You can initialize these randomly and the model will learn the initialization vectors by itself. This is showcased in this answer on SO
I'd say options one and three are the most straight forward and I've mostly seen the first method.
For reference, see this paper.

Could random initialization make sense if you don't want the LSTM to know where the sequence begins?

I think you cannot really say that in general: The space of the hidden state follows some distribution pattern/manifold that is learned. How exactly that space looks like is up to learning, so starting with the zero-vector can fall perfectly into that hidden space and does not 'appear' as something special to the LSTM, as does random initialization. However, if it is useful for the downstream task to treat the beginning as something special, the hidden space might develop in a way that allows identifying the starting token ....

... I could be wrong but I think you don't really have a lot of influence on that. What do you think about it?

Your question is related to the initial states of LSTM, where c(t-1) is the cell state (memory) and h(t-1) is the previous LSTM block output.
As pointed out here, it is reasonable to assume that those are random values.


text:
Where is the difference between a neural network mapping a problem space and learning a behaviour?
I've been looking at neural networks for control applications. Let's say I used an RL algorithm to train a controller for the cart pole balancing problem.
Assuming the neural network is simple and very small, I can pretty much deduce what exactly the network is doing. For instance, if the network takes inputs for pole angle and cart position and outputs a motor force, the neural network is approximating a function that will move the cart left if the pole is falling left etc. and I can forward propagate through the network manually, again assuming that it is simple. In this case however, I could say that the neural network isn't truly learning a behavior, and instead is just mapping the problem space.
However, what if I trained another, larger network for a similar problem, where there are environmental uncertainties that randomly occur (ie. oil patch on the ground so the cart dynamics change, or the ground is made of ice, or there are stochastic disturbances that simulate someone bumping the cart). If the training is successful, the resulting neural network would be learning the behaviour of balancing the cart for a variety of situations (robustness), instead of just pushing it left or right depending on the pole angle.
The cart pole problem may not be the best example for this since it's a relatively simple control problem, but for more complex behaviors (ie. autonomous driving), where does this inflection point between learning and mapping exist?
Is this even a valid question, or am I just completely mistaken and everything is technically just a function approximation and there is never any "true" robust learning happening?


text:
Why is the sigmoid function interpreted as a saturating firing rate of a neuron?
I've seen several people say that sigmoids are like a saturating firing rate of a neuron but I don't see how or why they interpret it as such. I especially don't see the relationship between a "rate" (so a number of something over time, I guess here it's the number that a neuron activates in a unit of time) and the sigmoid graph. For me it resembles more to the voltage output of an operational amplifier in some cases.

Sigmoid outputs a value between 0-1, this can be interpreted as the probability of the neuron ‘firing’.

@DavidIreland that could be an interpretation. Like the probability of a neuron "firing" when the input is x is $\sigma (x)$ (where $\sigma$ denotes the sigmoid function).

It would be nice if you could provide 1-2 references where people say "sigmoids are like a saturating firing rate of a neuron", just to have more context.

@nbro You're right, sorry for this imprecision. So a friend of mine told me about this interpretation (it was given by his professor and he found it cool) and when I searched for it I found this post: https://datascience.stackexchange.com/questions/14349/difference-of-activation-functions-in-neural-networks-in-general I agree it's not many people I just thought that since it's a professor who said it, he maybe have read/heard it somewhere else etc... ^^'


text:
How to have zero value or a value between 200 and 400 in the output of a deep learning model?
I want to implement a DDPG method and obviously, the action space will be continuous. I have three outputs. The first output should be zero or a value between 200 and 400, and the other outputs have similar conditions. I don't know how can I implement this condition in the layers and activation functions. Should I use a binary activation before the scaled sigmoid function? How can I scale the activation function for this example?

(a1 = 0) or (200 < a1 < 400)
(a2 = 0) or (100 < a2 < 500)
(a3 = 0) or (200 < a3 < 1000)

generally the approach is to have a separate head. For example, imagine you have latent vector $z_k$, you would output two values: $h(z_k)$ and $f(z_k)$ where $0 \leq h \leq 1$ and $b_0 \leq f \leq b_1$ where $b_0$ and $b_1$ are your bounds.
In thios setup, during inference you would check $h_k$ and if its greater than some threshold (usually .5), youd evaluate/output $f_k$.
In this case your loss would look something like $L_i = L_i^{(1)}(h_i, y_i)+y_i*L_i^{(2)}(f_i,v_i)$ where $(y,v)$ are your labels, and the $L$'s are your loss of choice.